You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "Akshat-Jain (via GitHub)" <gi...@apache.org> on 2024/05/21 17:16:59 UTC

[PR] Optimize S3 storage writing for MSQ durable storage (druid)

Akshat-Jain opened a new pull request, #16481:
URL: https://github.com/apache/druid/pull/16481

   ## Description
   
   This PR optimises S3 storage writing for MSQ durable storage by uploading the chunks to S3 in a separate threadpool.
   
   Currently, we were creating and uploading a chunk to S3 sequentially, which isn't optimal.
   
   This PR changes the logic such that the creation of chunks still happen in the same processing threadpool, but the uploading of chunks is offloaded to a separate threadpool.
   
   ## Test Plan
   
   1. For every operation, I verified the number of rows returned with this PR's changes match the number of rows returned without this PR's changes.
   2. For every operation, I verified the output files and their sizes (in S3) with this PR's changes match the corresponding files uploaded without this PR's changes.
   3. TODO: Need to add/update tests.
   
   ## Timing Data
   
   Following section summarizes the query duration comparison for a few different queries. We compare the original duration with the new duration across a few combinations of parameters.
   
   Note:
   1. `trips_xaa` is one of the sample datasources with 60M rows.
   3. All timing durations mentioned below are averages of 3 data points.
   4. All tasks used the following query context, unless a different value is explicitly mentioned in the `Other Notes` column:
   ```json
   {
     "maxNumTasks": 5,
     "durableShuffleStorage": true,
     "selectDestination": "durableStorage",
     "rowsPerPage": 10000000
   }
   ```
   
   ### Query 1
   ```sql
   select trip_id, sum(tip_amount) from "trips_xaa" group by trip_id
   ```
   
   Original timings:
   
   | Chunk Size | Time               | Other Notes          |
   | :--------- | :-------------- |  :-------------- | 
   | 100 MB     | 10:31 minutes   | |
   | 5 MB         | 10:51 minutes | |
   | 5 MB         | 9:53 minutes    | maxNumTasks = 2 |
   
   New timings:
   
   | Chunk Size | Threadpool size | Max concurrent chunks | Time          |
   | :--------- | :-------------- | :-------------------- | :------------ |
   | 100 MB     | 5               | 5                     | 8:56 minutes  |
   | 5 MB       | 5               | 5                     | 10:26 minutes |
   | 5 MB       | 10              | 10                    | 7:35 minutes  |
   | 5 MB       | 30              | 30                    | 6:52 minutes  |
   
   
   ### Query 2
   ```sql
   SELECT sum("commentLength"), "countryName" FROM "wikipedia_s3" group by "countryName" limit 4
   ```
   
   Original timings:
   
   | Chunk Size | Time               | 
   | :--------- | :-------------- | 
   | 100 MB     |  32 seconds | 
   | 5 MB         |  29 seconds | 
   
   New timings:
   
   | Chunk Size | Threadpool size | Max concurrent chunks | Time       |
   | :--------- | :-------------- | :-------------------- | :--------- |
   | 100 MB     | 5               | 5                     | 31 seconds |
   | 5 MB       | 5               | 5                     | 28 seconds |
   | 5 MB       | 10              | 10                    | 28 seconds |
   
   
   ### Query 3
   ```sql
   SELECT "trip_id", sum(tip_amount) FROM "trips_xaa" where "fare_amount" > 100 group by "trip_id", "tip_amount" order by "trip_id"
   ```
   
   Original timings:
   
   | Chunk Size | Time               | 
   | :--------- | :-------------- | 
   | 100 MB     | 6:22 minutes | 
   | 5 MB         | 4:56 minutes |
   
   New timings:
   
   | Chunk Size | Threadpool size | Max concurrent chunks | Time         |
   | :--------- | :-------------- | :-------------------- | :----------- |
   | 100 MB     | 5               | 5                     | 5:55 minutes |
   | 5 MB       | 5               | 5                     | 4:55 minutes |
   | 5 MB       | 10              | 10                    | 4:30 minutes |
   
   
   ### Query 4
   ```sql
   select trip_id, sum(tip_amount) from "trips_xaa" where __time < TIMESTAMP '2014-04-06 11:15:25' group by trip_id
   ```
   
   Original timings:
   
   | Chunk Size | Time               | 
   | :--------- | :-------------- | 
   | 100 MB     | 1:56 minutes | 
   | 5 MB         | 1:52 minutes | 
   
   New timings:
   
   | Chunk Size | Threadpool size | Max concurrent chunks | Time         |
   | :--------- | :-------------- | :-------------------- | :----------- |
   | 100 MB     | 5               | 5                     | 1:45 minutes |
   | 5 MB       | 5               | 5                     | 1:45 minutes |
   | 5 MB       | 10              | 10                    | 1:44 minutes |
   
   
   ### Query 5
   ```sql
   select * from "trips_xaa" where __time < TIMESTAMP '2013-11-06 11:15:25'
   ```
   
   The optimization is much more significant when we have I/O bound queries like this one with durable storage enabled for storing intermediary stage outputs as well as the query results. This query returned 56.8 MB of data.
   
   Original timings:
   
   | Chunk Size | Time               | Other Notes          |
   | :--------- | :-------------- |  :-------------- | 
   | 5 MB         | 2:11 minutes | maxNumTasks = 2 |
   
   New timings:
   
   | Chunk Size | Threadpool size | Max concurrent chunks | Time         | Other Notes          |
   | :--------- | :-------------- | :-------------------- | :----------- |  :-------------- | 
   | 5 MB       | 10              | 10                    | 56 seconds | maxNumTasks = 2 |
   
   
   ### Query 6
   ```sql
   select * from "trips_xaa" where __time < TIMESTAMP '2014-02-06 11:15:25'
   ```
   
   This is another I/O bound query like the previous one. This query returned 454 MB of data.
   
   Original timings:
   
   | Chunk Size | Time               | Other Notes          |
   | :--------- | :-------------- |  :-------------- | 
   | 5 MB         | 13:50 minutes | maxNumTasks = 2 |
   
   New timings:
   
   | Chunk Size | Threadpool size | Max concurrent chunks | Time         | Other Notes          |
   | :--------- | :-------------- | :-------------------- | :----------- |  :-------------- | 
   | 5 MB       | 10              | 10                    | 5:08 minutes | maxNumTasks = 2 |
   
   
   This PR has:
   
   - [x] been self-reviewed.
   - [ ] added documentation for new or modified features or behaviors.
   - [ ] a release note entry in the PR description.
   - [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
   - [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
   - [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
   - [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
   - [ ] added integration tests.
   - [ ] been tested in a test Druid cluster.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1608683803


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -103,13 +109,35 @@ public class RetryableS3OutputStream extends OutputStream
   private boolean error;
   private boolean closed;
 
+  /**
+   * A map to store number of files pending to be uploaded for a given uploadId.
+   */
+  private static final ConcurrentHashMap<String, AtomicInteger> PENDING_FILES = new ConcurrentHashMap<>();
+
+  /**
+   * A map containing the lock for a given uploadId used for notifying the main thread about the completion of
+   * s3.uploadPart() for all chunks for the uploadId, and hence starting the s3.completeMultipartUpload() for
+   * the uploadId.
+   */
+  private static final ConcurrentHashMap<String, Object> FILE_LOCK_MAP = new ConcurrentHashMap<>();
+
+  /**
+   * Semaphore to restrict the maximum number of simultaneous chunks on disk.
+   */
+  private static final int MAX_CONCURRENT_CHUNKS = 10;

Review Comment:
   @cryptoe How should this be configured? Is it fine to keep it constant? Or should we add another config parameter in S3OutputConfig? Thoughts?



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -103,13 +109,35 @@ public class RetryableS3OutputStream extends OutputStream
   private boolean error;
   private boolean closed;
 
+  /**
+   * A map to store number of files pending to be uploaded for a given uploadId.
+   */
+  private static final ConcurrentHashMap<String, AtomicInteger> PENDING_FILES = new ConcurrentHashMap<>();
+
+  /**
+   * A map containing the lock for a given uploadId used for notifying the main thread about the completion of
+   * s3.uploadPart() for all chunks for the uploadId, and hence starting the s3.completeMultipartUpload() for
+   * the uploadId.
+   */
+  private static final ConcurrentHashMap<String, Object> FILE_LOCK_MAP = new ConcurrentHashMap<>();
+
+  /**
+   * Semaphore to restrict the maximum number of simultaneous chunks on disk.
+   */
+  private static final int MAX_CONCURRENT_CHUNKS = 10;
+  private static final Semaphore SEMAPHORE = new Semaphore(MAX_CONCURRENT_CHUNKS);
+
+  /**
+   * Threadpool used for uploading the chunks asynchronously.
+   */
+  private static final ExecutorService UPLOAD_EXECUTOR = Executors.newFixedThreadPool(10);

Review Comment:
   @cryptoe Similar question here: How should the pool size be configured?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1609489542


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -199,15 +211,47 @@ private void pushCurrentChunk() throws IOException
   {
     currentChunk.close();
     final Chunk chunk = currentChunk;
-    try {
-      if (chunk.length() > 0) {
-        resultsSize += chunk.length();
+    if (chunk.length() > 0) {
+      try {
+        SEMAPHORE.acquire(); // Acquire a permit from the semaphore

Review Comment:
   > If we are trying to avoid running out of disk space, then instead of having semaphores, we just need to check some condition in write() that there is disk space available (this can be computed based on the formula that @cryptoe has suggested). If there isn't, we should wait for the condition to be satisfied before proceeding with the write.
   
   Do you suggest adding a retry wrapper over a new synchronized method assertEnoughDiskSpaceExists()? If yes, for what duration? I feel this would be more complicated to follow and more error prone, and would potentially increase the overall query duration a lot because of a bunch of retries across multiple threads, which defeats the purpose of the optimization. Appreciate your thoughts and clarification on this, thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1609650915


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -199,15 +211,47 @@ private void pushCurrentChunk() throws IOException
   {
     currentChunk.close();
     final Chunk chunk = currentChunk;
-    try {
-      if (chunk.length() > 0) {
-        resultsSize += chunk.length();
+    if (chunk.length() > 0) {
+      try {
+        SEMAPHORE.acquire(); // Acquire a permit from the semaphore

Review Comment:
   Interesting, makes sense. Will try this out, thank you!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "rohangarg (via GitHub)" <gi...@apache.org>.
rohangarg commented on PR #16481:
URL: https://github.com/apache/druid/pull/16481#issuecomment-2134521306

   > I have put some timing data in the PR description, where you can check queries 5 and 6 to get some idea of the degree to which concurrency helps in output heavy queries.
   
   Thanks, I checked earlier but I couldn't understand their timings much. For instance, were you able to reason about why the serial upload is taking ~ 8 minutes for 450MBs? Also, the chunk size in those queries seemed a bit de-generate to me.
   
   > I don't think earlier each writer got a thread instantly for writing. The writing was being done by the processing threadpool, but it's possible to have more workers than the processing threadpool size. I haven't tried such a scenario so far, but I think under this scenario, not all writers will get a thread instantly as there are more workers/writers than the size of processing threadpool. To answer your question about fairness in the PR changes, there's no such mechanism. But as I described, this seems like a separate problem than what this PR is trying to address.
   
   I was actually meaning that the threads which are running and are writing to the output stream currently are able to write it to S3 too on the same thread, which might not be the case which this change and the writes might have to wait. Are we solving that scenario in this change? Further, that brings me to another point I wanted to raise - have we considered supporting parallel uploads via the execution framework (maybe an operator to fan out writes) instead of doing it on a per storage basis? If so, can you please share the reasons for not choosing that? One more positive if it can be done via execution framework is that it will then automatically support all storage connectors.
   
   > That's a good point. I can add logs for wait times of all parts, but I felt it might be too much logging. Will think more about this. Happy to hear any suggestions as well, if you have any! :)
   
   You can create summary statistics over the wait times and post them in the logs - hopefully that's good enough :) 
   
   > Semaphore was used to make sure we restrict number of files locally to xyz number of files, to restrict disk space usage at any given point in time. Semaphore wasn't needed in the original code since the disk space was automatically restricted as the same thread was deleting the chunk before writing the next chunk.
   
   Yes, it makes sense to restrict the number of concurrent files. But I would do that in the executor related code itself instead of leaking it into the output stream.
   
   > I think the rationale here is that we don't know in advance how big the overall file would be. Hence we are doing multi-part uploads for everything.
   
   You can determine that the file is small when not even one local chunk is fully filled, and the output stream is closed. And once you've determined that a file is small, direct `PUT` can be used over multi-part upload.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on PR #16481:
URL: https://github.com/apache/druid/pull/16481#issuecomment-2134401814

   > > idea of the degree to which concurrency helps in output heavy queries
   > 
   > What is meant by output heavy query?
   > 
   > > but it's possible to have more workers than the processing threadpool size.
   > 
   > In which scenario is this possible?
   
   
   @LakshSingla 
   1. By output heavy query, I meant queries where we have a lot of data to upload to durable storage, and the processing time isn't a lot. Example: `select * from huge_datasource`
   2. I think our processing threadpool size is fixed based on number of cores of the system. For example, for my system it's 10 processing threads. But I can provide maxNumTasks higher than that? Or do we have a pre-flight check to not allow that? (I should try it out once)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "cryptoe (via GitHub)" <gi...@apache.org>.
cryptoe merged PR #16481:
URL: https://github.com/apache/druid/pull/16481


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1618143342


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -199,15 +221,44 @@ private void pushCurrentChunk() throws IOException
   {
     currentChunk.close();
     final Chunk chunk = currentChunk;
-    try {
-      if (chunk.length() > 0) {
-        resultsSize += chunk.length();
+    if (chunk.length() > 0) {
+      uploadManager.incrementCurrentNumChunks();
+      pendingFiles.incrementAndGet();
 
-        pushStopwatch.start();
-        pushResults.add(push(chunk));
-        pushStopwatch.stop();
-        numChunksPushed++;
+      uploadManager.submitTask(() -> {
+        try {
+          uploadChunk(chunk);
+        }
+        catch (Exception e) {
+          error = true;
+          LOG.error(e, e.getMessage());
+          throw new RuntimeException(e);
+        }
+        finally {
+          synchronized (maxChunksLock) {
+            uploadManager.decrementCurrentNumChunks();
+            maxChunksLock.notifyAll();
+          }
+          if (pendingFiles.decrementAndGet() == 0) {
+            synchronized (fileLock) {
+              fileLock.notifyAll();
+            }
+          }
+        }
+      });
+    }
+  }

Review Comment:
   @kfaraz I thought about that, but we have a couple of lock usages in RetryableS3OutputStream which can't be moved to S3UploadManager:
   1. `maxChunksLock.wait();` in `write()` method
   2. `fileLock.wait();` in `completeMultipartUpload()` method
   
   
   So would you suggest I move these locks to `S3UploadManager` and expose methods like `getMaxChunksLock()` and `getFileLock()` to replace the current usages in `RetryableS3OutputStream`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1616621024


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3StorageDruidModule.java:
##########
@@ -182,4 +187,20 @@ public Supplier<ServerSideEncryptingAmazonS3> getAmazonS3ClientSupplier(
   {
     return Suppliers.memoize(serverSideEncryptingAmazonS3Builder::build);
   }
+
+  @Provides
+  @LazySingleton
+  @Named("S3UploadThreadPool")
+  public ExecutorService getUploadPoolExecutorService(ScheduledExecutorFactory scheduledExecutorFactory, RuntimeInfo runtimeInfo)
+  {
+    int poolSize = Math.max(4, 3 * runtimeInfo.getAvailableProcessors());

Review Comment:
   @LakshSingla I made this change as per: https://github.com/apache/druid/pull/16481#discussion_r1609309586
   
   cc: @cryptoe Can you chime in? Appreciate your thoughts here, thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "LakshSingla (via GitHub)" <gi...@apache.org>.
LakshSingla commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1617126183


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3UploadManager.java:
##########


Review Comment:
   Note on the synchronization of the class - it is still not thread safe. Consider two threads T1 & T2 calling `updateChunkSizeIfGreater(100)` and `updateChunkSizeIfGreater(200)` respectively, and the current `chunkSize` is 50. 
   Both can simultaneously enter the method and at the end of the two operations, the `chunkSize` can be 100 instead of 200 (which was expected). 
   Making the integers atomic isn't enough since there are conditions in which multiple variables are getting updated/read. The easiest way of synchronizing it would be to leave all the variables as is, and mark all the methods `synchronized`. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1627919294


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -269,52 +213,74 @@ public void close() throws IOException
     // Closeables are closed in LIFO order
     closer.register(() -> {
       // This should be emitted as a metric
+      long totalChunkSize = (currentChunk.id - 1) * chunkSize + currentChunk.length();

Review Comment:
   > It allows us to not have a Map<Chunk, Future> unnecessarily
   
   To me, having the map seems cleaner as we know which future belongs to which chunk. I don't see how it would affect the rest of the code as we just need to put stuff in the map and then iterate over it in `completeMultipartUpload()` and then while determining the size (which could even just be returned by the `completeMultipartUpload()` method).
   
   That said, I don't really mind the current approach either. I guess the map is more of a personal preference in this case.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1628749775


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -103,27 +94,28 @@ public class RetryableS3OutputStream extends OutputStream
   private boolean error;
   private boolean closed;
 
-  public RetryableS3OutputStream(
-      S3OutputConfig config,
-      ServerSideEncryptingAmazonS3 s3,
-      String s3Key
-  ) throws IOException
-  {
+  /**
+   * Helper class for calculating maximum number of simultaneous chunks allowed on local disk.
+   */
+  private final S3UploadManager uploadManager;
 
-    this(config, s3, s3Key, true);
-  }
+  /**
+   * A list of futures to allow us to wait for completion of all uploadPart() calls
+   * before hitting {@link ServerSideEncryptingAmazonS3#completeMultipartUpload(CompleteMultipartUploadRequest)}.
+   */
+  private final List<Future<UploadPartResult>> futures = new ArrayList<>();

Review Comment:
   Marking this resolved as per the discussion at https://github.com/apache/druid/pull/16481#discussion_r1628218203



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1629278464


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3UploadManager.java:
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.inject.Inject;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.common.RetryUtils;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.storage.s3.S3Utils;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+import org.apache.druid.utils.RuntimeInfo;
+
+import java.io.File;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
+
+/**
+ * This class manages uploading files to S3 in chunks, while ensuring that the
+ * number of chunks currently present on local disk does not exceed a specific limit.
+ */
+@ManageLifecycle
+public class S3UploadManager
+{
+  private final ExecutorService uploadExecutor;
+
+  private static final Logger log = new Logger(S3UploadManager.class);
+
+  @Inject
+  public S3UploadManager(S3OutputConfig s3OutputConfig, S3ExportConfig s3ExportConfig, RuntimeInfo runtimeInfo)
+  {
+    int poolSize = Math.max(4, runtimeInfo.getAvailableProcessors());
+    int maxNumConcurrentChunks = computeMaxNumConcurrentChunks(s3OutputConfig, s3ExportConfig);
+    this.uploadExecutor = createExecutorService(poolSize, maxNumConcurrentChunks);
+    log.info("Initialized executor service for S3 multipart upload with pool size [%d] and work queue capacity [%d]",
+             poolSize, maxNumConcurrentChunks);
+  }
+
+  /**
+   * Computes the maximum number of concurrent chunks for an S3 multipart upload.
+   * We want to determine the maximum number of concurrent chunks on disk based on the maximum value of chunkSize
+   * between the 2 configs: S3OutputConfig and S3ExportConfig.
+   *
+   * @param s3OutputConfig  The S3 output configuration, which may specify a custom chunk size.
+   * @param s3ExportConfig  The S3 export configuration, which may also specify a custom chunk size.
+   * @return The maximum number of concurrent chunks.
+   */
+  @VisibleForTesting
+  int computeMaxNumConcurrentChunks(S3OutputConfig s3OutputConfig, S3ExportConfig s3ExportConfig)
+  {
+    long chunkSize = S3OutputConfig.S3_MULTIPART_UPLOAD_MIN_PART_SIZE_BYTES;
+    if (s3OutputConfig != null && s3OutputConfig.getChunkSize() != null) {
+      chunkSize = Math.max(chunkSize, s3OutputConfig.getChunkSize());
+    }
+    if (s3ExportConfig != null && s3ExportConfig.getChunkSize() != null) {
+      chunkSize = Math.max(chunkSize, s3ExportConfig.getChunkSize().getBytes());
+    }
+
+    return (int) (S3OutputConfig.S3_MULTIPART_UPLOAD_MAX_PART_SIZE_BYTES / chunkSize);

Review Comment:
   @kfaraz I had a discussion with Karan about it. The conclusion was that we don't need to block this PR for this review comment, and a follow-up task could be to add a metric for gathering data on how long are the processing threads actually blocked on this constraint. We could then tune this accordingly based on the gathered data.
   Hope that works!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1617223193


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3UploadManager.java:
##########


Review Comment:
   @LakshSingla Have made `updateChunkSizeIfGreater()` synchronized, and reverted back `maxConcurrentNumChunks` from `AtomicInteger` to `int`.
   
   All other methods don't need to be marked synchronized. `recomputeMaxConcurrentNumChunks()` is called only from the above synchronized method, and the other methods either read something, or update an atomic variable.
   
   Hope this works.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "rohangarg (via GitHub)" <gi...@apache.org>.
rohangarg commented on PR #16481:
URL: https://github.com/apache/druid/pull/16481#issuecomment-2134768598

   > The serial upload is uploading 5 MBs one-by-one, hence takes an extremely long time.
   
   Ok, although the bandwidth comes out to be around < 1 MB/sec with this calculation. So, I'm not sure about realistic nature of the test.
   
   > Actually it's the opposite. Previously the writes to the output stream (local file) had to wait for the previous chunk write+upload to finish. Now, the write doesn't have to wait for the previous uploads to finish (there's a backpressure semantic to restrict too many writes, but that's different from the current point). Please let me know if this answers the question, happy to clarify this further!
   
   Removing backpressure from the situation, previously if you have written a chunk, you'll upload it and wait for the upload to finish to start writing another chunk - and all that will happen without any extra wait time. Now, if you've written a chunk, you'll put it in an executor which might be consumed by the parts of other writers and you'll wait in the queue for your upload to be initiated. My question was that is the new behavior expected and fine?
   
   > Can you please clarify what do you mean by "leaking it into the output stream"? Sorry, I didn't get this part.
   
   I meant that the code for semaphores directly in the output stream seems more related to the behavior of the output stream rather than a property of the uploading mechanism.
   
   > But this would work only when there's a single chunk. How about when there are 5 chunks, for example? We'd have to wait for the last chunk to be processed before we can make the decision that the total number of chunks * chunkSize is small enough to go with PUT approach (and we would have to hold on to processing the initial 4 chunks until then as well). Also, supporting the PUT flow along with the multipart approach doesn't seem ideal to me, it would be one extra thing to maintain for not much benefits. That's my opinion on this atleast. Happy to discuss it further.
   
   Yes, this would only work for a single chunk. I've never recommended to extend it any further than a single chunk. Further, a multi-part request would involve 3 API calls (which are chargeable) for small uploads as compared to a PUT call which only makes 1 API call. Also, I think the multi-part API was intended more to be used as a resilient mechanism for large file uploads. So, it feels like unnecessary to me for small files. In any case, it was a suggestion for optimization and you can feel free to ignore it if you want.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1618816610


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -199,15 +221,44 @@ private void pushCurrentChunk() throws IOException
   {
     currentChunk.close();
     final Chunk chunk = currentChunk;
-    try {
-      if (chunk.length() > 0) {
-        resultsSize += chunk.length();
+    if (chunk.length() > 0) {
+      uploadManager.incrementCurrentNumChunks();
+      pendingFiles.incrementAndGet();
 
-        pushStopwatch.start();
-        pushResults.add(push(chunk));
-        pushStopwatch.stop();
-        numChunksPushed++;
+      uploadManager.submitTask(() -> {
+        try {
+          uploadChunk(chunk);
+        }
+        catch (Exception e) {
+          error = true;
+          LOG.error(e, e.getMessage());
+          throw new RuntimeException(e);
+        }
+        finally {
+          synchronized (maxChunksLock) {
+            uploadManager.decrementCurrentNumChunks();
+            maxChunksLock.notifyAll();
+          }
+          if (pendingFiles.decrementAndGet() == 0) {
+            synchronized (fileLock) {
+              fileLock.notifyAll();
+            }
+          }
+        }
+      });
+    }
+  }

Review Comment:
   @kfaraz I tried sketching out how it could look, and I don't think it'll be clean to move the upload functionality to outside of RetryableS3OutputStream file.
   
   RetryableS3OutputStream is a per-`uploadId` class. Most/All of its fields are per upload ID info.
   
   To move the upload functionality to a separate Singleton class like `S3UploadManager`, we would need a `ConcurrentHashMap: uploadId -> xyz` in the Singleton for every field like `pendingFiles`, `fileLock`, `resultsSize`, `numChunksPushed`, `pushStopwatch`, `pushResults`, `error`, `closed`, etc - which seems like an overkill to me.
   
   We'd also have to pass `S3OutputConfig`, `s3Key` and the s3 client to the Singleton class, that is, to S3UploadManager.
   
   Thoughts? Maybe we can rename `S3UploadManager` to something else indicating that it doesn't do the actual upload? The original intention was to keep and manage only the global (that is, non-uploadId-specific) components in the Singleton class, as RetryableS3OutputStream would've needed static fields to achieve it otherwise.
   
   I'll wait for your thoughts before tinkering further on this. Thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "LakshSingla (via GitHub)" <gi...@apache.org>.
LakshSingla commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1617126183


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3UploadManager.java:
##########


Review Comment:
   Note on the synchronization of the class - it is still not thread safe. Consider two threads T1 & T2 calling `updateChunkSizeIfGreater(100)` and `updateChunkSizeIfGreater(200)` respectively, and the current `chunkSize` is 50. 
   Both can simultaneously enter the method and at the end of the two operations, the `chunkSize` can be 100 instead of 200 (which was expected). 
   Making the integers atomic isn't enough since there are conditions in which multiple variables are getting updated/read. The easiest way of synchronizing it would be to leave all the variables as is (and not Atomic variables), and mark all the methods `synchronized`. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1616587787


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -103,13 +109,35 @@ public class RetryableS3OutputStream extends OutputStream
   private boolean error;
   private boolean closed;
 
+  /**
+   * A map to store number of files pending to be uploaded for a given uploadId.
+   */
+  private static final ConcurrentHashMap<String, AtomicInteger> PENDING_FILES = new ConcurrentHashMap<>();
+
+  /**
+   * A map containing the lock for a given uploadId used for notifying the main thread about the completion of
+   * s3.uploadPart() for all chunks for the uploadId, and hence starting the s3.completeMultipartUpload() for
+   * the uploadId.
+   */
+  private static final ConcurrentHashMap<String, Object> FILE_LOCK_MAP = new ConcurrentHashMap<>();
+
+  /**
+   * Semaphore to restrict the maximum number of simultaneous chunks on disk.
+   */
+  private static final int MAX_CONCURRENT_CHUNKS = 10;

Review Comment:
   @cryptoe As discussed yesterday, I've added a singleton class `S3UploadConfig` which maintains a `max(chunkSize)`, and uses that to dynamically update the `maxConcurrentNumChunks`. This is then used in `RetryableS3OutputStream` for backpressure semantics.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1629419675


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3UploadManager.java:
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.inject.Inject;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.common.RetryUtils;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.storage.s3.S3Utils;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+import org.apache.druid.utils.RuntimeInfo;
+
+import java.io.File;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
+
+/**
+ * This class manages uploading files to S3 in chunks, while ensuring that the
+ * number of chunks currently present on local disk does not exceed a specific limit.
+ */
+@ManageLifecycle
+public class S3UploadManager
+{
+  private final ExecutorService uploadExecutor;
+
+  private static final Logger log = new Logger(S3UploadManager.class);
+
+  @Inject
+  public S3UploadManager(S3OutputConfig s3OutputConfig, S3ExportConfig s3ExportConfig, RuntimeInfo runtimeInfo)
+  {
+    int poolSize = Math.max(4, runtimeInfo.getAvailableProcessors());
+    int maxNumConcurrentChunks = computeMaxNumConcurrentChunks(s3OutputConfig, s3ExportConfig);
+    this.uploadExecutor = createExecutorService(poolSize, maxNumConcurrentChunks);
+    log.info("Initialized executor service for S3 multipart upload with pool size [%d] and work queue capacity [%d]",
+             poolSize, maxNumConcurrentChunks);
+  }
+
+  /**
+   * Computes the maximum number of concurrent chunks for an S3 multipart upload.
+   * We want to determine the maximum number of concurrent chunks on disk based on the maximum value of chunkSize
+   * between the 2 configs: S3OutputConfig and S3ExportConfig.
+   *
+   * @param s3OutputConfig  The S3 output configuration, which may specify a custom chunk size.
+   * @param s3ExportConfig  The S3 export configuration, which may also specify a custom chunk size.
+   * @return The maximum number of concurrent chunks.
+   */
+  @VisibleForTesting
+  int computeMaxNumConcurrentChunks(S3OutputConfig s3OutputConfig, S3ExportConfig s3ExportConfig)
+  {
+    long chunkSize = S3OutputConfig.S3_MULTIPART_UPLOAD_MIN_PART_SIZE_BYTES;
+    if (s3OutputConfig != null && s3OutputConfig.getChunkSize() != null) {
+      chunkSize = Math.max(chunkSize, s3OutputConfig.getChunkSize());
+    }
+    if (s3ExportConfig != null && s3ExportConfig.getChunkSize() != null) {
+      chunkSize = Math.max(chunkSize, s3ExportConfig.getChunkSize().getBytes());
+    }
+
+    return (int) (S3OutputConfig.S3_MULTIPART_UPLOAD_MAX_PART_SIZE_BYTES / chunkSize);

Review Comment:
   Sounds good. 👍🏻 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1609581399


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -199,15 +211,47 @@ private void pushCurrentChunk() throws IOException
   {
     currentChunk.close();
     final Chunk chunk = currentChunk;
-    try {
-      if (chunk.length() > 0) {
-        resultsSize += chunk.length();
+    if (chunk.length() > 0) {
+      try {
+        SEMAPHORE.acquire(); // Acquire a permit from the semaphore

Review Comment:
   No, you can retain the `wait` and `notify` flow, but the condition that you are waiting on becomes the available disk space, which is what we really care about in the first place (unless there is any other condition that we want to wait on).
   
   It could look something like this:
   
   In `write()`:
   ```
   synchronized(diskSpaceAvailableLock) {
     while (availableDiskSpace < sizeToWrite) {
       diskSpaceAvailableLock.wait();
     }
   }
   ```
   
   After upload and delete:
   ```
   synchronized(diskSpaceAvailable) {
      // update availableDiskSpace
     diskspaceAvailableLock.notifyAll();
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1616589115


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -103,22 +107,32 @@ public class RetryableS3OutputStream extends OutputStream
   private boolean error;
   private boolean closed;
 
-  public RetryableS3OutputStream(
-      S3OutputConfig config,
-      ServerSideEncryptingAmazonS3 s3,
-      String s3Key
-  ) throws IOException
-  {
+  /**
+   * An atomic counter to store number of files pending to be uploaded for the particular uploadId.
+   */
+  private final AtomicInteger pendingFiles = new AtomicInteger(0);
 
-    this(config, s3, s3Key, true);
-  }
+  /**
+   * A lock used for notifying the main thread about the completion of s3.uploadPart() for all chunks
+   * and hence starting the s3.completeMultipartUpload() for the uploadId.
+   */
+  private final Object fileLock = new Object();
+
+  /**
+   * Semaphore to restrict the maximum number of simultaneous chunks on disk.
+   */
+  private static final int MAX_CONCURRENT_CHUNKS = 10;
+  private static final Semaphore SEMAPHORE = new Semaphore(MAX_CONCURRENT_CHUNKS);
 
-  @VisibleForTesting
-  protected RetryableS3OutputStream(
+  /**
+   * Threadpool used for uploading the chunks asynchronously.
+   */
+  private static final ExecutorService UPLOAD_EXECUTOR = Execs.multiThreaded(10, "UploadThreadPool-%d");

Review Comment:
   Have injected ExecutorService created via ScheduledExecutorFactory. Hope this works.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1616615904


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3UploadConfig.java:
##########
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import org.apache.druid.guice.LazySingleton;
+import org.apache.druid.java.util.common.HumanReadableBytes;
+import org.apache.druid.java.util.common.logger.Logger;
+
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * This class manages the configuration for uploading files to S3 in chunks.
+ * It tracks the current number of chunks written to local disk concurrently and ensures that the
+ * maximum number of chunks on disk does not exceed a specified limit at any given point in time.
+ */
+@LazySingleton
+public class S3UploadConfig
+{
+  /**
+   * The maximum chunk size based on injected values for {@link S3OutputConfig} and {@link S3ExportConfig} used for computing maximum number of chunks to save on disk at any given point in time.
+   * It is initialized to 5 MiB which is the minimum chunk size possible, denoted by {@link S3OutputConfig#S3_MULTIPART_UPLOAD_MIN_PART_SIZE_BYTES}.
+   */
+  private long chunkSize = new HumanReadableBytes("5MiB").getBytes();
+
+  /**
+   * An atomic counter to track the current number of chunks saved to local disk.
+   */
+  private final AtomicInteger currentNumChunks = new AtomicInteger(0);

Review Comment:
   @LakshSingla The naming of the class could be better, please let me know if you have any alternate naming suggestions.
   
   The intention here is to have a Singleton class which keeps a global state (across all instances of RetryableS3OutputStream) of how many chunks have been written to disk right now. It also updates the maxConcurrentNumChunks according to the chunk size.
   
   The other option to achieve this would have been to have a static counter in RetryableS3OutputStream layer, but Karan pointed out that it's not a good practice, and hence we should go with a Singleton class.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "LakshSingla (via GitHub)" <gi...@apache.org>.
LakshSingla commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1616595510


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3UploadConfig.java:
##########
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import org.apache.druid.guice.LazySingleton;
+import org.apache.druid.java.util.common.HumanReadableBytes;
+import org.apache.druid.java.util.common.logger.Logger;
+
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * This class manages the configuration for uploading files to S3 in chunks.
+ * It tracks the current number of chunks written to local disk concurrently and ensures that the
+ * maximum number of chunks on disk does not exceed a specified limit at any given point in time.
+ */
+@LazySingleton
+public class S3UploadConfig
+{
+  /**
+   * The maximum chunk size based on injected values for {@link S3OutputConfig} and {@link S3ExportConfig} used for computing maximum number of chunks to save on disk at any given point in time.
+   * It is initialized to 5 MiB which is the minimum chunk size possible, denoted by {@link S3OutputConfig#S3_MULTIPART_UPLOAD_MIN_PART_SIZE_BYTES}.
+   */
+  private long chunkSize = new HumanReadableBytes("5MiB").getBytes();
+
+  /**
+   * An atomic counter to track the current number of chunks saved to local disk.
+   */
+  private final AtomicInteger currentNumChunks = new AtomicInteger(0);
+
+  /**
+   * The maximum number of chunks that can be saved to local disk concurrently.
+   * This value is recalculated when the chunk size is updated in {@link #updateChunkSizeIfGreater(long)}.
+   */
+  private int maxConcurrentNumChunks = 100;

Review Comment:
   The multithreading model for this class is unclear to me. The class isn't thread-safe, however, `currentNumChunks` is marked as an atomic integer.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1627284471


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3UploadManager.java:
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.inject.Inject;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.common.RetryUtils;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.storage.s3.S3Utils;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+import org.apache.druid.utils.RuntimeInfo;
+
+import java.io.File;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
+
+/**
+ * This class manages uploading files to S3 in chunks, while ensuring that the
+ * number of chunks currently present on local disk does not exceed a specific limit.
+ */
+@ManageLifecycle
+public class S3UploadManager
+{
+  private final ExecutorService uploadExecutor;
+
+  private static final Logger log = new Logger(S3UploadManager.class);
+
+  @Inject
+  public S3UploadManager(S3OutputConfig s3OutputConfig, S3ExportConfig s3ExportConfig, RuntimeInfo runtimeInfo)
+  {
+    int poolSize = Math.max(4, runtimeInfo.getAvailableProcessors());
+    int maxNumConcurrentChunks = computeMaxNumConcurrentChunks(s3OutputConfig, s3ExportConfig);
+    this.uploadExecutor = createExecutorService(poolSize, maxNumConcurrentChunks);
+    log.info("Initialized executor service for S3 multipart upload with pool size [%d] and work queue capacity [%d]",
+             poolSize, maxNumConcurrentChunks);
+  }
+
+  /**
+   * Computes the maximum number of concurrent chunks for an S3 multipart upload.
+   * We want to determine the maximum number of concurrent chunks on disk based on the maximum value of chunkSize
+   * between the 2 configs: S3OutputConfig and S3ExportConfig.
+   *
+   * @param s3OutputConfig  The S3 output configuration, which may specify a custom chunk size.
+   * @param s3ExportConfig  The S3 export configuration, which may also specify a custom chunk size.
+   * @return The maximum number of concurrent chunks.
+   */
+  @VisibleForTesting
+  int computeMaxNumConcurrentChunks(S3OutputConfig s3OutputConfig, S3ExportConfig s3ExportConfig)
+  {
+    long chunkSize = S3OutputConfig.S3_MULTIPART_UPLOAD_MIN_PART_SIZE_BYTES;
+    if (s3OutputConfig != null && s3OutputConfig.getChunkSize() != null) {
+      chunkSize = Math.max(chunkSize, s3OutputConfig.getChunkSize());
+    }
+    if (s3ExportConfig != null && s3ExportConfig.getChunkSize() != null) {
+      chunkSize = Math.max(chunkSize, s3ExportConfig.getChunkSize().getBytes());
+    }
+
+    return (int) (S3OutputConfig.S3_MULTIPART_UPLOAD_MAX_PART_SIZE_BYTES / chunkSize);

Review Comment:
   @kfaraz I was trying to do it similar to how it's done in `WorkerStorageParameters`. IIUC, in `WorkerStorageParameters`, we have a hard-coded size of 1 GB (`MINIMUM_SUPER_SORTER_TMP_STORAGE_BYTES = 1_000_000_000L;`) which is used instead of computing the overall available disk space.
   
   Also, I don't think we should use the overall disk size available for this calculation, and should rather have a hard-limit? I chose 5 GB as that's the maximum allowed chunk size by S3. In the default case, chunk size is 100 MB, so this means around 50 chunks allowed on disk at once, which seems fine?
   
   Thoughts? cc: @cryptoe 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on PR #16481:
URL: https://github.com/apache/druid/pull/16481#issuecomment-2152891872

   Happy to help, @Akshat-Jain ! 🙂 
   Thank you for the changes and for the discussion!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1616512750


##########
extensions-core/s3-extensions/src/test/java/org/apache/druid/storage/s3/S3StorageConnectorProviderTest.java:
##########
@@ -146,6 +148,10 @@ public void configure(Binder binder)
             .addValue(
                 ServerSideEncryptingAmazonS3.class,
                 new ServerSideEncryptingAmazonS3(null, new NoopServerSideEncryption())
+            )
+            .addValue(
+                ExecutorService.class,
+                Execs.multiThreaded(10, "UploadThreadPool-%d")

Review Comment:
   Why do you need 10 threads in a test?



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3StorageDruidModule.java:
##########
@@ -182,4 +186,13 @@ public Supplier<ServerSideEncryptingAmazonS3> getAmazonS3ClientSupplier(
   {
     return Suppliers.memoize(serverSideEncryptingAmazonS3Builder::build);
   }
+
+  @Provides
+  @LazySingleton
+  @Named("S3UploadThreadPool")
+  public ExecutorService getUploadPoolExecutorService(ScheduledExecutorFactory scheduledExecutorFactory, RuntimeInfo runtimeInfo)
+  {
+    int poolSize = Math.max(4, 3 * runtimeInfo.getAvailableProcessors());

Review Comment:
   `3 x numProcessors` seems too many as there are going to be other threads too.
   I think just `numProcessors` is high enough.



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3StorageDruidModule.java:
##########
@@ -182,4 +186,13 @@ public Supplier<ServerSideEncryptingAmazonS3> getAmazonS3ClientSupplier(
   {
     return Suppliers.memoize(serverSideEncryptingAmazonS3Builder::build);
   }
+
+  @Provides
+  @LazySingleton
+  @Named("S3UploadThreadPool")

Review Comment:
   This string should be a constant somewhere.



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -199,15 +229,44 @@ private void pushCurrentChunk() throws IOException
   {
     currentChunk.close();
     final Chunk chunk = currentChunk;
-    try {
-      if (chunk.length() > 0) {
-        resultsSize += chunk.length();
+    if (chunk.length() > 0) {
+      s3UploadConfig.incrementCurrentNumChunks();

Review Comment:
   +1, I think this information can be tracked in a singleton called `S3UploadManager` or something. It is also a little weird to inject the `ExecutorService` everywhere. That executor service can live inside `S3UploadManager` so that we have a single place to manager the lifecycle of the executor. Also, individual `RetryableS3OutputStream` would submit jobs to the `S3UploadManager`.
   It would be this manager that is injected everywhere instead of the raw `ExecutorService`.



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3StorageDruidModule.java:
##########
@@ -182,4 +186,13 @@ public Supplier<ServerSideEncryptingAmazonS3> getAmazonS3ClientSupplier(
   {
     return Suppliers.memoize(serverSideEncryptingAmazonS3Builder::build);
   }
+
+  @Provides
+  @LazySingleton
+  @Named("S3UploadThreadPool")
+  public ExecutorService getUploadPoolExecutorService(ScheduledExecutorFactory scheduledExecutorFactory, RuntimeInfo runtimeInfo)

Review Comment:
   ```suggestion
     public ExecutorService getUploadExecutorService(ScheduledExecutorFactory scheduledExecutorFactory, RuntimeInfo runtimeInfo)
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1616615880


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -199,15 +229,44 @@ private void pushCurrentChunk() throws IOException
   {
     currentChunk.close();
     final Chunk chunk = currentChunk;
-    try {
-      if (chunk.length() > 0) {
-        resultsSize += chunk.length();
+    if (chunk.length() > 0) {
+      s3UploadConfig.incrementCurrentNumChunks();

Review Comment:
   +1, I think this information can be tracked in a singleton called `S3UploadManager` or something. It is also a little weird to inject the `ExecutorService` everywhere. That executor service can live inside `S3UploadManager` so that we have a single place to manage the lifecycle of the executor. Also, individual `RetryableS3OutputStream` would submit jobs to the `S3UploadManager`.
   It would be this manager that is injected everywhere instead of the raw `ExecutorService`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1616976275


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3UploadConfig.java:
##########
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import org.apache.druid.guice.LazySingleton;
+import org.apache.druid.java.util.common.HumanReadableBytes;
+import org.apache.druid.java.util.common.logger.Logger;
+
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * This class manages the configuration for uploading files to S3 in chunks.
+ * It tracks the current number of chunks written to local disk concurrently and ensures that the
+ * maximum number of chunks on disk does not exceed a specified limit at any given point in time.
+ */
+@LazySingleton
+public class S3UploadConfig
+{
+  /**
+   * The maximum chunk size based on injected values for {@link S3OutputConfig} and {@link S3ExportConfig} used for computing maximum number of chunks to save on disk at any given point in time.
+   * It is initialized to 5 MiB which is the minimum chunk size possible, denoted by {@link S3OutputConfig#S3_MULTIPART_UPLOAD_MIN_PART_SIZE_BYTES}.
+   */
+  private long chunkSize = new HumanReadableBytes("5MiB").getBytes();
+
+  /**
+   * An atomic counter to track the current number of chunks saved to local disk.
+   */
+  private final AtomicInteger currentNumChunks = new AtomicInteger(0);
+
+  /**
+   * The maximum number of chunks that can be saved to local disk concurrently.
+   * This value is recalculated when the chunk size is updated in {@link #updateChunkSizeIfGreater(long)}.
+   */
+  private int maxConcurrentNumChunks = 100;

Review Comment:
   @LakshSingla Changed it into an atomic integer.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "LakshSingla (via GitHub)" <gi...@apache.org>.
LakshSingla commented on PR #16481:
URL: https://github.com/apache/druid/pull/16481#issuecomment-2132663033

   Were the tests run on a middle manager or an indexer? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1615463749


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -103,13 +109,35 @@ public class RetryableS3OutputStream extends OutputStream
   private boolean error;
   private boolean closed;
 
+  /**
+   * A map to store number of files pending to be uploaded for a given uploadId.
+   */
+  private static final ConcurrentHashMap<String, AtomicInteger> PENDING_FILES = new ConcurrentHashMap<>();
+
+  /**
+   * A map containing the lock for a given uploadId used for notifying the main thread about the completion of
+   * s3.uploadPart() for all chunks for the uploadId, and hence starting the s3.completeMultipartUpload() for
+   * the uploadId.
+   */
+  private static final ConcurrentHashMap<String, Object> FILE_LOCK_MAP = new ConcurrentHashMap<>();
+
+  /**
+   * Semaphore to restrict the maximum number of simultaneous chunks on disk.
+   */
+  private static final int MAX_CONCURRENT_CHUNKS = 10;

Review Comment:
   @cryptoe I'm a bit confused on this. The `maxConcurrentChunks` should be the same across all instances of `RetryableS3OutputStream` (IIUC??), but `SIze of the chunk` is a per-instance property for RetryableS3OutputStream.
   
   Similar confusion for adding a dependency on `Number of upload threads`.
   
   Appreciate your inputs, thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1616625803


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -271,50 +328,72 @@ public void close() throws IOException
       // This should be emitted as a metric
       LOG.info(
           "Pushed total [%d] parts containing [%d] bytes in [%d]ms.",
-          numChunksPushed,
-          resultsSize,
+          numChunksPushed.get(),
+          resultsSize.get(),
           pushStopwatch.elapsed(TimeUnit.MILLISECONDS)
       );
     });
 
-    closer.register(() -> org.apache.commons.io.FileUtils.forceDelete(chunkStorePath));
+    try (Closer ignored = closer) {
+      if (!error) {
+        pushCurrentChunk();
+        completeMultipartUpload();
+      }
+    }
+  }
 
-    closer.register(() -> {
-      try {
-        if (resultsSize > 0 && isAllPushSucceeded()) {
-          RetryUtils.retry(
-              () -> s3.completeMultipartUpload(
-                  new CompleteMultipartUploadRequest(config.getBucket(), s3Key, uploadId, pushResults)
-              ),
-              S3Utils.S3RETRY,
-              config.getMaxRetry()
-          );
-        } else {
-          RetryUtils.retry(
-              () -> {
-                s3.cancelMultiPartUpload(new AbortMultipartUploadRequest(config.getBucket(), s3Key, uploadId));
-                return null;
-              },
-              S3Utils.S3RETRY,
-              config.getMaxRetry()
-          );
+  private void completeMultipartUpload()
+  {
+    synchronized (fileLock) {
+      while (pendingFiles.get() > 0) {
+        try {
+          LOG.info("Waiting for lock for completing multipart task for uploadId [%s].", uploadId);

Review Comment:
   Just to clarify: This is one log line per file, not per "part".
   
   Each worker uploads its stage output (comprising of several parts, less than 10 in whatever testing I did) and a success file.
   
   So if we have 10 workers, 3 stages, 5 parts per stage, this line will be roughly logged around 10*3*5 + 10*3*1 == 150 part files + 30 success files == 180 times for such a query.
   
   This doesn't seem like a huge number to me. I felt it might help in some debugging situations down the line. Thoughts?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "LakshSingla (via GitHub)" <gi...@apache.org>.
LakshSingla commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1616633265


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3UploadConfig.java:
##########
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import org.apache.druid.guice.LazySingleton;
+import org.apache.druid.java.util.common.HumanReadableBytes;
+import org.apache.druid.java.util.common.logger.Logger;
+
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * This class manages the configuration for uploading files to S3 in chunks.
+ * It tracks the current number of chunks written to local disk concurrently and ensures that the
+ * maximum number of chunks on disk does not exceed a specified limit at any given point in time.
+ */
+@LazySingleton
+public class S3UploadConfig
+{
+  /**
+   * The maximum chunk size based on injected values for {@link S3OutputConfig} and {@link S3ExportConfig} used for computing maximum number of chunks to save on disk at any given point in time.
+   * It is initialized to 5 MiB which is the minimum chunk size possible, denoted by {@link S3OutputConfig#S3_MULTIPART_UPLOAD_MIN_PART_SIZE_BYTES}.
+   */
+  private long chunkSize = new HumanReadableBytes("5MiB").getBytes();
+
+  /**
+   * An atomic counter to track the current number of chunks saved to local disk.
+   */
+  private final AtomicInteger currentNumChunks = new AtomicInteger(0);
+
+  /**
+   * The maximum number of chunks that can be saved to local disk concurrently.
+   * This value is recalculated when the chunk size is updated in {@link #updateChunkSizeIfGreater(long)}.
+   */
+  private int maxConcurrentNumChunks = 100;

Review Comment:
   As a reader, this isn't clear at all. The class works for the current use case, however it isn't thread-safe. This is pretty error-prone if multiple threads decide to call `updateChunkSizeIfGreater` later, and we are expecting this class to be called with multiple threads. I'd either make it thread-safe throughout or make it thread-unsafe and expect the callers to do the synchronization. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1609645900


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -103,22 +107,32 @@ public class RetryableS3OutputStream extends OutputStream
   private boolean error;
   private boolean closed;
 
-  public RetryableS3OutputStream(
-      S3OutputConfig config,
-      ServerSideEncryptingAmazonS3 s3,
-      String s3Key
-  ) throws IOException
-  {
+  /**
+   * An atomic counter to store number of files pending to be uploaded for the particular uploadId.
+   */
+  private final AtomicInteger pendingFiles = new AtomicInteger(0);
 
-    this(config, s3, s3Key, true);
-  }
+  /**
+   * A lock used for notifying the main thread about the completion of s3.uploadPart() for all chunks
+   * and hence starting the s3.completeMultipartUpload() for the uploadId.
+   */
+  private final Object fileLock = new Object();
+
+  /**
+   * Semaphore to restrict the maximum number of simultaneous chunks on disk.
+   */
+  private static final int MAX_CONCURRENT_CHUNKS = 10;
+  private static final Semaphore SEMAPHORE = new Semaphore(MAX_CONCURRENT_CHUNKS);
 
-  @VisibleForTesting
-  protected RetryableS3OutputStream(
+  /**
+   * Threadpool used for uploading the chunks asynchronously.
+   */
+  private static final ExecutorService UPLOAD_EXECUTOR = Execs.multiThreaded(10, "UploadThreadPool-%d");

Review Comment:
   Thanks for the explanation, makes sense!
   
   > All Druid services are injected with a ScheduledExecutorFactory which can be used for creating an executor
   
   I don't see any occurrence of this in the `druid-s3-extensions`? Can you suggest how to use this?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1626257336


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3UploadManager.java:
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.inject.Inject;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.common.RetryUtils;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.storage.s3.S3Utils;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+import org.apache.druid.utils.RuntimeInfo;
+
+import java.io.File;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
+
+/**
+ * This class manages uploading files to S3 in chunks, while ensuring that the
+ * number of chunks currently present on local disk does not exceed a specific limit.
+ */
+@ManageLifecycle
+public class S3UploadManager
+{
+  private final ExecutorService uploadExecutor;
+
+  private static final Logger log = new Logger(S3UploadManager.class);
+
+  @Inject
+  public S3UploadManager(S3OutputConfig s3OutputConfig, S3ExportConfig s3ExportConfig, RuntimeInfo runtimeInfo)
+  {
+    int poolSize = Math.max(4, runtimeInfo.getAvailableProcessors());
+    int maxNumConcurrentChunks = computeMaxNumConcurrentChunks(s3OutputConfig, s3ExportConfig);
+    this.uploadExecutor = createExecutorService(poolSize, maxNumConcurrentChunks);
+    log.info("Initialized executor service for S3 multipart upload with pool size [%d] and work queue capacity [%d]",
+             poolSize, maxNumConcurrentChunks);
+  }
+
+  /**
+   * Computes the maximum number of concurrent chunks for an S3 multipart upload.
+   * We want to determine the maximum number of concurrent chunks on disk based on the maximum value of chunkSize
+   * between the 2 configs: S3OutputConfig and S3ExportConfig.
+   *
+   * @param s3OutputConfig  The S3 output configuration, which may specify a custom chunk size.
+   * @param s3ExportConfig  The S3 export configuration, which may also specify a custom chunk size.
+   * @return The maximum number of concurrent chunks.
+   */
+  @VisibleForTesting
+  int computeMaxNumConcurrentChunks(S3OutputConfig s3OutputConfig, S3ExportConfig s3ExportConfig)
+  {
+    long chunkSize = S3OutputConfig.S3_MULTIPART_UPLOAD_MIN_PART_SIZE_BYTES;
+    if (s3OutputConfig != null && s3OutputConfig.getChunkSize() != null) {
+      chunkSize = Math.max(chunkSize, s3OutputConfig.getChunkSize());
+    }
+    if (s3ExportConfig != null && s3ExportConfig.getChunkSize() != null) {
+      chunkSize = Math.max(chunkSize, s3ExportConfig.getChunkSize().getBytes());
+    }
+
+    return (int) (S3OutputConfig.S3_MULTIPART_UPLOAD_MAX_PART_SIZE_BYTES / chunkSize);

Review Comment:
   Shouldn't the numerator be the overall disk size available for keeping chunks?
   The value used here `S3OutputConfig.S3_MULTIPART_UPLOAD_MAX_PART_SIZE_BYTES` is 5gb which just seems to be the maximum object size limit imposed by S3.



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3UploadManager.java:
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.inject.Inject;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.common.RetryUtils;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.storage.s3.S3Utils;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+import org.apache.druid.utils.RuntimeInfo;
+
+import java.io.File;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
+
+/**
+ * This class manages uploading files to S3 in chunks, while ensuring that the
+ * number of chunks currently present on local disk does not exceed a specific limit.
+ */
+@ManageLifecycle
+public class S3UploadManager
+{
+  private final ExecutorService uploadExecutor;
+
+  private static final Logger log = new Logger(S3UploadManager.class);
+
+  @Inject
+  public S3UploadManager(S3OutputConfig s3OutputConfig, S3ExportConfig s3ExportConfig, RuntimeInfo runtimeInfo)
+  {
+    int poolSize = Math.max(4, runtimeInfo.getAvailableProcessors());
+    int maxNumConcurrentChunks = computeMaxNumConcurrentChunks(s3OutputConfig, s3ExportConfig);
+    this.uploadExecutor = createExecutorService(poolSize, maxNumConcurrentChunks);
+    log.info("Initialized executor service for S3 multipart upload with pool size [%d] and work queue capacity [%d]",
+             poolSize, maxNumConcurrentChunks);
+  }
+
+  /**
+   * Computes the maximum number of concurrent chunks for an S3 multipart upload.
+   * We want to determine the maximum number of concurrent chunks on disk based on the maximum value of chunkSize
+   * between the 2 configs: S3OutputConfig and S3ExportConfig.
+   *
+   * @param s3OutputConfig  The S3 output configuration, which may specify a custom chunk size.
+   * @param s3ExportConfig  The S3 export configuration, which may also specify a custom chunk size.
+   * @return The maximum number of concurrent chunks.
+   */
+  @VisibleForTesting
+  int computeMaxNumConcurrentChunks(S3OutputConfig s3OutputConfig, S3ExportConfig s3ExportConfig)
+  {
+    long chunkSize = S3OutputConfig.S3_MULTIPART_UPLOAD_MIN_PART_SIZE_BYTES;

Review Comment:
   ```suggestion
       long maxChunkSize = S3OutputConfig.S3_MULTIPART_UPLOAD_MIN_PART_SIZE_BYTES;
   ```



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -269,52 +213,74 @@ public void close() throws IOException
     // Closeables are closed in LIFO order
     closer.register(() -> {
       // This should be emitted as a metric
+      long totalChunkSize = (currentChunk.id - 1) * chunkSize + currentChunk.length();
       LOG.info(
           "Pushed total [%d] parts containing [%d] bytes in [%d]ms.",
-          numChunksPushed,
-          resultsSize,
+          futures.size(),
+          totalChunkSize,
           pushStopwatch.elapsed(TimeUnit.MILLISECONDS)
       );
     });
 
-    closer.register(() -> org.apache.commons.io.FileUtils.forceDelete(chunkStorePath));
+    try (Closer ignored = closer) {
+      if (!error) {
+        pushCurrentChunk();
+        completeMultipartUpload();
+      }
+    }
+  }
 
-    closer.register(() -> {
+  private void completeMultipartUpload()
+  {
+    for (Future<UploadPartResult> future : futures) {
       try {
-        if (resultsSize > 0 && isAllPushSucceeded()) {
-          RetryUtils.retry(
-              () -> s3.completeMultipartUpload(
-                  new CompleteMultipartUploadRequest(config.getBucket(), s3Key, uploadId, pushResults)
-              ),
-              S3Utils.S3RETRY,
-              config.getMaxRetry()
-          );
-        } else {
-          RetryUtils.retry(
-              () -> {
-                s3.cancelMultiPartUpload(new AbortMultipartUploadRequest(config.getBucket(), s3Key, uploadId));
-                return null;
-              },
-              S3Utils.S3RETRY,
-              config.getMaxRetry()
-          );
-        }
+        UploadPartResult result = future.get();
+        pushResults.add(result.getPartETag());
       }
       catch (Exception e) {
-        throw new IOException(e);
+        error = true;
+        LOG.error(e, "Error in uploading part for upload ID [%s]", uploadId);
+        break;
       }
-    });
+    }
 
-    try (Closer ignored = closer) {
-      if (!error) {
-        pushCurrentChunk();
+    try {
+      if (isAllPushSucceeded()) {

Review Comment:
   I don't think `isAllPushSucceeded()` needs to be a separate method anymore as all the relevant fields i.e `error` and `pushResults` are being updated in this method itself.
   
   On that note, AFAICT, `pushResults` need not be a member field anymore and can just be a local variable in this method.
   
   So, this if-else will simply reduce to:
   
   ```
   if (error)
       cancel multipart upload
   else
       finish multipart upload
   ```



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -103,27 +94,28 @@ public class RetryableS3OutputStream extends OutputStream
   private boolean error;
   private boolean closed;
 
-  public RetryableS3OutputStream(
-      S3OutputConfig config,
-      ServerSideEncryptingAmazonS3 s3,
-      String s3Key
-  ) throws IOException
-  {
+  /**
+   * Helper class for calculating maximum number of simultaneous chunks allowed on local disk.
+   */
+  private final S3UploadManager uploadManager;
 
-    this(config, s3, s3Key, true);
-  }
+  /**
+   * A list of futures to allow us to wait for completion of all uploadPart() calls
+   * before hitting {@link ServerSideEncryptingAmazonS3#completeMultipartUpload(CompleteMultipartUploadRequest)}.
+   */
+  private final List<Future<UploadPartResult>> futures = new ArrayList<>();

Review Comment:
   This could even be a map from `Chunk` to `Future`



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -269,52 +213,74 @@ public void close() throws IOException
     // Closeables are closed in LIFO order
     closer.register(() -> {
       // This should be emitted as a metric
+      long totalChunkSize = (currentChunk.id - 1) * chunkSize + currentChunk.length();
       LOG.info(
           "Pushed total [%d] parts containing [%d] bytes in [%d]ms.",
-          numChunksPushed,
-          resultsSize,
+          futures.size(),
+          totalChunkSize,
           pushStopwatch.elapsed(TimeUnit.MILLISECONDS)
       );
     });
 
-    closer.register(() -> org.apache.commons.io.FileUtils.forceDelete(chunkStorePath));
+    try (Closer ignored = closer) {
+      if (!error) {
+        pushCurrentChunk();
+        completeMultipartUpload();
+      }
+    }
+  }
 
-    closer.register(() -> {
+  private void completeMultipartUpload()
+  {
+    for (Future<UploadPartResult> future : futures) {
       try {
-        if (resultsSize > 0 && isAllPushSucceeded()) {
-          RetryUtils.retry(
-              () -> s3.completeMultipartUpload(
-                  new CompleteMultipartUploadRequest(config.getBucket(), s3Key, uploadId, pushResults)
-              ),
-              S3Utils.S3RETRY,
-              config.getMaxRetry()
-          );
-        } else {
-          RetryUtils.retry(
-              () -> {
-                s3.cancelMultiPartUpload(new AbortMultipartUploadRequest(config.getBucket(), s3Key, uploadId));
-                return null;
-              },
-              S3Utils.S3RETRY,
-              config.getMaxRetry()
-          );
-        }
+        UploadPartResult result = future.get();
+        pushResults.add(result.getPartETag());
       }
       catch (Exception e) {
-        throw new IOException(e);
+        error = true;
+        LOG.error(e, "Error in uploading part for upload ID [%s]", uploadId);
+        break;
       }
-    });
+    }
 
-    try (Closer ignored = closer) {
-      if (!error) {
-        pushCurrentChunk();
+    try {
+      if (isAllPushSucceeded()) {
+        RetryUtils.retry(
+            () -> s3.completeMultipartUpload(
+                new CompleteMultipartUploadRequest(config.getBucket(), s3Key, uploadId, pushResults)
+            ),
+            S3Utils.S3RETRY,
+            config.getMaxRetry()
+        );
+      } else {
+        RetryUtils.retry(
+            () -> {
+              s3.cancelMultiPartUpload(new AbortMultipartUploadRequest(config.getBucket(), s3Key, uploadId));
+              return null;
+            },
+            S3Utils.S3RETRY,
+            config.getMaxRetry()
+        );
+      }
+    }
+    catch (Exception e) {
+      throw new RuntimeException(e);
+    }
+    finally {
+      try {
+        org.apache.commons.io.FileUtils.forceDelete(chunkStorePath);
+        pushStopwatch.stop();

Review Comment:
   It would be better to do this in `close()`. No particular reason, just for symmetry. The stopwatch is started in the constructor so it should be stopped in the last action performed on the object i.e. `close()`. In fact, the stopwatch doesn't even need to be stopped as there are no more time consuming operations after this anyway.



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3UploadManager.java:
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.inject.Inject;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.common.RetryUtils;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.storage.s3.S3Utils;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+import org.apache.druid.utils.RuntimeInfo;
+
+import java.io.File;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
+
+/**
+ * This class manages uploading files to S3 in chunks, while ensuring that the
+ * number of chunks currently present on local disk does not exceed a specific limit.
+ */
+@ManageLifecycle
+public class S3UploadManager
+{
+  private final ExecutorService uploadExecutor;
+
+  private static final Logger log = new Logger(S3UploadManager.class);
+
+  @Inject
+  public S3UploadManager(S3OutputConfig s3OutputConfig, S3ExportConfig s3ExportConfig, RuntimeInfo runtimeInfo)
+  {
+    int poolSize = Math.max(4, runtimeInfo.getAvailableProcessors());
+    int maxNumConcurrentChunks = computeMaxNumConcurrentChunks(s3OutputConfig, s3ExportConfig);
+    this.uploadExecutor = createExecutorService(poolSize, maxNumConcurrentChunks);
+    log.info("Initialized executor service for S3 multipart upload with pool size [%d] and work queue capacity [%d]",
+             poolSize, maxNumConcurrentChunks);
+  }
+
+  /**
+   * Computes the maximum number of concurrent chunks for an S3 multipart upload.
+   * We want to determine the maximum number of concurrent chunks on disk based on the maximum value of chunkSize
+   * between the 2 configs: S3OutputConfig and S3ExportConfig.
+   *
+   * @param s3OutputConfig  The S3 output configuration, which may specify a custom chunk size.
+   * @param s3ExportConfig  The S3 export configuration, which may also specify a custom chunk size.
+   * @return The maximum number of concurrent chunks.
+   */
+  @VisibleForTesting
+  int computeMaxNumConcurrentChunks(S3OutputConfig s3OutputConfig, S3ExportConfig s3ExportConfig)
+  {
+    long chunkSize = S3OutputConfig.S3_MULTIPART_UPLOAD_MIN_PART_SIZE_BYTES;
+    if (s3OutputConfig != null && s3OutputConfig.getChunkSize() != null) {
+      chunkSize = Math.max(chunkSize, s3OutputConfig.getChunkSize());
+    }
+    if (s3ExportConfig != null && s3ExportConfig.getChunkSize() != null) {
+      chunkSize = Math.max(chunkSize, s3ExportConfig.getChunkSize().getBytes());
+    }
+
+    return (int) (S3OutputConfig.S3_MULTIPART_UPLOAD_MAX_PART_SIZE_BYTES / chunkSize);
+  }
+
+  /**
+   * Queues a chunk of a file for upload to S3 as part of a multipart upload.
+   */
+  public Future<UploadPartResult> queueChunkForUpload(ServerSideEncryptingAmazonS3 s3Client, String key, int chunkNumber, File chunkFile, String uploadId, S3OutputConfig config)

Review Comment:
   Please put the method args in separate lines.



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3UploadManager.java:
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.inject.Inject;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.common.RetryUtils;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.storage.s3.S3Utils;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+import org.apache.druid.utils.RuntimeInfo;
+
+import java.io.File;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
+
+/**
+ * This class manages uploading files to S3 in chunks, while ensuring that the
+ * number of chunks currently present on local disk does not exceed a specific limit.
+ */
+@ManageLifecycle
+public class S3UploadManager
+{
+  private final ExecutorService uploadExecutor;
+
+  private static final Logger log = new Logger(S3UploadManager.class);
+
+  @Inject
+  public S3UploadManager(S3OutputConfig s3OutputConfig, S3ExportConfig s3ExportConfig, RuntimeInfo runtimeInfo)
+  {
+    int poolSize = Math.max(4, runtimeInfo.getAvailableProcessors());
+    int maxNumConcurrentChunks = computeMaxNumConcurrentChunks(s3OutputConfig, s3ExportConfig);

Review Comment:
   ```suggestion
       int maxNumChunksOnDisk = computeMaxNumChunksOnDisk(s3OutputConfig, s3ExportConfig);
   ```



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -269,52 +213,74 @@ public void close() throws IOException
     // Closeables are closed in LIFO order
     closer.register(() -> {
       // This should be emitted as a metric
+      long totalChunkSize = (currentChunk.id - 1) * chunkSize + currentChunk.length();
       LOG.info(
           "Pushed total [%d] parts containing [%d] bytes in [%d]ms.",
-          numChunksPushed,
-          resultsSize,
+          futures.size(),
+          totalChunkSize,
           pushStopwatch.elapsed(TimeUnit.MILLISECONDS)
       );
     });
 
-    closer.register(() -> org.apache.commons.io.FileUtils.forceDelete(chunkStorePath));
+    try (Closer ignored = closer) {
+      if (!error) {
+        pushCurrentChunk();
+        completeMultipartUpload();
+      }
+    }
+  }
 
-    closer.register(() -> {
+  private void completeMultipartUpload()
+  {
+    for (Future<UploadPartResult> future : futures) {
       try {
-        if (resultsSize > 0 && isAllPushSucceeded()) {
-          RetryUtils.retry(
-              () -> s3.completeMultipartUpload(
-                  new CompleteMultipartUploadRequest(config.getBucket(), s3Key, uploadId, pushResults)
-              ),
-              S3Utils.S3RETRY,
-              config.getMaxRetry()
-          );
-        } else {
-          RetryUtils.retry(
-              () -> {
-                s3.cancelMultiPartUpload(new AbortMultipartUploadRequest(config.getBucket(), s3Key, uploadId));
-                return null;
-              },
-              S3Utils.S3RETRY,
-              config.getMaxRetry()
-          );
-        }
+        UploadPartResult result = future.get();

Review Comment:
   Better to use `future.get()` with some timeout i.e `future.get(timeout, unit)` rather than blocking indefinitely in case the executor is misbehaving. To start with, the timeout maybe a large enough value, say 1 hour.



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -199,64 +188,19 @@ private void pushCurrentChunk() throws IOException
   {
     currentChunk.close();
     final Chunk chunk = currentChunk;
-    try {
-      if (chunk.length() > 0) {
-        resultsSize += chunk.length();
-
-        pushStopwatch.start();
-        pushResults.add(push(chunk));
-        pushStopwatch.stop();
-        numChunksPushed++;
-      }
-    }
-    finally {
-      if (!chunk.delete()) {
-        LOG.warn("Failed to delete chunk [%s]", chunk.getAbsolutePath());
-      }
-    }
-  }
-
-  private PartETag push(Chunk chunk) throws IOException
-  {
-    try {
-      return RetryUtils.retry(
-          () -> uploadPartIfPossible(uploadId, config.getBucket(), s3Key, chunk),
-          S3Utils.S3RETRY,
-          config.getMaxRetry()
+    if (chunk.length() > 0) {
+      Future<UploadPartResult> uploadPartResultFuture = uploadManager.queueChunkForUpload(
+          s3,
+          s3Key,
+          chunk.id,
+          chunk.file,
+          uploadId,
+          config
       );
-    }
-    catch (AmazonServiceException e) {
-      throw new IOException(e);
-    }
-    catch (Exception e) {
-      throw new RuntimeException(e);
+      futures.add(uploadPartResultFuture);

Review Comment:
   Nit: Probably more readable as
   
   ```java
   futures.add(
       uploadManager.queueChunkForUpload(s3, s3Key, chunk.id, chunk.file, uploadId, config)
   );
   ```



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3UploadManager.java:
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.inject.Inject;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.common.RetryUtils;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.storage.s3.S3Utils;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+import org.apache.druid.utils.RuntimeInfo;
+
+import java.io.File;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
+
+/**
+ * This class manages uploading files to S3 in chunks, while ensuring that the
+ * number of chunks currently present on local disk does not exceed a specific limit.
+ */
+@ManageLifecycle
+public class S3UploadManager
+{
+  private final ExecutorService uploadExecutor;
+
+  private static final Logger log = new Logger(S3UploadManager.class);
+
+  @Inject
+  public S3UploadManager(S3OutputConfig s3OutputConfig, S3ExportConfig s3ExportConfig, RuntimeInfo runtimeInfo)
+  {
+    int poolSize = Math.max(4, runtimeInfo.getAvailableProcessors());
+    int maxNumConcurrentChunks = computeMaxNumConcurrentChunks(s3OutputConfig, s3ExportConfig);
+    this.uploadExecutor = createExecutorService(poolSize, maxNumConcurrentChunks);
+    log.info("Initialized executor service for S3 multipart upload with pool size [%d] and work queue capacity [%d]",
+             poolSize, maxNumConcurrentChunks);
+  }
+
+  /**
+   * Computes the maximum number of concurrent chunks for an S3 multipart upload.
+   * We want to determine the maximum number of concurrent chunks on disk based on the maximum value of chunkSize
+   * between the 2 configs: S3OutputConfig and S3ExportConfig.
+   *
+   * @param s3OutputConfig  The S3 output configuration, which may specify a custom chunk size.
+   * @param s3ExportConfig  The S3 export configuration, which may also specify a custom chunk size.
+   * @return The maximum number of concurrent chunks.

Review Comment:
   Not really needed.



##########
extensions-core/s3-extensions/src/test/java/org/apache/druid/storage/s3/output/RetryableS3OutputStreamTest.java:
##########
@@ -198,17 +205,15 @@ public void testFailToUploadAfterRetries() throws IOException
         config,
         s3,
         path,
-        false
+        s3UploadManager
     )) {
       for (int i = 0; i < 2; i++) {
         bb.clear();
         bb.putInt(i);
         out.write(bb.array());
       }
 
-      expectedException.expect(RuntimeException.class);
-      expectedException.expectCause(CoreMatchers.instanceOf(AmazonClientException.class));
-      expectedException.expectMessage("Upload failure test. Remaining failures [1]");
+      // No exception is thrown in the main thread, since the upload now happens in a separate threadpool.

Review Comment:
   not needed.



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -269,52 +213,74 @@ public void close() throws IOException
     // Closeables are closed in LIFO order
     closer.register(() -> {
       // This should be emitted as a metric
+      long totalChunkSize = (currentChunk.id - 1) * chunkSize + currentChunk.length();

Review Comment:
   Hmm, while this is correct, it seems better to simply add up the size of every chunk and not rely on the chunkSize here. If you use a map to maintain the futures, you could the total size of the successfully uploaded chunks.



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3UploadManager.java:
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.inject.Inject;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.common.RetryUtils;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.storage.s3.S3Utils;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+import org.apache.druid.utils.RuntimeInfo;
+
+import java.io.File;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
+
+/**
+ * This class manages uploading files to S3 in chunks, while ensuring that the
+ * number of chunks currently present on local disk does not exceed a specific limit.
+ */
+@ManageLifecycle
+public class S3UploadManager
+{
+  private final ExecutorService uploadExecutor;
+
+  private static final Logger log = new Logger(S3UploadManager.class);
+
+  @Inject
+  public S3UploadManager(S3OutputConfig s3OutputConfig, S3ExportConfig s3ExportConfig, RuntimeInfo runtimeInfo)
+  {
+    int poolSize = Math.max(4, runtimeInfo.getAvailableProcessors());
+    int maxNumConcurrentChunks = computeMaxNumConcurrentChunks(s3OutputConfig, s3ExportConfig);
+    this.uploadExecutor = createExecutorService(poolSize, maxNumConcurrentChunks);
+    log.info("Initialized executor service for S3 multipart upload with pool size [%d] and work queue capacity [%d]",
+             poolSize, maxNumConcurrentChunks);
+  }
+
+  /**
+   * Computes the maximum number of concurrent chunks for an S3 multipart upload.
+   * We want to determine the maximum number of concurrent chunks on disk based on the maximum value of chunkSize
+   * between the 2 configs: S3OutputConfig and S3ExportConfig.
+   *
+   * @param s3OutputConfig  The S3 output configuration, which may specify a custom chunk size.
+   * @param s3ExportConfig  The S3 export configuration, which may also specify a custom chunk size.
+   * @return The maximum number of concurrent chunks.
+   */
+  @VisibleForTesting
+  int computeMaxNumConcurrentChunks(S3OutputConfig s3OutputConfig, S3ExportConfig s3ExportConfig)
+  {
+    long chunkSize = S3OutputConfig.S3_MULTIPART_UPLOAD_MIN_PART_SIZE_BYTES;
+    if (s3OutputConfig != null && s3OutputConfig.getChunkSize() != null) {
+      chunkSize = Math.max(chunkSize, s3OutputConfig.getChunkSize());
+    }
+    if (s3ExportConfig != null && s3ExportConfig.getChunkSize() != null) {
+      chunkSize = Math.max(chunkSize, s3ExportConfig.getChunkSize().getBytes());
+    }
+
+    return (int) (S3OutputConfig.S3_MULTIPART_UPLOAD_MAX_PART_SIZE_BYTES / chunkSize);
+  }
+
+  /**
+   * Queues a chunk of a file for upload to S3 as part of a multipart upload.
+   */
+  public Future<UploadPartResult> queueChunkForUpload(ServerSideEncryptingAmazonS3 s3Client, String key, int chunkNumber, File chunkFile, String uploadId, S3OutputConfig config)
+  {
+    return uploadExecutor.submit(() -> RetryUtils.retry(
+        () -> {
+          log.info("Uploading chunk [%d] for uploadId [%s].", chunkNumber, uploadId);
+          UploadPartResult uploadPartResult = uploadPartIfPossible(
+              s3Client,
+              uploadId,
+              config.getBucket(),
+              key,
+              chunkNumber,
+              chunkFile
+          );
+          if (!chunkFile.delete()) {
+            log.warn("Failed to delete chunk [%s]", chunkFile.getAbsolutePath());
+          }
+          return uploadPartResult;
+        },
+        S3Utils.S3RETRY,
+        config.getMaxRetry()
+    ));
+  }
+
+  @VisibleForTesting
+  UploadPartResult uploadPartIfPossible(
+      ServerSideEncryptingAmazonS3 s3Client,
+      String uploadId,
+      String bucket,
+      String key,
+      int chunkNumber,
+      File chunkFile
+  )
+  {
+    final UploadPartRequest uploadPartRequest = new UploadPartRequest()
+        .withUploadId(uploadId)
+        .withBucketName(bucket)
+        .withKey(key)
+        .withFile(chunkFile)
+        .withPartNumber(chunkNumber)
+        .withPartSize(chunkFile.length());
+
+    if (log.isDebugEnabled()) {
+      log.debug("Pushing chunk [%s] to bucket[%s] and key[%s].", chunkNumber, bucket, key);
+    }
+    return s3Client.uploadPart(uploadPartRequest);
+  }
+
+  @VisibleForTesting
+  ExecutorService createExecutorService(int poolSize, int maxNumConcurrentChunks)
+  {
+    return Execs.newBlockingThreaded("S3UploadThreadPool-%d", poolSize, maxNumConcurrentChunks);
+  }
+
+  @LifecycleStart
+  public void start()
+  {
+    // No state startup required
+  }
+
+  @LifecycleStop
+  public void stop()
+  {
+    log.info("Stopping S3UploadManager");
+    uploadExecutor.shutdown();
+    log.info("Stopped S3UploadManager");

Review Comment:
   The logs are not needed as we are simply shutting down an executor.



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -269,52 +213,74 @@ public void close() throws IOException
     // Closeables are closed in LIFO order
     closer.register(() -> {
       // This should be emitted as a metric
+      long totalChunkSize = (currentChunk.id - 1) * chunkSize + currentChunk.length();
       LOG.info(
           "Pushed total [%d] parts containing [%d] bytes in [%d]ms.",
-          numChunksPushed,
-          resultsSize,
+          futures.size(),
+          totalChunkSize,
           pushStopwatch.elapsed(TimeUnit.MILLISECONDS)
       );
     });
 
-    closer.register(() -> org.apache.commons.io.FileUtils.forceDelete(chunkStorePath));
+    try (Closer ignored = closer) {
+      if (!error) {
+        pushCurrentChunk();
+        completeMultipartUpload();
+      }
+    }
+  }
 
-    closer.register(() -> {
+  private void completeMultipartUpload()
+  {
+    for (Future<UploadPartResult> future : futures) {
       try {
-        if (resultsSize > 0 && isAllPushSucceeded()) {
-          RetryUtils.retry(
-              () -> s3.completeMultipartUpload(
-                  new CompleteMultipartUploadRequest(config.getBucket(), s3Key, uploadId, pushResults)
-              ),
-              S3Utils.S3RETRY,
-              config.getMaxRetry()
-          );
-        } else {
-          RetryUtils.retry(
-              () -> {
-                s3.cancelMultiPartUpload(new AbortMultipartUploadRequest(config.getBucket(), s3Key, uploadId));
-                return null;
-              },
-              S3Utils.S3RETRY,
-              config.getMaxRetry()
-          );
-        }
+        UploadPartResult result = future.get();
+        pushResults.add(result.getPartETag());
       }
       catch (Exception e) {
-        throw new IOException(e);
+        error = true;
+        LOG.error(e, "Error in uploading part for upload ID [%s]", uploadId);
+        break;
       }
-    });
+    }
 
-    try (Closer ignored = closer) {
-      if (!error) {
-        pushCurrentChunk();
+    try {
+      if (isAllPushSucceeded()) {
+        RetryUtils.retry(
+            () -> s3.completeMultipartUpload(
+                new CompleteMultipartUploadRequest(config.getBucket(), s3Key, uploadId, pushResults)
+            ),
+            S3Utils.S3RETRY,
+            config.getMaxRetry()
+        );
+      } else {
+        RetryUtils.retry(
+            () -> {
+              s3.cancelMultiPartUpload(new AbortMultipartUploadRequest(config.getBucket(), s3Key, uploadId));
+              return null;
+            },
+            S3Utils.S3RETRY,
+            config.getMaxRetry()
+        );
+      }
+    }
+    catch (Exception e) {
+      throw new RuntimeException(e);
+    }
+    finally {
+      try {
+        org.apache.commons.io.FileUtils.forceDelete(chunkStorePath);

Review Comment:
   this should also happen in `close()`.
   The `completeMultipartUpload()` method should be concerned only with finishing up (or canceling) the upload work.



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3UploadManager.java:
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.inject.Inject;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.common.RetryUtils;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.storage.s3.S3Utils;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+import org.apache.druid.utils.RuntimeInfo;
+
+import java.io.File;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
+
+/**
+ * This class manages uploading files to S3 in chunks, while ensuring that the
+ * number of chunks currently present on local disk does not exceed a specific limit.
+ */
+@ManageLifecycle
+public class S3UploadManager
+{
+  private final ExecutorService uploadExecutor;
+
+  private static final Logger log = new Logger(S3UploadManager.class);
+
+  @Inject
+  public S3UploadManager(S3OutputConfig s3OutputConfig, S3ExportConfig s3ExportConfig, RuntimeInfo runtimeInfo)
+  {
+    int poolSize = Math.max(4, runtimeInfo.getAvailableProcessors());
+    int maxNumConcurrentChunks = computeMaxNumConcurrentChunks(s3OutputConfig, s3ExportConfig);
+    this.uploadExecutor = createExecutorService(poolSize, maxNumConcurrentChunks);
+    log.info("Initialized executor service for S3 multipart upload with pool size [%d] and work queue capacity [%d]",
+             poolSize, maxNumConcurrentChunks);
+  }
+
+  /**
+   * Computes the maximum number of concurrent chunks for an S3 multipart upload.
+   * We want to determine the maximum number of concurrent chunks on disk based on the maximum value of chunkSize
+   * between the 2 configs: S3OutputConfig and S3ExportConfig.
+   *
+   * @param s3OutputConfig  The S3 output configuration, which may specify a custom chunk size.
+   * @param s3ExportConfig  The S3 export configuration, which may also specify a custom chunk size.
+   * @return The maximum number of concurrent chunks.
+   */
+  @VisibleForTesting

Review Comment:
   Let's not do this, it really defeats the purpose of keeping this as an internal implementation detail.
   You might as well make this method `public static`.



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3UploadManager.java:
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.inject.Inject;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.common.RetryUtils;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.storage.s3.S3Utils;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+import org.apache.druid.utils.RuntimeInfo;
+
+import java.io.File;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
+
+/**
+ * This class manages uploading files to S3 in chunks, while ensuring that the
+ * number of chunks currently present on local disk does not exceed a specific limit.
+ */
+@ManageLifecycle
+public class S3UploadManager
+{
+  private final ExecutorService uploadExecutor;
+
+  private static final Logger log = new Logger(S3UploadManager.class);
+
+  @Inject
+  public S3UploadManager(S3OutputConfig s3OutputConfig, S3ExportConfig s3ExportConfig, RuntimeInfo runtimeInfo)
+  {
+    int poolSize = Math.max(4, runtimeInfo.getAvailableProcessors());
+    int maxNumConcurrentChunks = computeMaxNumConcurrentChunks(s3OutputConfig, s3ExportConfig);
+    this.uploadExecutor = createExecutorService(poolSize, maxNumConcurrentChunks);
+    log.info("Initialized executor service for S3 multipart upload with pool size [%d] and work queue capacity [%d]",
+             poolSize, maxNumConcurrentChunks);
+  }
+
+  /**
+   * Computes the maximum number of concurrent chunks for an S3 multipart upload.
+   * We want to determine the maximum number of concurrent chunks on disk based on the maximum value of chunkSize
+   * between the 2 configs: S3OutputConfig and S3ExportConfig.
+   *
+   * @param s3OutputConfig  The S3 output configuration, which may specify a custom chunk size.
+   * @param s3ExportConfig  The S3 export configuration, which may also specify a custom chunk size.
+   * @return The maximum number of concurrent chunks.
+   */
+  @VisibleForTesting
+  int computeMaxNumConcurrentChunks(S3OutputConfig s3OutputConfig, S3ExportConfig s3ExportConfig)

Review Comment:
   Seems like this method should also take an argument for maximum disk space allocated for storing chunks.



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3UploadManager.java:
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.inject.Inject;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.common.RetryUtils;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.storage.s3.S3Utils;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+import org.apache.druid.utils.RuntimeInfo;
+
+import java.io.File;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
+
+/**
+ * This class manages uploading files to S3 in chunks, while ensuring that the
+ * number of chunks currently present on local disk does not exceed a specific limit.
+ */
+@ManageLifecycle
+public class S3UploadManager
+{
+  private final ExecutorService uploadExecutor;
+
+  private static final Logger log = new Logger(S3UploadManager.class);
+
+  @Inject
+  public S3UploadManager(S3OutputConfig s3OutputConfig, S3ExportConfig s3ExportConfig, RuntimeInfo runtimeInfo)
+  {
+    int poolSize = Math.max(4, runtimeInfo.getAvailableProcessors());
+    int maxNumConcurrentChunks = computeMaxNumConcurrentChunks(s3OutputConfig, s3ExportConfig);
+    this.uploadExecutor = createExecutorService(poolSize, maxNumConcurrentChunks);
+    log.info("Initialized executor service for S3 multipart upload with pool size [%d] and work queue capacity [%d]",
+             poolSize, maxNumConcurrentChunks);
+  }
+
+  /**
+   * Computes the maximum number of concurrent chunks for an S3 multipart upload.
+   * We want to determine the maximum number of concurrent chunks on disk based on the maximum value of chunkSize
+   * between the 2 configs: S3OutputConfig and S3ExportConfig.

Review Comment:
   ```suggestion
      * Computes the maximum number of S3 upload chunks that can be kept on disk using the
      * maximum chunk size specified in {@link S3OutputConfig} and {@link S3ExportConfig}.
   ```



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3UploadManager.java:
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.inject.Inject;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.common.RetryUtils;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.storage.s3.S3Utils;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+import org.apache.druid.utils.RuntimeInfo;
+
+import java.io.File;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
+
+/**
+ * This class manages uploading files to S3 in chunks, while ensuring that the
+ * number of chunks currently present on local disk does not exceed a specific limit.
+ */
+@ManageLifecycle
+public class S3UploadManager
+{
+  private final ExecutorService uploadExecutor;
+
+  private static final Logger log = new Logger(S3UploadManager.class);
+
+  @Inject
+  public S3UploadManager(S3OutputConfig s3OutputConfig, S3ExportConfig s3ExportConfig, RuntimeInfo runtimeInfo)
+  {
+    int poolSize = Math.max(4, runtimeInfo.getAvailableProcessors());
+    int maxNumConcurrentChunks = computeMaxNumConcurrentChunks(s3OutputConfig, s3ExportConfig);
+    this.uploadExecutor = createExecutorService(poolSize, maxNumConcurrentChunks);
+    log.info("Initialized executor service for S3 multipart upload with pool size [%d] and work queue capacity [%d]",
+             poolSize, maxNumConcurrentChunks);
+  }
+
+  /**
+   * Computes the maximum number of concurrent chunks for an S3 multipart upload.
+   * We want to determine the maximum number of concurrent chunks on disk based on the maximum value of chunkSize
+   * between the 2 configs: S3OutputConfig and S3ExportConfig.
+   *
+   * @param s3OutputConfig  The S3 output configuration, which may specify a custom chunk size.
+   * @param s3ExportConfig  The S3 export configuration, which may also specify a custom chunk size.
+   * @return The maximum number of concurrent chunks.
+   */
+  @VisibleForTesting
+  int computeMaxNumConcurrentChunks(S3OutputConfig s3OutputConfig, S3ExportConfig s3ExportConfig)
+  {
+    long chunkSize = S3OutputConfig.S3_MULTIPART_UPLOAD_MIN_PART_SIZE_BYTES;
+    if (s3OutputConfig != null && s3OutputConfig.getChunkSize() != null) {
+      chunkSize = Math.max(chunkSize, s3OutputConfig.getChunkSize());
+    }
+    if (s3ExportConfig != null && s3ExportConfig.getChunkSize() != null) {
+      chunkSize = Math.max(chunkSize, s3ExportConfig.getChunkSize().getBytes());
+    }
+
+    return (int) (S3OutputConfig.S3_MULTIPART_UPLOAD_MAX_PART_SIZE_BYTES / chunkSize);
+  }
+
+  /**
+   * Queues a chunk of a file for upload to S3 as part of a multipart upload.
+   */
+  public Future<UploadPartResult> queueChunkForUpload(ServerSideEncryptingAmazonS3 s3Client, String key, int chunkNumber, File chunkFile, String uploadId, S3OutputConfig config)
+  {
+    return uploadExecutor.submit(() -> RetryUtils.retry(
+        () -> {
+          log.info("Uploading chunk [%d] for uploadId [%s].", chunkNumber, uploadId);
+          UploadPartResult uploadPartResult = uploadPartIfPossible(
+              s3Client,
+              uploadId,
+              config.getBucket(),
+              key,
+              chunkNumber,
+              chunkFile
+          );
+          if (!chunkFile.delete()) {
+            log.warn("Failed to delete chunk [%s]", chunkFile.getAbsolutePath());
+          }
+          return uploadPartResult;
+        },
+        S3Utils.S3RETRY,
+        config.getMaxRetry()
+    ));
+  }
+
+  @VisibleForTesting
+  UploadPartResult uploadPartIfPossible(
+      ServerSideEncryptingAmazonS3 s3Client,
+      String uploadId,
+      String bucket,
+      String key,
+      int chunkNumber,
+      File chunkFile
+  )
+  {
+    final UploadPartRequest uploadPartRequest = new UploadPartRequest()
+        .withUploadId(uploadId)
+        .withBucketName(bucket)
+        .withKey(key)
+        .withFile(chunkFile)
+        .withPartNumber(chunkNumber)
+        .withPartSize(chunkFile.length());
+
+    if (log.isDebugEnabled()) {
+      log.debug("Pushing chunk [%s] to bucket[%s] and key[%s].", chunkNumber, bucket, key);

Review Comment:
   ```suggestion
         log.debug("Pushing chunk[%s] to bucket[%s] and key[%s].", chunkNumber, bucket, key);
   ```



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -269,52 +213,74 @@ public void close() throws IOException
     // Closeables are closed in LIFO order
     closer.register(() -> {
       // This should be emitted as a metric
+      long totalChunkSize = (currentChunk.id - 1) * chunkSize + currentChunk.length();
       LOG.info(
           "Pushed total [%d] parts containing [%d] bytes in [%d]ms.",
-          numChunksPushed,
-          resultsSize,
+          futures.size(),
+          totalChunkSize,
           pushStopwatch.elapsed(TimeUnit.MILLISECONDS)
       );
     });
 
-    closer.register(() -> org.apache.commons.io.FileUtils.forceDelete(chunkStorePath));
+    try (Closer ignored = closer) {
+      if (!error) {
+        pushCurrentChunk();
+        completeMultipartUpload();
+      }
+    }
+  }
 
-    closer.register(() -> {
+  private void completeMultipartUpload()
+  {
+    for (Future<UploadPartResult> future : futures) {
       try {
-        if (resultsSize > 0 && isAllPushSucceeded()) {
-          RetryUtils.retry(
-              () -> s3.completeMultipartUpload(
-                  new CompleteMultipartUploadRequest(config.getBucket(), s3Key, uploadId, pushResults)
-              ),
-              S3Utils.S3RETRY,
-              config.getMaxRetry()
-          );
-        } else {
-          RetryUtils.retry(
-              () -> {
-                s3.cancelMultiPartUpload(new AbortMultipartUploadRequest(config.getBucket(), s3Key, uploadId));
-                return null;
-              },
-              S3Utils.S3RETRY,
-              config.getMaxRetry()
-          );
-        }
+        UploadPartResult result = future.get();
+        pushResults.add(result.getPartETag());
       }
       catch (Exception e) {
-        throw new IOException(e);
+        error = true;
+        LOG.error(e, "Error in uploading part for upload ID [%s]", uploadId);
+        break;
       }
-    });
+    }
 
-    try (Closer ignored = closer) {
-      if (!error) {
-        pushCurrentChunk();
+    try {
+      if (isAllPushSucceeded()) {

Review Comment:
   In case of error, we must make sure to cancel any pending futures.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1629424508


##########
extensions-core/s3-extensions/src/test/java/org/apache/druid/storage/s3/output/S3UploadManagerTest.java:
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import org.apache.druid.java.util.common.HumanReadableBytes;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+import org.apache.druid.utils.RuntimeInfo;
+import org.easymock.Capture;
+import org.easymock.EasyMock;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.File;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+public class S3UploadManagerTest
+{
+
+  private S3UploadManager s3UploadManager;
+  private S3OutputConfig s3OutputConfig;
+  private S3ExportConfig s3ExportConfig;
+  private static ExecutorService uploadExecutor;

Review Comment:
   this should not be static.



##########
extensions-core/s3-extensions/src/test/java/org/apache/druid/storage/s3/output/S3UploadManagerTest.java:
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import org.apache.druid.java.util.common.HumanReadableBytes;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+import org.apache.druid.utils.RuntimeInfo;
+import org.easymock.Capture;
+import org.easymock.EasyMock;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.File;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+public class S3UploadManagerTest
+{
+
+  private S3UploadManager s3UploadManager;
+  private S3OutputConfig s3OutputConfig;
+  private S3ExportConfig s3ExportConfig;
+  private static ExecutorService uploadExecutor;
+
+  @Before
+  public void setUp()
+  {
+    s3OutputConfig = EasyMock.mock(S3OutputConfig.class);
+    s3ExportConfig = EasyMock.mock(S3ExportConfig.class);

Review Comment:
   Please use concrete objects instead of mocks. `S3OutputConfig` and `S3ExportConfig` are plain POJOs that can be easily constructed.



##########
extensions-core/s3-extensions/src/test/java/org/apache/druid/storage/s3/output/S3UploadManagerTest.java:
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import org.apache.druid.java.util.common.HumanReadableBytes;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+import org.apache.druid.utils.RuntimeInfo;
+import org.easymock.Capture;
+import org.easymock.EasyMock;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.File;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+public class S3UploadManagerTest
+{
+
+  private S3UploadManager s3UploadManager;
+  private S3OutputConfig s3OutputConfig;
+  private S3ExportConfig s3ExportConfig;
+  private static ExecutorService uploadExecutor;
+
+  @Before
+  public void setUp()
+  {
+    s3OutputConfig = EasyMock.mock(S3OutputConfig.class);
+    s3ExportConfig = EasyMock.mock(S3ExportConfig.class);
+    final RuntimeInfo runtimeInfo = EasyMock.mock(RuntimeInfo.class);
+    uploadExecutor = Execs.singleThreaded("UploadThreadPool-%d");
+
+    EasyMock.expect(runtimeInfo.getAvailableProcessors()).andReturn(8).anyTimes();
+    EasyMock.expect(s3OutputConfig.getChunkSize()).andReturn(100L * 1024 * 1024).anyTimes(); // 100 MB
+    EasyMock.expect(s3OutputConfig.getMaxRetry()).andReturn(1).anyTimes();
+    EasyMock.expect(s3OutputConfig.getBucket()).andReturn("bucket").anyTimes();
+    EasyMock.expect(s3ExportConfig.getChunkSize()).andReturn(HumanReadableBytes.valueOf(200L * 1024 * 1024)).anyTimes(); // 200 MB
+
+    EasyMock.replay(runtimeInfo, s3OutputConfig, s3ExportConfig);
+
+    s3UploadManager = new TestS3UploadManager(s3OutputConfig, s3ExportConfig, runtimeInfo);
+  }
+
+  @Test
+  public void testQueueChunkForUpload() throws Exception
+  {
+    ServerSideEncryptingAmazonS3 s3Client = EasyMock.mock(ServerSideEncryptingAmazonS3.class);
+
+    File chunkFile = EasyMock.mock(File.class);
+    EasyMock.expect(chunkFile.length()).andReturn(1024L).anyTimes();
+    EasyMock.expect(chunkFile.delete()).andReturn(true).anyTimes();
+
+    int chunkId = 42;
+    UploadPartResult uploadPartResult = new UploadPartResult();
+    uploadPartResult.setPartNumber(chunkId);
+    uploadPartResult.setETag("etag");
+    EasyMock.expect(s3Client.uploadPart(EasyMock.anyObject(UploadPartRequest.class))).andReturn(uploadPartResult);
+
+    EasyMock.replay(chunkFile, s3Client);
+
+    Future<UploadPartResult> result = s3UploadManager.queueChunkForUpload(s3Client, "test-key", chunkId, chunkFile, "upload-id", s3OutputConfig);
+
+    UploadPartResult futureResult = result.get();
+    Assert.assertEquals(chunkId, futureResult.getPartNumber());
+    Assert.assertEquals("etag", futureResult.getETag());
+  }
+
+  @Test
+  public void testComputeMaxNumChunksOnDisk()
+  {
+    int maxNumConcurrentChunks = S3UploadManager.computeMaxNumChunksOnDisk(s3OutputConfig, s3ExportConfig);
+    int expectedMaxNumConcurrentChunks = 25; // maxChunkSizePossible/200 MB
+    assertEquals(expectedMaxNumConcurrentChunks, maxNumConcurrentChunks);
+  }
+
+  @Test
+  public void testComputeMaxNumChunksOnDiskWithNullOutputConfig()
+  {
+    // Null S3OutputConfig
+    int maxNumConcurrentChunks = S3UploadManager.computeMaxNumChunksOnDisk(null, s3ExportConfig);
+    int expectedMaxNumConcurrentChunks = 25; // maxChunkSizePossible / s3ExportConfig's chunk size
+    assertEquals(expectedMaxNumConcurrentChunks, maxNumConcurrentChunks);
+
+    // Null S3OutputConfig#getChunkSize()
+    maxNumConcurrentChunks = S3UploadManager.computeMaxNumChunksOnDisk(EasyMock.mock(S3OutputConfig.class), s3ExportConfig);
+    assertEquals(expectedMaxNumConcurrentChunks, maxNumConcurrentChunks);
+  }
+
+  @Test
+  public void testComputeMaxNumChunksOnDiskWithNullExportConfig()
+  {
+    // Null S3ExportConfig
+    int maxNumConcurrentChunks = S3UploadManager.computeMaxNumChunksOnDisk(s3OutputConfig, null);
+    int expectedMaxNumConcurrentChunks = 51; // maxChunkSizePossible / s3OutputConfig's chunk size
+    assertEquals(expectedMaxNumConcurrentChunks, maxNumConcurrentChunks);
+
+    // Null S3ExportConfig#getChunkSize()
+    maxNumConcurrentChunks = S3UploadManager.computeMaxNumChunksOnDisk(s3OutputConfig, EasyMock.mock(S3ExportConfig.class));
+    assertEquals(expectedMaxNumConcurrentChunks, maxNumConcurrentChunks);
+  }
+
+  @Test
+  public void testUploadPartIfPossible()
+  {
+    ServerSideEncryptingAmazonS3 s3Client = EasyMock.mock(ServerSideEncryptingAmazonS3.class);
+
+    File chunkFile = EasyMock.mock(File.class);
+    EasyMock.expect(chunkFile.length()).andReturn(1024L).anyTimes();
+
+    UploadPartResult uploadPartResult = new UploadPartResult();
+    Capture<UploadPartRequest> partRequestCapture = EasyMock.newCapture();
+    EasyMock.expect(s3Client.uploadPart(EasyMock.capture(partRequestCapture))).andReturn(uploadPartResult);
+    EasyMock.replay(s3Client, chunkFile);
+
+    UploadPartResult result = s3UploadManager.uploadPartIfPossible(s3Client, "upload-id", "bucket", "key", 1, chunkFile);
+
+    UploadPartRequest capturedRequest = partRequestCapture.getValue();
+    assertEquals("upload-id", capturedRequest.getUploadId());
+    assertEquals("bucket", capturedRequest.getBucketName());
+    assertEquals("key", capturedRequest.getKey());
+    assertEquals(1, capturedRequest.getPartNumber());
+    assertEquals(chunkFile, capturedRequest.getFile());
+    assertEquals(1024L, capturedRequest.getPartSize());
+
+    assertEquals(uploadPartResult, result);
+  }
+
+  @Test
+  public void testStartAndStop()
+  {
+    s3UploadManager.start();
+    s3UploadManager.stop();
+
+    assertTrue(uploadExecutor.isShutdown());
+  }
+
+  private static class TestS3UploadManager extends S3UploadManager

Review Comment:
   I don't think this is needed.



##########
extensions-core/s3-extensions/src/test/java/org/apache/druid/storage/s3/output/S3UploadManagerTest.java:
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import org.apache.druid.java.util.common.HumanReadableBytes;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+import org.apache.druid.utils.RuntimeInfo;
+import org.easymock.Capture;
+import org.easymock.EasyMock;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.File;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+public class S3UploadManagerTest
+{
+
+  private S3UploadManager s3UploadManager;
+  private S3OutputConfig s3OutputConfig;
+  private S3ExportConfig s3ExportConfig;
+  private static ExecutorService uploadExecutor;
+
+  @Before
+  public void setUp()
+  {
+    s3OutputConfig = EasyMock.mock(S3OutputConfig.class);
+    s3ExportConfig = EasyMock.mock(S3ExportConfig.class);
+    final RuntimeInfo runtimeInfo = EasyMock.mock(RuntimeInfo.class);
+    uploadExecutor = Execs.singleThreaded("UploadThreadPool-%d");
+
+    EasyMock.expect(runtimeInfo.getAvailableProcessors()).andReturn(8).anyTimes();
+    EasyMock.expect(s3OutputConfig.getChunkSize()).andReturn(100L * 1024 * 1024).anyTimes(); // 100 MB
+    EasyMock.expect(s3OutputConfig.getMaxRetry()).andReturn(1).anyTimes();
+    EasyMock.expect(s3OutputConfig.getBucket()).andReturn("bucket").anyTimes();
+    EasyMock.expect(s3ExportConfig.getChunkSize()).andReturn(HumanReadableBytes.valueOf(200L * 1024 * 1024)).anyTimes(); // 200 MB
+
+    EasyMock.replay(runtimeInfo, s3OutputConfig, s3ExportConfig);
+
+    s3UploadManager = new TestS3UploadManager(s3OutputConfig, s3ExportConfig, runtimeInfo);
+  }
+
+  @Test
+  public void testQueueChunkForUpload() throws Exception
+  {
+    ServerSideEncryptingAmazonS3 s3Client = EasyMock.mock(ServerSideEncryptingAmazonS3.class);
+
+    File chunkFile = EasyMock.mock(File.class);
+    EasyMock.expect(chunkFile.length()).andReturn(1024L).anyTimes();
+    EasyMock.expect(chunkFile.delete()).andReturn(true).anyTimes();
+
+    int chunkId = 42;
+    UploadPartResult uploadPartResult = new UploadPartResult();
+    uploadPartResult.setPartNumber(chunkId);
+    uploadPartResult.setETag("etag");
+    EasyMock.expect(s3Client.uploadPart(EasyMock.anyObject(UploadPartRequest.class))).andReturn(uploadPartResult);
+
+    EasyMock.replay(chunkFile, s3Client);
+
+    Future<UploadPartResult> result = s3UploadManager.queueChunkForUpload(s3Client, "test-key", chunkId, chunkFile, "upload-id", s3OutputConfig);
+
+    UploadPartResult futureResult = result.get();
+    Assert.assertEquals(chunkId, futureResult.getPartNumber());
+    Assert.assertEquals("etag", futureResult.getETag());
+  }
+
+  @Test
+  public void testComputeMaxNumChunksOnDisk()
+  {
+    int maxNumConcurrentChunks = S3UploadManager.computeMaxNumChunksOnDisk(s3OutputConfig, s3ExportConfig);
+    int expectedMaxNumConcurrentChunks = 25; // maxChunkSizePossible/200 MB
+    assertEquals(expectedMaxNumConcurrentChunks, maxNumConcurrentChunks);
+  }
+
+  @Test
+  public void testComputeMaxNumChunksOnDiskWithNullOutputConfig()
+  {
+    // Null S3OutputConfig
+    int maxNumConcurrentChunks = S3UploadManager.computeMaxNumChunksOnDisk(null, s3ExportConfig);
+    int expectedMaxNumConcurrentChunks = 25; // maxChunkSizePossible / s3ExportConfig's chunk size
+    assertEquals(expectedMaxNumConcurrentChunks, maxNumConcurrentChunks);
+
+    // Null S3OutputConfig#getChunkSize()
+    maxNumConcurrentChunks = S3UploadManager.computeMaxNumChunksOnDisk(EasyMock.mock(S3OutputConfig.class), s3ExportConfig);
+    assertEquals(expectedMaxNumConcurrentChunks, maxNumConcurrentChunks);
+  }
+
+  @Test
+  public void testComputeMaxNumChunksOnDiskWithNullExportConfig()
+  {
+    // Null S3ExportConfig
+    int maxNumConcurrentChunks = S3UploadManager.computeMaxNumChunksOnDisk(s3OutputConfig, null);
+    int expectedMaxNumConcurrentChunks = 51; // maxChunkSizePossible / s3OutputConfig's chunk size
+    assertEquals(expectedMaxNumConcurrentChunks, maxNumConcurrentChunks);
+
+    // Null S3ExportConfig#getChunkSize()
+    maxNumConcurrentChunks = S3UploadManager.computeMaxNumChunksOnDisk(s3OutputConfig, EasyMock.mock(S3ExportConfig.class));
+    assertEquals(expectedMaxNumConcurrentChunks, maxNumConcurrentChunks);
+  }
+
+  @Test
+  public void testUploadPartIfPossible()
+  {
+    ServerSideEncryptingAmazonS3 s3Client = EasyMock.mock(ServerSideEncryptingAmazonS3.class);
+
+    File chunkFile = EasyMock.mock(File.class);
+    EasyMock.expect(chunkFile.length()).andReturn(1024L).anyTimes();
+
+    UploadPartResult uploadPartResult = new UploadPartResult();
+    Capture<UploadPartRequest> partRequestCapture = EasyMock.newCapture();
+    EasyMock.expect(s3Client.uploadPart(EasyMock.capture(partRequestCapture))).andReturn(uploadPartResult);
+    EasyMock.replay(s3Client, chunkFile);
+
+    UploadPartResult result = s3UploadManager.uploadPartIfPossible(s3Client, "upload-id", "bucket", "key", 1, chunkFile);
+
+    UploadPartRequest capturedRequest = partRequestCapture.getValue();
+    assertEquals("upload-id", capturedRequest.getUploadId());
+    assertEquals("bucket", capturedRequest.getBucketName());
+    assertEquals("key", capturedRequest.getKey());
+    assertEquals(1, capturedRequest.getPartNumber());
+    assertEquals(chunkFile, capturedRequest.getFile());
+    assertEquals(1024L, capturedRequest.getPartSize());
+
+    assertEquals(uploadPartResult, result);
+  }
+
+  @Test
+  public void testStartAndStop()
+  {
+    s3UploadManager.start();
+    s3UploadManager.stop();
+
+    assertTrue(uploadExecutor.isShutdown());
+  }
+
+  private static class TestS3UploadManager extends S3UploadManager
+  {
+    public TestS3UploadManager(S3OutputConfig s3OutputConfig, S3ExportConfig s3ExportConfig, RuntimeInfo runtimeInfo)
+    {
+      super(s3OutputConfig, s3ExportConfig, runtimeInfo);
+    }
+
+    @Override
+    ExecutorService createExecutorService(int poolSize, int maxNumConcurrentChunks)
+    {
+      return uploadExecutor;
+    }

Review Comment:
   this should not be needed, let the `S3UploadManager` create its own executor. Just make sure to add a `tearDown` method annotated with `@After` so that it runs after every test and calls `S3UploadManager.stop()`.



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3UploadManager.java:
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.inject.Inject;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.common.RetryUtils;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.storage.s3.S3Utils;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+import org.apache.druid.utils.RuntimeInfo;
+
+import java.io.File;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
+
+/**
+ * This class manages uploading files to S3 in chunks, while ensuring that the
+ * number of chunks currently present on local disk does not exceed a specific limit.
+ */
+@ManageLifecycle
+public class S3UploadManager
+{
+  private final ExecutorService uploadExecutor;
+
+  private static final Logger log = new Logger(S3UploadManager.class);
+
+  @Inject
+  public S3UploadManager(S3OutputConfig s3OutputConfig, S3ExportConfig s3ExportConfig, RuntimeInfo runtimeInfo)
+  {
+    int poolSize = Math.max(4, runtimeInfo.getAvailableProcessors());
+    int maxNumChunksOnDisk = computeMaxNumChunksOnDisk(s3OutputConfig, s3ExportConfig);
+    this.uploadExecutor = createExecutorService(poolSize, maxNumChunksOnDisk);
+    log.info("Initialized executor service for S3 multipart upload with pool size [%d] and work queue capacity [%d]",
+             poolSize, maxNumChunksOnDisk);
+  }
+
+  /**
+   * Computes the maximum number of S3 upload chunks that can be kept on disk using the
+   * maximum chunk size specified in {@link S3OutputConfig} and {@link S3ExportConfig}.
+   */
+  public static int computeMaxNumChunksOnDisk(S3OutputConfig s3OutputConfig, S3ExportConfig s3ExportConfig)
+  {
+    long maxChunkSize = S3OutputConfig.S3_MULTIPART_UPLOAD_MIN_PART_SIZE_BYTES;
+    if (s3OutputConfig != null && s3OutputConfig.getChunkSize() != null) {
+      maxChunkSize = Math.max(maxChunkSize, s3OutputConfig.getChunkSize());
+    }
+    if (s3ExportConfig != null && s3ExportConfig.getChunkSize() != null) {
+      maxChunkSize = Math.max(maxChunkSize, s3ExportConfig.getChunkSize().getBytes());
+    }
+
+    return (int) (S3OutputConfig.S3_MULTIPART_UPLOAD_MAX_PART_SIZE_BYTES / maxChunkSize);
+  }
+
+  /**
+   * Queues a chunk of a file for upload to S3 as part of a multipart upload.
+   */
+  public Future<UploadPartResult> queueChunkForUpload(
+      ServerSideEncryptingAmazonS3 s3Client,
+      String key,
+      int chunkNumber,
+      File chunkFile,
+      String uploadId,
+      S3OutputConfig config
+  )
+  {
+    return uploadExecutor.submit(() -> RetryUtils.retry(
+        () -> {
+          log.debug("Uploading chunk [%d] for uploadId [%s].", chunkNumber, uploadId);
+          UploadPartResult uploadPartResult = uploadPartIfPossible(
+              s3Client,
+              uploadId,
+              config.getBucket(),
+              key,
+              chunkNumber,
+              chunkFile
+          );
+          if (!chunkFile.delete()) {
+            log.warn("Failed to delete chunk [%s]", chunkFile.getAbsolutePath());
+          }
+          return uploadPartResult;
+        },
+        S3Utils.S3RETRY,
+        config.getMaxRetry()
+    ));
+  }
+
+  @VisibleForTesting
+  UploadPartResult uploadPartIfPossible(
+      ServerSideEncryptingAmazonS3 s3Client,
+      String uploadId,
+      String bucket,
+      String key,
+      int chunkNumber,
+      File chunkFile
+  )
+  {
+    final UploadPartRequest uploadPartRequest = new UploadPartRequest()
+        .withUploadId(uploadId)
+        .withBucketName(bucket)
+        .withKey(key)
+        .withFile(chunkFile)
+        .withPartNumber(chunkNumber)
+        .withPartSize(chunkFile.length());
+
+    if (log.isDebugEnabled()) {
+      log.debug("Pushing chunk[%s] to bucket[%s] and key[%s].", chunkNumber, bucket, key);
+    }
+    return s3Client.uploadPart(uploadPartRequest);
+  }
+
+  @VisibleForTesting

Review Comment:
   This method should be private, need not be visible for testing.



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3UploadManager.java:
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.inject.Inject;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.common.RetryUtils;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.storage.s3.S3Utils;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+import org.apache.druid.utils.RuntimeInfo;
+
+import java.io.File;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
+
+/**
+ * This class manages uploading files to S3 in chunks, while ensuring that the
+ * number of chunks currently present on local disk does not exceed a specific limit.
+ */
+@ManageLifecycle
+public class S3UploadManager
+{
+  private final ExecutorService uploadExecutor;
+
+  private static final Logger log = new Logger(S3UploadManager.class);
+
+  @Inject
+  public S3UploadManager(S3OutputConfig s3OutputConfig, S3ExportConfig s3ExportConfig, RuntimeInfo runtimeInfo)
+  {
+    int poolSize = Math.max(4, runtimeInfo.getAvailableProcessors());
+    int maxNumChunksOnDisk = computeMaxNumChunksOnDisk(s3OutputConfig, s3ExportConfig);
+    this.uploadExecutor = createExecutorService(poolSize, maxNumChunksOnDisk);
+    log.info("Initialized executor service for S3 multipart upload with pool size [%d] and work queue capacity [%d]",
+             poolSize, maxNumChunksOnDisk);
+  }
+
+  /**
+   * Computes the maximum number of S3 upload chunks that can be kept on disk using the
+   * maximum chunk size specified in {@link S3OutputConfig} and {@link S3ExportConfig}.
+   */
+  public static int computeMaxNumChunksOnDisk(S3OutputConfig s3OutputConfig, S3ExportConfig s3ExportConfig)
+  {
+    long maxChunkSize = S3OutputConfig.S3_MULTIPART_UPLOAD_MIN_PART_SIZE_BYTES;
+    if (s3OutputConfig != null && s3OutputConfig.getChunkSize() != null) {
+      maxChunkSize = Math.max(maxChunkSize, s3OutputConfig.getChunkSize());
+    }
+    if (s3ExportConfig != null && s3ExportConfig.getChunkSize() != null) {
+      maxChunkSize = Math.max(maxChunkSize, s3ExportConfig.getChunkSize().getBytes());
+    }
+
+    return (int) (S3OutputConfig.S3_MULTIPART_UPLOAD_MAX_PART_SIZE_BYTES / maxChunkSize);
+  }
+
+  /**
+   * Queues a chunk of a file for upload to S3 as part of a multipart upload.
+   */
+  public Future<UploadPartResult> queueChunkForUpload(
+      ServerSideEncryptingAmazonS3 s3Client,
+      String key,
+      int chunkNumber,
+      File chunkFile,
+      String uploadId,
+      S3OutputConfig config
+  )
+  {
+    return uploadExecutor.submit(() -> RetryUtils.retry(
+        () -> {
+          log.debug("Uploading chunk [%d] for uploadId [%s].", chunkNumber, uploadId);

Review Comment:
   ```suggestion
             log.debug("Uploading chunk[%d] for uploadId[%s].", chunkNumber, uploadId);
   ```



##########
extensions-core/s3-extensions/src/test/java/org/apache/druid/storage/s3/output/S3UploadManagerTest.java:
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import org.apache.druid.java.util.common.HumanReadableBytes;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+import org.apache.druid.utils.RuntimeInfo;
+import org.easymock.Capture;
+import org.easymock.EasyMock;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.File;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+public class S3UploadManagerTest
+{
+
+  private S3UploadManager s3UploadManager;
+  private S3OutputConfig s3OutputConfig;
+  private S3ExportConfig s3ExportConfig;
+  private static ExecutorService uploadExecutor;
+
+  @Before
+  public void setUp()
+  {
+    s3OutputConfig = EasyMock.mock(S3OutputConfig.class);
+    s3ExportConfig = EasyMock.mock(S3ExportConfig.class);
+    final RuntimeInfo runtimeInfo = EasyMock.mock(RuntimeInfo.class);
+    uploadExecutor = Execs.singleThreaded("UploadThreadPool-%d");
+
+    EasyMock.expect(runtimeInfo.getAvailableProcessors()).andReturn(8).anyTimes();
+    EasyMock.expect(s3OutputConfig.getChunkSize()).andReturn(100L * 1024 * 1024).anyTimes(); // 100 MB
+    EasyMock.expect(s3OutputConfig.getMaxRetry()).andReturn(1).anyTimes();
+    EasyMock.expect(s3OutputConfig.getBucket()).andReturn("bucket").anyTimes();
+    EasyMock.expect(s3ExportConfig.getChunkSize()).andReturn(HumanReadableBytes.valueOf(200L * 1024 * 1024)).anyTimes(); // 200 MB
+
+    EasyMock.replay(runtimeInfo, s3OutputConfig, s3ExportConfig);
+
+    s3UploadManager = new TestS3UploadManager(s3OutputConfig, s3ExportConfig, runtimeInfo);
+  }
+
+  @Test
+  public void testQueueChunkForUpload() throws Exception
+  {
+    ServerSideEncryptingAmazonS3 s3Client = EasyMock.mock(ServerSideEncryptingAmazonS3.class);
+
+    File chunkFile = EasyMock.mock(File.class);
+    EasyMock.expect(chunkFile.length()).andReturn(1024L).anyTimes();
+    EasyMock.expect(chunkFile.delete()).andReturn(true).anyTimes();
+
+    int chunkId = 42;
+    UploadPartResult uploadPartResult = new UploadPartResult();
+    uploadPartResult.setPartNumber(chunkId);
+    uploadPartResult.setETag("etag");
+    EasyMock.expect(s3Client.uploadPart(EasyMock.anyObject(UploadPartRequest.class))).andReturn(uploadPartResult);
+
+    EasyMock.replay(chunkFile, s3Client);
+
+    Future<UploadPartResult> result = s3UploadManager.queueChunkForUpload(s3Client, "test-key", chunkId, chunkFile, "upload-id", s3OutputConfig);
+
+    UploadPartResult futureResult = result.get();
+    Assert.assertEquals(chunkId, futureResult.getPartNumber());
+    Assert.assertEquals("etag", futureResult.getETag());
+  }
+
+  @Test
+  public void testComputeMaxNumChunksOnDisk()
+  {
+    int maxNumConcurrentChunks = S3UploadManager.computeMaxNumChunksOnDisk(s3OutputConfig, s3ExportConfig);
+    int expectedMaxNumConcurrentChunks = 25; // maxChunkSizePossible/200 MB
+    assertEquals(expectedMaxNumConcurrentChunks, maxNumConcurrentChunks);
+  }
+
+  @Test
+  public void testComputeMaxNumChunksOnDiskWithNullOutputConfig()
+  {
+    // Null S3OutputConfig
+    int maxNumConcurrentChunks = S3UploadManager.computeMaxNumChunksOnDisk(null, s3ExportConfig);
+    int expectedMaxNumConcurrentChunks = 25; // maxChunkSizePossible / s3ExportConfig's chunk size
+    assertEquals(expectedMaxNumConcurrentChunks, maxNumConcurrentChunks);
+
+    // Null S3OutputConfig#getChunkSize()
+    maxNumConcurrentChunks = S3UploadManager.computeMaxNumChunksOnDisk(EasyMock.mock(S3OutputConfig.class), s3ExportConfig);
+    assertEquals(expectedMaxNumConcurrentChunks, maxNumConcurrentChunks);
+  }
+
+  @Test
+  public void testComputeMaxNumChunksOnDiskWithNullExportConfig()
+  {
+    // Null S3ExportConfig
+    int maxNumConcurrentChunks = S3UploadManager.computeMaxNumChunksOnDisk(s3OutputConfig, null);
+    int expectedMaxNumConcurrentChunks = 51; // maxChunkSizePossible / s3OutputConfig's chunk size
+    assertEquals(expectedMaxNumConcurrentChunks, maxNumConcurrentChunks);
+
+    // Null S3ExportConfig#getChunkSize()
+    maxNumConcurrentChunks = S3UploadManager.computeMaxNumChunksOnDisk(s3OutputConfig, EasyMock.mock(S3ExportConfig.class));
+    assertEquals(expectedMaxNumConcurrentChunks, maxNumConcurrentChunks);
+  }
+
+  @Test
+  public void testUploadPartIfPossible()
+  {
+    ServerSideEncryptingAmazonS3 s3Client = EasyMock.mock(ServerSideEncryptingAmazonS3.class);
+
+    File chunkFile = EasyMock.mock(File.class);
+    EasyMock.expect(chunkFile.length()).andReturn(1024L).anyTimes();
+
+    UploadPartResult uploadPartResult = new UploadPartResult();
+    Capture<UploadPartRequest> partRequestCapture = EasyMock.newCapture();
+    EasyMock.expect(s3Client.uploadPart(EasyMock.capture(partRequestCapture))).andReturn(uploadPartResult);
+    EasyMock.replay(s3Client, chunkFile);
+
+    UploadPartResult result = s3UploadManager.uploadPartIfPossible(s3Client, "upload-id", "bucket", "key", 1, chunkFile);
+
+    UploadPartRequest capturedRequest = partRequestCapture.getValue();
+    assertEquals("upload-id", capturedRequest.getUploadId());
+    assertEquals("bucket", capturedRequest.getBucketName());
+    assertEquals("key", capturedRequest.getKey());
+    assertEquals(1, capturedRequest.getPartNumber());
+    assertEquals(chunkFile, capturedRequest.getFile());
+    assertEquals(1024L, capturedRequest.getPartSize());
+
+    assertEquals(uploadPartResult, result);
+  }
+
+  @Test
+  public void testStartAndStop()
+  {
+    s3UploadManager.start();
+    s3UploadManager.stop();
+
+    assertTrue(uploadExecutor.isShutdown());

Review Comment:
   this part can go in the suggested `tearDown` method



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on PR #16481:
URL: https://github.com/apache/druid/pull/16481#issuecomment-2133300113

   > @Akshat-Jain : I had some doubts regarding the changes :
   > 
   > 1. Is there an example of how much time does the S3 writes take in real jobs? That can include the fraction of the overall time being consumed as well. Also, maybe that varies for small jobs vs large jobs and the concurrency in the system.
   > 2. I see that parallelism is introduced via a thread-pool in the system. Does that pool ensure some fairness for each writer? Earlier each writer used to get atleast one thread instantly for writing - what are your thoughts on that requirement?
   > 3. Also, adding things arbitrarily in thread-pool might mean that the some parts for a writer may have to wait to be uploaded. Is there any mechanism to know a summary of the wait times of the parts? I am asking this since I'm not sure how would a person evaluate the performance of a write in a concurrent system.
   > 4. I see some thread safe things being done with semaphores in the output stream - is the output stream expected to be thread safe? Or is that done for coordination with the executor among multiple writers. If it for coordination, then should that code reside inside the thread-pool executor somehow? That is also attached to the backpressure mechanism being built per-writer.
   > 5. I find it weird that we're doing `ALL` uploads in the output stream using multipart-uploads. And that includes having `initiateMultipart` call in the output stream's constructor which I personally don't like. Are there any thoughts on improving that by not using multi-part uploads (and rather use plain `PUT` request) for small uploads?
   > 
   > Please let me know your thoughts, if that's possible! 👍
   
   @rohangarg Thanks for the detailed questions!
   
   1. I have put some timing data in the PR description, where you can check queries 5 and 6 to get some idea of the degree to which concurrency helps in output heavy queries. With that said, I can work on getting timing data for queries across these scenarios, which should help us deduce how much time was taken in the durable storage part of the overall thing:
       1. MSQ without this PR's code changes + durable storage disabled
       2. MSQ without this PR's code changes + durable storage enabled
       3. MSQ with this PR's code changes + durable storage enabled
   2. I don't think earlier each writer got a thread instantly for writing. The writing was being done by the processing threadpool, but it's possible to have more workers than the processing threadpool size. I haven't tried such a scenario so far, but I think under this scenario, not all writers will get a thread instantly as there are more workers/writers than the size of processing threadpool. To answer your question about fairness in the PR changes, there's no such mechanism. But as I described, this seems like a separate problem than what this PR is trying to address.
   3. That's a good point. I can add logs for wait times of all parts, but I felt it might be too much logging. Will think more about this. Happy to hear any suggestions as well, if you have any! :)
   4. Semaphore was used to make sure we restrict number of files locally to xyz number of files, to restrict disk space usage at any given point in time. Semaphore wasn't needed in the original code since the disk space was automatically restricted as the same thread was deleting the chunk before writing the next chunk.
   5. I think the rationale here is that we don't know in advance how big the overall file would be. Hence we are doing multi-part uploads for everything. cc: @cryptoe to confirm this point though.
   
   Hope these answer your questions. Happy to discuss further on any of the above points if needed, thanks! :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1616622617


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -199,15 +229,44 @@ private void pushCurrentChunk() throws IOException
   {
     currentChunk.close();
     final Chunk chunk = currentChunk;
-    try {
-      if (chunk.length() > 0) {
-        resultsSize += chunk.length();
+    if (chunk.length() > 0) {
+      s3UploadConfig.incrementCurrentNumChunks();

Review Comment:
   Yeah, I couldn't decide a good name for it. I initially named it S3UploadHelper, but that seemed confusing as well as it doesn't have the upload logic. Hence I settled for S3UploadConfig, as it does define the configuration for the upload mechanism.
   Please let me know if you have any better naming suggestions for this, definitely open to it!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "github-advanced-security[bot] (via GitHub)" <gi...@apache.org>.
github-advanced-security[bot] commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1609316591


##########
extensions-core/s3-extensions/src/test/java/org/apache/druid/storage/s3/output/RetryableS3OutputStreamTest.java:
##########
@@ -286,12 +286,12 @@
       final List<PartETag> eTags = completeRequest.getPartETags();
       Assert.assertEquals(partRequests.size(), eTags.size());
       Assert.assertEquals(
-          partRequests.stream().map(UploadPartRequest::getPartNumber).collect(Collectors.toList()),
-          eTags.stream().map(PartETag::getPartNumber).collect(Collectors.toList())
+          partNumbersFromRequest,
+          eTags.stream().map(PartETag::getPartNumber).collect(Collectors.toSet())
       );
       Assert.assertEquals(
-          partRequests.stream().map(UploadPartRequest::getPartNumber).collect(Collectors.toList()),
-          eTags.stream().map(tag -> Integer.parseInt(tag.getETag())).collect(Collectors.toList())
+          partNumbersFromRequest,
+          eTags.stream().map(tag -> Integer.parseInt(tag.getETag())).collect(Collectors.toSet())

Review Comment:
   ## Missing catch of NumberFormatException
   
   Potential uncaught 'java.lang.NumberFormatException'.
   
   [Show more details](https://github.com/apache/druid/security/code-scanning/7391)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1609455161


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -199,15 +211,47 @@ private void pushCurrentChunk() throws IOException
   {
     currentChunk.close();
     final Chunk chunk = currentChunk;
-    try {
-      if (chunk.length() > 0) {
-        resultsSize += chunk.length();
+    if (chunk.length() > 0) {
+      try {
+        SEMAPHORE.acquire(); // Acquire a permit from the semaphore

Review Comment:
   Between the static semaphore, the executor threads and other locks, the code has become very difficult to follow and clearly error prone.
   
   If we are trying to avoid running out of disk space, then instead of having semaphores, we just need to check some condition in `write()` that there is disk space available (this can be computed based on the formula that @cryptoe has suggested). If there isn't, we should wait for the condition to be satisfied before proceeding with the write. 
   
   There should be a javadoc in `write()` method which also calls this out clearly.
   The class level javadoc might need to be updated too based on the changes here.
   



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -103,22 +107,32 @@ public class RetryableS3OutputStream extends OutputStream
   private boolean error;
   private boolean closed;
 
-  public RetryableS3OutputStream(
-      S3OutputConfig config,
-      ServerSideEncryptingAmazonS3 s3,
-      String s3Key
-  ) throws IOException
-  {
+  /**
+   * An atomic counter to store number of files pending to be uploaded for the particular uploadId.
+   */
+  private final AtomicInteger pendingFiles = new AtomicInteger(0);
 
-    this(config, s3, s3Key, true);
-  }
+  /**
+   * A lock used for notifying the main thread about the completion of s3.uploadPart() for all chunks
+   * and hence starting the s3.completeMultipartUpload() for the uploadId.
+   */
+  private final Object fileLock = new Object();
+
+  /**
+   * Semaphore to restrict the maximum number of simultaneous chunks on disk.
+   */
+  private static final int MAX_CONCURRENT_CHUNKS = 10;
+  private static final Semaphore SEMAPHORE = new Semaphore(MAX_CONCURRENT_CHUNKS);
 
-  @VisibleForTesting
-  protected RetryableS3OutputStream(
+  /**
+   * Threadpool used for uploading the chunks asynchronously.
+   */
+  private static final ExecutorService UPLOAD_EXECUTOR = Execs.multiThreaded(10, "UploadThreadPool-%d");

Review Comment:
   This executor should not be static. If the intention is to share the executor, it should be passed in as an argument to the constructor. Also, there should be a well defined lifecycle of the executor, i.e. when it is started and stopped.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1616511509


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3StorageDruidModule.java:
##########
@@ -182,4 +186,13 @@ public Supplier<ServerSideEncryptingAmazonS3> getAmazonS3ClientSupplier(
   {
     return Suppliers.memoize(serverSideEncryptingAmazonS3Builder::build);
   }
+
+  @Provides
+  @LazySingleton
+  @Named("S3UploadThreadPool")
+  public ExecutorService getUploadPoolExecutorService(ScheduledExecutorFactory scheduledExecutorFactory, RuntimeInfo runtimeInfo)
+  {
+    int poolSize = Math.max(4, 3 * runtimeInfo.getAvailableProcessors());

Review Comment:
   `3 x numProcessors` seems too many as there are going to be other threads doing other work too. I think just `numProcessors` is high enough.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "LakshSingla (via GitHub)" <gi...@apache.org>.
LakshSingla commented on PR #16481:
URL: https://github.com/apache/druid/pull/16481#issuecomment-2134383078

   > idea of the degree to which concurrency helps in output heavy queries
   What is meant by output heavy query? 
   
   > but it's possible to have more workers than the processing threadpool size.
   In which scenario is this possible? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1616716176


##########
extensions-core/s3-extensions/src/test/java/org/apache/druid/storage/s3/S3StorageConnectorProviderTest.java:
##########
@@ -146,6 +148,10 @@ public void configure(Binder binder)
             .addValue(
                 ServerSideEncryptingAmazonS3.class,
                 new ServerSideEncryptingAmazonS3(null, new NoopServerSideEncryption())
+            )
+            .addValue(
+                ExecutorService.class,
+                Execs.multiThreaded(10, "UploadThreadPool-%d")

Review Comment:
   Fair point, changed it to `Execs.singleThreaded("UploadThreadPool-%d")`



##########
extensions-core/s3-extensions/src/test/java/org/apache/druid/storage/s3/S3StorageConnectorProviderTest.java:
##########
@@ -146,6 +148,10 @@ public void configure(Binder binder)
             .addValue(
                 ServerSideEncryptingAmazonS3.class,
                 new ServerSideEncryptingAmazonS3(null, new NoopServerSideEncryption())
+            )
+            .addValue(
+                ExecutorService.class,
+                Execs.multiThreaded(10, "UploadThreadPool-%d")

Review Comment:
   Fair point, will change it to `Execs.singleThreaded("UploadThreadPool-%d")`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on PR #16481:
URL: https://github.com/apache/druid/pull/16481#issuecomment-2132663759

   > Were the tests run on a middle manager or an indexer?
   @LakshSingla On Indexer


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1615486183


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -271,50 +313,72 @@ public void close() throws IOException
       // This should be emitted as a metric
       LOG.info(
           "Pushed total [%d] parts containing [%d] bytes in [%d]ms.",
-          numChunksPushed,
-          resultsSize,
+          numChunksPushed.get(),
+          resultsSize.get(),
           pushStopwatch.elapsed(TimeUnit.MILLISECONDS)
       );
     });
 
-    closer.register(() -> org.apache.commons.io.FileUtils.forceDelete(chunkStorePath));
+    try (Closer ignored = closer) {
+      if (!error) {

Review Comment:
   @LakshSingla We do throw, but the error can get thrown by one of the upload threads, and this if-condition is run by the main thread.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1618124099


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3UploadManager.java:
##########
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import org.apache.druid.java.util.common.HumanReadableBytes;
+import org.apache.druid.java.util.common.logger.Logger;
+
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * This class manages the configuration for uploading files to S3 in chunks.
+ * It tracks the current number of chunks written to local disk concurrently and ensures that the
+ * maximum number of chunks on disk does not exceed a specified limit at any given point in time.
+ */
+public class S3UploadManager
+{
+  /**
+   * The maximum chunk size based on injected values for {@link S3OutputConfig} and {@link S3ExportConfig} used for computing maximum number of chunks to save on disk at any given point in time.
+   * It is initialized to 5 MiB which is the minimum chunk size possible, denoted by {@link S3OutputConfig#S3_MULTIPART_UPLOAD_MIN_PART_SIZE_BYTES}.
+   */
+  private long chunkSize = new HumanReadableBytes("5MiB").getBytes();
+
+  /**
+   * An atomic counter to track the current number of chunks saved to local disk.
+   */
+  private final AtomicInteger currentNumChunks = new AtomicInteger(0);
+
+  /**
+   * The maximum number of chunks that can be saved to local disk concurrently.
+   * This value is recalculated when the chunk size is updated in {@link #updateChunkSizeIfGreater(long)}.
+   */
+  private int maxConcurrentNumChunks = 100;
+
+  /**
+   * Threadpool used for uploading the chunks asynchronously.
+   */

Review Comment:
   Not needed.



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3UploadManager.java:
##########
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import org.apache.druid.java.util.common.HumanReadableBytes;
+import org.apache.druid.java.util.common.logger.Logger;
+
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * This class manages the configuration for uploading files to S3 in chunks.
+ * It tracks the current number of chunks written to local disk concurrently and ensures that the
+ * maximum number of chunks on disk does not exceed a specified limit at any given point in time.
+ */
+public class S3UploadManager
+{
+  /**
+   * The maximum chunk size based on injected values for {@link S3OutputConfig} and {@link S3ExportConfig} used for computing maximum number of chunks to save on disk at any given point in time.
+   * It is initialized to 5 MiB which is the minimum chunk size possible, denoted by {@link S3OutputConfig#S3_MULTIPART_UPLOAD_MIN_PART_SIZE_BYTES}.
+   */
+  private long chunkSize = new HumanReadableBytes("5MiB").getBytes();
+
+  /**
+   * An atomic counter to track the current number of chunks saved to local disk.
+   */

Review Comment:
   Not really needed, just rename the field `currentNumChunksOnDisk`.



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3UploadManager.java:
##########
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import org.apache.druid.java.util.common.HumanReadableBytes;
+import org.apache.druid.java.util.common.logger.Logger;
+
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * This class manages the configuration for uploading files to S3 in chunks.
+ * It tracks the current number of chunks written to local disk concurrently and ensures that the
+ * maximum number of chunks on disk does not exceed a specified limit at any given point in time.

Review Comment:
   ```suggestion
    * It tracks the number of chunks currently present on local disk and ensures that
    * it does not exceed a specified limit.
   ```



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3StorageDruidModule.java:
##########
@@ -182,4 +186,16 @@ public Supplier<ServerSideEncryptingAmazonS3> getAmazonS3ClientSupplier(
   {
     return Suppliers.memoize(serverSideEncryptingAmazonS3Builder::build);
   }
+
+  @Provides
+  @LazySingleton
+  public S3UploadManager getS3UploadManager(ScheduledExecutorFactory scheduledExecutorFactory, RuntimeInfo runtimeInfo)
+  {
+    int poolSize = Math.max(4, runtimeInfo.getAvailableProcessors());
+    ScheduledExecutorService executorService = scheduledExecutorFactory.create(
+        poolSize,
+        "UploadThreadPool-%d"
+    );
+    return new S3UploadManager(executorService);
+  }

Review Comment:
   Instead of this, I guess it would be better to just have an `@Inject` constructor for `S3UploadManager`. That way, the executor service would be created inside the `S3UploadManager`, giving it complete control over its lifecycle.



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -103,27 +104,38 @@ public class RetryableS3OutputStream extends OutputStream
   private boolean error;
   private boolean closed;
 
-  public RetryableS3OutputStream(
-      S3OutputConfig config,
-      ServerSideEncryptingAmazonS3 s3,
-      String s3Key
-  ) throws IOException
-  {
+  /**
+   * An atomic counter to store number of files pending to be uploaded for the particular uploadId.
+   */
+  private final AtomicInteger pendingFiles = new AtomicInteger(0);
 
-    this(config, s3, s3Key, true);
-  }
+  /**
+   * A lock used for notifying the main thread about the completion of s3.uploadPart() for all chunks
+   * and hence starting the s3.completeMultipartUpload() for the uploadId.
+   */
+  private final Object fileLock = new Object();
+
+  /**
+   * Helper class for calculating maximum number of simultaneous chunks allowed on local disk.
+   */
+  private final S3UploadManager uploadManager;
 
-  @VisibleForTesting
-  protected RetryableS3OutputStream(
+  /**
+   * A lock to restrict the maximum number of chunks on disk at any given point in time.
+   */
+  private final Object maxChunksLock = new Object();

Review Comment:
   Shouldn't this restriction be enforced by `S3UploadManager` instead?



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -199,15 +221,44 @@ private void pushCurrentChunk() throws IOException
   {
     currentChunk.close();
     final Chunk chunk = currentChunk;
-    try {
-      if (chunk.length() > 0) {
-        resultsSize += chunk.length();
+    if (chunk.length() > 0) {
+      uploadManager.incrementCurrentNumChunks();
+      pendingFiles.incrementAndGet();
 
-        pushStopwatch.start();
-        pushResults.add(push(chunk));
-        pushStopwatch.stop();
-        numChunksPushed++;
+      uploadManager.submitTask(() -> {
+        try {
+          uploadChunk(chunk);
+        }
+        catch (Exception e) {
+          error = true;
+          LOG.error(e, e.getMessage());
+          throw new RuntimeException(e);
+        }
+        finally {
+          synchronized (maxChunksLock) {
+            uploadManager.decrementCurrentNumChunks();
+            maxChunksLock.notifyAll();
+          }
+          if (pendingFiles.decrementAndGet() == 0) {
+            synchronized (fileLock) {
+              fileLock.notifyAll();
+            }
+          }
+        }
+      });
+    }
+  }

Review Comment:
   All the upload related logic should live in `S3UploadManager` itself, otherwise it is just an `ExecutorService`. Let me know if you need any assistance with this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1628226834


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3UploadManager.java:
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.inject.Inject;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.common.RetryUtils;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.storage.s3.S3Utils;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+import org.apache.druid.utils.RuntimeInfo;
+
+import java.io.File;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
+
+/**
+ * This class manages uploading files to S3 in chunks, while ensuring that the
+ * number of chunks currently present on local disk does not exceed a specific limit.
+ */
+@ManageLifecycle
+public class S3UploadManager
+{
+  private final ExecutorService uploadExecutor;
+
+  private static final Logger log = new Logger(S3UploadManager.class);
+
+  @Inject
+  public S3UploadManager(S3OutputConfig s3OutputConfig, S3ExportConfig s3ExportConfig, RuntimeInfo runtimeInfo)
+  {
+    int poolSize = Math.max(4, runtimeInfo.getAvailableProcessors());
+    int maxNumConcurrentChunks = computeMaxNumConcurrentChunks(s3OutputConfig, s3ExportConfig);
+    this.uploadExecutor = createExecutorService(poolSize, maxNumConcurrentChunks);
+    log.info("Initialized executor service for S3 multipart upload with pool size [%d] and work queue capacity [%d]",
+             poolSize, maxNumConcurrentChunks);
+  }
+
+  /**
+   * Computes the maximum number of concurrent chunks for an S3 multipart upload.
+   * We want to determine the maximum number of concurrent chunks on disk based on the maximum value of chunkSize
+   * between the 2 configs: S3OutputConfig and S3ExportConfig.
+   *
+   * @param s3OutputConfig  The S3 output configuration, which may specify a custom chunk size.
+   * @param s3ExportConfig  The S3 export configuration, which may also specify a custom chunk size.
+   * @return The maximum number of concurrent chunks.
+   */
+  @VisibleForTesting
+  int computeMaxNumConcurrentChunks(S3OutputConfig s3OutputConfig, S3ExportConfig s3ExportConfig)
+  {
+    long chunkSize = S3OutputConfig.S3_MULTIPART_UPLOAD_MIN_PART_SIZE_BYTES;
+    if (s3OutputConfig != null && s3OutputConfig.getChunkSize() != null) {
+      chunkSize = Math.max(chunkSize, s3OutputConfig.getChunkSize());
+    }
+    if (s3ExportConfig != null && s3ExportConfig.getChunkSize() != null) {
+      chunkSize = Math.max(chunkSize, s3ExportConfig.getChunkSize().getBytes());
+    }
+
+    return (int) (S3OutputConfig.S3_MULTIPART_UPLOAD_MAX_PART_SIZE_BYTES / chunkSize);

Review Comment:
   5GB doesn't seem like a bad default but limiting it to 5GB even when we might have much more disk available seems weird. Also, if 5GB is the limit we want to use, we should define a separate constant rather than reuse the constant for max size.
   
   > we should use the overall disk size available for this calculation
   
   Just to clarify, I have not suggested using the entire disk space for storing chunks, rather the portion actually allocated for this purpose. That allocation may either be a hard-coded value, a fixed percentage of the overall disk space or even a config.
   
   In any case, that amount of disk space should be passed as an argument to the `computeMaxNumChunksOnDisk` method.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1609576287


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -103,22 +107,32 @@ public class RetryableS3OutputStream extends OutputStream
   private boolean error;
   private boolean closed;
 
-  public RetryableS3OutputStream(
-      S3OutputConfig config,
-      ServerSideEncryptingAmazonS3 s3,
-      String s3Key
-  ) throws IOException
-  {
+  /**
+   * An atomic counter to store number of files pending to be uploaded for the particular uploadId.
+   */
+  private final AtomicInteger pendingFiles = new AtomicInteger(0);
 
-    this(config, s3, s3Key, true);
-  }
+  /**
+   * A lock used for notifying the main thread about the completion of s3.uploadPart() for all chunks
+   * and hence starting the s3.completeMultipartUpload() for the uploadId.
+   */
+  private final Object fileLock = new Object();
+
+  /**
+   * Semaphore to restrict the maximum number of simultaneous chunks on disk.
+   */
+  private static final int MAX_CONCURRENT_CHUNKS = 10;
+  private static final Semaphore SEMAPHORE = new Semaphore(MAX_CONCURRENT_CHUNKS);
 
-  @VisibleForTesting
-  protected RetryableS3OutputStream(
+  /**
+   * Threadpool used for uploading the chunks asynchronously.
+   */
+  private static final ExecutorService UPLOAD_EXECUTOR = Execs.multiThreaded(10, "UploadThreadPool-%d");

Review Comment:
   > Where would you suggest to shutdown() the executor?
   
   If the executor is meant to be used throughout the lifecycle of the underlying service, shutting it down on service shutdown would make sense. All Druid services are injected with a `ScheduledExecutorFactory` which can be used for creating an executor which is tied to the lifecycle of the service.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1616625803


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -271,50 +328,72 @@ public void close() throws IOException
       // This should be emitted as a metric
       LOG.info(
           "Pushed total [%d] parts containing [%d] bytes in [%d]ms.",
-          numChunksPushed,
-          resultsSize,
+          numChunksPushed.get(),
+          resultsSize.get(),
           pushStopwatch.elapsed(TimeUnit.MILLISECONDS)
       );
     });
 
-    closer.register(() -> org.apache.commons.io.FileUtils.forceDelete(chunkStorePath));
+    try (Closer ignored = closer) {
+      if (!error) {
+        pushCurrentChunk();
+        completeMultipartUpload();
+      }
+    }
+  }
 
-    closer.register(() -> {
-      try {
-        if (resultsSize > 0 && isAllPushSucceeded()) {
-          RetryUtils.retry(
-              () -> s3.completeMultipartUpload(
-                  new CompleteMultipartUploadRequest(config.getBucket(), s3Key, uploadId, pushResults)
-              ),
-              S3Utils.S3RETRY,
-              config.getMaxRetry()
-          );
-        } else {
-          RetryUtils.retry(
-              () -> {
-                s3.cancelMultiPartUpload(new AbortMultipartUploadRequest(config.getBucket(), s3Key, uploadId));
-                return null;
-              },
-              S3Utils.S3RETRY,
-              config.getMaxRetry()
-          );
+  private void completeMultipartUpload()
+  {
+    synchronized (fileLock) {
+      while (pendingFiles.get() > 0) {
+        try {
+          LOG.info("Waiting for lock for completing multipart task for uploadId [%s].", uploadId);

Review Comment:
   Just to clarify: This is one log line per file, not per "part".
   
   Each worker uploads its stage output (comprising of several parts, less than 10 in whatever testing I did) and a success file.
   
   So if we have 10 workers, 3 stages, 5 parts per stage, this line will be roughly logged around `10*3*5 + 10*3*1 == 150 part files + 30 success files == 180 times` for such a query.
   
   This doesn't seem like a huge number to me. I felt it might help in some debugging situations down the line. Thoughts?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "LakshSingla (via GitHub)" <gi...@apache.org>.
LakshSingla commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1615480836


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -271,50 +313,72 @@ public void close() throws IOException
       // This should be emitted as a metric
       LOG.info(
           "Pushed total [%d] parts containing [%d] bytes in [%d]ms.",
-          numChunksPushed,
-          resultsSize,
+          numChunksPushed.get(),
+          resultsSize.get(),
           pushStopwatch.elapsed(TimeUnit.MILLISECONDS)
       );
     });
 
-    closer.register(() -> org.apache.commons.io.FileUtils.forceDelete(chunkStorePath));
+    try (Closer ignored = closer) {
+      if (!error) {

Review Comment:
   Would we ever reach here if error = true? Don't we throw in that case? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1616751823


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3StorageDruidModule.java:
##########
@@ -182,4 +187,20 @@ public Supplier<ServerSideEncryptingAmazonS3> getAmazonS3ClientSupplier(
   {
     return Suppliers.memoize(serverSideEncryptingAmazonS3Builder::build);
   }
+
+  @Provides
+  @LazySingleton
+  @Named("S3UploadThreadPool")
+  public ExecutorService getUploadPoolExecutorService(ScheduledExecutorFactory scheduledExecutorFactory, RuntimeInfo runtimeInfo)
+  {
+    int poolSize = Math.max(4, 3 * runtimeInfo.getAvailableProcessors());

Review Comment:
   I've changed it to `Math.max(4, runtimeInfo.getAvailableProcessors());` as Kashif also suggested to reduce this. Hope that works!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1616588296


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -103,13 +109,35 @@ public class RetryableS3OutputStream extends OutputStream
   private boolean error;
   private boolean closed;
 
+  /**
+   * A map to store number of files pending to be uploaded for a given uploadId.
+   */
+  private static final ConcurrentHashMap<String, AtomicInteger> PENDING_FILES = new ConcurrentHashMap<>();
+
+  /**
+   * A map containing the lock for a given uploadId used for notifying the main thread about the completion of
+   * s3.uploadPart() for all chunks for the uploadId, and hence starting the s3.completeMultipartUpload() for
+   * the uploadId.
+   */
+  private static final ConcurrentHashMap<String, Object> FILE_LOCK_MAP = new ConcurrentHashMap<>();
+
+  /**
+   * Semaphore to restrict the maximum number of simultaneous chunks on disk.
+   */
+  private static final int MAX_CONCURRENT_CHUNKS = 10;
+  private static final Semaphore SEMAPHORE = new Semaphore(MAX_CONCURRENT_CHUNKS);
+
+  /**
+   * Threadpool used for uploading the chunks asynchronously.
+   */
+  private static final ExecutorService UPLOAD_EXECUTOR = Executors.newFixedThreadPool(10);

Review Comment:
   @cryptoe Have made the change. We can benchmark further on this by varying it a bit if needed, but appreciate if you check if the overall changes look good.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1629670678


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3UploadManager.java:
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.inject.Inject;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.common.RetryUtils;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.storage.s3.S3Utils;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+import org.apache.druid.utils.RuntimeInfo;
+
+import java.io.File;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
+
+/**
+ * This class manages uploading files to S3 in chunks, while ensuring that the
+ * number of chunks currently present on local disk does not exceed a specific limit.
+ */
+@ManageLifecycle
+public class S3UploadManager
+{
+  private final ExecutorService uploadExecutor;
+
+  private static final Logger log = new Logger(S3UploadManager.class);
+
+  @Inject
+  public S3UploadManager(S3OutputConfig s3OutputConfig, S3ExportConfig s3ExportConfig, RuntimeInfo runtimeInfo)
+  {
+    int poolSize = Math.max(4, runtimeInfo.getAvailableProcessors());
+    int maxNumConcurrentChunks = computeMaxNumConcurrentChunks(s3OutputConfig, s3ExportConfig);
+    this.uploadExecutor = createExecutorService(poolSize, maxNumConcurrentChunks);
+    log.info("Initialized executor service for S3 multipart upload with pool size [%d] and work queue capacity [%d]",
+             poolSize, maxNumConcurrentChunks);
+  }
+
+  /**
+   * Computes the maximum number of concurrent chunks for an S3 multipart upload.
+   * We want to determine the maximum number of concurrent chunks on disk based on the maximum value of chunkSize
+   * between the 2 configs: S3OutputConfig and S3ExportConfig.
+   *
+   * @param s3OutputConfig  The S3 output configuration, which may specify a custom chunk size.
+   * @param s3ExportConfig  The S3 export configuration, which may also specify a custom chunk size.
+   * @return The maximum number of concurrent chunks.
+   */
+  @VisibleForTesting
+  int computeMaxNumConcurrentChunks(S3OutputConfig s3OutputConfig, S3ExportConfig s3ExportConfig)

Review Comment:
   Resolving this comment as per the discussion at https://github.com/apache/druid/pull/16481#discussion_r1629278464



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "cryptoe (via GitHub)" <gi...@apache.org>.
cryptoe commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1609306271


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -103,13 +109,35 @@ public class RetryableS3OutputStream extends OutputStream
   private boolean error;
   private boolean closed;
 
+  /**
+   * A map to store number of files pending to be uploaded for a given uploadId.
+   */
+  private static final ConcurrentHashMap<String, AtomicInteger> PENDING_FILES = new ConcurrentHashMap<>();
+
+  /**
+   * A map containing the lock for a given uploadId used for notifying the main thread about the completion of
+   * s3.uploadPart() for all chunks for the uploadId, and hence starting the s3.completeMultipartUpload() for
+   * the uploadId.
+   */
+  private static final ConcurrentHashMap<String, Object> FILE_LOCK_MAP = new ConcurrentHashMap<>();
+
+  /**
+   * Semaphore to restrict the maximum number of simultaneous chunks on disk.
+   */
+  private static final int MAX_CONCURRENT_CHUNKS = 10;

Review Comment:
   This is dependent on 
   * Size of the local disk we have
   * SIze of the chunk
   * Number of upload threads 
   So we need to come up with a formula like `WorkerStorageParameters`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1609474708


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -103,22 +107,32 @@ public class RetryableS3OutputStream extends OutputStream
   private boolean error;
   private boolean closed;
 
-  public RetryableS3OutputStream(
-      S3OutputConfig config,
-      ServerSideEncryptingAmazonS3 s3,
-      String s3Key
-  ) throws IOException
-  {
+  /**
+   * An atomic counter to store number of files pending to be uploaded for the particular uploadId.
+   */
+  private final AtomicInteger pendingFiles = new AtomicInteger(0);
 
-    this(config, s3, s3Key, true);
-  }
+  /**
+   * A lock used for notifying the main thread about the completion of s3.uploadPart() for all chunks
+   * and hence starting the s3.completeMultipartUpload() for the uploadId.
+   */
+  private final Object fileLock = new Object();
+
+  /**
+   * Semaphore to restrict the maximum number of simultaneous chunks on disk.
+   */
+  private static final int MAX_CONCURRENT_CHUNKS = 10;
+  private static final Semaphore SEMAPHORE = new Semaphore(MAX_CONCURRENT_CHUNKS);
 
-  @VisibleForTesting
-  protected RetryableS3OutputStream(
+  /**
+   * Threadpool used for uploading the chunks asynchronously.
+   */
+  private static final ExecutorService UPLOAD_EXECUTOR = Execs.multiThreaded(10, "UploadThreadPool-%d");

Review Comment:
   > This executor should not be static. If the intention is to share the executor, it should be passed in as an argument to the constructor.
   
   But then we'd have to inject it and pass it through a bunch of files: S3StorageConnectorModule, S3ExportStorageProvider, S3StorageConnectorProvider, StorageConnector, RetryableS3OutputStream.
   
   Seems unnecessary to me? For my understanding, what benefit does it provide over the current approach?
   
   > Also, there should be a well defined lifecycle of the executor, i.e. when it is started and stopped.
   
   Where would you suggest to shutdown() the executor?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1609685088


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -103,22 +107,32 @@ public class RetryableS3OutputStream extends OutputStream
   private boolean error;
   private boolean closed;
 
-  public RetryableS3OutputStream(
-      S3OutputConfig config,
-      ServerSideEncryptingAmazonS3 s3,
-      String s3Key
-  ) throws IOException
-  {
+  /**
+   * An atomic counter to store number of files pending to be uploaded for the particular uploadId.
+   */
+  private final AtomicInteger pendingFiles = new AtomicInteger(0);
 
-    this(config, s3, s3Key, true);
-  }
+  /**
+   * A lock used for notifying the main thread about the completion of s3.uploadPart() for all chunks
+   * and hence starting the s3.completeMultipartUpload() for the uploadId.
+   */
+  private final Object fileLock = new Object();
+
+  /**
+   * Semaphore to restrict the maximum number of simultaneous chunks on disk.
+   */
+  private static final int MAX_CONCURRENT_CHUNKS = 10;
+  private static final Semaphore SEMAPHORE = new Semaphore(MAX_CONCURRENT_CHUNKS);
 
-  @VisibleForTesting
-  protected RetryableS3OutputStream(
+  /**
+   * Threadpool used for uploading the chunks asynchronously.
+   */
+  private static final ExecutorService UPLOAD_EXECUTOR = Execs.multiThreaded(10, "UploadThreadPool-%d");

Review Comment:
   I tried adding the following in `S3StorageConnectorModule`:
   ```java
   @Provides
     @LazySingleton
     public ScheduledExecutorFactory getScheduledExecutorFactory(Lifecycle lifecycle)
     {
       return ScheduledExecutors.createFactory(lifecycle);
     }
   ```
   ...and then `@Inject ScheduledExecutorFactory scheduledExecutorFactory;` in `S3StorageConnectorProvider`, but it's null.
   
   Can you advise on how/where to inject it to be available in the `druid-s3-extensions`? Thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "cryptoe (via GitHub)" <gi...@apache.org>.
cryptoe commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1621695546


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -166,13 +176,25 @@ public void write(byte[] b, int off, int len) throws IOException
       return;
     }
 
+    synchronized (maxChunksLock) {
+      while (uploadManager.getCurrentNumChunksOnDisk() > uploadManager.getMaxConcurrentNumChunks()) {
+        try {
+          LOG.debug("Waiting for lock for writing further chunks to local disk.");
+          maxChunksLock.wait();
+        }
+        catch (InterruptedException e) {
+          Thread.currentThread().interrupt();

Review Comment:
   Clearing the current thread of the interupt flag is not needed IMHO . 



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3UploadManager.java:
##########
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.google.inject.Inject;
+import org.apache.druid.java.util.common.HumanReadableBytes;
+import org.apache.druid.java.util.common.concurrent.ScheduledExecutorFactory;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.utils.RuntimeInfo;
+
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * This class manages the configuration for uploading files to S3 in chunks.
+ * It tracks the number of chunks currently present on local disk and ensures that
+ * it does not exceed a specified limit.
+ */
+public class S3UploadManager
+{
+  /**
+   * The maximum chunk size based on injected values for {@link S3OutputConfig} and {@link S3ExportConfig} used for computing maximum number of chunks to save on disk at any given point in time.
+   * It is initialized to 5 MiB which is the minimum chunk size possible, denoted by {@link S3OutputConfig#S3_MULTIPART_UPLOAD_MIN_PART_SIZE_BYTES}.
+   */
+  private long chunkSize = new HumanReadableBytes("5MiB").getBytes();

Review Comment:
   The default should not be 5 but something like 100 MB I think



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3UploadManager.java:
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.inject.Inject;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.common.RetryUtils;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.storage.s3.S3Utils;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+import org.apache.druid.utils.RuntimeInfo;
+
+import java.io.File;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
+
+/**
+ * This class manages uploading files to S3 in chunks, while ensuring that the
+ * number of chunks currently present on local disk does not exceed a specific limit.
+ */
+@ManageLifecycle
+public class S3UploadManager
+{
+  private final ExecutorService uploadExecutor;
+
+  private static final Logger log = new Logger(S3UploadManager.class);
+
+  @Inject
+  public S3UploadManager(S3OutputConfig s3OutputConfig, S3ExportConfig s3ExportConfig, RuntimeInfo runtimeInfo)
+  {
+    int poolSize = Math.max(4, runtimeInfo.getAvailableProcessors());
+    int maxNumConcurrentChunks = computeMaxNumConcurrentChunks(s3OutputConfig, s3ExportConfig);
+    this.uploadExecutor = createExecutorService(poolSize, maxNumConcurrentChunks);
+    log.info("Initialized executor service for S3 multipart upload with pool size [%d] and work queue capacity [%d]",
+             poolSize, maxNumConcurrentChunks);
+  }
+
+  /**
+   * Computes the maximum number of concurrent chunks for an S3 multipart upload.
+   * We want to determine the maximum number of concurrent chunks on disk based on the maximum value of chunkSize
+   * between the 2 configs: S3OutputConfig and S3ExportConfig.
+   *
+   * @param s3OutputConfig  The S3 output configuration, which may specify a custom chunk size.
+   * @param s3ExportConfig  The S3 export configuration, which may also specify a custom chunk size.
+   * @return The maximum number of concurrent chunks.
+   */
+  @VisibleForTesting
+  int computeMaxNumConcurrentChunks(S3OutputConfig s3OutputConfig, S3ExportConfig s3ExportConfig)
+  {
+    long chunkSize = S3OutputConfig.S3_MULTIPART_UPLOAD_MIN_PART_SIZE_BYTES;
+    if (s3OutputConfig != null && s3OutputConfig.getChunkSize() != null) {
+      chunkSize = Math.max(chunkSize, s3OutputConfig.getChunkSize());
+    }
+    if (s3ExportConfig != null && s3ExportConfig.getChunkSize() != null) {
+      chunkSize = Math.max(chunkSize, s3ExportConfig.getChunkSize().getBytes());
+    }
+
+    return (int) (S3OutputConfig.S3_MULTIPART_UPLOAD_MAX_PART_SIZE_BYTES / chunkSize);
+  }
+
+  /**
+   * Queues a chunk of a file for upload to S3 as part of a multipart upload.
+   */
+  public Future<UploadPartResult> queueChunkForUpload(ServerSideEncryptingAmazonS3 s3Client, String key, int chunkNumber, File chunkFile, String uploadId, S3OutputConfig config)
+  {
+    return uploadExecutor.submit(() -> RetryUtils.retry(
+        () -> {
+          log.info("Uploading chunk [%d] for uploadId [%s].", chunkNumber, uploadId);
+          UploadPartResult uploadPartResult = uploadPartIfPossible(
+              s3Client,
+              uploadId,
+              config.getBucket(),
+              key,
+              chunkNumber,
+              chunkFile
+          );
+          if (!chunkFile.delete()) {
+            log.warn("Failed to delete chunk [%s]", chunkFile.getAbsolutePath());
+          }
+          return uploadPartResult;
+        },
+        S3Utils.S3RETRY,
+        config.getMaxRetry()
+    ));
+  }
+
+  @VisibleForTesting
+  UploadPartResult uploadPartIfPossible(
+      ServerSideEncryptingAmazonS3 s3Client,
+      String uploadId,
+      String bucket,
+      String key,
+      int chunkNumber,
+      File chunkFile
+  )
+  {
+    final UploadPartRequest uploadPartRequest = new UploadPartRequest()
+        .withUploadId(uploadId)
+        .withBucketName(bucket)
+        .withKey(key)
+        .withFile(chunkFile)
+        .withPartNumber(chunkNumber)
+        .withPartSize(chunkFile.length());
+
+    if (log.isDebugEnabled()) {
+      log.debug("Pushing chunk [%s] to bucket[%s] and key[%s].", chunkNumber, bucket, key);
+    }
+    return s3Client.uploadPart(uploadPartRequest);
+  }
+
+  @VisibleForTesting
+  ExecutorService createExecutorService(int poolSize, int maxNumConcurrentChunks)
+  {
+    return Execs.newBlockingThreaded("UploadThreadPool-%d", poolSize, maxNumConcurrentChunks);
+  }
+
+  @LifecycleStart
+  public void start()
+  {
+    // No state startup required
+  }
+
+  @LifecycleStop
+  public void stop()
+  {
+    log.debug("Stopping S3UploadManager");

Review Comment:
   This can be info since this is stopped only once in a jvm. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "cryptoe (via GitHub)" <gi...@apache.org>.
cryptoe commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1609309586


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -103,13 +109,35 @@ public class RetryableS3OutputStream extends OutputStream
   private boolean error;
   private boolean closed;
 
+  /**
+   * A map to store number of files pending to be uploaded for a given uploadId.
+   */
+  private static final ConcurrentHashMap<String, AtomicInteger> PENDING_FILES = new ConcurrentHashMap<>();
+
+  /**
+   * A map containing the lock for a given uploadId used for notifying the main thread about the completion of
+   * s3.uploadPart() for all chunks for the uploadId, and hence starting the s3.completeMultipartUpload() for
+   * the uploadId.
+   */
+  private static final ConcurrentHashMap<String, Object> FILE_LOCK_MAP = new ConcurrentHashMap<>();
+
+  /**
+   * Semaphore to restrict the maximum number of simultaneous chunks on disk.
+   */
+  private static final int MAX_CONCURRENT_CHUNKS = 10;
+  private static final Semaphore SEMAPHORE = new Semaphore(MAX_CONCURRENT_CHUNKS);
+
+  /**
+   * Threadpool used for uploading the chunks asynchronously.
+   */
+  private static final ExecutorService UPLOAD_EXECUTOR = Executors.newFixedThreadPool(10);

Review Comment:
   This should be dependent on the number of cores. 
   Checkout `WorkerMemoryParameters` to figure out how we calculate number of cores. 
   I think `Math.max(4,3*numberOfCores)` maybe but we should benchmark. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "rohangarg (via GitHub)" <gi...@apache.org>.
rohangarg commented on PR #16481:
URL: https://github.com/apache/druid/pull/16481#issuecomment-2133077298

   @Akshat-Jain : I had some doubts regarding the changes : 
   1. Is there an example of how much time does the S3 writes take in real jobs? That can include the fraction of the overall time being consumed as well. Also, maybe that varies for small jobs vs large jobs and the concurrency in the system.
   2. I see that parallelism is introduced via a thread-pool in the system. Does that pool ensure some fairness for each writer? Earlier each writer used to get atleast one thread instantly for writing - what are your thoughts on that requirement?
   3. Also, adding things arbitrarily in thread-pool might mean that the some parts for a writer may have to wait to be uploaded. Is there any mechanism to know a summary of the wait times of the parts? I am asking this since I'm not sure how would a person evaluate the performance of a write in a concurrent system.
   4. I see some thread safe things being done with semaphores in the output stream - is the output stream expected to be thread safe? Or is that done for coordination with the executor among multiple writers. If it for coordination, then should that code reside inside the thread-pool executor somehow? That is also attached to the backpressure mechanism being built per-writer.
   5. I find it weird that we're doing `ALL` uploads in the output stream using multipart-uploads. And that includes having `initiateMultipart` call in the output stream's constructor which I personally don't like. Are there any thoughts on improving that by not using multi-part uploads (and rather use plain `PUT` request) for small uploads? 
   
   
   Please let me know your thoughts, if that's possible! :+1:  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1616592364


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -199,15 +211,47 @@ private void pushCurrentChunk() throws IOException
   {
     currentChunk.close();
     final Chunk chunk = currentChunk;
-    try {
-      if (chunk.length() > 0) {
-        resultsSize += chunk.length();
+    if (chunk.length() > 0) {
+      try {
+        SEMAPHORE.acquire(); // Acquire a permit from the semaphore

Review Comment:
   @kfaraz I had a discussion with @cryptoe yesterday about this approach.
   While it would theoretically be possible, this approach will get very complicated and error prone to manage considering how our code is structured, since there are way too many edge scenarios that can go wrong.
   
   `write()` method is called concurrently by multiple threads, and we run into edge scenarios like:
   ```
   say chunk size is 5 MB, and disk space available is 10 MB.
   
   thread 1 has written 3 MB
   thread 2 has written 3 MB
   thread 3 has written 3 MB
   
   Deadlock!!! --- since each thread has written enough data to reach the disk space threshold, but not enough data to reach the chunk size threshold, and hence none of them trigger an upload(), and hence all of them are stuck in deadlock.
   ```
   
   Based on yesterday's discussion with Karan, we concluded that we can go with `number of chunks` criteria instead of bytes stored, and hence I've implemented that approach via S3UploadConfig class.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1616755692


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3UploadConfig.java:
##########
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import org.apache.druid.guice.LazySingleton;
+import org.apache.druid.java.util.common.HumanReadableBytes;
+import org.apache.druid.java.util.common.logger.Logger;
+
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * This class manages the configuration for uploading files to S3 in chunks.
+ * It tracks the current number of chunks written to local disk concurrently and ensures that the
+ * maximum number of chunks on disk does not exceed a specified limit at any given point in time.
+ */
+@LazySingleton
+public class S3UploadConfig
+{
+  /**
+   * The maximum chunk size based on injected values for {@link S3OutputConfig} and {@link S3ExportConfig} used for computing maximum number of chunks to save on disk at any given point in time.
+   * It is initialized to 5 MiB which is the minimum chunk size possible, denoted by {@link S3OutputConfig#S3_MULTIPART_UPLOAD_MIN_PART_SIZE_BYTES}.
+   */
+  private long chunkSize = new HumanReadableBytes("5MiB").getBytes();
+
+  /**
+   * An atomic counter to track the current number of chunks saved to local disk.
+   */
+  private final AtomicInteger currentNumChunks = new AtomicInteger(0);

Review Comment:
   @LakshSingla I'm closing this comment in favor of this other comment which is kind of a superset: https://github.com/apache/druid/pull/16481#discussion_r1616615880
   
   I'm working on the other comment, will leave that comment open until then for tracking. Hope that works!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on PR #16481:
URL: https://github.com/apache/druid/pull/16481#issuecomment-2134806610

   @rohangarg 
   
   > Ok, although the bandwidth comes out to be around < 1 MB/sec with this calculation. So, I'm not sure about realistic nature of the test.
   
   The data I mentioned was average of 3+ readings. I can do it again (once all review comments are addressed and everyone is okay with the implementation).
   
   > Now, if you've written a chunk, you'll put it in an executor which might be consumed by the parts of other writers and you'll wait in the queue for your upload to be initiated. My question was that is the new behavior expected and fine?
   
   Yes, the upload will happen sooner (despite the potential wait) than it would have happened in the original sequential code.
   
   > I meant that the code for semaphores directly in the output stream seems more related to the behavior of the output stream rather than a property of the uploading mechanism.
   
   We don't want the main threads to be blocked by the upload threads (except for the last chunk where we have to complete the multipart upload). Hence, the main thread cannot release the semaphore post finish of upload(), if that's what the suggestion was.
   
   > Yes, this would only work for a single chunk.
   
   Cool, I agree then. The only counter point I have would be the maintainability angle. I'll discuss it with Karan offline if it makes sense to go for it. Either way, this can go in a separate follow-up PR, if it has to. Hope that works!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1618168687


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -103,27 +104,38 @@ public class RetryableS3OutputStream extends OutputStream
   private boolean error;
   private boolean closed;
 
-  public RetryableS3OutputStream(
-      S3OutputConfig config,
-      ServerSideEncryptingAmazonS3 s3,
-      String s3Key
-  ) throws IOException
-  {
+  /**
+   * An atomic counter to store number of files pending to be uploaded for the particular uploadId.
+   */
+  private final AtomicInteger pendingFiles = new AtomicInteger(0);
 
-    this(config, s3, s3Key, true);
-  }
+  /**
+   * A lock used for notifying the main thread about the completion of s3.uploadPart() for all chunks
+   * and hence starting the s3.completeMultipartUpload() for the uploadId.
+   */
+  private final Object fileLock = new Object();
+
+  /**
+   * Helper class for calculating maximum number of simultaneous chunks allowed on local disk.
+   */
+  private final S3UploadManager uploadManager;
 
-  @VisibleForTesting
-  protected RetryableS3OutputStream(
+  /**
+   * A lock to restrict the maximum number of chunks on disk at any given point in time.
+   */
+  private final Object maxChunksLock = new Object();

Review Comment:
   @kfaraz This lock is to make the processing thread wait before writing further chunks on disk. The `S3UploadManager` would only deal with the upload threads. This relates to my comment here on the refactoring suggestion: https://github.com/apache/druid/pull/16481#discussion_r1618143342



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1618175763


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -199,15 +221,44 @@ private void pushCurrentChunk() throws IOException
   {
     currentChunk.close();
     final Chunk chunk = currentChunk;
-    try {
-      if (chunk.length() > 0) {
-        resultsSize += chunk.length();
+    if (chunk.length() > 0) {
+      uploadManager.incrementCurrentNumChunks();
+      pendingFiles.incrementAndGet();
 
-        pushStopwatch.start();
-        pushResults.add(push(chunk));
-        pushStopwatch.stop();
-        numChunksPushed++;
+      uploadManager.submitTask(() -> {
+        try {
+          uploadChunk(chunk);
+        }
+        catch (Exception e) {
+          error = true;
+          LOG.error(e, e.getMessage());
+          throw new RuntimeException(e);
+        }
+        finally {
+          synchronized (maxChunksLock) {
+            uploadManager.decrementCurrentNumChunks();
+            maxChunksLock.notifyAll();
+          }
+          if (pendingFiles.decrementAndGet() == 0) {
+            synchronized (fileLock) {
+              fileLock.notifyAll();
+            }
+          }
+        }
+      });
+    }
+  }

Review Comment:
   Or maybe exposing a method like this in S3UploadManager?
   ```java
   public boolean canWriteMoreChunksOnDisk()
     {
       synchronized (maxChunksLock) {
         while (uploadManager.getCurrentNumChunksOnDisk() > uploadManager.getMaxConcurrentNumChunks()) {
           try {
             LOG.debug("Waiting for lock for writing further chunks to local disk.");
             maxChunksLock.wait();
           }
           catch (InterruptedException e) {
             Thread.currentThread().interrupt();
             throw new RuntimeException(e);
           }
         }
       }
     }
   ```
   
   And replacing the above code in RetryableS3OutputStream with `canWriteMoreChunksOnDisk()` 🤔 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1617223785


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -271,50 +328,72 @@ public void close() throws IOException
       // This should be emitted as a metric
       LOG.info(
           "Pushed total [%d] parts containing [%d] bytes in [%d]ms.",
-          numChunksPushed,
-          resultsSize,
+          numChunksPushed.get(),
+          resultsSize.get(),
           pushStopwatch.elapsed(TimeUnit.MILLISECONDS)
       );
     });
 
-    closer.register(() -> org.apache.commons.io.FileUtils.forceDelete(chunkStorePath));
+    try (Closer ignored = closer) {
+      if (!error) {
+        pushCurrentChunk();
+        completeMultipartUpload();
+      }
+    }
+  }
 
-    closer.register(() -> {
-      try {
-        if (resultsSize > 0 && isAllPushSucceeded()) {
-          RetryUtils.retry(
-              () -> s3.completeMultipartUpload(
-                  new CompleteMultipartUploadRequest(config.getBucket(), s3Key, uploadId, pushResults)
-              ),
-              S3Utils.S3RETRY,
-              config.getMaxRetry()
-          );
-        } else {
-          RetryUtils.retry(
-              () -> {
-                s3.cancelMultiPartUpload(new AbortMultipartUploadRequest(config.getBucket(), s3Key, uploadId));
-                return null;
-              },
-              S3Utils.S3RETRY,
-              config.getMaxRetry()
-          );
+  private void completeMultipartUpload()
+  {
+    synchronized (fileLock) {
+      while (pendingFiles.get() > 0) {
+        try {
+          LOG.info("Waiting for lock for completing multipart task for uploadId [%s].", uploadId);

Review Comment:
   Fair enough, changed it to `LOG.debug()`
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1618128051


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3UploadManager.java:
##########
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import org.apache.druid.java.util.common.HumanReadableBytes;
+import org.apache.druid.java.util.common.logger.Logger;
+
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * This class manages the configuration for uploading files to S3 in chunks.
+ * It tracks the current number of chunks written to local disk concurrently and ensures that the
+ * maximum number of chunks on disk does not exceed a specified limit at any given point in time.
+ */
+public class S3UploadManager
+{
+  /**
+   * The maximum chunk size based on injected values for {@link S3OutputConfig} and {@link S3ExportConfig} used for computing maximum number of chunks to save on disk at any given point in time.
+   * It is initialized to 5 MiB which is the minimum chunk size possible, denoted by {@link S3OutputConfig#S3_MULTIPART_UPLOAD_MIN_PART_SIZE_BYTES}.
+   */
+  private long chunkSize = new HumanReadableBytes("5MiB").getBytes();
+
+  /**
+   * An atomic counter to track the current number of chunks saved to local disk.
+   */
+  private final AtomicInteger currentNumChunks = new AtomicInteger(0);
+
+  /**
+   * The maximum number of chunks that can be saved to local disk concurrently.
+   * This value is recalculated when the chunk size is updated in {@link #updateChunkSizeIfGreater(long)}.
+   */
+  private int maxConcurrentNumChunks = 100;
+
+  /**
+   * Threadpool used for uploading the chunks asynchronously.
+   */
+  private final ExecutorService uploadExecutor;
+
+  private static final Logger log = new Logger(S3UploadManager.class);
+
+  public S3UploadManager(ExecutorService executorService)
+  {
+    this.uploadExecutor = executorService;
+  }
+
+  /**
+   * Increments the counter for the current number of chunks saved on disk.
+   */
+  public void incrementCurrentNumChunks()
+  {
+    currentNumChunks.incrementAndGet();
+  }
+
+  /**
+   * Decrements the counter for the current number of chunks saved on disk.
+   */
+  public void decrementCurrentNumChunks()
+  {
+    currentNumChunks.decrementAndGet();
+  }
+
+  /**
+   * Gets the current number of chunks saved to local disk.
+   *
+   * @return the current number of chunks saved to local disk.
+   */
+  public int getCurrentNumChunks()
+  {
+    return currentNumChunks.get();
+  }

Review Comment:
   This class should ideally only expose a `queueChunkUpload()` method. All other logic should be internal to this method.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1616620124


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3UploadConfig.java:
##########
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import org.apache.druid.guice.LazySingleton;
+import org.apache.druid.java.util.common.HumanReadableBytes;
+import org.apache.druid.java.util.common.logger.Logger;
+
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * This class manages the configuration for uploading files to S3 in chunks.
+ * It tracks the current number of chunks written to local disk concurrently and ensures that the
+ * maximum number of chunks on disk does not exceed a specified limit at any given point in time.
+ */
+@LazySingleton
+public class S3UploadConfig
+{
+  /**
+   * The maximum chunk size based on injected values for {@link S3OutputConfig} and {@link S3ExportConfig} used for computing maximum number of chunks to save on disk at any given point in time.
+   * It is initialized to 5 MiB which is the minimum chunk size possible, denoted by {@link S3OutputConfig#S3_MULTIPART_UPLOAD_MIN_PART_SIZE_BYTES}.
+   */
+  private long chunkSize = new HumanReadableBytes("5MiB").getBytes();
+
+  /**
+   * An atomic counter to track the current number of chunks saved to local disk.
+   */
+  private final AtomicInteger currentNumChunks = new AtomicInteger(0);
+
+  /**
+   * The maximum number of chunks that can be saved to local disk concurrently.
+   * This value is recalculated when the chunk size is updated in {@link #updateChunkSizeIfGreater(long)}.
+   */
+  private int maxConcurrentNumChunks = 100;

Review Comment:
   > Also, why is it not an atomic integer?
   
   It doesn't really need to be IIUC.
   
   Answering your other comment about how often we would end up recomputing this (as it's relevant here): `recomputeMaxConcurrentNumChunks()` would be called only when `S3OutputConfig` and `S3ExportConfig` are injected initially. We needed a global place to calculate the max chunk size (out of the two configs), and use that to calculate the maximum number of chunks we want to allow on the disk simultaneously.
   
   In comparison, `currentNumChunks` is marked as an atomic integer since it's updated by the upload threads.
   
   Hope this makes sense.



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3UploadConfig.java:
##########
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import org.apache.druid.guice.LazySingleton;
+import org.apache.druid.java.util.common.HumanReadableBytes;
+import org.apache.druid.java.util.common.logger.Logger;
+
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * This class manages the configuration for uploading files to S3 in chunks.
+ * It tracks the current number of chunks written to local disk concurrently and ensures that the
+ * maximum number of chunks on disk does not exceed a specified limit at any given point in time.
+ */
+@LazySingleton
+public class S3UploadConfig
+{
+  /**
+   * The maximum chunk size based on injected values for {@link S3OutputConfig} and {@link S3ExportConfig} used for computing maximum number of chunks to save on disk at any given point in time.
+   * It is initialized to 5 MiB which is the minimum chunk size possible, denoted by {@link S3OutputConfig#S3_MULTIPART_UPLOAD_MIN_PART_SIZE_BYTES}.
+   */
+  private long chunkSize = new HumanReadableBytes("5MiB").getBytes();
+
+  /**
+   * An atomic counter to track the current number of chunks saved to local disk.
+   */
+  private final AtomicInteger currentNumChunks = new AtomicInteger(0);
+
+  /**
+   * The maximum number of chunks that can be saved to local disk concurrently.
+   * This value is recalculated when the chunk size is updated in {@link #updateChunkSizeIfGreater(long)}.
+   */
+  private int maxConcurrentNumChunks = 100;
+
+  private static final Logger log = new Logger(S3UploadConfig.class);
+
+  /**
+   * Increments the counter for the current number of chunks saved on disk.
+   */
+  public void incrementCurrentNumChunks()
+  {
+    currentNumChunks.incrementAndGet();
+  }
+
+  /**
+   * Decrements the counter for the current number of chunks saved on disk.
+   */
+  public void decrementCurrentNumChunks()
+  {
+    currentNumChunks.decrementAndGet();
+  }
+
+  /**
+   * Gets the current number of chunks saved to local disk.
+   *
+   * @return the current number of chunks saved to local disk.
+   */
+  public int getCurrentNumChunks()
+  {
+    return currentNumChunks.get();
+  }
+
+  /**
+   * Gets the maximum number of concurrent chunks that can be saved to local disk.
+   *
+   * @return the maximum number of concurrent chunks that can be saved to local disk.
+   */
+  public int getMaxConcurrentNumChunks()
+  {
+    return maxConcurrentNumChunks;
+  }
+
+  /**
+   * Updates the chunk size if the provided size is greater than the current chunk size.
+   * Recomputes the maximum number of concurrent chunks that can be saved to local disk based on the new chunk size.
+   *
+   * @param chunkSize the new chunk size to be set if it is greater than the current chunk size.
+   */
+  public void updateChunkSizeIfGreater(long chunkSize)
+  {
+    if (this.chunkSize < chunkSize) {
+      this.chunkSize = chunkSize;
+      recomputeMaxConcurrentNumChunks();
+    }
+  }
+
+  /**
+   * Recomputes the maximum number of concurrent chunks that can be saved to local disk based on the current chunk size.
+   * The maximum allowed chunk size is {@link S3OutputConfig#S3_MULTIPART_UPLOAD_MAX_PART_SIZE_BYTES} which is quite big,
+   * so we restrict the maximum disk space used to the same, at any given point in time.
+   */
+  private void recomputeMaxConcurrentNumChunks()
+  {
+    maxConcurrentNumChunks = (int) (S3OutputConfig.S3_MULTIPART_UPLOAD_MAX_PART_SIZE_BYTES / chunkSize);
+    log.info("Recomputed maxConcurrentNumChunks: %d", maxConcurrentNumChunks);

Review Comment:
   Twice. Please see: https://github.com/apache/druid/pull/16481#discussion_r1616620124



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1627923604


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -269,52 +213,74 @@ public void close() throws IOException
     // Closeables are closed in LIFO order
     closer.register(() -> {
       // This should be emitted as a metric
+      long totalChunkSize = (currentChunk.id - 1) * chunkSize + currentChunk.length();

Review Comment:
   > To me, having the map seems cleaner as we know which future belongs to which chunk
   
   @kfaraz The returned future (if successful) contains UploadPartResult which contains the partNumber which is the chunk ID, if that helps.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1628218203


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -269,52 +213,74 @@ public void close() throws IOException
     // Closeables are closed in LIFO order
     closer.register(() -> {
       // This should be emitted as a metric
+      long totalChunkSize = (currentChunk.id - 1) * chunkSize + currentChunk.length();

Review Comment:
   It's fine for now. We can worry about it when we really need to do something with the chunk. For now, let's just stick with the list.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "github-advanced-security[bot] (via GitHub)" <gi...@apache.org>.
github-advanced-security[bot] commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1622450984


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3UploadManager.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import com.google.inject.Inject;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.common.RetryUtils;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.storage.s3.S3Utils;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+import org.apache.druid.utils.RuntimeInfo;
+
+import java.io.File;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
+
+/**
+ * This class manages uploading files to S3 in chunks, while ensuring that the
+ * number of chunks currently present on local disk does not exceed a specific limit.
+ */
+@ManageLifecycle
+public class S3UploadManager
+{
+  private final ExecutorService uploadExecutor;
+
+  private static final Logger log = new Logger(S3UploadManager.class);
+
+  @Inject
+  public S3UploadManager(S3OutputConfig s3OutputConfig, S3ExportConfig s3ExportConfig, RuntimeInfo runtimeInfo)

Review Comment:
   ## Useless parameter
   
   The parameter 's3OutputConfig' is never used.
   
   [Show more details](https://github.com/apache/druid/security/code-scanning/7428)



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3UploadManager.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import com.google.inject.Inject;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.common.RetryUtils;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.storage.s3.S3Utils;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+import org.apache.druid.utils.RuntimeInfo;
+
+import java.io.File;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
+
+/**
+ * This class manages uploading files to S3 in chunks, while ensuring that the
+ * number of chunks currently present on local disk does not exceed a specific limit.
+ */
+@ManageLifecycle
+public class S3UploadManager
+{
+  private final ExecutorService uploadExecutor;
+
+  private static final Logger log = new Logger(S3UploadManager.class);
+
+  @Inject
+  public S3UploadManager(S3OutputConfig s3OutputConfig, S3ExportConfig s3ExportConfig, RuntimeInfo runtimeInfo)

Review Comment:
   ## Useless parameter
   
   The parameter 's3ExportConfig' is never used.
   
   [Show more details](https://github.com/apache/druid/security/code-scanning/7429)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1622418794


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3UploadManager.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import com.amazonaws.services.s3.model.UploadPartRequest;
+import com.amazonaws.services.s3.model.UploadPartResult;
+import com.google.inject.Inject;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.common.RetryUtils;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.storage.s3.S3Utils;
+import org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3;
+import org.apache.druid.utils.RuntimeInfo;
+
+import java.io.File;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
+
+/**
+ * This class manages uploading files to S3 in chunks, while ensuring that the
+ * number of chunks currently present on local disk does not exceed a specific limit.
+ */
+@ManageLifecycle
+public class S3UploadManager
+{
+  private final ExecutorService uploadExecutor;
+
+  private static final Logger log = new Logger(S3UploadManager.class);
+
+  @Inject
+  public S3UploadManager(S3OutputConfig s3OutputConfig, S3ExportConfig s3ExportConfig, RuntimeInfo runtimeInfo)
+  {
+    int poolSize = Math.max(4, runtimeInfo.getAvailableProcessors());
+    int maxNumConcurrentChunks = 10;

Review Comment:
   TODO: This needs to be updated, but I was running into issues getting proper values for the chunk sizes.
   
   We need to calculate maxNumConcurrentChunks based on `max(s3OutputConfig.getChunkSize(), s3ExportConfig.getChunkSize())`.
   
   But `s3OutputConfig.getChunkSize()` returns the default value hardcoded in `S3OutputConfig`, and doesn't get the value from `druid.msq.intermediate.storage.chunkSize`. But I could see `druid.msq.intermediate.storage.chunkSize` value populated in `S3OutputConfig`.
   
   It's probably some injection ordering issue, since S3UploadManager is also injected in S3UploadConfig and S3ExportConfig.
   
   cc: @cryptoe @kfaraz I was attempting this change based on our earlier discussion to inject both the configs into the file, so that we can do a max(x,y) on those instead of having methods like `updateChunkSizeIfGreater()`. Appreciate your guidance here on the injection issue. Thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1627296496


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -269,52 +213,74 @@ public void close() throws IOException
     // Closeables are closed in LIFO order
     closer.register(() -> {
       // This should be emitted as a metric
+      long totalChunkSize = (currentChunk.id - 1) * chunkSize + currentChunk.length();

Review Comment:
   It allows us to not have a Map<Chunk, Future> unnecessarily and helps the rest of the code be cleaner and simpler, which I really liked. Can you please elaborate more on why you'd prefer adding them individually? Thanks!
   
   PS: Can the chunkSize change in the middle of the operation? (in case of node failures or rolling upgrades where we update the chunk size or something along those lines?) 🤔 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on PR #16481:
URL: https://github.com/apache/druid/pull/16481#issuecomment-2152681440

   @kfaraz Thank you for the extremely exhaustive review comments, got to learn a LOT from them throughout this PR! 😄 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1609573706


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -103,22 +107,32 @@ public class RetryableS3OutputStream extends OutputStream
   private boolean error;
   private boolean closed;
 
-  public RetryableS3OutputStream(
-      S3OutputConfig config,
-      ServerSideEncryptingAmazonS3 s3,
-      String s3Key
-  ) throws IOException
-  {
+  /**
+   * An atomic counter to store number of files pending to be uploaded for the particular uploadId.
+   */
+  private final AtomicInteger pendingFiles = new AtomicInteger(0);
 
-    this(config, s3, s3Key, true);
-  }
+  /**
+   * A lock used for notifying the main thread about the completion of s3.uploadPart() for all chunks
+   * and hence starting the s3.completeMultipartUpload() for the uploadId.
+   */
+  private final Object fileLock = new Object();
+
+  /**
+   * Semaphore to restrict the maximum number of simultaneous chunks on disk.
+   */
+  private static final int MAX_CONCURRENT_CHUNKS = 10;
+  private static final Semaphore SEMAPHORE = new Semaphore(MAX_CONCURRENT_CHUNKS);
 
-  @VisibleForTesting
-  protected RetryableS3OutputStream(
+  /**
+   * Threadpool used for uploading the chunks asynchronously.
+   */
+  private static final ExecutorService UPLOAD_EXECUTOR = Execs.multiThreaded(10, "UploadThreadPool-%d");

Review Comment:
   > Seems unnecessary to me? For my understanding, what benefit does it provide over the current approach?
   
   It is not unnecessary, it is the right approach 🙂 . `static` should be used for constants, static variables and static classes. It doesn't make sense to make executors and other resources as static for the sole purpose of sharing them across instances. It makes the code difficult to follow and interferes with the lifecycle of such resources. You never know when to clean up such a resource as you don't know if an instance still exists in the JVM.
   
   An executor is a resource and if it is meant to be shared, it must be explicitly passed into the constructor as a dependency.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1615441116


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -103,22 +107,32 @@ public class RetryableS3OutputStream extends OutputStream
   private boolean error;
   private boolean closed;
 
-  public RetryableS3OutputStream(
-      S3OutputConfig config,
-      ServerSideEncryptingAmazonS3 s3,
-      String s3Key
-  ) throws IOException
-  {
+  /**
+   * An atomic counter to store number of files pending to be uploaded for the particular uploadId.
+   */
+  private final AtomicInteger pendingFiles = new AtomicInteger(0);
 
-    this(config, s3, s3Key, true);
-  }
+  /**
+   * A lock used for notifying the main thread about the completion of s3.uploadPart() for all chunks
+   * and hence starting the s3.completeMultipartUpload() for the uploadId.
+   */
+  private final Object fileLock = new Object();
+
+  /**
+   * Semaphore to restrict the maximum number of simultaneous chunks on disk.
+   */
+  private static final int MAX_CONCURRENT_CHUNKS = 10;
+  private static final Semaphore SEMAPHORE = new Semaphore(MAX_CONCURRENT_CHUNKS);
 
-  @VisibleForTesting
-  protected RetryableS3OutputStream(
+  /**
+   * Threadpool used for uploading the chunks asynchronously.
+   */
+  private static final ExecutorService UPLOAD_EXECUTOR = Execs.multiThreaded(10, "UploadThreadPool-%d");

Review Comment:
   In `S3StorageConnectorProvider`, you would need to use `@JacksonInject` instead of plain `@Inject`.
   Also, you probably don't need to add the provider method `getScheduledExecutorFactory` as it is already being provided in `ServerModule`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "LakshSingla (via GitHub)" <gi...@apache.org>.
LakshSingla commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1615476668


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -199,15 +211,47 @@ private void pushCurrentChunk() throws IOException
   {
     currentChunk.close();
     final Chunk chunk = currentChunk;
-    try {
-      if (chunk.length() > 0) {
-        resultsSize += chunk.length();
+    if (chunk.length() > 0) {
+      try {
+        SEMAPHORE.acquire(); // Acquire a permit from the semaphore
+        pendingFiles.incrementAndGet();
+
+        UPLOAD_EXECUTOR.submit(() -> {
+          try {
+            uploadChunk(chunk);
+          }
+          catch (Exception e) {
+            error = true;
+            LOG.error(e, e.getMessage());
+            throw new RuntimeException(e);
+          }
+          finally {
+            SEMAPHORE.release(); // Release the permit after upload is completed

Review Comment:
   QQ: Wouldn't it be nice if the semaphore was acquired in the thread calling the release on it, instead of the one helping in scheduling the jobs? 



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -271,50 +313,72 @@ public void close() throws IOException
       // This should be emitted as a metric
       LOG.info(
           "Pushed total [%d] parts containing [%d] bytes in [%d]ms.",
-          numChunksPushed,
-          resultsSize,
+          numChunksPushed.get(),
+          resultsSize.get(),
           pushStopwatch.elapsed(TimeUnit.MILLISECONDS)
       );
     });
 
-    closer.register(() -> org.apache.commons.io.FileUtils.forceDelete(chunkStorePath));
+    try (Closer ignored = closer) {
+      if (!error) {

Review Comment:
   Would we ever reach here if error = true? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1615484400


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -199,15 +211,47 @@ private void pushCurrentChunk() throws IOException
   {
     currentChunk.close();
     final Chunk chunk = currentChunk;
-    try {
-      if (chunk.length() > 0) {
-        resultsSize += chunk.length();
+    if (chunk.length() > 0) {
+      try {
+        SEMAPHORE.acquire(); // Acquire a permit from the semaphore
+        pendingFiles.incrementAndGet();
+
+        UPLOAD_EXECUTOR.submit(() -> {
+          try {
+            uploadChunk(chunk);
+          }
+          catch (Exception e) {
+            error = true;
+            LOG.error(e, e.getMessage());
+            throw new RuntimeException(e);
+          }
+          finally {
+            SEMAPHORE.release(); // Release the permit after upload is completed

Review Comment:
   @LakshSingla `acquire` needs to block the main thread (not the upload threads), hence it's acquired by the main thread. And since we don't block the main thread by waiting for results of the individual upload threads (except the last part when we complete the multipart upload), the semaphore is released by the individual upload threads.
   
   Also, Kashif suggested to move away from semaphore in favor of a different approach, and I'm working on making that change currently: https://github.com/apache/druid/pull/16481#discussion_r1609581399



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "LakshSingla (via GitHub)" <gi...@apache.org>.
LakshSingla commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1616590921


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3UploadConfig.java:
##########
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import org.apache.druid.guice.LazySingleton;
+import org.apache.druid.java.util.common.HumanReadableBytes;
+import org.apache.druid.java.util.common.logger.Logger;
+
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * This class manages the configuration for uploading files to S3 in chunks.
+ * It tracks the current number of chunks written to local disk concurrently and ensures that the
+ * maximum number of chunks on disk does not exceed a specified limit at any given point in time.
+ */
+@LazySingleton

Review Comment:
   I don't think it's required here. 



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -166,13 +184,25 @@ public void write(byte[] b, int off, int len) throws IOException
       return;
     }
 
+    synchronized (maxChunksLock) {
+      while (s3UploadConfig.getCurrentNumChunks() > s3UploadConfig.getMaxConcurrentNumChunks()) {
+        try {
+          LOG.info("Waiting for lock for writing further chunks to local disk.");

Review Comment:
   ```suggestion
             LOG.debug("Waiting for lock for writing further chunks to local disk.");
   ```



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3StorageDruidModule.java:
##########
@@ -182,4 +187,20 @@ public Supplier<ServerSideEncryptingAmazonS3> getAmazonS3ClientSupplier(
   {
     return Suppliers.memoize(serverSideEncryptingAmazonS3Builder::build);
   }
+
+  @Provides
+  @LazySingleton
+  @Named("S3UploadThreadPool")
+  public ExecutorService getUploadPoolExecutorService(ScheduledExecutorFactory scheduledExecutorFactory, RuntimeInfo runtimeInfo)
+  {
+    int poolSize = Math.max(4, 3 * runtimeInfo.getAvailableProcessors());

Review Comment:
   What's the rationale behind this? It seems super weird that we are allocating more threads to uploading than the number of processors. Check the basic cluster tuning section: https://druid.apache.org/docs/latest/operations/basic-cluster-tuning/#processing-threads-and-buffers. 
   The number of processing threads is limited to the number of processors to avoid over utilisation. 
   
   In any case, this value should be extracted somewhere and a Javadoc should be present behind this logic. 
   



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -199,15 +229,44 @@ private void pushCurrentChunk() throws IOException
   {
     currentChunk.close();
     final Chunk chunk = currentChunk;
-    try {
-      if (chunk.length() > 0) {
-        resultsSize += chunk.length();
+    if (chunk.length() > 0) {
+      s3UploadConfig.incrementCurrentNumChunks();

Review Comment:
   It really doesn't sit well with me that we are able to modify a "config" class. Perhaps the nomenclature is off, however configs aren't supposed to be modified during the execution. They can be dynamically updated, however I'd still expect it to be stateless. 



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3StorageDruidModule.java:
##########
@@ -182,4 +187,20 @@ public Supplier<ServerSideEncryptingAmazonS3> getAmazonS3ClientSupplier(
   {
     return Suppliers.memoize(serverSideEncryptingAmazonS3Builder::build);
   }
+
+  @Provides
+  @LazySingleton
+  @Named("S3UploadThreadPool")

Review Comment:
   "S3UploadThreadPool" should be extracted out somewhere. 



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3UploadConfig.java:
##########
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import org.apache.druid.guice.LazySingleton;
+import org.apache.druid.java.util.common.HumanReadableBytes;
+import org.apache.druid.java.util.common.logger.Logger;
+
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * This class manages the configuration for uploading files to S3 in chunks.
+ * It tracks the current number of chunks written to local disk concurrently and ensures that the
+ * maximum number of chunks on disk does not exceed a specified limit at any given point in time.
+ */
+@LazySingleton
+public class S3UploadConfig
+{
+  /**
+   * The maximum chunk size based on injected values for {@link S3OutputConfig} and {@link S3ExportConfig} used for computing maximum number of chunks to save on disk at any given point in time.
+   * It is initialized to 5 MiB which is the minimum chunk size possible, denoted by {@link S3OutputConfig#S3_MULTIPART_UPLOAD_MIN_PART_SIZE_BYTES}.
+   */
+  private long chunkSize = new HumanReadableBytes("5MiB").getBytes();
+
+  /**
+   * An atomic counter to track the current number of chunks saved to local disk.
+   */
+  private final AtomicInteger currentNumChunks = new AtomicInteger(0);
+
+  /**
+   * The maximum number of chunks that can be saved to local disk concurrently.
+   * This value is recalculated when the chunk size is updated in {@link #updateChunkSizeIfGreater(long)}.
+   */
+  private int maxConcurrentNumChunks = 100;

Review Comment:
   Same comment as above. Also, why is it not an atomic integer? 



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -271,50 +328,72 @@ public void close() throws IOException
       // This should be emitted as a metric
       LOG.info(
           "Pushed total [%d] parts containing [%d] bytes in [%d]ms.",
-          numChunksPushed,
-          resultsSize,
+          numChunksPushed.get(),
+          resultsSize.get(),
           pushStopwatch.elapsed(TimeUnit.MILLISECONDS)
       );
     });
 
-    closer.register(() -> org.apache.commons.io.FileUtils.forceDelete(chunkStorePath));
+    try (Closer ignored = closer) {
+      if (!error) {
+        pushCurrentChunk();
+        completeMultipartUpload();
+      }
+    }
+  }
 
-    closer.register(() -> {
-      try {
-        if (resultsSize > 0 && isAllPushSucceeded()) {
-          RetryUtils.retry(
-              () -> s3.completeMultipartUpload(
-                  new CompleteMultipartUploadRequest(config.getBucket(), s3Key, uploadId, pushResults)
-              ),
-              S3Utils.S3RETRY,
-              config.getMaxRetry()
-          );
-        } else {
-          RetryUtils.retry(
-              () -> {
-                s3.cancelMultiPartUpload(new AbortMultipartUploadRequest(config.getBucket(), s3Key, uploadId));
-                return null;
-              },
-              S3Utils.S3RETRY,
-              config.getMaxRetry()
-          );
+  private void completeMultipartUpload()
+  {
+    synchronized (fileLock) {
+      while (pendingFiles.get() > 0) {
+        try {
+          LOG.info("Waiting for lock for completing multipart task for uploadId [%s].", uploadId);

Review Comment:
   Would this not pollute the logs, given that we can have one log line per upload? 



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3UploadConfig.java:
##########
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import org.apache.druid.guice.LazySingleton;
+import org.apache.druid.java.util.common.HumanReadableBytes;
+import org.apache.druid.java.util.common.logger.Logger;
+
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * This class manages the configuration for uploading files to S3 in chunks.
+ * It tracks the current number of chunks written to local disk concurrently and ensures that the
+ * maximum number of chunks on disk does not exceed a specified limit at any given point in time.
+ */
+@LazySingleton
+public class S3UploadConfig
+{
+  /**
+   * The maximum chunk size based on injected values for {@link S3OutputConfig} and {@link S3ExportConfig} used for computing maximum number of chunks to save on disk at any given point in time.
+   * It is initialized to 5 MiB which is the minimum chunk size possible, denoted by {@link S3OutputConfig#S3_MULTIPART_UPLOAD_MIN_PART_SIZE_BYTES}.
+   */
+  private long chunkSize = new HumanReadableBytes("5MiB").getBytes();
+
+  /**
+   * An atomic counter to track the current number of chunks saved to local disk.
+   */
+  private final AtomicInteger currentNumChunks = new AtomicInteger(0);
+
+  /**
+   * The maximum number of chunks that can be saved to local disk concurrently.
+   * This value is recalculated when the chunk size is updated in {@link #updateChunkSizeIfGreater(long)}.
+   */
+  private int maxConcurrentNumChunks = 100;
+
+  private static final Logger log = new Logger(S3UploadConfig.class);
+
+  /**
+   * Increments the counter for the current number of chunks saved on disk.
+   */
+  public void incrementCurrentNumChunks()
+  {
+    currentNumChunks.incrementAndGet();
+  }
+
+  /**
+   * Decrements the counter for the current number of chunks saved on disk.
+   */
+  public void decrementCurrentNumChunks()
+  {
+    currentNumChunks.decrementAndGet();
+  }
+
+  /**
+   * Gets the current number of chunks saved to local disk.
+   *
+   * @return the current number of chunks saved to local disk.
+   */
+  public int getCurrentNumChunks()
+  {
+    return currentNumChunks.get();
+  }
+
+  /**
+   * Gets the maximum number of concurrent chunks that can be saved to local disk.
+   *
+   * @return the maximum number of concurrent chunks that can be saved to local disk.
+   */
+  public int getMaxConcurrentNumChunks()
+  {
+    return maxConcurrentNumChunks;
+  }
+
+  /**
+   * Updates the chunk size if the provided size is greater than the current chunk size.
+   * Recomputes the maximum number of concurrent chunks that can be saved to local disk based on the new chunk size.
+   *
+   * @param chunkSize the new chunk size to be set if it is greater than the current chunk size.
+   */
+  public void updateChunkSizeIfGreater(long chunkSize)
+  {
+    if (this.chunkSize < chunkSize) {
+      this.chunkSize = chunkSize;
+      recomputeMaxConcurrentNumChunks();
+    }
+  }
+
+  /**
+   * Recomputes the maximum number of concurrent chunks that can be saved to local disk based on the current chunk size.
+   * The maximum allowed chunk size is {@link S3OutputConfig#S3_MULTIPART_UPLOAD_MAX_PART_SIZE_BYTES} which is quite big,
+   * so we restrict the maximum disk space used to the same, at any given point in time.
+   */
+  private void recomputeMaxConcurrentNumChunks()
+  {
+    maxConcurrentNumChunks = (int) (S3OutputConfig.S3_MULTIPART_UPLOAD_MAX_PART_SIZE_BYTES / chunkSize);
+    log.info("Recomputed maxConcurrentNumChunks: %d", maxConcurrentNumChunks);

Review Comment:
   How often do we expect this to be called?



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3UploadConfig.java:
##########
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import org.apache.druid.guice.LazySingleton;
+import org.apache.druid.java.util.common.HumanReadableBytes;
+import org.apache.druid.java.util.common.logger.Logger;
+
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * This class manages the configuration for uploading files to S3 in chunks.
+ * It tracks the current number of chunks written to local disk concurrently and ensures that the
+ * maximum number of chunks on disk does not exceed a specified limit at any given point in time.
+ */
+@LazySingleton
+public class S3UploadConfig
+{
+  /**
+   * The maximum chunk size based on injected values for {@link S3OutputConfig} and {@link S3ExportConfig} used for computing maximum number of chunks to save on disk at any given point in time.
+   * It is initialized to 5 MiB which is the minimum chunk size possible, denoted by {@link S3OutputConfig#S3_MULTIPART_UPLOAD_MIN_PART_SIZE_BYTES}.
+   */
+  private long chunkSize = new HumanReadableBytes("5MiB").getBytes();
+
+  /**
+   * An atomic counter to track the current number of chunks saved to local disk.
+   */
+  private final AtomicInteger currentNumChunks = new AtomicInteger(0);

Review Comment:
   Why are we tracking something in a config? This doesn't seem very cool for a singleton. I would have expected it to be stateless. If this needs to be updated dynamically, it should mention that somewhere, `DynamicS3UploadConfig`, and a global instance of this should be avoided.
   All the callers that need to share this must receive it explicitly instead of being injected via Guice. 
   This seems error-prone if someone decides to use the storage connector down the line for some other task and they both share the same config. 



##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/S3UploadConfig.java:
##########
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.storage.s3.output;
+
+import org.apache.druid.guice.LazySingleton;
+import org.apache.druid.java.util.common.HumanReadableBytes;
+import org.apache.druid.java.util.common.logger.Logger;
+
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * This class manages the configuration for uploading files to S3 in chunks.
+ * It tracks the current number of chunks written to local disk concurrently and ensures that the
+ * maximum number of chunks on disk does not exceed a specified limit at any given point in time.
+ */
+@LazySingleton
+public class S3UploadConfig
+{
+  /**
+   * The maximum chunk size based on injected values for {@link S3OutputConfig} and {@link S3ExportConfig} used for computing maximum number of chunks to save on disk at any given point in time.
+   * It is initialized to 5 MiB which is the minimum chunk size possible, denoted by {@link S3OutputConfig#S3_MULTIPART_UPLOAD_MIN_PART_SIZE_BYTES}.
+   */
+  private long chunkSize = new HumanReadableBytes("5MiB").getBytes();
+
+  /**
+   * An atomic counter to track the current number of chunks saved to local disk.
+   */
+  private final AtomicInteger currentNumChunks = new AtomicInteger(0);
+
+  /**
+   * The maximum number of chunks that can be saved to local disk concurrently.
+   * This value is recalculated when the chunk size is updated in {@link #updateChunkSizeIfGreater(long)}.
+   */
+  private int maxConcurrentNumChunks = 100;

Review Comment:
   The multithreading model for this class is unclear to me. It isn't thread-safe, however, `currentNumChunks` is marked as an atomic integer.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1616622617


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -199,15 +229,44 @@ private void pushCurrentChunk() throws IOException
   {
     currentChunk.close();
     final Chunk chunk = currentChunk;
-    try {
-      if (chunk.length() > 0) {
-        resultsSize += chunk.length();
+    if (chunk.length() > 0) {
+      s3UploadConfig.incrementCurrentNumChunks();

Review Comment:
   @LakshSingla Yeah, I couldn't decide a good name for it. I initially named it S3UploadHelper, but that seemed confusing as well as it doesn't have the upload logic. Hence I settled for S3UploadConfig, as it does define the configuration for the upload mechanism.
   Please let me know if you have any better naming suggestions for this, definitely open to it!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on PR #16481:
URL: https://github.com/apache/druid/pull/16481#issuecomment-2134545306

   @rohangarg 
   
   > Thanks, I checked earlier but I couldn't understand their timings much. For instance, were you able to reason about why the serial upload is taking ~ 8 minutes for 450MBs?
   
   The serial upload is uploading 5 MBs one-by-one, hence takes an extremely long time.
   
   > Also, the chunk size in those queries seemed a bit de-generate to me.
   
   So, I had to reduce the chunk size to lowest possible value to be able to simulate upload of a large number of chunks, as I can't do it otherwise due to limited system resources. But it shouldn't affect the comparison/improvement trend, hence we can still rely on this data to get some idea on the degree of savings.
   
   > I was actually meaning that the threads which are running and are writing to the output stream currently are able to write it to S3 too on the same thread, which might not be the case which this change and the writes might have to wait. Are we solving that scenario in this change?
   
   Actually it's the opposite. Previously the writes to the output stream (local file) had to wait for the previous chunk write+upload to finish. Now, the write doesn't have to wait for the previous uploads to finish (there's a backpressure semantic to restrict too many writes, but that's different from the current point). Please let me know if this answers the question, happy to clarify this further!
   
   > have we considered supporting parallel uploads via the execution framework (maybe an operator to fan out writes) instead of doing it on a per storage basis? 
   
   No, we hadn't discussed any approach like that. cc: @cryptoe 
   
   > But I would do that in the executor related code itself instead of leaking it into the output stream.
   
   Can you please clarify what do you mean by "leaking it into the output stream"? Sorry, I didn't get this part.
   
   > You can determine that the file is small when not even one local chunk is fully filled, and the output stream is closed. And once you've determined that a file is small, direct PUT can be used over multi-part upload.
   
   But this would work only when there's a single chunk. How about when there are 5 chunks, for example? We'd have to wait for the last chunk to be processed before we can make the decision that the `total number of chunks * chunkSize` is small enough to go with PUT approach. Also, supporting the PUT flow along with the multipart approach doesn't seem ideal to me, it would be one extra thing to maintain for not much benefits. That's my opinion on this atleast. Happy to discuss it further.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "Akshat-Jain (via GitHub)" <gi...@apache.org>.
Akshat-Jain commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1616971538


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -199,15 +229,44 @@ private void pushCurrentChunk() throws IOException
   {
     currentChunk.close();
     final Chunk chunk = currentChunk;
-    try {
-      if (chunk.length() > 0) {
-        resultsSize += chunk.length();
+    if (chunk.length() > 0) {
+      s3UploadConfig.incrementCurrentNumChunks();

Review Comment:
   @kfaraz Have made the suggested change. Refactored S3UploadConfig + ExecutorService into S3UploadManager. Please have a look, thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize S3 storage writing for MSQ durable storage (druid)

Posted by "LakshSingla (via GitHub)" <gi...@apache.org>.
LakshSingla commented on code in PR #16481:
URL: https://github.com/apache/druid/pull/16481#discussion_r1617112881


##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/output/RetryableS3OutputStream.java:
##########
@@ -271,50 +328,72 @@ public void close() throws IOException
       // This should be emitted as a metric
       LOG.info(
           "Pushed total [%d] parts containing [%d] bytes in [%d]ms.",
-          numChunksPushed,
-          resultsSize,
+          numChunksPushed.get(),
+          resultsSize.get(),
           pushStopwatch.elapsed(TimeUnit.MILLISECONDS)
       );
     });
 
-    closer.register(() -> org.apache.commons.io.FileUtils.forceDelete(chunkStorePath));
+    try (Closer ignored = closer) {
+      if (!error) {
+        pushCurrentChunk();
+        completeMultipartUpload();
+      }
+    }
+  }
 
-    closer.register(() -> {
-      try {
-        if (resultsSize > 0 && isAllPushSucceeded()) {
-          RetryUtils.retry(
-              () -> s3.completeMultipartUpload(
-                  new CompleteMultipartUploadRequest(config.getBucket(), s3Key, uploadId, pushResults)
-              ),
-              S3Utils.S3RETRY,
-              config.getMaxRetry()
-          );
-        } else {
-          RetryUtils.retry(
-              () -> {
-                s3.cancelMultiPartUpload(new AbortMultipartUploadRequest(config.getBucket(), s3Key, uploadId));
-                return null;
-              },
-              S3Utils.S3RETRY,
-              config.getMaxRetry()
-          );
+  private void completeMultipartUpload()
+  {
+    synchronized (fileLock) {
+      while (pendingFiles.get() > 0) {
+        try {
+          LOG.info("Waiting for lock for completing multipart task for uploadId [%s].", uploadId);

Review Comment:
   I think this is high. How is parts per stage determined? How many parts per stage would it be if there was 1TB data ingested? And how much would it be if there was 10TB data ingested? Also consider situations when there are 4-5 layers of nested joins, and their are 10 stages. (Note: Ignore the cumulative number of logging lines, only the number of log lines per worker) The number of log lines per worker that mention  "Waiting for the lock for completing multipart task..." would run pretty large. Also, in your test case, I don't think we would have started using durable storage for the super sorter, which has drastically reduced the number of log lines.
   
   Also, what potential conditions will it help in debugging? Let's say for a medium job, there are 200 log lines (which doesn't include all the cases that I have mentioned above). How can those 200 lines aid in debugging the job (and not the connector)? Note: If you worry about a deadlock, we have jstacks which are a much better tool for debugging. 
   
   If you think that a) The number of log lines will stay (say) <100 for the largest of jobs and b) All those logged lines will aid someone debugging the job we can keep it as INFO. Else I feel that the benefit of logging about acquisition/waiting/release of locks isn't going to help someone looking through the logs, and will most likely sift over it. DEBUG is a much better place for such logs. 
   
   If we do choose to keep it at INFO level, we should reword it in a way that helps the reader of the logs identify how to use these.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org