[GitHub] [druid] cryptoe commented on a diff in pull request #13960: Reworking s3 connector with various improvements

["cryptoe (via GitHub)" <gi...@apache.org>]

cryptoe commented on code in PR #13960:
URL: https://github.com/apache/druid/pull/13960#discussion_r1146101160


##########
processing/src/main/java/org/apache/druid/storage/StorageConnector.java:
##########
@@ -105,26 +105,39 @@
    * with a basePath.
    * If the path is a directory, this method throws an exception.
    *
-   * @param path
-   * @throws IOException
+   * @param path to delete
+   * @throws IOException thrown in case of errors.
    */
   void deleteFile(String path) throws IOException;
 
+
+  /**
+   * Delete files present at the input paths. Most implementations prepend all the input paths
+   * with the basePath.
+   * <br/>
+   * This method is <b>recommended</b> in case we need to delete a batch of files.
+   * If the path is a directory, this method throws an exception.
+   *
+   * @param paths Iterable of the paths to delete.
+   * @throws IOException thrown in case of errors.
+   */
+  void deleteFiles(Iterable<String> paths) throws IOException;
+
   /**
    * Delete a directory pointed to by the path and also recursively deletes all files/directories in said directory.
    * Most implementations prepend the input path with a basePath.
    *
    * @param path path
-   * @throws IOException
+   * @throws IOException thrown in case of errors.
    */
   void deleteRecursively(String path) throws IOException;
 
   /**
-   * Returns a list containing all the files present in the path. The returned filenames should be such that joining
+   * Returns a lazy iterator containing all the files present in the path. The returned filenames should be such that joining
    * the dirName and the file name form the full path that can be used as the arguments for other methods of the storage
    * connector.
-   * For example, for a S3 path such as s3://bucket/parent1/parent2/child, the filename returned for the path
-   * "parent1/parent2" should be "child" and for "parent1" should be "parent2"
+   * For example, for a S3 path such as s3://bucket/parent1/parent2/child, the filename returned for the input path
+   * "parent1/parent2" should be "child" and for input "parent1" should be "parent2/child"
    */
-  List<String> listDir(String dirName);
+  Iterator<String> listDir(String dirName) throws IOException;

Review Comment:
   Since we do not want to materialize all the returned items in memory I used an iterator so that on hasNext we can fetch the next items if required. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org