You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "yihua (via GitHub)" <gi...@apache.org> on 2024/01/26 02:59:17 UTC

[PR] [HUDI-7336] Introduce new HoodieStorage abstraction [hudi]

yihua opened a new pull request, #10567:
URL: https://github.com/apache/hudi/pull/10567

   ### Change Logs
   
   This PR introduces `HoodieStorage` abstraction and tries to remove the usage of Hadoop File System classes (`org.apache.hadoop.fs.`[`FileSystem`, `Path`, `FileStatus`], etc.) in core Hudi APIs, so it's much easier to plugin different file system implementation.
   
   ### Impact
   
   No impact as this PR does not have the integration.
   
   ### Risk level
   
   none
   
   ### Documentation Update
   
   N/A
   
   ### Contributor's checklist
   
   - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [ ] Change Logs and Impact were stated clearly
   - [ ] Adequate tests were added if applicable
   - [ ] CI passed
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7336] Introduce new HoodieStorage abstraction [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10567:
URL: https://github.com/apache/hudi/pull/10567#issuecomment-1911417321

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "ca3cb838f94d96e02860e4587e035b903b7746d6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ca3cb838f94d96e02860e4587e035b903b7746d6",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * ca3cb838f94d96e02860e4587e035b903b7746d6 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7336][RFR|DNM] Introduce new HoodieStorage abstraction [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #10567:
URL: https://github.com/apache/hudi/pull/10567#discussion_r1467316121


##########
hudi-io/src/main/java/org/apache/hudi/io/storage/HoodieStorage.java:
##########
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.io.storage;
+
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieIOException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.Closeable;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Provides I/O APIs on files and directories on storage.
+ */
+public interface HoodieStorage extends Closeable {
+  Logger LOG = LoggerFactory.getLogger(HoodieStorage.class);
+  String TMP_PATH_POSTFIX = ".tmp";
+
+  /**
+   * @return the scheme of the storage.
+   */
+  String getScheme();
+
+  /**
+   * Creates an OutputStream at the indicated location.
+   *
+   * @param location  the file to create.
+   * @param overwrite if a file with this name already exists, then if {@code true},
+   *                  the file will be overwritten, and if {@code false} an exception will be thrown.
+   * @return the OutputStream to write to.
+   * @throws IOException IO error.
+   */
+  OutputStream create(HoodieLocation location, boolean overwrite) throws IOException;
+
+  /**
+   * Opens an InputStream at the indicated location.
+   *
+   * @param location the file to open.
+   * @return the InputStream to read from.
+   * @throws IOException IO error.
+   */
+  InputStream open(HoodieLocation location) throws IOException;
+
+  /**
+   * Appends to an existing file (optional operation).
+   *
+   * @param location the file to append.
+   * @return the OutputStream to write to.
+   * @throws IOException IO error.
+   */
+  OutputStream append(HoodieLocation location) throws IOException;
+
+  /**
+   * Checks if a location exists.
+   *
+   * @param location location to check.
+   * @return {@code true} if the location exists.
+   * @throws IOException IO error.
+   */
+  boolean exists(HoodieLocation location) throws IOException;
+
+  /**
+   * Returns a file status object that represents the location.
+   *
+   * @param location location to check.
+   * @return a {@link HoodieFileStatus} object.
+   * @throws FileNotFoundException when the path does not exist.
+   * @throws IOException           IO error.
+   */
+  HoodieFileStatus getFileStatus(HoodieLocation location) throws IOException;
+
+  /**
+   * Creates the directory and non-existent parent directories.
+   *
+   * @param location location to create.
+   * @return {@code true} if the directory was created.
+   * @throws IOException IO error.
+   */
+  boolean createDirectory(HoodieLocation location) throws IOException;
+
+  /**
+   * Lists the statuses of the direct files/directories in the given location if the path is a directory.
+   *
+   * @param location given location.
+   * @return the statuses of the files/directories in the given location.
+   * @throws FileNotFoundException when the location does not exist.
+   * @throws IOException           IO error.
+   */
+  List<HoodieFileStatus> listDirectEntries(HoodieLocation location) throws IOException;
+
+  /**
+   * Lists the statuses of all files under the give location recursively.
+   *
+   * @param location given location.
+   * @return the statuses of the files under the given location.
+   * @throws FileNotFoundException when the location does not exist.
+   * @throws IOException           IO error.
+   */
+  List<HoodieFileStatus> listFiles(HoodieLocation location) throws IOException;
+
+  /**
+   * Lists the statuses of the direct files/directories in the given location
+   * and filters the results, if the path is a directory.
+   *
+   * @param location given location.
+   * @param filter   filter to apply.
+   * @return the statuses of the files/directories in the given location.
+   * @throws FileNotFoundException when the location does not exist.
+   * @throws IOException           IO error.
+   */
+  List<HoodieFileStatus> listDirectEntries(HoodieLocation location,
+                                           HoodieLocationFilter filter) throws IOException;
+
+  /**
+   * Returns all the files that match the locationPattern and are not checksum files,
+   * and filters the results.
+   *
+   * @param locationPattern given pattern.
+   * @param filter          filter to apply.
+   * @return the statuses of the files.
+   * @throws IOException IO error.
+   */
+  List<HoodieFileStatus> globEntries(HoodieLocation locationPattern, HoodieLocationFilter filter)
+      throws IOException;
+
+  /**
+   * Renames the location from old to new.
+   *
+   * @param oldLocation source location.
+   * @param newLocation destination location.
+   * @return {@true} if rename is successful.
+   * @throws IOException IO error.
+   */
+  boolean rename(HoodieLocation oldLocation, HoodieLocation newLocation) throws IOException;
+
+  /**
+   * Deletes a directory at location.
+   *
+   * @param location directory to delete.
+   * @return {@code true} if successful.
+   * @throws IOException IO error.
+   */
+  boolean deleteDirectory(HoodieLocation location) throws IOException;
+
+  /**
+   * Deletes a file at location.
+   *
+   * @param location file to delete.
+   * @return {@code true} if successful.
+   * @throws IOException IO error.
+   */
+  boolean deleteFile(HoodieLocation location) throws IOException;
+
+  /**
+   * Qualifies a path to one which uses this storage and, if relative, made absolute.
+   *
+   * @param location to qualify.
+   * @return Qualified location.
+   */
+  HoodieLocation makeQualified(HoodieLocation location);
+
+  /**
+   * @return the underlying file system instance if exists.
+   */
+  Object getFileSystem();

Review Comment:
   After complete refactoring using HoodieStorage, this method should not be used anymore.  However, to make incremental progress on refactoring, in some places I have to get the internal FileSystem instance.  So for now, I think this is fine.



##########
hudi-io/src/main/java/org/apache/hudi/io/storage/HoodieStorage.java:
##########
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.io.storage;
+
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieIOException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.Closeable;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Provides I/O APIs on files and directories on storage.
+ */
+public interface HoodieStorage extends Closeable {
+  Logger LOG = LoggerFactory.getLogger(HoodieStorage.class);
+  String TMP_PATH_POSTFIX = ".tmp";
+
+  /**
+   * @return the scheme of the storage.
+   */
+  String getScheme();
+
+  /**
+   * Creates an OutputStream at the indicated location.
+   *
+   * @param location  the file to create.
+   * @param overwrite if a file with this name already exists, then if {@code true},
+   *                  the file will be overwritten, and if {@code false} an exception will be thrown.
+   * @return the OutputStream to write to.
+   * @throws IOException IO error.
+   */
+  OutputStream create(HoodieLocation location, boolean overwrite) throws IOException;
+
+  /**
+   * Opens an InputStream at the indicated location.
+   *
+   * @param location the file to open.
+   * @return the InputStream to read from.
+   * @throws IOException IO error.
+   */
+  InputStream open(HoodieLocation location) throws IOException;
+
+  /**
+   * Appends to an existing file (optional operation).
+   *
+   * @param location the file to append.
+   * @return the OutputStream to write to.
+   * @throws IOException IO error.
+   */
+  OutputStream append(HoodieLocation location) throws IOException;
+
+  /**
+   * Checks if a location exists.
+   *
+   * @param location location to check.
+   * @return {@code true} if the location exists.
+   * @throws IOException IO error.
+   */
+  boolean exists(HoodieLocation location) throws IOException;
+
+  /**
+   * Returns a file status object that represents the location.
+   *
+   * @param location location to check.
+   * @return a {@link HoodieFileStatus} object.
+   * @throws FileNotFoundException when the path does not exist.
+   * @throws IOException           IO error.
+   */
+  HoodieFileStatus getFileStatus(HoodieLocation location) throws IOException;
+
+  /**
+   * Creates the directory and non-existent parent directories.
+   *
+   * @param location location to create.
+   * @return {@code true} if the directory was created.
+   * @throws IOException IO error.
+   */
+  boolean createDirectory(HoodieLocation location) throws IOException;
+
+  /**
+   * Lists the statuses of the direct files/directories in the given location if the path is a directory.
+   *
+   * @param location given location.
+   * @return the statuses of the files/directories in the given location.
+   * @throws FileNotFoundException when the location does not exist.
+   * @throws IOException           IO error.
+   */
+  List<HoodieFileStatus> listDirectEntries(HoodieLocation location) throws IOException;
+
+  /**
+   * Lists the statuses of all files under the give location recursively.
+   *
+   * @param location given location.
+   * @return the statuses of the files under the given location.
+   * @throws FileNotFoundException when the location does not exist.
+   * @throws IOException           IO error.
+   */
+  List<HoodieFileStatus> listFiles(HoodieLocation location) throws IOException;
+
+  /**
+   * Lists the statuses of the direct files/directories in the given location
+   * and filters the results, if the path is a directory.
+   *
+   * @param location given location.
+   * @param filter   filter to apply.
+   * @return the statuses of the files/directories in the given location.
+   * @throws FileNotFoundException when the location does not exist.
+   * @throws IOException           IO error.
+   */
+  List<HoodieFileStatus> listDirectEntries(HoodieLocation location,
+                                           HoodieLocationFilter filter) throws IOException;
+
+  /**
+   * Returns all the files that match the locationPattern and are not checksum files,
+   * and filters the results.
+   *
+   * @param locationPattern given pattern.
+   * @param filter          filter to apply.
+   * @return the statuses of the files.
+   * @throws IOException IO error.
+   */
+  List<HoodieFileStatus> globEntries(HoodieLocation locationPattern, HoodieLocationFilter filter)
+      throws IOException;
+
+  /**
+   * Renames the location from old to new.
+   *
+   * @param oldLocation source location.
+   * @param newLocation destination location.
+   * @return {@true} if rename is successful.
+   * @throws IOException IO error.
+   */
+  boolean rename(HoodieLocation oldLocation, HoodieLocation newLocation) throws IOException;
+
+  /**
+   * Deletes a directory at location.
+   *
+   * @param location directory to delete.
+   * @return {@code true} if successful.
+   * @throws IOException IO error.
+   */
+  boolean deleteDirectory(HoodieLocation location) throws IOException;
+
+  /**
+   * Deletes a file at location.
+   *
+   * @param location file to delete.
+   * @return {@code true} if successful.
+   * @throws IOException IO error.
+   */
+  boolean deleteFile(HoodieLocation location) throws IOException;
+
+  /**
+   * Qualifies a path to one which uses this storage and, if relative, made absolute.
+   *
+   * @param location to qualify.
+   * @return Qualified location.
+   */
+  HoodieLocation makeQualified(HoodieLocation location);
+
+  /**
+   * @return the underlying file system instance if exists.
+   */
+  Object getFileSystem();
+
+  /**
+   * @return the underlying configuration instance if exists.
+   */
+  Object getConf();
+
+  /**
+   * Create an OutputStream at the indicated location.
+   * The file is overwritten by default.
+   *
+   * @param location the file to create.
+   * @return the OutputStream to write to.
+   * @throws IOException IO error.
+   */
+  default OutputStream create(HoodieLocation location) throws IOException {
+    return create(location, true);
+  }
+
+  /**
+   * Creates a new file with overwrite set to false. This ensures files are created
+   * only once and never rewritten, also, here we take care if the content is not
+   * empty, will first write the content to a temp file if {needCreateTempFile} is
+   * true, and then rename it back after the content is written.
+   *
+   * @param location file Path.
+   * @param content  content to be stored.
+   */
+  default void createImmutableFileInPath(HoodieLocation location,

Review Comment:
   Makes sense.  Fixed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7336][RFR|DNM] Introduce new HoodieStorage abstraction [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10567:
URL: https://github.com/apache/hudi/pull/10567#issuecomment-1913331895

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "ca3cb838f94d96e02860e4587e035b903b7746d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22168",
       "triggerID" : "ca3cb838f94d96e02860e4587e035b903b7746d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8b765537863e155372fed079ea520f0a91a45b84",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22170",
       "triggerID" : "8b765537863e155372fed079ea520f0a91a45b84",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ea050a0d021273313ef3d9e5e3c566186e0b28ee",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22186",
       "triggerID" : "ea050a0d021273313ef3d9e5e3c566186e0b28ee",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8b765537863e155372fed079ea520f0a91a45b84 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22170) 
   * ea050a0d021273313ef3d9e5e3c566186e0b28ee Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22186) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7336] Introduce new HoodieStorage abstraction [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on PR #10567:
URL: https://github.com/apache/hudi/pull/10567#issuecomment-1913987587

   Azure CI is green.
   <img width="1647" alt="Screenshot 2024-01-28 at 21 26 22" src="https://github.com/apache/hudi/assets/2497195/24e1e3ae-042a-4821-a0ce-311e90325238">
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7336][RFR|DNM] Introduce new HoodieStorage abstraction [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10567:
URL: https://github.com/apache/hudi/pull/10567#issuecomment-1911675531

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "ca3cb838f94d96e02860e4587e035b903b7746d6",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22168",
       "triggerID" : "ca3cb838f94d96e02860e4587e035b903b7746d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8b765537863e155372fed079ea520f0a91a45b84",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8b765537863e155372fed079ea520f0a91a45b84",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * ca3cb838f94d96e02860e4587e035b903b7746d6 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22168) 
   * 8b765537863e155372fed079ea520f0a91a45b84 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7336][RFR|DNM] Introduce new HoodieStorage abstraction [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10567:
URL: https://github.com/apache/hudi/pull/10567#issuecomment-1911684372

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "ca3cb838f94d96e02860e4587e035b903b7746d6",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22168",
       "triggerID" : "ca3cb838f94d96e02860e4587e035b903b7746d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8b765537863e155372fed079ea520f0a91a45b84",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22170",
       "triggerID" : "8b765537863e155372fed079ea520f0a91a45b84",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * ca3cb838f94d96e02860e4587e035b903b7746d6 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22168) 
   * 8b765537863e155372fed079ea520f0a91a45b84 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22170) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7336] Introduce new HoodieStorage abstraction [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10567:
URL: https://github.com/apache/hudi/pull/10567#issuecomment-1913866101

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "ca3cb838f94d96e02860e4587e035b903b7746d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22168",
       "triggerID" : "ca3cb838f94d96e02860e4587e035b903b7746d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8b765537863e155372fed079ea520f0a91a45b84",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22170",
       "triggerID" : "8b765537863e155372fed079ea520f0a91a45b84",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ea050a0d021273313ef3d9e5e3c566186e0b28ee",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22186",
       "triggerID" : "ea050a0d021273313ef3d9e5e3c566186e0b28ee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c5ebd502865830bcd840f904b6ea780ab1175058",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "c5ebd502865830bcd840f904b6ea780ab1175058",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * ea050a0d021273313ef3d9e5e3c566186e0b28ee Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22186) 
   * c5ebd502865830bcd840f904b6ea780ab1175058 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7336] Introduce new HoodieStorage abstraction [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10567:
URL: https://github.com/apache/hudi/pull/10567#issuecomment-1911554997

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "ca3cb838f94d96e02860e4587e035b903b7746d6",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22168",
       "triggerID" : "ca3cb838f94d96e02860e4587e035b903b7746d6",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * ca3cb838f94d96e02860e4587e035b903b7746d6 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22168) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7336][RFR|DNM] Introduce new HoodieStorage abstraction [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10567:
URL: https://github.com/apache/hudi/pull/10567#issuecomment-1911928434

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "ca3cb838f94d96e02860e4587e035b903b7746d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22168",
       "triggerID" : "ca3cb838f94d96e02860e4587e035b903b7746d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8b765537863e155372fed079ea520f0a91a45b84",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22170",
       "triggerID" : "8b765537863e155372fed079ea520f0a91a45b84",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8b765537863e155372fed079ea520f0a91a45b84 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22170) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7336] Introduce new HoodieStorage abstraction [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #10567:
URL: https://github.com/apache/hudi/pull/10567


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7336][RFR|DNM] Introduce new HoodieStorage abstraction [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on PR #10567:
URL: https://github.com/apache/hudi/pull/10567#issuecomment-1911616198

   > Looks good in general, I'm wondering do we have referenced system for the `HoodieStorage` abstration?
   
   Yes, it's mainly based on hadoop's `FileSystem` class, so that we can replace any FS calls with HoodieStorage calls.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7336] Introduce new HoodieStorage abstraction [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10567:
URL: https://github.com/apache/hudi/pull/10567#issuecomment-1913871098

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "ca3cb838f94d96e02860e4587e035b903b7746d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22168",
       "triggerID" : "ca3cb838f94d96e02860e4587e035b903b7746d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8b765537863e155372fed079ea520f0a91a45b84",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22170",
       "triggerID" : "8b765537863e155372fed079ea520f0a91a45b84",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ea050a0d021273313ef3d9e5e3c566186e0b28ee",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22186",
       "triggerID" : "ea050a0d021273313ef3d9e5e3c566186e0b28ee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c5ebd502865830bcd840f904b6ea780ab1175058",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22192",
       "triggerID" : "c5ebd502865830bcd840f904b6ea780ab1175058",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * ea050a0d021273313ef3d9e5e3c566186e0b28ee Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22186) 
   * c5ebd502865830bcd840f904b6ea780ab1175058 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22192) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7336] Introduce new HoodieStorage abstraction [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on PR #10567:
URL: https://github.com/apache/hudi/pull/10567#issuecomment-1911363910

   Currently, this PR contains some changes from #10564. The PR is reviewable. Please do not merge until #10564 is merged.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7336] Introduce new HoodieStorage abstraction [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #10567:
URL: https://github.com/apache/hudi/pull/10567#discussion_r1467198940


##########
hudi-io/src/main/java/org/apache/hudi/io/storage/HoodieStorage.java:
##########
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.io.storage;
+
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieIOException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.Closeable;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Provides I/O APIs on files and directories on storage.
+ */
+public interface HoodieStorage extends Closeable {
+  Logger LOG = LoggerFactory.getLogger(HoodieStorage.class);
+  String TMP_PATH_POSTFIX = ".tmp";
+
+  /**
+   * @return the scheme of the storage.
+   */
+  String getScheme();
+
+  /**
+   * Creates an OutputStream at the indicated location.
+   *
+   * @param location  the file to create.
+   * @param overwrite if a file with this name already exists, then if {@code true},
+   *                  the file will be overwritten, and if {@code false} an exception will be thrown.
+   * @return the OutputStream to write to.
+   * @throws IOException IO error.
+   */
+  OutputStream create(HoodieLocation location, boolean overwrite) throws IOException;
+
+  /**
+   * Opens an InputStream at the indicated location.
+   *
+   * @param location the file to open.
+   * @return the InputStream to read from.
+   * @throws IOException IO error.
+   */
+  InputStream open(HoodieLocation location) throws IOException;
+
+  /**
+   * Appends to an existing file (optional operation).
+   *
+   * @param location the file to append.
+   * @return the OutputStream to write to.
+   * @throws IOException IO error.
+   */
+  OutputStream append(HoodieLocation location) throws IOException;
+
+  /**
+   * Checks if a location exists.
+   *
+   * @param location location to check.
+   * @return {@code true} if the location exists.
+   * @throws IOException IO error.
+   */
+  boolean exists(HoodieLocation location) throws IOException;
+
+  /**
+   * Returns a file status object that represents the location.
+   *
+   * @param location location to check.
+   * @return a {@link HoodieFileStatus} object.
+   * @throws FileNotFoundException when the path does not exist.
+   * @throws IOException           IO error.
+   */
+  HoodieFileStatus getFileStatus(HoodieLocation location) throws IOException;
+
+  /**
+   * Creates the directory and non-existent parent directories.
+   *
+   * @param location location to create.
+   * @return {@code true} if the directory was created.
+   * @throws IOException IO error.
+   */
+  boolean createDirectory(HoodieLocation location) throws IOException;
+
+  /**
+   * Lists the statuses of the direct files/directories in the given location if the path is a directory.
+   *
+   * @param location given location.
+   * @return the statuses of the files/directories in the given location.
+   * @throws FileNotFoundException when the location does not exist.
+   * @throws IOException           IO error.
+   */
+  List<HoodieFileStatus> listDirectEntries(HoodieLocation location) throws IOException;
+
+  /**
+   * Lists the statuses of all files under the give location recursively.
+   *
+   * @param location given location.
+   * @return the statuses of the files under the given location.
+   * @throws FileNotFoundException when the location does not exist.
+   * @throws IOException           IO error.
+   */
+  List<HoodieFileStatus> listFiles(HoodieLocation location) throws IOException;
+
+  /**
+   * Lists the statuses of the direct files/directories in the given location
+   * and filters the results, if the path is a directory.
+   *
+   * @param location given location.
+   * @param filter   filter to apply.
+   * @return the statuses of the files/directories in the given location.
+   * @throws FileNotFoundException when the location does not exist.
+   * @throws IOException           IO error.
+   */
+  List<HoodieFileStatus> listDirectEntries(HoodieLocation location,
+                                           HoodieLocationFilter filter) throws IOException;
+
+  /**
+   * Returns all the files that match the locationPattern and are not checksum files,
+   * and filters the results.
+   *
+   * @param locationPattern given pattern.
+   * @param filter          filter to apply.
+   * @return the statuses of the files.
+   * @throws IOException IO error.
+   */
+  List<HoodieFileStatus> globEntries(HoodieLocation locationPattern, HoodieLocationFilter filter)
+      throws IOException;
+
+  /**
+   * Renames the location from old to new.
+   *
+   * @param oldLocation source location.
+   * @param newLocation destination location.
+   * @return {@true} if rename is successful.
+   * @throws IOException IO error.
+   */
+  boolean rename(HoodieLocation oldLocation, HoodieLocation newLocation) throws IOException;
+
+  /**
+   * Deletes a directory at location.
+   *
+   * @param location directory to delete.
+   * @return {@code true} if successful.
+   * @throws IOException IO error.
+   */
+  boolean deleteDirectory(HoodieLocation location) throws IOException;
+
+  /**
+   * Deletes a file at location.
+   *
+   * @param location file to delete.
+   * @return {@code true} if successful.
+   * @throws IOException IO error.
+   */
+  boolean deleteFile(HoodieLocation location) throws IOException;
+
+  /**
+   * Qualifies a path to one which uses this storage and, if relative, made absolute.
+   *
+   * @param location to qualify.
+   * @return Qualified location.
+   */
+  HoodieLocation makeQualified(HoodieLocation location);
+
+  /**
+   * @return the underlying file system instance if exists.
+   */
+  Object getFileSystem();
+
+  /**
+   * @return the underlying configuration instance if exists.
+   */
+  Object getConf();
+
+  /**
+   * Create an OutputStream at the indicated location.
+   * The file is overwritten by default.
+   *
+   * @param location the file to create.
+   * @return the OutputStream to write to.
+   * @throws IOException IO error.
+   */
+  default OutputStream create(HoodieLocation location) throws IOException {
+    return create(location, true);
+  }
+
+  /**
+   * Creates a new file with overwrite set to false. This ensures files are created
+   * only once and never rewritten, also, here we take care if the content is not
+   * empty, will first write the content to a temp file if {needCreateTempFile} is
+   * true, and then rename it back after the content is written.
+   *
+   * @param location file Path.
+   * @param content  content to be stored.
+   */
+  default void createImmutableFileInPath(HoodieLocation location,

Review Comment:
   We can make it final if non overwrite is expected.



##########
hudi-io/src/main/java/org/apache/hudi/io/storage/HoodieStorage.java:
##########
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.io.storage;
+
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieIOException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.Closeable;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Provides I/O APIs on files and directories on storage.
+ */
+public interface HoodieStorage extends Closeable {
+  Logger LOG = LoggerFactory.getLogger(HoodieStorage.class);
+  String TMP_PATH_POSTFIX = ".tmp";
+
+  /**
+   * @return the scheme of the storage.
+   */
+  String getScheme();
+
+  /**
+   * Creates an OutputStream at the indicated location.
+   *
+   * @param location  the file to create.
+   * @param overwrite if a file with this name already exists, then if {@code true},
+   *                  the file will be overwritten, and if {@code false} an exception will be thrown.
+   * @return the OutputStream to write to.
+   * @throws IOException IO error.
+   */
+  OutputStream create(HoodieLocation location, boolean overwrite) throws IOException;
+
+  /**
+   * Opens an InputStream at the indicated location.
+   *
+   * @param location the file to open.
+   * @return the InputStream to read from.
+   * @throws IOException IO error.
+   */
+  InputStream open(HoodieLocation location) throws IOException;
+
+  /**
+   * Appends to an existing file (optional operation).
+   *
+   * @param location the file to append.
+   * @return the OutputStream to write to.
+   * @throws IOException IO error.
+   */
+  OutputStream append(HoodieLocation location) throws IOException;
+
+  /**
+   * Checks if a location exists.
+   *
+   * @param location location to check.
+   * @return {@code true} if the location exists.
+   * @throws IOException IO error.
+   */
+  boolean exists(HoodieLocation location) throws IOException;
+
+  /**
+   * Returns a file status object that represents the location.
+   *
+   * @param location location to check.
+   * @return a {@link HoodieFileStatus} object.
+   * @throws FileNotFoundException when the path does not exist.
+   * @throws IOException           IO error.
+   */
+  HoodieFileStatus getFileStatus(HoodieLocation location) throws IOException;
+
+  /**
+   * Creates the directory and non-existent parent directories.
+   *
+   * @param location location to create.
+   * @return {@code true} if the directory was created.
+   * @throws IOException IO error.
+   */
+  boolean createDirectory(HoodieLocation location) throws IOException;
+
+  /**
+   * Lists the statuses of the direct files/directories in the given location if the path is a directory.
+   *
+   * @param location given location.
+   * @return the statuses of the files/directories in the given location.
+   * @throws FileNotFoundException when the location does not exist.
+   * @throws IOException           IO error.
+   */
+  List<HoodieFileStatus> listDirectEntries(HoodieLocation location) throws IOException;
+
+  /**
+   * Lists the statuses of all files under the give location recursively.
+   *
+   * @param location given location.
+   * @return the statuses of the files under the given location.
+   * @throws FileNotFoundException when the location does not exist.
+   * @throws IOException           IO error.
+   */
+  List<HoodieFileStatus> listFiles(HoodieLocation location) throws IOException;
+
+  /**
+   * Lists the statuses of the direct files/directories in the given location
+   * and filters the results, if the path is a directory.
+   *
+   * @param location given location.
+   * @param filter   filter to apply.
+   * @return the statuses of the files/directories in the given location.
+   * @throws FileNotFoundException when the location does not exist.
+   * @throws IOException           IO error.
+   */
+  List<HoodieFileStatus> listDirectEntries(HoodieLocation location,
+                                           HoodieLocationFilter filter) throws IOException;
+
+  /**
+   * Returns all the files that match the locationPattern and are not checksum files,
+   * and filters the results.
+   *
+   * @param locationPattern given pattern.
+   * @param filter          filter to apply.
+   * @return the statuses of the files.
+   * @throws IOException IO error.
+   */
+  List<HoodieFileStatus> globEntries(HoodieLocation locationPattern, HoodieLocationFilter filter)
+      throws IOException;
+
+  /**
+   * Renames the location from old to new.
+   *
+   * @param oldLocation source location.
+   * @param newLocation destination location.
+   * @return {@true} if rename is successful.
+   * @throws IOException IO error.
+   */
+  boolean rename(HoodieLocation oldLocation, HoodieLocation newLocation) throws IOException;
+
+  /**
+   * Deletes a directory at location.
+   *
+   * @param location directory to delete.
+   * @return {@code true} if successful.
+   * @throws IOException IO error.
+   */
+  boolean deleteDirectory(HoodieLocation location) throws IOException;
+
+  /**
+   * Deletes a file at location.
+   *
+   * @param location file to delete.
+   * @return {@code true} if successful.
+   * @throws IOException IO error.
+   */
+  boolean deleteFile(HoodieLocation location) throws IOException;
+
+  /**
+   * Qualifies a path to one which uses this storage and, if relative, made absolute.
+   *
+   * @param location to qualify.
+   * @return Qualified location.
+   */
+  HoodieLocation makeQualified(HoodieLocation location);
+
+  /**
+   * @return the underlying file system instance if exists.
+   */
+  Object getFileSystem();

Review Comment:
   Not sure if we should make the return type generic.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7336] Introduce new HoodieStorage abstraction [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10567:
URL: https://github.com/apache/hudi/pull/10567#issuecomment-1911433875

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "ca3cb838f94d96e02860e4587e035b903b7746d6",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22168",
       "triggerID" : "ca3cb838f94d96e02860e4587e035b903b7746d6",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * ca3cb838f94d96e02860e4587e035b903b7746d6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22168) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7336][RFR|DNM] Introduce new HoodieStorage abstraction [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10567:
URL: https://github.com/apache/hudi/pull/10567#issuecomment-1913368388

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "ca3cb838f94d96e02860e4587e035b903b7746d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22168",
       "triggerID" : "ca3cb838f94d96e02860e4587e035b903b7746d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8b765537863e155372fed079ea520f0a91a45b84",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22170",
       "triggerID" : "8b765537863e155372fed079ea520f0a91a45b84",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ea050a0d021273313ef3d9e5e3c566186e0b28ee",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22186",
       "triggerID" : "ea050a0d021273313ef3d9e5e3c566186e0b28ee",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * ea050a0d021273313ef3d9e5e3c566186e0b28ee Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22186) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7336][RFR|DNM] Introduce new HoodieStorage abstraction [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10567:
URL: https://github.com/apache/hudi/pull/10567#issuecomment-1913330268

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "ca3cb838f94d96e02860e4587e035b903b7746d6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22168",
       "triggerID" : "ca3cb838f94d96e02860e4587e035b903b7746d6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8b765537863e155372fed079ea520f0a91a45b84",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22170",
       "triggerID" : "8b765537863e155372fed079ea520f0a91a45b84",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ea050a0d021273313ef3d9e5e3c566186e0b28ee",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ea050a0d021273313ef3d9e5e3c566186e0b28ee",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8b765537863e155372fed079ea520f0a91a45b84 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22170) 
   * ea050a0d021273313ef3d9e5e3c566186e0b28ee UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org