You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "zhangyue19921010 (via GitHub)" <gi...@apache.org> on 2024/12/03 07:23:19 UTC

[PR] [HUDI-3625][RFC-60] Update RFC-60 With Federated Storage Layout [hudi]

zhangyue19921010 opened a new pull request, #12407:
URL: https://github.com/apache/hudi/pull/12407

   ### Change Logs
   
   During the landing Hudi lakehouse technology at JD, we encountered issues with the instability of the shared HDFS cluster, which led to frequent timeouts and restarts of Flink ingestion tasks, significantly affecting the performance and stability of data ingestion. To address this issue, we developed a solution based on the open-source community version, similar to RFC-60's Federated Storage Layout. This allows Hudi to support multiple pluggable physical storage systems and build logical table views on top of them, exposing these logical views externally.
   
   **Based on this design, we implemented a cache layer capability for hudi tables, where hot data is first written to a high-performance, dedicated small HDFS cluster to ensure strong performance and stability.** 
   
   Subsequently, through modified Clustering/Compaction table services, we achieve the migration of data from the cache layer to the persistent layer while organizing hudi table data.
   
   **_The implementation has shown significant results and has been successfully deployed to production, enduring multiple rounds of challenges during major shopping festivals like 618 and 11.11 (similar to Black Friday)_**. 
   
   For instance, in JD's traffic data ingestion business, with a single table containing 180 fields:
   
   After enabling JD's Federated Storage Layout sourced from PRD env:
   
   1. Flink writing performance(5000C) improved by 150%, reaching 770 million records per minute .
   2. Consumption throughput reached 27 GB/s.
   3. Flink checkpoint duration improved 90.3% from 31.8s to 3s
   4. Flink stability improved by 97%.
   5. Data freshness was optimized from T+1 to a minute-level.
   
   Here is PRD monitor 
   <img width="1549" alt="image" src="https://github.com/user-attachments/assets/89c654f6-b5c8-4b1d-aa54-18880bc47c7a">
   
   Here is our benchmark result using 2000C
   
   <img width="849" alt="image" src="https://github.com/user-attachments/assets/9404c789-08d7-4257-9f6f-9df8d2771a1c">
   
   This has comprehensively enhanced the user experience of traffic data utilization.
   
   After discussions with Ethan and CTTY, we are collaboratively building the Open Source Federated Storage Layout. Based on my production experience, I have provided some improvement suggestions regarding interface design, engine integration methods, etc. Please review.
   
   ### Impact
   
   no
   
   ### Risk level (write none, low medium or high below)
   low
   
   ### Documentation Update
   
   _Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none"._
   
   - _The config description must be updated if new configs are added or the default value of the configs are changed_
   - _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
     ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
     changes to the website._
   
   ### Contributor's checklist
   
   - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [ ] Change Logs and Impact were stated clearly
   - [ ] Adequate tests were added if applicable
   - [ ] CI passed
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-3625][RFC-60] Update RFC-60 With Federated Storage Layout [hudi]

Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on PR #12407:
URL: https://github.com/apache/hudi/pull/12407#issuecomment-2533546180

   @CTTY @yihua https://github.com/apache/hudi/pull/12460 poc for OSS storage strategy is finished. data and meta distribution are all as expected. PTAL


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-3625][RFC-60] Update RFC-60 With Federated Storage Layout [hudi]

Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #12407:
URL: https://github.com/apache/hudi/pull/12407#discussion_r1870575752


##########
rfc/rfc-60/rfc-60.md:
##########
@@ -198,46 +419,68 @@ for metadata table to be populated.
 
 4. If there is an error reading from Metadata table, we will not fall back listing from file system.
 
+After enabling the Federated Storage Layout feature, under certain strategies such as the "data cache layer," 
+data from different lake tables may be stored on different physical media, resulting in different schemes. 
+For example, cache layer data may be stored on hdfs://ns1/, while persistent layer data is stored on hdfs://ns2/. 
+In this case, we need to add a new field named "scheme" in MDT HoodieMetadataFileInfo to store the scheme information for different files, 
+which will be used for path restoration.
+
+```avro schema
+        {
+            "doc": "Contains information about partitions and files within the dataset",
+            "name": "filesystemMetadata",
+            "type": [
+                "null",
+                {
+                    "type": "map",
+                    "values": {
+                        "type": "record",
+                        "name": "HoodieMetadataFileInfo",
+                        "fields": [
+                            {
+                                "name": "size",
+                                "type": "long",
+                                "doc": "Size of the file"
+                            },
+                            {
+                                "name": "isDeleted",
+                                "type": "boolean",
+                                "doc": "True if this file has been deleted"
+                            },
+                            {
+                                "name":"scheme",

Review Comment:
   For example  `s3://<table_storage_bucket>/hudi_location/0bfb3d6e/<hudi_table_name>/9320889c-8537-4aa7-a63e-ef088b9a21ce-0_9-11-51_20220301005056692.parquet`
   
   we need to record `s3://<table_storage_bucket>/`. rename it to prefix



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-3625][RFC-60] Update RFC-60 With Federated Storage Layout [hudi]

Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #12407:
URL: https://github.com/apache/hudi/pull/12407#discussion_r1870571856


##########
rfc/rfc-60/rfc-60.md:
##########
@@ -84,23 +119,85 @@ public interface HoodieStorageStrategy extends Serializable {
   /**
    * Return a storage location for the given filename.
    *
-   * @param fileId data file ID
+   * @param path fileName
    * @return a storage location string for a data file
    */
-  String storageLocation(String fileId);
+  StoragePath storageLocation(String path, String instantTime);
 
   /**
-   * Return a storage location for the given partition and filename.
+   * Return all possible StoragePaths
    *
-   * @param partitionPath partition path for the file
-   * @param fileId data file ID
-   * @return a storage location string for a data file
+   * @param partitionPath
+   * @param checkExist check if StoragePath is truly existed or not. 
+   * @return a st of storage partition path
+   */
+  Set<StoragePath> getAllLocations(String partitionPath, boolean checkExist);
+
+  /**
+   * Return RelativePath base on path and locations.
+   *
+   * @param path
+   * @return relative path
+   */
+  String getRelativePath(Path path);
+}
+```
+
+```java
+public class HoodieStorageStrategyFactory {
+  private HoodieStorageStrategyFactory() {
+  }
+
+  public static HoodieStorageStrategy getInstant(HoodieTableMetaClient metaClient, Boolean reset) {
+    HoodieTableConfig config = metaClient.getTableConfig();
+    if (reset) {
+      return getInstant(HoodieStorageStrategyType.DEFAULT.value, metaClient.getBasePath(), config.getStoragePath(), metaClient);
+    }
+    return getInstant(config.getStorageStrategy(), metaClient.getBasePath(), config.getStoragePath(), metaClient);
+  }
+
+  public static HoodieStorageStrategy getInstant(HoodieTableMetaClient metaClient) {
+    return getInstant(metaClient, false);
+  }
+
+  /**
+   *  Just for HoodieParquetInputFormatBase
+   * @param config
+   * @param basePath
+   * @return
    */
-  String storageLocation(String partitionPath, String fileId);
+  public static HoodieStorageStrategy getInstant(HoodieTableConfig config, String basePath, Boolean reset) {
+    if (reset) {
+      return getInstant(HoodieStorageStrategyType.DEFAULT.value, basePath, config.getStoragePath(), null);
+    }
+    return getInstant(config.getStorageStrategy(), basePath, config.getStoragePath(), null);
+  }
+
+  private static HoodieStorageStrategy getInstant(
+      String storageStrategyClass,
+      String basePath,
+      String storagePath, HoodieTableMetaClient metaClient) {
+    return (HoodieStorageStrategy) ReflectionUtils.loadClass(storageStrategyClass,
+        basePath, storagePath, metaClient == null ? Option.empty() : Option.of(metaClient));
+  }
 }
 ```
 
-### Generating File Paths for Object Store Optimized Layout
+```java
+public enum HoodieStorageStrategyType {

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-3625][RFC-60] Update RFC-60 With Federated Storage Layout [hudi]

Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on PR #12407:
URL: https://github.com/apache/hudi/pull/12407#issuecomment-2513743753

   @yihua @CTTY PTAL


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-3625][RFC-60] Update RFC-60 With Federated Storage Layout [hudi]

Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #12407:
URL: https://github.com/apache/hudi/pull/12407#discussion_r1868808333


##########
rfc/rfc-60/rfc-60.md:
##########
@@ -84,23 +119,85 @@ public interface HoodieStorageStrategy extends Serializable {
   /**
    * Return a storage location for the given filename.
    *
-   * @param fileId data file ID
+   * @param path fileName

Review Comment:
   This path is `partitionPath` with scheme actually. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-3625][RFC-60] Update RFC-60 With Federated Storage Layout [hudi]

Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #12407:
URL: https://github.com/apache/hudi/pull/12407#discussion_r1870573138


##########
rfc/rfc-60/rfc-60.md:
##########
@@ -178,6 +275,130 @@ The hashing function should be made user configurable for use cases like bucketi
 sub-partitioning/re-hash to reduce the number of hash prefixes. Having too many unique hash prefixes
 would make files too dispersed, and affect performance on other operations such as listing.
 
+### Case2: Hudi Storage Cache Layer
+
+The cache layer of a lake table is a specific implementation scenario for Federated Storage Layout in hudi tables. 
+It divides the physical storage of the lake table into a high-performance HDFS-based cache layer and a shared HDFS-based persistent layer. 
+Hot data is first written to the cache layer and later moved to the persistent layer through table services like Compaction and Clustering. 
+This setup addresses the strong demands for performance and stability in scenarios involving massive data ingestion into the lake. 
+It is important to note that once data is written to the cache layer and committed, it becomes visible to downstream processes, regardless of when 
+the "relocation work" starts or finishes. Additionally, since the data relocation from the cache layer to the persistent layer leverages the lake table's 
+own Compaction and Clustering capabilities, this process adheres to the lake table's commit mechanism and MVCC snapshot isolation design. Therefore, 
+after enabling the cache layer, the lake table maintains atomicity, transactional guarantees, and Exactly Once semantics.
+
+Below is a comparison between the normal read/write process of a lake table and the process after enabling the lake table cache layer. 
+Green arrows indicate data reads, and red arrows indicate data writes. Before enabling the cache layer, the compute engine directly writes 
+data to the shared HDFS and commits, including Parquet base files, log files, and metadata files. The compute engine queries the lake table 
+by directly reading data from the shared HDFS. Additionally, the lake table services for Clustering and Compaction also directly query data from 
+the shared HDFS, process it, and write it back to the shared HDFS. After enabling the lake table cache layer, the compute engine first writes hot 
+data to the high-performance HDFS and commits, including Parquet files and log files. During queries, a unified logical view of both the cache layer 
+and the persistent layer is constructed to meet query demands. The Clustering and Compaction table services, while performing regular lake table file 
+organization, also facilitate data relocation from the cache layer to the persistent layer. Notably, regardless of when Clustering and Compaction jobs 
+start or finish, the data visible to downstream processes is always complete and timely.
+
+Original Read/Write workflow
+![originalReadWrite.png](originalReadWrite.png)
+
+Read/Write workflow with hudi cache layer enabled
+![ReadWriteWithCacheLayer.png](ReadWriteWithCacheLayer.png)
+
+#### HoodieCacheLayerStorageStrategy
+Based on the HoodieActiveTimeline and the current write instant, determine the specific write path. For common commit operations 
+in COW (Copy on Write) tables and delta commit operations in MOR (Merge on Read) tables, we generate a CacheLayer-related Storage 
+Path to write/read. This type of I/O is targeted at the cache layer.
+
+As for commit action in mor table and replace commit in cow table, we will generate a persistent Storage Path which will let 
+Compaction/Clustering do the data migration works from cache layer to persistent layer
+
+Note: It is required that Clustering is enabled for COW tables and Compaction is enabled for MOR tables; otherwise, there is a 
+risk of storage overflow in the cache layer.
+
+```java
+/**
+ * When using Storage Cache Layer make sure that table service is enabled :
+ * 1. MOR + Upsert + Compaction
+ * 2. COW + Insert + Clustering
+ */
+public class HoodieCacheLayerStorageStrategy extends HoodieDefaultStorageStrategy {
+
+  private HoodieTableType tableType;
+  private String hoodieStorageStrategyModifyTime;

Review Comment:
   no need actually. As we discuss, we only  focus on abstraction, so removed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-3625][RFC-60] Update RFC-60 With Federated Storage Layout [hudi]

Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #12407:
URL: https://github.com/apache/hudi/pull/12407#discussion_r1868858025


##########
rfc/rfc-60/rfc-60.md:
##########
@@ -84,23 +119,85 @@ public interface HoodieStorageStrategy extends Serializable {
   /**
    * Return a storage location for the given filename.
    *
-   * @param fileId data file ID
+   * @param path fileName

Review Comment:
   > Also could you help me understand why instantTime is needed in this API by adding it to the comment?
   
   The instantTime is a necessary hint for HoodieCacheLayerStorageStrategy to determine whether the current data should be written to/coming from the cache layer or the persistent layer. Therefore, it needs to be passed in. Similarly, HoodieObjectStorageStrategy might require fileID. I thought about it and maybe we can abstract this into a map. Different strategies might need different hints, so we can pass them all into the map.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-3625][RFC-60] Update RFC-60 With Federated Storage Layout [hudi]

Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #12407:
URL: https://github.com/apache/hudi/pull/12407#discussion_r1868799544


##########
rfc/rfc-60/rfc-60.md:
##########
@@ -18,33 +18,68 @@
 # RFC-60: Federated Storage Layout
 
 ## Proposers
+- @zhangyue19921010
+- @CTTY
 - @umehrot2
 
 ## Approvers
 - @vinoth
 - @shivnarayan
+- @yihua
 
 ## Status
 
 JIRA: [https://issues.apache.org/jira/browse/HUDI-3625](https://issues.apache.org/jira/browse/HUDI-3625)
 
 ## Abstract
+In this RFC, we will support the Federated Storage Layout for Hudi tables, enabling Hudi to support multiple pluggable physical 
+storage systems. By combining Hudi's own metadata, we can construct logical table views and expose these views externally,
+making them transparent to the engine.
 
+![](FederatedStorageLayout.png)
+
+After implementing the Hudi Federated Storage Layout, we can develop many interesting new features for Hudi lake tables based on this, such as:
+
+### Object Store Optimized Layout
 As you scale your Apache Hudi workloads over cloud object stores like Amazon S3, there is potential of hitting request
 throttling limits which in-turn impacts performance. In this RFC, we are proposing to support an alternate storage
 layout that is optimized for Amazon S3 and other cloud object stores, which helps achieve maximum throughput and
 significantly reduce throttling.
 
+![](ObjectStoreOptimizedLayout.png)
+
+### Hudi Storage Cache Layer
+The Hudi lake table data cache layer involves dividing the lake table physical storage into a high-performance HDFS-based data 
+cache layer and a shared HDFS-based data persistence layer. Hot data is initially written to the cache layer and later moved to 
+the persistence layer through table services like Compaction and Clustering. This approach meets the strong demands for performance 
+and stability in scenarios involving massive data ingestion into the lake. It is important to note that once data is written to the 
+cache layer and committed, it becomes visible to downstream consumers, regardless of when the subsequent "moving operations" start 
+or finish, ensuring data visibility and timeliness are unaffected. Additionally, since the data movement from the cache layer to 
+the persistence layer leverages the lake table's own Compaction and Clustering table service capabilities, this process adheres to 
+the lake table's commit mechanism and MVCC snapshot isolation design. Therefore, with the data cache layer enabled, the lake table 
+maintains atomicity, transactional guarantees, and Exactly Once semantics.
+
+![](HudiStorageCacheLayer.png)
+
+### Hudi Table Second-Level Latency

Review Comment:
   Hey @CTTY . Thanks for your review. My thought is that Case 3 is another implementation of the Federated Storage Layout, rather than just a concept of a caching layer. In Case 3, we effectively use Kafka as a log storage (since both follow an append-only model like avro) and implement a complete set of HoodieKafkaLogScanner and corresponding Compactor. Regarding data visibility in Case 3, the data should be visible externally as soon as it enters Kafka, without a strong dependency on Compaction/Clustering operations. For the query side, it will return a union of data from Kafka and the base files. Of course, it also needs to provide the capability to read/stream Kafka logs in a read-only manner. In summary, this approach aims to achieve second-level latency and visibility for lake tables based on the Hudi Federated Storage Layout. Also update to rfc.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-3625][RFC-60] Update RFC-60 With Federated Storage Layout [hudi]

Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #12407:
URL: https://github.com/apache/hudi/pull/12407#discussion_r1869051330


##########
rfc/rfc-60/rfc-60.md:
##########
@@ -84,23 +119,85 @@ public interface HoodieStorageStrategy extends Serializable {
   /**
    * Return a storage location for the given filename.
    *
-   * @param fileId data file ID
+   * @param path fileName
    * @return a storage location string for a data file
    */
-  String storageLocation(String fileId);
+  StoragePath storageLocation(String path, String instantTime);
 
   /**
-   * Return a storage location for the given partition and filename.
+   * Return all possible StoragePaths
    *
-   * @param partitionPath partition path for the file
-   * @param fileId data file ID
-   * @return a storage location string for a data file
+   * @param partitionPath
+   * @param checkExist check if StoragePath is truly existed or not. 
+   * @return a st of storage partition path
+   */
+  Set<StoragePath> getAllLocations(String partitionPath, boolean checkExist);

Review Comment:
   For `HoodieObjectStoreStrategy`, can we design it such that `HoodieObjectStoreStrategy` specifies `hoodie.storage.path` as `s3://<table_storage_bucket>/hudi_location/`, and then all data is written under `s3://<table_storage_bucket>/hudi_location/` like
   ```
   s3://<table_storage_bucket>/hudi_location/0bfb3d6e/<hudi_table_name>/9320889c-8537-4aa7-a63e-ef088b9a21ce-0_9-11-51_20220301005056692.parquet
   s3://<table_storage_bucket>/hudi_location/0bfb3d6e/<hudi_table_name>/a62aa56b-d55e-4a2b-88a6-d603ef26775c-0_8-11-50_20220301005056692.parquet
   s3://<table_storage_bucket>/hudi_location/0bfb3d6e/<hudi_table_name>/.4b0c6b40-2ac0-4a1c-a26f-6338aa4db22e-0_6-11-48_20220301005056692.log.1_0-22-26
   s3://<table_storage_bucket>/hudi_location/0bfb3d6e/<hudi_table_name>/.075f3295-def8-4a42-a927-07fd2dd2976c-0_7-11-49_20220301005056692.log.1_0-22-26
   ```
   In the `HoodieObjectStoreStrategy#getAllLocations()` method, first list the `hoodie.storage.path`, and then append the relative path.
   
   getAllLocations() is not in `hot path`, so list cost is not very expensive.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-3625][RFC-60] Update RFC-60 With Federated Storage Layout [hudi]

Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #12407:
URL: https://github.com/apache/hudi/pull/12407#discussion_r1870571369


##########
rfc/rfc-60/rfc-60.md:
##########
@@ -84,23 +119,85 @@ public interface HoodieStorageStrategy extends Serializable {
   /**
    * Return a storage location for the given filename.
    *
-   * @param fileId data file ID
+   * @param path fileName
    * @return a storage location string for a data file
    */
-  String storageLocation(String fileId);
+  StoragePath storageLocation(String path, String instantTime);
 
   /**
-   * Return a storage location for the given partition and filename.
+   * Return all possible StoragePaths
    *
-   * @param partitionPath partition path for the file
-   * @param fileId data file ID
-   * @return a storage location string for a data file
+   * @param partitionPath
+   * @param checkExist check if StoragePath is truly existed or not. 
+   * @return a st of storage partition path
+   */
+  Set<StoragePath> getAllLocations(String partitionPath, boolean checkExist);
+
+  /**
+   * Return RelativePath base on path and locations.
+   *
+   * @param path
+   * @return relative path
+   */
+  String getRelativePath(Path path);
+}
+```
+
+```java
+public class HoodieStorageStrategyFactory {
+  private HoodieStorageStrategyFactory() {
+  }
+
+  public static HoodieStorageStrategy getInstant(HoodieTableMetaClient metaClient, Boolean reset) {

Review Comment:
   removed. We can  focus on the abstraction



##########
rfc/rfc-60/rfc-60.md:
##########
@@ -84,23 +119,85 @@ public interface HoodieStorageStrategy extends Serializable {
   /**
    * Return a storage location for the given filename.
    *
-   * @param fileId data file ID
+   * @param path fileName
    * @return a storage location string for a data file
    */
-  String storageLocation(String fileId);
+  StoragePath storageLocation(String path, String instantTime);
 
   /**
-   * Return a storage location for the given partition and filename.
+   * Return all possible StoragePaths
    *
-   * @param partitionPath partition path for the file
-   * @param fileId data file ID
-   * @return a storage location string for a data file
+   * @param partitionPath
+   * @param checkExist check if StoragePath is truly existed or not. 
+   * @return a st of storage partition path
+   */
+  Set<StoragePath> getAllLocations(String partitionPath, boolean checkExist);
+
+  /**
+   * Return RelativePath base on path and locations.
+   *
+   * @param path
+   * @return relative path
+   */
+  String getRelativePath(Path path);
+}
+```
+
+```java
+public class HoodieStorageStrategyFactory {

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-3625][RFC-60] Update RFC-60 With Federated Storage Layout [hudi]

Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #12407:
URL: https://github.com/apache/hudi/pull/12407#discussion_r1869051330


##########
rfc/rfc-60/rfc-60.md:
##########
@@ -84,23 +119,85 @@ public interface HoodieStorageStrategy extends Serializable {
   /**
    * Return a storage location for the given filename.
    *
-   * @param fileId data file ID
+   * @param path fileName
    * @return a storage location string for a data file
    */
-  String storageLocation(String fileId);
+  StoragePath storageLocation(String path, String instantTime);
 
   /**
-   * Return a storage location for the given partition and filename.
+   * Return all possible StoragePaths
    *
-   * @param partitionPath partition path for the file
-   * @param fileId data file ID
-   * @return a storage location string for a data file
+   * @param partitionPath
+   * @param checkExist check if StoragePath is truly existed or not. 
+   * @return a st of storage partition path
+   */
+  Set<StoragePath> getAllLocations(String partitionPath, boolean checkExist);

Review Comment:
   For `HoodieObjectStoreStrategy`, we might need to strongly rely on MDT. For the given relative partition path, use MDT to query all the corresponding files, calculate hash value and then append their schemes.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-3625][RFC-60] Update RFC-60 With Federated Storage Layout [hudi]

Posted by "voonhous (via GitHub) via commits" <co...@hudi.apache.org>.
voonhous commented on PR #12407:
URL: https://github.com/apache/hudi/pull/12407#issuecomment-6700451287

   Temporarily closing this PR while we debug a GitHub settings error on master (#20471); it will be reopened shortly, no action needed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-3625][RFC-60] Update RFC-60 With Federated Storage Layout [hudi]

Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #12407:
URL: https://github.com/apache/hudi/pull/12407#discussion_r1869005485


##########
rfc/rfc-60/rfc-60.md:
##########
@@ -84,23 +119,85 @@ public interface HoodieStorageStrategy extends Serializable {
   /**
    * Return a storage location for the given filename.
    *
-   * @param fileId data file ID
+   * @param path fileName
    * @return a storage location string for a data file
    */
-  String storageLocation(String fileId);
+  StoragePath storageLocation(String path, String instantTime);
 
   /**
-   * Return a storage location for the given partition and filename.
+   * Return all possible StoragePaths
    *
-   * @param partitionPath partition path for the file
-   * @param fileId data file ID
-   * @return a storage location string for a data file
+   * @param partitionPath
+   * @param checkExist check if StoragePath is truly existed or not. 
+   * @return a st of storage partition path
+   */
+  Set<StoragePath> getAllLocations(String partitionPath, boolean checkExist);

Review Comment:
   This is needed, for example, in the `AbstractTableFileSystemView#ensurePartitionLoadedCorrectly(String partition)` method. 
   
   An external relative path partition is passed in, and currently, the base path is directly prefixed to it before performing the listPartition(Path partitionPath) operation. 
   
   For Hudi tables with the Federated Storage Layout feature enabled, the same relative path might map to multiple physical paths. Here, the HoodieStorageStrategy#getAllLocations method is required.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-3625][RFC-60] Update RFC-60 With Federated Storage Layout [hudi]

Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #12407:
URL: https://github.com/apache/hudi/pull/12407#discussion_r1870570603


##########
rfc/rfc-60/rfc-60.md:
##########
@@ -84,23 +119,85 @@ public interface HoodieStorageStrategy extends Serializable {
   /**
    * Return a storage location for the given filename.
    *
-   * @param fileId data file ID
+   * @param path fileName
    * @return a storage location string for a data file
    */
-  String storageLocation(String fileId);
+  StoragePath storageLocation(String path, String instantTime);
 
   /**
-   * Return a storage location for the given partition and filename.
+   * Return all possible StoragePaths
    *
-   * @param partitionPath partition path for the file
-   * @param fileId data file ID
-   * @return a storage location string for a data file
+   * @param partitionPath
+   * @param checkExist check if StoragePath is truly existed or not. 
+   * @return a st of storage partition path
+   */
+  Set<StoragePath> getAllLocations(String partitionPath, boolean checkExist);
+
+  /**
+   * Return RelativePath base on path and locations.
+   *
+   * @param path
+   * @return relative path
+   */
+  String getRelativePath(Path path);

Review Comment:
   Yes the same as storageLocation



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-3625][RFC-60] Update RFC-60 With Federated Storage Layout [hudi]

Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #12407:
URL: https://github.com/apache/hudi/pull/12407#discussion_r1870577157


##########
rfc/rfc-60/rfc-60.md:
##########
@@ -198,46 +419,68 @@ for metadata table to be populated.
 
 4. If there is an error reading from Metadata table, we will not fall back listing from file system.
 
+After enabling the Federated Storage Layout feature, under certain strategies such as the "data cache layer," 
+data from different lake tables may be stored on different physical media, resulting in different schemes. 
+For example, cache layer data may be stored on hdfs://ns1/, while persistent layer data is stored on hdfs://ns2/. 
+In this case, we need to add a new field named "scheme" in MDT HoodieMetadataFileInfo to store the scheme information for different files, 
+which will be used for path restoration.
+
+```avro schema
+        {
+            "doc": "Contains information about partitions and files within the dataset",
+            "name": "filesystemMetadata",
+            "type": [
+                "null",
+                {
+                    "type": "map",
+                    "values": {
+                        "type": "record",
+                        "name": "HoodieMetadataFileInfo",
+                        "fields": [
+                            {
+                                "name": "size",
+                                "type": "long",
+                                "doc": "Size of the file"
+                            },
+                            {
+                                "name": "isDeleted",
+                                "type": "boolean",
+                                "doc": "True if this file has been deleted"
+                            },
+                            {
+                                "name":"scheme",
+                                "type": ["null","string"],
+                                "default":null
+                            }
+                        ]
+                    }
+                }
+            ]
+        }
+```
+
+Note: For lake tables that do not have the Federated Storage Layout enabled, the value of this "scheme" field will be null.
+
 ### Integration
 This section mainly describes how storage strategy is integrated with other components and how read/write
-would look like from Hudi side with object storage layout.
-
-We propose integrating the storage strategy at the filesystem level, specifically within `HoodieWrapperFileSystem`. 
-This way, only file read/write operations undergo path conversion and we can limit the usage of 
-storage strategy to only filesystem level so other upper-level components don't need to be aware of physical paths.
-
-This also mandates that `HoodieWrapperFileSystem` is the filesystem of choice for all upper-level Hudi components.
-Getting filesystem from `Path` or such won't be allowed anymore as using raw filesystem may not reach 
-to physical locations without storage strategy. Hudi components can simply call `HoodieMetaClient#getFs` 
-to get `HoodieWrapperFileSystem`, and this needs to be the only allowed way for any filesystem-related operation. 
-The only exception is when we need to interact with metadata that's still stored under the original table path, 
-and we should call `HoodieMetaClient#getRawFs` in this case so `HoodieMetaClient` can still be the single entry
-for getting filesystem.
-
-![](wrapper_fs.png)
-
-When conducting a read operation, Hudi would: 
-1. Access filesystem view, `HoodieMetadataFileSystemView` specifically
-2. Scan metadata table via filesystem view to compose `HoodieMetadataPayload`
-3. Call `HoodieMetadataPayload#getFileStatuses` and employ `HoodieWrapperFileSystem` to get 
-file statuses with physical locations
-
-This flow can be concluded in the chart below.
-
-![](read_flow.png)
-
-#### Considerations
-- Path conversion happens on the fly when reading/writing files. This saves Hudi from storing physical locations,
-and adds the cost of hashing, but the performance burden should be negligible.
-- Since table path and data path will most likely have different top-level folders/authorities,
-`HoodieWrapperFileSystem` should maintain at least two `FileSystem` objects: one to access table path and another
-to access storage path. `HoodieWrapperFileSystem` should intelligently tell if it needs
-to convert the path by checking the path on the fly.
-- When using Hudi file reader/writer implementation, we will need to pass `HoodieWrapperFileSystem` down
-to parent reader. For instance, when using `HoodieAvroHFileReader`, we will need to pass `HoodieWrapperFileSystem`
-to `HFile.Reader` so it can have access to storage strategy. If reader/writer doesn't take filesystem
-directly (e.g. `ParquetFileReader` only takes `Configuration` and `Path` for reading), then we will
-need to register `HoodieWrapperFileSystem` to `Configuration` so it can be initialized/used later.
+would look like from Hudi side with Federated Storage Layout.
+
+We already have the abstractions HoodieStorage, StoragePath, and HoodieWrapperFileSystem. Here, we need to add a 

Review Comment:
   emmmm,actually  in our practice, we also need to do some changing in hoodiewrapperfs,because For HoodieWrapperFileSystem, after enabling the Federated Storage Layout feature, a single HoodieWrapperFileSystem 
   instance will handle file objects with different schemes. Therefore, we need to cache the inner file systems corresponding 
   to different schemes within HoodieWrapperFileSystem. When necessary, the correct inner file system object can be retrieved 
   from the cache based on the input path scheme.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-3625][RFC-60] Update RFC-60 With Federated Storage Layout [hudi]

Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #12407:
URL: https://github.com/apache/hudi/pull/12407#discussion_r1870595188


##########
rfc/rfc-60/rfc-60.md:
##########
@@ -84,23 +119,85 @@ public interface HoodieStorageStrategy extends Serializable {
   /**
    * Return a storage location for the given filename.
    *
-   * @param fileId data file ID
+   * @param path fileName
    * @return a storage location string for a data file
    */
-  String storageLocation(String fileId);
+  StoragePath storageLocation(String path, String instantTime);
 
   /**
-   * Return a storage location for the given partition and filename.
+   * Return all possible StoragePaths
    *
-   * @param partitionPath partition path for the file
-   * @param fileId data file ID
-   * @return a storage location string for a data file
+   * @param partitionPath
+   * @param checkExist check if StoragePath is truly existed or not. 
+   * @return a st of storage partition path
+   */
+  Set<StoragePath> getAllLocations(String partitionPath, boolean checkExist);
+
+  /**
+   * Return RelativePath base on path and locations.
+   *
+   * @param path
+   * @return relative path
+   */
+  String getRelativePath(Path path);
+}
+```
+
+```java
+public class HoodieStorageStrategyFactory {
+  private HoodieStorageStrategyFactory() {
+  }
+
+  public static HoodieStorageStrategy getInstant(HoodieTableMetaClient metaClient, Boolean reset) {
+    HoodieTableConfig config = metaClient.getTableConfig();
+    if (reset) {
+      return getInstant(HoodieStorageStrategyType.DEFAULT.value, metaClient.getBasePath(), config.getStoragePath(), metaClient);
+    }
+    return getInstant(config.getStorageStrategy(), metaClient.getBasePath(), config.getStoragePath(), metaClient);
+  }
+
+  public static HoodieStorageStrategy getInstant(HoodieTableMetaClient metaClient) {
+    return getInstant(metaClient, false);
+  }
+
+  /**
+   *  Just for HoodieParquetInputFormatBase
+   * @param config
+   * @param basePath
+   * @return
    */
-  String storageLocation(String partitionPath, String fileId);
+  public static HoodieStorageStrategy getInstant(HoodieTableConfig config, String basePath, Boolean reset) {
+    if (reset) {
+      return getInstant(HoodieStorageStrategyType.DEFAULT.value, basePath, config.getStoragePath(), null);
+    }
+    return getInstant(config.getStorageStrategy(), basePath, config.getStoragePath(), null);
+  }
+
+  private static HoodieStorageStrategy getInstant(
+      String storageStrategyClass,
+      String basePath,
+      String storagePath, HoodieTableMetaClient metaClient) {
+    return (HoodieStorageStrategy) ReflectionUtils.loadClass(storageStrategyClass,
+        basePath, storagePath, metaClient == null ? Option.empty() : Option.of(metaClient));
+  }
 }
 ```
 
-### Generating File Paths for Object Store Optimized Layout
+```java
+public enum HoodieStorageStrategyType {
+  DEFAULT(HoodieDefaultStorageStrategy.class.getName()),
+  CACHE_LAYER(HoodieCacheLayerStorageStrategy.class.getName()),
+  OBJECT_STORAGE_STRATEGY(ObjectStorageStrategy.class.getName());
+
+  public final String value;
+
+  HoodieStorageStrategyType(String strategy) {
+    this.value = strategy;
+  }
+}
+```
+
+### Case1: Generating File Paths for Object Store Optimized Layout

Review Comment:
   Done!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-3625][RFC-60] Update RFC-60 With Federated Storage Layout [hudi]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on code in PR #12407:
URL: https://github.com/apache/hudi/pull/12407#discussion_r1881347866


##########
rfc/rfc-60/rfc-60.md:
##########
@@ -74,37 +116,78 @@ not meet their use-case.
 
 ## Design
 
-### Interface
+### Config
+We will add two table-level parameters, namely Hoodie Table Config:
+
+The first one, `hoodie.storage.path` allows users to specify additional physical storage locations apart from the Base Path, separated by commas if there are multiple.
+
+The second one, `hoodie.storage.strategy.class` allows users to specify the storage strategy for the current Hudi table, such as 
+HoodieDefaultStorageStrategy (default), HoodieCacheLayerStorageStrategy, and HoodieObjectStorageStrategy.
+Take HoodieCacheLayerStorageStrategy as example, related table config should like this
+```text
+[sss@ssss ~]$ hdfs dfs -cat hdfs://nsxxx/user/jdr_lakehouse/gdm.db/xxx/.hoodie/hoodie.properties
+#Updated at 2024-08-07T11:49:10.252Z
+#Wed Aug 07 19:49:10 CST 2024
+hoodie.table.keygenerator.class=org.apache.hudi.keygen.ComplexAvroKeyGenerator
+hoodie.table.precombine.field=receive_ts
+hoodie.table.version=5
+hoodie.datasource.write.hive_style_partitioning=true
+hoodie.table.checksum=318067208
+hoodie.table.create.schema={xxxx}
+hoodie.table.cdc.enabled=false
+hoodie.archivelog.folder=archived
+hoodie.table.name=xxxx
+hoodie.table.type=COPY_ON_WRITE
+hoodie.datasource.write.partitionpath.urlencode=false
+hoodie.datasource.write.drop.partition.columns=false
+hoodie.timeline.layout.version=1
+hoodie.table.recordkey.fields=mid,dt
+hoodie.table.partition.fields=dt,application,business
+hoodie.storage.path=hdfs\://nsxxxx/user/jdr_lakehouse/jdr_lakehouse_traffic/gdm.db/gdm_jdr_xxxxx
+hoodie.storage.strategy.class=org.apache.hudi.common.storage.HoodieCacheLayerStorageStrategy
+```
+
+
+### Core Abstraction
 
 ```java
 /**
  * Interface for providing storage file locations.
  */
 public interface HoodieStorageStrategy extends Serializable {
   /**
-   * Return a storage location for the given filename.
+   * Return a storage location for the given path
    *
-   * @param fileId data file ID
-   * @return a storage location string for a data file
+   * @param path
+   * @param configMap
+   * @return Append the appropriate prefix based on the Path and return
    */
-  String storageLocation(String fileId);
+  StoragePath storageLocation(String path, Map<String, String> configMap);

Review Comment:
   I took a look at #12460 and I'm still hesitant on passing a config map to storage strategy API instead of fixed configs. Passing a config map makes it harder to enforce what should be passed in, and will eventually cause failures for certain strategy implementations that require specific props. 
   
   One way to solve this is to add a new class `StrategySpec` as a container and enforces certain props to be passed in when creating an instance of it. But even this is worse than simply asking for props like `fileId` and `instantTime` in the API directly.
   
   I have some rough thoughts on using `StorageStrategFactory` to dynamicly create strategies instead of passing props/configs when calling API, something like below: 
   ```
   Factory {
     getInstance(String path, tableConfig, Map<> props) {
       // some logic to infer strategy from path and tableConfig
       Reflection.new(className, props)
     }
   }
   ```
   
   I haven't got the time to dig deeper though, will need to circle back on this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-3625][RFC-60] Update RFC-60 With Federated Storage Layout [hudi]

Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #12407:
URL: https://github.com/apache/hudi/pull/12407#discussion_r1870572657


##########
rfc/rfc-60/rfc-60.md:
##########
@@ -178,6 +275,130 @@ The hashing function should be made user configurable for use cases like bucketi
 sub-partitioning/re-hash to reduce the number of hash prefixes. Having too many unique hash prefixes
 would make files too dispersed, and affect performance on other operations such as listing.
 
+### Case2: Hudi Storage Cache Layer
+
+The cache layer of a lake table is a specific implementation scenario for Federated Storage Layout in hudi tables. 
+It divides the physical storage of the lake table into a high-performance HDFS-based cache layer and a shared HDFS-based persistent layer. 
+Hot data is first written to the cache layer and later moved to the persistent layer through table services like Compaction and Clustering. 
+This setup addresses the strong demands for performance and stability in scenarios involving massive data ingestion into the lake. 
+It is important to note that once data is written to the cache layer and committed, it becomes visible to downstream processes, regardless of when 
+the "relocation work" starts or finishes. Additionally, since the data relocation from the cache layer to the persistent layer leverages the lake table's 
+own Compaction and Clustering capabilities, this process adheres to the lake table's commit mechanism and MVCC snapshot isolation design. Therefore, 
+after enabling the cache layer, the lake table maintains atomicity, transactional guarantees, and Exactly Once semantics.
+
+Below is a comparison between the normal read/write process of a lake table and the process after enabling the lake table cache layer. 
+Green arrows indicate data reads, and red arrows indicate data writes. Before enabling the cache layer, the compute engine directly writes 
+data to the shared HDFS and commits, including Parquet base files, log files, and metadata files. The compute engine queries the lake table 
+by directly reading data from the shared HDFS. Additionally, the lake table services for Clustering and Compaction also directly query data from 
+the shared HDFS, process it, and write it back to the shared HDFS. After enabling the lake table cache layer, the compute engine first writes hot 
+data to the high-performance HDFS and commits, including Parquet files and log files. During queries, a unified logical view of both the cache layer 
+and the persistent layer is constructed to meet query demands. The Clustering and Compaction table services, while performing regular lake table file 
+organization, also facilitate data relocation from the cache layer to the persistent layer. Notably, regardless of when Clustering and Compaction jobs 
+start or finish, the data visible to downstream processes is always complete and timely.
+
+Original Read/Write workflow
+![originalReadWrite.png](originalReadWrite.png)
+
+Read/Write workflow with hudi cache layer enabled
+![ReadWriteWithCacheLayer.png](ReadWriteWithCacheLayer.png)
+
+#### HoodieCacheLayerStorageStrategy
+Based on the HoodieActiveTimeline and the current write instant, determine the specific write path. For common commit operations 
+in COW (Copy on Write) tables and delta commit operations in MOR (Merge on Read) tables, we generate a CacheLayer-related Storage 
+Path to write/read. This type of I/O is targeted at the cache layer.
+
+As for commit action in mor table and replace commit in cow table, we will generate a persistent Storage Path which will let 
+Compaction/Clustering do the data migration works from cache layer to persistent layer
+
+Note: It is required that Clustering is enabled for COW tables and Compaction is enabled for MOR tables; otherwise, there is a 
+risk of storage overflow in the cache layer.
+
+```java
+/**
+ * When using Storage Cache Layer make sure that table service is enabled :
+ * 1. MOR + Upsert + Compaction
+ * 2. COW + Insert + Clustering
+ */
+public class HoodieCacheLayerStorageStrategy extends HoodieDefaultStorageStrategy {

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-3625][RFC-60] Update RFC-60 With Federated Storage Layout [hudi]

Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on PR #12407:
URL: https://github.com/apache/hudi/pull/12407#issuecomment-2519056459

   > Hi @zhangyue19921010 , thanks for picking this up! The update is informative amd the cache storage separation is very insteresting and it's obviously performant as well.
   > 
   > I've left some comments, and below are some high-level suggestions:
   > 
   > * We can omit the implementation details and focus on the `StorageStrategy` abstraction in this RFC
   > * Please also update the case for `Object Storage Strategy`, I'm afraid some of the assumptions we made back then may not hold true anymore and will need an update
   > * I remember we discussed offline that we want to keep a map in `HoodieStorage` to track storage locations on different storages, but I don't see it in this PR. Is that merged into the idea of storing prefixes in metadata table?
   
   Comments are all addressed. PTAL!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-3625][RFC-60] Update RFC-60 With Federated Storage Layout [hudi]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on code in PR #12407:
URL: https://github.com/apache/hudi/pull/12407#discussion_r1881342233


##########
rfc/rfc-60/rfc-60.md:
##########
@@ -84,23 +119,85 @@ public interface HoodieStorageStrategy extends Serializable {
   /**
    * Return a storage location for the given filename.
    *
-   * @param fileId data file ID
+   * @param path fileName
    * @return a storage location string for a data file
    */
-  String storageLocation(String fileId);
+  StoragePath storageLocation(String path, String instantTime);
 
   /**
-   * Return a storage location for the given partition and filename.
+   * Return all possible StoragePaths
    *
-   * @param partitionPath partition path for the file
-   * @param fileId data file ID
-   * @return a storage location string for a data file
+   * @param partitionPath
+   * @param checkExist check if StoragePath is truly existed or not. 
+   * @return a st of storage partition path
+   */
+  Set<StoragePath> getAllLocations(String partitionPath, boolean checkExist);

Review Comment:
   This API was changed in #12460, could you keep this updated as well?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-3625][RFC-60] Update RFC-60 With Federated Storage Layout [hudi]

Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #12407:
URL: https://github.com/apache/hudi/pull/12407#discussion_r1870573339


##########
rfc/rfc-60/rfc-60.md:
##########
@@ -178,6 +275,130 @@ The hashing function should be made user configurable for use cases like bucketi
 sub-partitioning/re-hash to reduce the number of hash prefixes. Having too many unique hash prefixes
 would make files too dispersed, and affect performance on other operations such as listing.
 
+### Case2: Hudi Storage Cache Layer
+
+The cache layer of a lake table is a specific implementation scenario for Federated Storage Layout in hudi tables. 
+It divides the physical storage of the lake table into a high-performance HDFS-based cache layer and a shared HDFS-based persistent layer. 
+Hot data is first written to the cache layer and later moved to the persistent layer through table services like Compaction and Clustering. 
+This setup addresses the strong demands for performance and stability in scenarios involving massive data ingestion into the lake. 
+It is important to note that once data is written to the cache layer and committed, it becomes visible to downstream processes, regardless of when 
+the "relocation work" starts or finishes. Additionally, since the data relocation from the cache layer to the persistent layer leverages the lake table's 
+own Compaction and Clustering capabilities, this process adheres to the lake table's commit mechanism and MVCC snapshot isolation design. Therefore, 
+after enabling the cache layer, the lake table maintains atomicity, transactional guarantees, and Exactly Once semantics.
+
+Below is a comparison between the normal read/write process of a lake table and the process after enabling the lake table cache layer. 
+Green arrows indicate data reads, and red arrows indicate data writes. Before enabling the cache layer, the compute engine directly writes 
+data to the shared HDFS and commits, including Parquet base files, log files, and metadata files. The compute engine queries the lake table 
+by directly reading data from the shared HDFS. Additionally, the lake table services for Clustering and Compaction also directly query data from 
+the shared HDFS, process it, and write it back to the shared HDFS. After enabling the lake table cache layer, the compute engine first writes hot 
+data to the high-performance HDFS and commits, including Parquet files and log files. During queries, a unified logical view of both the cache layer 
+and the persistent layer is constructed to meet query demands. The Clustering and Compaction table services, while performing regular lake table file 
+organization, also facilitate data relocation from the cache layer to the persistent layer. Notably, regardless of when Clustering and Compaction jobs 
+start or finish, the data visible to downstream processes is always complete and timely.
+
+Original Read/Write workflow
+![originalReadWrite.png](originalReadWrite.png)
+
+Read/Write workflow with hudi cache layer enabled
+![ReadWriteWithCacheLayer.png](ReadWriteWithCacheLayer.png)
+
+#### HoodieCacheLayerStorageStrategy
+Based on the HoodieActiveTimeline and the current write instant, determine the specific write path. For common commit operations 
+in COW (Copy on Write) tables and delta commit operations in MOR (Merge on Read) tables, we generate a CacheLayer-related Storage 
+Path to write/read. This type of I/O is targeted at the cache layer.
+
+As for commit action in mor table and replace commit in cow table, we will generate a persistent Storage Path which will let 
+Compaction/Clustering do the data migration works from cache layer to persistent layer
+
+Note: It is required that Clustering is enabled for COW tables and Compaction is enabled for MOR tables; otherwise, there is a 
+risk of storage overflow in the cache layer.
+
+```java
+/**
+ * When using Storage Cache Layer make sure that table service is enabled :
+ * 1. MOR + Upsert + Compaction
+ * 2. COW + Insert + Clustering
+ */
+public class HoodieCacheLayerStorageStrategy extends HoodieDefaultStorageStrategy {
+
+  private HoodieTableType tableType;
+  private String hoodieStorageStrategyModifyTime;
+
+  /**
+   * Only support on Storage Path as cache layer
+   * @param basePath
+   * @param storagePath
+   * @param metaClient
+   */
+  public HoodieCacheLayerStorageStrategy(String basePath,
+                                         String storagePath,
+                                         Option<HoodieTableMetaClient> metaClient) {
+    // init
+  }
+
+  /**
+   * Generate StoragePath based on active instant time
+   * for common write instant :
+   * 1. commit for cow table 
+   * 2. delta commit for mor table
+   * We will generate a CacheLayer related Storage Path to write/read
+   * 
+   * As for commit action in mor table and replace commit in cow table,
+   * we will generate a persistent Storage Path which will let Compaction/Clustering
+   * do the data migration works from cache layer to persistent layer
+   */
+  @Override
+  public StoragePath storageLocation(String path, String instantTime) {
+    if (isCommonCommit(instantTime)) {
+      return FSUtils.getPartitionPath(storagePaths.get(0), path);

Review Comment:
   As we discuss, we only focus on abstraction, so removed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-3625][RFC-60] Update RFC-60 With Federated Storage Layout [hudi]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on code in PR #12407:
URL: https://github.com/apache/hudi/pull/12407#discussion_r1868589645


##########
rfc/rfc-60/rfc-60.md:
##########
@@ -84,23 +119,85 @@ public interface HoodieStorageStrategy extends Serializable {
   /**
    * Return a storage location for the given filename.
    *
-   * @param fileId data file ID
+   * @param path fileName

Review Comment:
   Is this a path or just the file name? Would appreciate if we can add an example of the exact string that gets passed in here.
   
   Also could you help me understand why `instantTime` is needed in this API by adding it to the comment?



##########
rfc/rfc-60/rfc-60.md:
##########
@@ -84,23 +119,85 @@ public interface HoodieStorageStrategy extends Serializable {
   /**
    * Return a storage location for the given filename.
    *
-   * @param fileId data file ID
+   * @param path fileName
    * @return a storage location string for a data file
    */
-  String storageLocation(String fileId);
+  StoragePath storageLocation(String path, String instantTime);
 
   /**
-   * Return a storage location for the given partition and filename.
+   * Return all possible StoragePaths
    *
-   * @param partitionPath partition path for the file
-   * @param fileId data file ID
-   * @return a storage location string for a data file
+   * @param partitionPath
+   * @param checkExist check if StoragePath is truly existed or not. 
+   * @return a st of storage partition path
+   */
+  Set<StoragePath> getAllLocations(String partitionPath, boolean checkExist);

Review Comment:
   Where can this API be useful? 
   
   I think we need to think twice about it. Adding this API means Hudi should be able to infer relative partition locations solely based on partition path. And for `Object Store Strategy` this means all the data under the same partition would be stored under the same physical location again, because we can only calculate the hash based on two arguments that got passed in.



##########
rfc/rfc-60/rfc-60.md:
##########
@@ -198,46 +419,68 @@ for metadata table to be populated.
 
 4. If there is an error reading from Metadata table, we will not fall back listing from file system.
 
+After enabling the Federated Storage Layout feature, under certain strategies such as the "data cache layer," 
+data from different lake tables may be stored on different physical media, resulting in different schemes. 
+For example, cache layer data may be stored on hdfs://ns1/, while persistent layer data is stored on hdfs://ns2/. 
+In this case, we need to add a new field named "scheme" in MDT HoodieMetadataFileInfo to store the scheme information for different files, 
+which will be used for path restoration.
+
+```avro schema
+        {
+            "doc": "Contains information about partitions and files within the dataset",
+            "name": "filesystemMetadata",
+            "type": [
+                "null",
+                {
+                    "type": "map",
+                    "values": {
+                        "type": "record",
+                        "name": "HoodieMetadataFileInfo",
+                        "fields": [
+                            {
+                                "name": "size",
+                                "type": "long",
+                                "doc": "Size of the file"
+                            },
+                            {
+                                "name": "isDeleted",
+                                "type": "boolean",
+                                "doc": "True if this file has been deleted"
+                            },
+                            {
+                                "name":"scheme",

Review Comment:
   We should use `prefix` instead to avoid confusion. The "scheme" here can also contain the root path/authority



##########
rfc/rfc-60/rfc-60.md:
##########
@@ -178,6 +275,130 @@ The hashing function should be made user configurable for use cases like bucketi
 sub-partitioning/re-hash to reduce the number of hash prefixes. Having too many unique hash prefixes
 would make files too dispersed, and affect performance on other operations such as listing.
 
+### Case2: Hudi Storage Cache Layer
+
+The cache layer of a lake table is a specific implementation scenario for Federated Storage Layout in hudi tables. 
+It divides the physical storage of the lake table into a high-performance HDFS-based cache layer and a shared HDFS-based persistent layer. 
+Hot data is first written to the cache layer and later moved to the persistent layer through table services like Compaction and Clustering. 
+This setup addresses the strong demands for performance and stability in scenarios involving massive data ingestion into the lake. 
+It is important to note that once data is written to the cache layer and committed, it becomes visible to downstream processes, regardless of when 
+the "relocation work" starts or finishes. Additionally, since the data relocation from the cache layer to the persistent layer leverages the lake table's 
+own Compaction and Clustering capabilities, this process adheres to the lake table's commit mechanism and MVCC snapshot isolation design. Therefore, 
+after enabling the cache layer, the lake table maintains atomicity, transactional guarantees, and Exactly Once semantics.
+
+Below is a comparison between the normal read/write process of a lake table and the process after enabling the lake table cache layer. 
+Green arrows indicate data reads, and red arrows indicate data writes. Before enabling the cache layer, the compute engine directly writes 
+data to the shared HDFS and commits, including Parquet base files, log files, and metadata files. The compute engine queries the lake table 
+by directly reading data from the shared HDFS. Additionally, the lake table services for Clustering and Compaction also directly query data from 
+the shared HDFS, process it, and write it back to the shared HDFS. After enabling the lake table cache layer, the compute engine first writes hot 
+data to the high-performance HDFS and commits, including Parquet files and log files. During queries, a unified logical view of both the cache layer 
+and the persistent layer is constructed to meet query demands. The Clustering and Compaction table services, while performing regular lake table file 
+organization, also facilitate data relocation from the cache layer to the persistent layer. Notably, regardless of when Clustering and Compaction jobs 
+start or finish, the data visible to downstream processes is always complete and timely.
+
+Original Read/Write workflow
+![originalReadWrite.png](originalReadWrite.png)
+
+Read/Write workflow with hudi cache layer enabled
+![ReadWriteWithCacheLayer.png](ReadWriteWithCacheLayer.png)
+
+#### HoodieCacheLayerStorageStrategy
+Based on the HoodieActiveTimeline and the current write instant, determine the specific write path. For common commit operations 
+in COW (Copy on Write) tables and delta commit operations in MOR (Merge on Read) tables, we generate a CacheLayer-related Storage 
+Path to write/read. This type of I/O is targeted at the cache layer.
+
+As for commit action in mor table and replace commit in cow table, we will generate a persistent Storage Path which will let 
+Compaction/Clustering do the data migration works from cache layer to persistent layer
+
+Note: It is required that Clustering is enabled for COW tables and Compaction is enabled for MOR tables; otherwise, there is a 
+risk of storage overflow in the cache layer.
+
+```java
+/**
+ * When using Storage Cache Layer make sure that table service is enabled :
+ * 1. MOR + Upsert + Compaction
+ * 2. COW + Insert + Clustering
+ */
+public class HoodieCacheLayerStorageStrategy extends HoodieDefaultStorageStrategy {

Review Comment:
   We should omit the implementation details to focus on discussing the abstraction API in this RFC



##########
rfc/rfc-60/rfc-60.md:
##########
@@ -198,46 +419,68 @@ for metadata table to be populated.
 
 4. If there is an error reading from Metadata table, we will not fall back listing from file system.
 
+After enabling the Federated Storage Layout feature, under certain strategies such as the "data cache layer," 
+data from different lake tables may be stored on different physical media, resulting in different schemes. 
+For example, cache layer data may be stored on hdfs://ns1/, while persistent layer data is stored on hdfs://ns2/. 
+In this case, we need to add a new field named "scheme" in MDT HoodieMetadataFileInfo to store the scheme information for different files, 
+which will be used for path restoration.
+
+```avro schema
+        {
+            "doc": "Contains information about partitions and files within the dataset",
+            "name": "filesystemMetadata",
+            "type": [
+                "null",
+                {
+                    "type": "map",
+                    "values": {
+                        "type": "record",
+                        "name": "HoodieMetadataFileInfo",
+                        "fields": [
+                            {
+                                "name": "size",
+                                "type": "long",
+                                "doc": "Size of the file"
+                            },
+                            {
+                                "name": "isDeleted",
+                                "type": "boolean",
+                                "doc": "True if this file has been deleted"
+                            },
+                            {
+                                "name":"scheme",
+                                "type": ["null","string"],
+                                "default":null
+                            }
+                        ]
+                    }
+                }
+            ]
+        }
+```
+
+Note: For lake tables that do not have the Federated Storage Layout enabled, the value of this "scheme" field will be null.
+
 ### Integration
 This section mainly describes how storage strategy is integrated with other components and how read/write
-would look like from Hudi side with object storage layout.
-
-We propose integrating the storage strategy at the filesystem level, specifically within `HoodieWrapperFileSystem`. 
-This way, only file read/write operations undergo path conversion and we can limit the usage of 
-storage strategy to only filesystem level so other upper-level components don't need to be aware of physical paths.
-
-This also mandates that `HoodieWrapperFileSystem` is the filesystem of choice for all upper-level Hudi components.
-Getting filesystem from `Path` or such won't be allowed anymore as using raw filesystem may not reach 
-to physical locations without storage strategy. Hudi components can simply call `HoodieMetaClient#getFs` 
-to get `HoodieWrapperFileSystem`, and this needs to be the only allowed way for any filesystem-related operation. 
-The only exception is when we need to interact with metadata that's still stored under the original table path, 
-and we should call `HoodieMetaClient#getRawFs` in this case so `HoodieMetaClient` can still be the single entry
-for getting filesystem.
-
-![](wrapper_fs.png)
-
-When conducting a read operation, Hudi would: 
-1. Access filesystem view, `HoodieMetadataFileSystemView` specifically
-2. Scan metadata table via filesystem view to compose `HoodieMetadataPayload`
-3. Call `HoodieMetadataPayload#getFileStatuses` and employ `HoodieWrapperFileSystem` to get 
-file statuses with physical locations
-
-This flow can be concluded in the chart below.
-
-![](read_flow.png)
-
-#### Considerations
-- Path conversion happens on the fly when reading/writing files. This saves Hudi from storing physical locations,
-and adds the cost of hashing, but the performance burden should be negligible.
-- Since table path and data path will most likely have different top-level folders/authorities,
-`HoodieWrapperFileSystem` should maintain at least two `FileSystem` objects: one to access table path and another
-to access storage path. `HoodieWrapperFileSystem` should intelligently tell if it needs
-to convert the path by checking the path on the fly.
-- When using Hudi file reader/writer implementation, we will need to pass `HoodieWrapperFileSystem` down
-to parent reader. For instance, when using `HoodieAvroHFileReader`, we will need to pass `HoodieWrapperFileSystem`
-to `HFile.Reader` so it can have access to storage strategy. If reader/writer doesn't take filesystem
-directly (e.g. `ParquetFileReader` only takes `Configuration` and `Path` for reading), then we will
-need to register `HoodieWrapperFileSystem` to `Configuration` so it can be initialized/used later.
+would look like from Hudi side with Federated Storage Layout.
+
+We already have the abstractions HoodieStorage, StoragePath, and HoodieWrapperFileSystem. Here, we need to add a 

Review Comment:
   It would be cleaner if we could not use `HoodieWrapperFileSystem` at all and concentrate all the logic to `HoodieStorage` or `StorageStrategy`



##########
rfc/rfc-60/rfc-60.md:
##########
@@ -18,33 +18,68 @@
 # RFC-60: Federated Storage Layout
 
 ## Proposers
+- @zhangyue19921010
+- @CTTY
 - @umehrot2
 
 ## Approvers
 - @vinoth
 - @shivnarayan
+- @yihua
 
 ## Status
 
 JIRA: [https://issues.apache.org/jira/browse/HUDI-3625](https://issues.apache.org/jira/browse/HUDI-3625)
 
 ## Abstract
+In this RFC, we will support the Federated Storage Layout for Hudi tables, enabling Hudi to support multiple pluggable physical 
+storage systems. By combining Hudi's own metadata, we can construct logical table views and expose these views externally,
+making them transparent to the engine.
 
+![](FederatedStorageLayout.png)
+
+After implementing the Hudi Federated Storage Layout, we can develop many interesting new features for Hudi lake tables based on this, such as:
+
+### Object Store Optimized Layout
 As you scale your Apache Hudi workloads over cloud object stores like Amazon S3, there is potential of hitting request
 throttling limits which in-turn impacts performance. In this RFC, we are proposing to support an alternate storage
 layout that is optimized for Amazon S3 and other cloud object stores, which helps achieve maximum throughput and
 significantly reduce throttling.
 
+![](ObjectStoreOptimizedLayout.png)
+
+### Hudi Storage Cache Layer
+The Hudi lake table data cache layer involves dividing the lake table physical storage into a high-performance HDFS-based data 
+cache layer and a shared HDFS-based data persistence layer. Hot data is initially written to the cache layer and later moved to 
+the persistence layer through table services like Compaction and Clustering. This approach meets the strong demands for performance 
+and stability in scenarios involving massive data ingestion into the lake. It is important to note that once data is written to the 
+cache layer and committed, it becomes visible to downstream consumers, regardless of when the subsequent "moving operations" start 
+or finish, ensuring data visibility and timeliness are unaffected. Additionally, since the data movement from the cache layer to 
+the persistence layer leverages the lake table's own Compaction and Clustering table service capabilities, this process adheres to 
+the lake table's commit mechanism and MVCC snapshot isolation design. Therefore, with the data cache layer enabled, the lake table 
+maintains atomicity, transactional guarantees, and Exactly Once semantics.
+
+![](HudiStorageCacheLayer.png)
+
+### Hudi Table Second-Level Latency
+When data is ingested into the lake, it is first written to data files and metadata, and only becomes visible after the transaction 
+is successfully committed. Since writing data files is relatively slow, it typically involves minute-level commits. Here, the capabilities 
+of the Federated Storage Layout can be extended by writing data to high-speed storage systems like Kafka, HBase, or Redis, and recording 
+metadata such as offsets or keys in the lake table's metadata. During reads, the content from both persistent storage and high-speed storage 
+can be combined, enabling Hudi to achieve second-level latency.
+
 In addition, we are proposing an interface that would allow users to implement their own custom strategy to allow them
 to distribute the data files across cloud stores, hdfs or on prem based on their specific use-cases.
 
+![](HudiTableSecondLevelLatency.png)

Review Comment:
   nit: The image size is much smaller comparing to other images, can we fix this



##########
rfc/rfc-60/rfc-60.md:
##########
@@ -84,23 +119,85 @@ public interface HoodieStorageStrategy extends Serializable {
   /**
    * Return a storage location for the given filename.
    *
-   * @param fileId data file ID
+   * @param path fileName
    * @return a storage location string for a data file
    */
-  String storageLocation(String fileId);
+  StoragePath storageLocation(String path, String instantTime);
 
   /**
-   * Return a storage location for the given partition and filename.
+   * Return all possible StoragePaths
    *
-   * @param partitionPath partition path for the file
-   * @param fileId data file ID
-   * @return a storage location string for a data file
+   * @param partitionPath
+   * @param checkExist check if StoragePath is truly existed or not. 
+   * @return a st of storage partition path
+   */
+  Set<StoragePath> getAllLocations(String partitionPath, boolean checkExist);
+
+  /**
+   * Return RelativePath base on path and locations.
+   *
+   * @param path
+   * @return relative path
+   */
+  String getRelativePath(Path path);
+}
+```
+
+```java
+public class HoodieStorageStrategyFactory {

Review Comment:
   We can omit the factory class in RFC to focus on the abstraction



##########
rfc/rfc-60/rfc-60.md:
##########
@@ -84,23 +119,85 @@ public interface HoodieStorageStrategy extends Serializable {
   /**
    * Return a storage location for the given filename.
    *
-   * @param fileId data file ID
+   * @param path fileName
    * @return a storage location string for a data file
    */
-  String storageLocation(String fileId);
+  StoragePath storageLocation(String path, String instantTime);
 
   /**
-   * Return a storage location for the given partition and filename.
+   * Return all possible StoragePaths
    *
-   * @param partitionPath partition path for the file
-   * @param fileId data file ID
-   * @return a storage location string for a data file
+   * @param partitionPath
+   * @param checkExist check if StoragePath is truly existed or not. 
+   * @return a st of storage partition path
+   */
+  Set<StoragePath> getAllLocations(String partitionPath, boolean checkExist);
+
+  /**
+   * Return RelativePath base on path and locations.
+   *
+   * @param path
+   * @return relative path
+   */
+  String getRelativePath(Path path);
+}
+```
+
+```java
+public class HoodieStorageStrategyFactory {
+  private HoodieStorageStrategyFactory() {
+  }
+
+  public static HoodieStorageStrategy getInstant(HoodieTableMetaClient metaClient, Boolean reset) {

Review Comment:
   nit: Do you mean `getInstance`? Or we can just say `getStorageStrategy`. `reset` can be replaced with `useDefault`



##########
rfc/rfc-60/rfc-60.md:
##########
@@ -18,33 +18,68 @@
 # RFC-60: Federated Storage Layout
 
 ## Proposers
+- @zhangyue19921010
+- @CTTY
 - @umehrot2
 
 ## Approvers
 - @vinoth
 - @shivnarayan
+- @yihua
 
 ## Status
 
 JIRA: [https://issues.apache.org/jira/browse/HUDI-3625](https://issues.apache.org/jira/browse/HUDI-3625)
 
 ## Abstract
+In this RFC, we will support the Federated Storage Layout for Hudi tables, enabling Hudi to support multiple pluggable physical 
+storage systems. By combining Hudi's own metadata, we can construct logical table views and expose these views externally,
+making them transparent to the engine.
 
+![](FederatedStorageLayout.png)
+
+After implementing the Hudi Federated Storage Layout, we can develop many interesting new features for Hudi lake tables based on this, such as:
+
+### Object Store Optimized Layout
 As you scale your Apache Hudi workloads over cloud object stores like Amazon S3, there is potential of hitting request
 throttling limits which in-turn impacts performance. In this RFC, we are proposing to support an alternate storage
 layout that is optimized for Amazon S3 and other cloud object stores, which helps achieve maximum throughput and
 significantly reduce throttling.
 
+![](ObjectStoreOptimizedLayout.png)
+
+### Hudi Storage Cache Layer
+The Hudi lake table data cache layer involves dividing the lake table physical storage into a high-performance HDFS-based data 
+cache layer and a shared HDFS-based data persistence layer. Hot data is initially written to the cache layer and later moved to 
+the persistence layer through table services like Compaction and Clustering. This approach meets the strong demands for performance 
+and stability in scenarios involving massive data ingestion into the lake. It is important to note that once data is written to the 
+cache layer and committed, it becomes visible to downstream consumers, regardless of when the subsequent "moving operations" start 
+or finish, ensuring data visibility and timeliness are unaffected. Additionally, since the data movement from the cache layer to 
+the persistence layer leverages the lake table's own Compaction and Clustering table service capabilities, this process adheres to 
+the lake table's commit mechanism and MVCC snapshot isolation design. Therefore, with the data cache layer enabled, the lake table 
+maintains atomicity, transactional guarantees, and Exactly Once semantics.
+
+![](HudiStorageCacheLayer.png)
+
+### Hudi Table Second-Level Latency

Review Comment:
   CMIIW: I think the low-level logic of this case is very similar to the cache layer, which is splitting storage into a cache level and a persistent level. The only difference is that in the third case, cache layer's data would be visible to consumers only after being persisted. 
   
   If so, can we combining these two cases into one by abstracting them more?



##########
rfc/rfc-60/rfc-60.md:
##########
@@ -84,23 +119,85 @@ public interface HoodieStorageStrategy extends Serializable {
   /**
    * Return a storage location for the given filename.
    *
-   * @param fileId data file ID
+   * @param path fileName
    * @return a storage location string for a data file
    */
-  String storageLocation(String fileId);
+  StoragePath storageLocation(String path, String instantTime);
 
   /**
-   * Return a storage location for the given partition and filename.
+   * Return all possible StoragePaths
    *
-   * @param partitionPath partition path for the file
-   * @param fileId data file ID
-   * @return a storage location string for a data file
+   * @param partitionPath
+   * @param checkExist check if StoragePath is truly existed or not. 
+   * @return a st of storage partition path
+   */
+  Set<StoragePath> getAllLocations(String partitionPath, boolean checkExist);
+
+  /**
+   * Return RelativePath base on path and locations.
+   *
+   * @param path
+   * @return relative path
+   */
+  String getRelativePath(Path path);

Review Comment:
   Change to `StoragePath`
   
   Is the `path` here the same as the `path` in `storageLocation`?



##########
rfc/rfc-60/rfc-60.md:
##########
@@ -178,6 +275,130 @@ The hashing function should be made user configurable for use cases like bucketi
 sub-partitioning/re-hash to reduce the number of hash prefixes. Having too many unique hash prefixes
 would make files too dispersed, and affect performance on other operations such as listing.
 
+### Case2: Hudi Storage Cache Layer
+
+The cache layer of a lake table is a specific implementation scenario for Federated Storage Layout in hudi tables. 
+It divides the physical storage of the lake table into a high-performance HDFS-based cache layer and a shared HDFS-based persistent layer. 
+Hot data is first written to the cache layer and later moved to the persistent layer through table services like Compaction and Clustering. 
+This setup addresses the strong demands for performance and stability in scenarios involving massive data ingestion into the lake. 
+It is important to note that once data is written to the cache layer and committed, it becomes visible to downstream processes, regardless of when 
+the "relocation work" starts or finishes. Additionally, since the data relocation from the cache layer to the persistent layer leverages the lake table's 
+own Compaction and Clustering capabilities, this process adheres to the lake table's commit mechanism and MVCC snapshot isolation design. Therefore, 
+after enabling the cache layer, the lake table maintains atomicity, transactional guarantees, and Exactly Once semantics.
+
+Below is a comparison between the normal read/write process of a lake table and the process after enabling the lake table cache layer. 
+Green arrows indicate data reads, and red arrows indicate data writes. Before enabling the cache layer, the compute engine directly writes 
+data to the shared HDFS and commits, including Parquet base files, log files, and metadata files. The compute engine queries the lake table 
+by directly reading data from the shared HDFS. Additionally, the lake table services for Clustering and Compaction also directly query data from 
+the shared HDFS, process it, and write it back to the shared HDFS. After enabling the lake table cache layer, the compute engine first writes hot 
+data to the high-performance HDFS and commits, including Parquet files and log files. During queries, a unified logical view of both the cache layer 
+and the persistent layer is constructed to meet query demands. The Clustering and Compaction table services, while performing regular lake table file 
+organization, also facilitate data relocation from the cache layer to the persistent layer. Notably, regardless of when Clustering and Compaction jobs 
+start or finish, the data visible to downstream processes is always complete and timely.
+
+Original Read/Write workflow
+![originalReadWrite.png](originalReadWrite.png)
+
+Read/Write workflow with hudi cache layer enabled
+![ReadWriteWithCacheLayer.png](ReadWriteWithCacheLayer.png)
+
+#### HoodieCacheLayerStorageStrategy
+Based on the HoodieActiveTimeline and the current write instant, determine the specific write path. For common commit operations 
+in COW (Copy on Write) tables and delta commit operations in MOR (Merge on Read) tables, we generate a CacheLayer-related Storage 
+Path to write/read. This type of I/O is targeted at the cache layer.
+
+As for commit action in mor table and replace commit in cow table, we will generate a persistent Storage Path which will let 
+Compaction/Clustering do the data migration works from cache layer to persistent layer
+
+Note: It is required that Clustering is enabled for COW tables and Compaction is enabled for MOR tables; otherwise, there is a 
+risk of storage overflow in the cache layer.
+
+```java
+/**
+ * When using Storage Cache Layer make sure that table service is enabled :
+ * 1. MOR + Upsert + Compaction
+ * 2. COW + Insert + Clustering
+ */
+public class HoodieCacheLayerStorageStrategy extends HoodieDefaultStorageStrategy {
+
+  private HoodieTableType tableType;
+  private String hoodieStorageStrategyModifyTime;

Review Comment:
   Is this going to be updated often?



##########
rfc/rfc-60/rfc-60.md:
##########
@@ -84,23 +119,85 @@ public interface HoodieStorageStrategy extends Serializable {
   /**
    * Return a storage location for the given filename.
    *
-   * @param fileId data file ID
+   * @param path fileName
    * @return a storage location string for a data file
    */
-  String storageLocation(String fileId);
+  StoragePath storageLocation(String path, String instantTime);
 
   /**
-   * Return a storage location for the given partition and filename.
+   * Return all possible StoragePaths
    *
-   * @param partitionPath partition path for the file
-   * @param fileId data file ID
-   * @return a storage location string for a data file
+   * @param partitionPath
+   * @param checkExist check if StoragePath is truly existed or not. 
+   * @return a st of storage partition path
+   */
+  Set<StoragePath> getAllLocations(String partitionPath, boolean checkExist);
+
+  /**
+   * Return RelativePath base on path and locations.
+   *
+   * @param path
+   * @return relative path
+   */
+  String getRelativePath(Path path);
+}
+```
+
+```java
+public class HoodieStorageStrategyFactory {
+  private HoodieStorageStrategyFactory() {
+  }
+
+  public static HoodieStorageStrategy getInstant(HoodieTableMetaClient metaClient, Boolean reset) {
+    HoodieTableConfig config = metaClient.getTableConfig();
+    if (reset) {
+      return getInstant(HoodieStorageStrategyType.DEFAULT.value, metaClient.getBasePath(), config.getStoragePath(), metaClient);
+    }
+    return getInstant(config.getStorageStrategy(), metaClient.getBasePath(), config.getStoragePath(), metaClient);
+  }
+
+  public static HoodieStorageStrategy getInstant(HoodieTableMetaClient metaClient) {
+    return getInstant(metaClient, false);
+  }
+
+  /**
+   *  Just for HoodieParquetInputFormatBase
+   * @param config
+   * @param basePath
+   * @return
    */
-  String storageLocation(String partitionPath, String fileId);
+  public static HoodieStorageStrategy getInstant(HoodieTableConfig config, String basePath, Boolean reset) {
+    if (reset) {
+      return getInstant(HoodieStorageStrategyType.DEFAULT.value, basePath, config.getStoragePath(), null);
+    }
+    return getInstant(config.getStorageStrategy(), basePath, config.getStoragePath(), null);
+  }
+
+  private static HoodieStorageStrategy getInstant(
+      String storageStrategyClass,
+      String basePath,
+      String storagePath, HoodieTableMetaClient metaClient) {
+    return (HoodieStorageStrategy) ReflectionUtils.loadClass(storageStrategyClass,
+        basePath, storagePath, metaClient == null ? Option.empty() : Option.of(metaClient));
+  }
 }
 ```
 
-### Generating File Paths for Object Store Optimized Layout
+```java
+public enum HoodieStorageStrategyType {

Review Comment:
   Same here, we can omit this class to focus on the API abstraction



##########
rfc/rfc-60/rfc-60.md:
##########
@@ -84,23 +119,85 @@ public interface HoodieStorageStrategy extends Serializable {
   /**
    * Return a storage location for the given filename.
    *
-   * @param fileId data file ID
+   * @param path fileName
    * @return a storage location string for a data file
    */
-  String storageLocation(String fileId);
+  StoragePath storageLocation(String path, String instantTime);
 
   /**
-   * Return a storage location for the given partition and filename.
+   * Return all possible StoragePaths
    *
-   * @param partitionPath partition path for the file
-   * @param fileId data file ID
-   * @return a storage location string for a data file
+   * @param partitionPath
+   * @param checkExist check if StoragePath is truly existed or not. 
+   * @return a st of storage partition path
+   */
+  Set<StoragePath> getAllLocations(String partitionPath, boolean checkExist);
+
+  /**
+   * Return RelativePath base on path and locations.
+   *
+   * @param path
+   * @return relative path
+   */
+  String getRelativePath(Path path);
+}
+```
+
+```java
+public class HoodieStorageStrategyFactory {
+  private HoodieStorageStrategyFactory() {
+  }
+
+  public static HoodieStorageStrategy getInstant(HoodieTableMetaClient metaClient, Boolean reset) {
+    HoodieTableConfig config = metaClient.getTableConfig();
+    if (reset) {
+      return getInstant(HoodieStorageStrategyType.DEFAULT.value, metaClient.getBasePath(), config.getStoragePath(), metaClient);
+    }
+    return getInstant(config.getStorageStrategy(), metaClient.getBasePath(), config.getStoragePath(), metaClient);
+  }
+
+  public static HoodieStorageStrategy getInstant(HoodieTableMetaClient metaClient) {
+    return getInstant(metaClient, false);
+  }
+
+  /**
+   *  Just for HoodieParquetInputFormatBase
+   * @param config
+   * @param basePath
+   * @return
    */
-  String storageLocation(String partitionPath, String fileId);
+  public static HoodieStorageStrategy getInstant(HoodieTableConfig config, String basePath, Boolean reset) {
+    if (reset) {
+      return getInstant(HoodieStorageStrategyType.DEFAULT.value, basePath, config.getStoragePath(), null);
+    }
+    return getInstant(config.getStorageStrategy(), basePath, config.getStoragePath(), null);
+  }
+
+  private static HoodieStorageStrategy getInstant(
+      String storageStrategyClass,
+      String basePath,
+      String storagePath, HoodieTableMetaClient metaClient) {
+    return (HoodieStorageStrategy) ReflectionUtils.loadClass(storageStrategyClass,
+        basePath, storagePath, metaClient == null ? Option.empty() : Option.of(metaClient));
+  }
 }
 ```
 
-### Generating File Paths for Object Store Optimized Layout
+```java
+public enum HoodieStorageStrategyType {
+  DEFAULT(HoodieDefaultStorageStrategy.class.getName()),
+  CACHE_LAYER(HoodieCacheLayerStorageStrategy.class.getName()),
+  OBJECT_STORAGE_STRATEGY(ObjectStorageStrategy.class.getName());
+
+  public final String value;
+
+  HoodieStorageStrategyType(String strategy) {
+    this.value = strategy;
+  }
+}
+```
+
+### Case1: Generating File Paths for Object Store Optimized Layout

Review Comment:
   Could you help update the `Case 1` section as well? I think some of the assumptions we made originally may not hold true anymore



##########
rfc/rfc-60/rfc-60.md:
##########
@@ -178,6 +275,130 @@ The hashing function should be made user configurable for use cases like bucketi
 sub-partitioning/re-hash to reduce the number of hash prefixes. Having too many unique hash prefixes
 would make files too dispersed, and affect performance on other operations such as listing.
 
+### Case2: Hudi Storage Cache Layer
+
+The cache layer of a lake table is a specific implementation scenario for Federated Storage Layout in hudi tables. 
+It divides the physical storage of the lake table into a high-performance HDFS-based cache layer and a shared HDFS-based persistent layer. 
+Hot data is first written to the cache layer and later moved to the persistent layer through table services like Compaction and Clustering. 
+This setup addresses the strong demands for performance and stability in scenarios involving massive data ingestion into the lake. 
+It is important to note that once data is written to the cache layer and committed, it becomes visible to downstream processes, regardless of when 
+the "relocation work" starts or finishes. Additionally, since the data relocation from the cache layer to the persistent layer leverages the lake table's 
+own Compaction and Clustering capabilities, this process adheres to the lake table's commit mechanism and MVCC snapshot isolation design. Therefore, 
+after enabling the cache layer, the lake table maintains atomicity, transactional guarantees, and Exactly Once semantics.
+
+Below is a comparison between the normal read/write process of a lake table and the process after enabling the lake table cache layer. 
+Green arrows indicate data reads, and red arrows indicate data writes. Before enabling the cache layer, the compute engine directly writes 
+data to the shared HDFS and commits, including Parquet base files, log files, and metadata files. The compute engine queries the lake table 
+by directly reading data from the shared HDFS. Additionally, the lake table services for Clustering and Compaction also directly query data from 
+the shared HDFS, process it, and write it back to the shared HDFS. After enabling the lake table cache layer, the compute engine first writes hot 
+data to the high-performance HDFS and commits, including Parquet files and log files. During queries, a unified logical view of both the cache layer 
+and the persistent layer is constructed to meet query demands. The Clustering and Compaction table services, while performing regular lake table file 
+organization, also facilitate data relocation from the cache layer to the persistent layer. Notably, regardless of when Clustering and Compaction jobs 
+start or finish, the data visible to downstream processes is always complete and timely.
+
+Original Read/Write workflow
+![originalReadWrite.png](originalReadWrite.png)
+
+Read/Write workflow with hudi cache layer enabled
+![ReadWriteWithCacheLayer.png](ReadWriteWithCacheLayer.png)
+
+#### HoodieCacheLayerStorageStrategy
+Based on the HoodieActiveTimeline and the current write instant, determine the specific write path. For common commit operations 
+in COW (Copy on Write) tables and delta commit operations in MOR (Merge on Read) tables, we generate a CacheLayer-related Storage 
+Path to write/read. This type of I/O is targeted at the cache layer.
+
+As for commit action in mor table and replace commit in cow table, we will generate a persistent Storage Path which will let 
+Compaction/Clustering do the data migration works from cache layer to persistent layer
+
+Note: It is required that Clustering is enabled for COW tables and Compaction is enabled for MOR tables; otherwise, there is a 
+risk of storage overflow in the cache layer.
+
+```java
+/**
+ * When using Storage Cache Layer make sure that table service is enabled :
+ * 1. MOR + Upsert + Compaction
+ * 2. COW + Insert + Clustering
+ */
+public class HoodieCacheLayerStorageStrategy extends HoodieDefaultStorageStrategy {
+
+  private HoodieTableType tableType;
+  private String hoodieStorageStrategyModifyTime;
+
+  /**
+   * Only support on Storage Path as cache layer
+   * @param basePath
+   * @param storagePath
+   * @param metaClient
+   */
+  public HoodieCacheLayerStorageStrategy(String basePath,
+                                         String storagePath,
+                                         Option<HoodieTableMetaClient> metaClient) {
+    // init
+  }
+
+  /**
+   * Generate StoragePath based on active instant time
+   * for common write instant :
+   * 1. commit for cow table 
+   * 2. delta commit for mor table
+   * We will generate a CacheLayer related Storage Path to write/read
+   * 
+   * As for commit action in mor table and replace commit in cow table,
+   * we will generate a persistent Storage Path which will let Compaction/Clustering
+   * do the data migration works from cache layer to persistent layer
+   */
+  @Override
+  public StoragePath storageLocation(String path, String instantTime) {
+    if (isCommonCommit(instantTime)) {
+      return FSUtils.getPartitionPath(storagePaths.get(0), path);

Review Comment:
   I'm assuming `storagePaths` here is populated by `getAllLocations`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-3625][RFC-60] Update RFC-60 With Federated Storage Layout [hudi]

Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #12407:
URL: https://github.com/apache/hudi/pull/12407#discussion_r1868853356


##########
rfc/rfc-60/rfc-60.md:
##########
@@ -84,23 +119,85 @@ public interface HoodieStorageStrategy extends Serializable {
   /**
    * Return a storage location for the given filename.
    *
-   * @param fileId data file ID
+   * @param path fileName

Review Comment:
   The path can be a `relative partition path` or a `relative file path with a filename`.
   
   When a path is passed in externally, the storageLocation is processed by different storage strategies to prepend the appropriate prefix to the path.
   
   For `HoodieWriteHandle`, the input is a relative partition path.
   
   For example, in HoodieCacheLayerStorageStrategy, a relative path like `tableName/dt=2024-12-04` is passed into HoodieWriteHandle. After processing by HoodieCacheLayerStorageStrategy, it determines that the current write should go to the cache layer and returns a full path with the cache layer prefix, such as `hdfs://nsxxx/user/jdr_lakehouse_cache_layer/jdr_lakehouse_traffic/gdm.db/gdm_jdrxxxxx/`
   
   Similarly, in HoodieObjectStorageStrategy, a relative path like `tableName/dt=2024-12-04` is passed into HoodieWriteHandle. After processing by HoodieObjectStorageStrategy, it returns a full path like `s3://<table_storage_bucket>/0bfb3d6e/<hudi_table_name>/country=india/`
   
   For `HoodieCommitMetaData#getFileIdAndFullPaths`, the input is a relative path with a file, such as dt=2021-01-05/00000000-3fd6-4d4f-b1e7-4b9768ee0911-0_0-8-7_20241104162125935.parquet.
   
   ```
     public HashMap<String, String> getFileIdAndFullPaths(String instantTime, HoodieStorageStrategy strategy) {
       HashMap<String, String> fullPaths = new HashMap<>();
       for (Map.Entry<String, String> entry : getFileIdAndRelativePaths().entrySet()) {
         String fullPath = entry.getValue() != null
             ? strategy.storageLocation(entry.getValue(), instantTime).toString() : null;
         fullPaths.put(entry.getKey(), fullPath);
       }
       return fullPaths;
     }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-3625][RFC-60] Update RFC-60 With Federated Storage Layout [hudi]

Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #12407:
URL: https://github.com/apache/hudi/pull/12407#discussion_r1869005485


##########
rfc/rfc-60/rfc-60.md:
##########
@@ -84,23 +119,85 @@ public interface HoodieStorageStrategy extends Serializable {
   /**
    * Return a storage location for the given filename.
    *
-   * @param fileId data file ID
+   * @param path fileName
    * @return a storage location string for a data file
    */
-  String storageLocation(String fileId);
+  StoragePath storageLocation(String path, String instantTime);
 
   /**
-   * Return a storage location for the given partition and filename.
+   * Return all possible StoragePaths
    *
-   * @param partitionPath partition path for the file
-   * @param fileId data file ID
-   * @return a storage location string for a data file
+   * @param partitionPath
+   * @param checkExist check if StoragePath is truly existed or not. 
+   * @return a st of storage partition path
+   */
+  Set<StoragePath> getAllLocations(String partitionPath, boolean checkExist);

Review Comment:
   This is needed, for example, in the `AbstractTableFileSystemView#ensurePartitionLoadedCorrectly(String partition)` method.  (Hive query side has same logic)
   
   An external relative path partition is passed in, and currently, the base path is directly prefixed to it before performing the listPartition(Path partitionPath) operation. 
   
   For Hudi tables with the Federated Storage Layout feature enabled, the same relative path might map to multiple physical paths. Here, the HoodieStorageStrategy#getAllLocations method is required.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-3625][RFC-60] Update RFC-60 With Federated Storage Layout [hudi]

Posted by "voonhous (via GitHub) via commits" <co...@hudi.apache.org>.
voonhous closed pull request #12407: [HUDI-3625][RFC-60] Update RFC-60 With Federated Storage Layout
URL: https://github.com/apache/hudi/pull/12407


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org