You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "CTTY (via GitHub)" <gi...@apache.org> on 2023/08/17 00:11:08 UTC
[GitHub] [hudi] CTTY opened a new pull request, #9462: [HUDI-3625] Update RFC-60
CTTY opened a new pull request, #9462:
URL: https://github.com/apache/hudi/pull/9462
### Change Logs
_Describe context and summary for this change. Highlight if any code was copied._
### Impact
_Describe any public API or user-facing feature change or any performance impact._
### Risk level (write none, low medium or high below)
_If medium or high, explain what verification was done to mitigate the risks._
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change_
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-3625] Update RFC-60 [hudi]
Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on code in PR #9462:
URL: https://github.com/apache/hudi/pull/9462#discussion_r1510428841
##########
rfc/rfc-60/rfc-60.md:
##########
@@ -196,13 +195,75 @@ for metadata table to be populated.
4. If there is an error reading from Metadata table, we will not fall back listing from file system.
-5. In case of metadata table getting corrupted or lost, we need to have a solution here to reconstruct metadata table
-from the files which distributed using federated storage. We will likely have to implement a file system listing
-logic, that can get all the partition to files mapping by listing all the prefixes under the `Table Storage Path`.
-Following the folder structure of adding table name/partitions under the prefix will help in getting the listing and
-identifying the table/partition they belong to.
+### Integration
+This section mainly describes how storage strategy is integrated with other components and how read/write
+would look like from Hudi side with object storage layout.
+
+We propose integrating the storage strategy at the filesystem level, specifically within `HoodieWrapperFileSystem`.
+This way, only file read/write operations undergo path conversion and we can limit the usage of
+storage strategy to only filesystem level so other upper-level components don't need to be aware of physical paths.
+
+This also mandates that `HoodieWrapperFileSystem` is the filesystem of choice for all upper-level Hudi components.
+Getting filesystem from `Path` or such won't be allowed anymore as using raw filesystem may not reach
+to physical locations without storage strategy. Hudi components can simply call `HoodieMetaClient#getFs`
+to get `HoodieWrapperFileSystem`, and this needs to be the only allowed way for any filesystem-related operation.
+The only exception is when we need to interact with metadata that's still stored under the original table path,
+and we should call `HoodieMetaClient#getRawFs` in this case so `HoodieMetaClient` can still be the single entry
+for getting filesystem.
+
+
+
+When conducting a read operation, Hudi would:
+1. Access filesystem view, `HoodieMetadataFileSystemView` specifically
+2. Scan metadata table via filesystem view to compose `HoodieMetadataPayload`
+3. Call `HoodieMetadataPayload#getFileStatuses` and employ `HoodieWrapperFileSystem` to get
+file statuses with physical locations
+
+This flow can be concluded in the chart below.
+
+
+
+#### Considerations
+- Path conversion happens on the fly when reading/writing files. This saves Hudi from storing physical locations
+but it also means extra performance burden, even though it may be negligible.
+- Since table path and data path will most likely have different top-level folders/authorities,
+`HoodieWrapperFileSystem` should maintain at least two `FileSystem` objects: one to access table path and another
+to access storage path. `HoodieWrapperFileSystem` should intelligently tell if it needs
+to convert the path by checking the path on the fly.
+- When using Hudi file reader/writer implementation, we will need to pass `HoodieWrapperFileSystem` down
+to parent reader. For instance, when using `HoodieAvroHFileReader`, we will need to pass `HoodieWrapperFileSystem`
+to `HFile.Reader` so it can have access to storage strategy. If reader/writer doesn't take filesystem
+directly (e.g. `ParquetFileReader` only takes `Configuration` and `Path` for reading), then we will
+need to register `HoodieWrapperFileSystem` to `Configuration` so it can be initialized/used later.
+
+### Repair Tool
Review Comment:
Yes, it should work with custom strategies. But the how custom strategies would be implemented is a big black box so we plan to support basic cases where there is only one storage path (the algo used for hashing doesn't matter here).
This may be implemented differently with the new `HoodieStorage` and I'll explore that and update this part in separate PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9462: [HUDI-3625] Update RFC-60
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9462:
URL: https://github.com/apache/hudi/pull/9462#issuecomment-1681639968
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9ba2ca5c4f4bd4a2af5c621e72592baa77b2262d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19328",
"triggerID" : "9ba2ca5c4f4bd4a2af5c621e72592baa77b2262d",
"triggerType" : "PUSH"
}, {
"hash" : "6891d53be9579bc698e6266781fc69c43d08db27",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19331",
"triggerID" : "6891d53be9579bc698e6266781fc69c43d08db27",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6891d53be9579bc698e6266781fc69c43d08db27 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19331)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-3625] Update RFC-60 [hudi]
Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on code in PR #9462:
URL: https://github.com/apache/hudi/pull/9462#discussion_r1510431710
##########
rfc/rfc-60/rfc-60.md:
##########
@@ -196,13 +195,75 @@ for metadata table to be populated.
4. If there is an error reading from Metadata table, we will not fall back listing from file system.
-5. In case of metadata table getting corrupted or lost, we need to have a solution here to reconstruct metadata table
-from the files which distributed using federated storage. We will likely have to implement a file system listing
-logic, that can get all the partition to files mapping by listing all the prefixes under the `Table Storage Path`.
-Following the folder structure of adding table name/partitions under the prefix will help in getting the listing and
-identifying the table/partition they belong to.
+### Integration
+This section mainly describes how storage strategy is integrated with other components and how read/write
+would look like from Hudi side with object storage layout.
+
+We propose integrating the storage strategy at the filesystem level, specifically within `HoodieWrapperFileSystem`.
+This way, only file read/write operations undergo path conversion and we can limit the usage of
+storage strategy to only filesystem level so other upper-level components don't need to be aware of physical paths.
+
+This also mandates that `HoodieWrapperFileSystem` is the filesystem of choice for all upper-level Hudi components.
+Getting filesystem from `Path` or such won't be allowed anymore as using raw filesystem may not reach
+to physical locations without storage strategy. Hudi components can simply call `HoodieMetaClient#getFs`
+to get `HoodieWrapperFileSystem`, and this needs to be the only allowed way for any filesystem-related operation.
+The only exception is when we need to interact with metadata that's still stored under the original table path,
+and we should call `HoodieMetaClient#getRawFs` in this case so `HoodieMetaClient` can still be the single entry
+for getting filesystem.
+
+
+
+When conducting a read operation, Hudi would:
Review Comment:
This original idea is to register HoodieWrapperFileSystem on file writers like `HoodieSparkParquetWriter`. But with the recent addition of `HoodieStorage`, I believe there is an easier & cleaner way to achieve that. I'll leave this RFC as it is for now and will update this RFC with the idea of `HoodieStorage` in a separate PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9462: [HUDI-3625] Update RFC-60
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9462:
URL: https://github.com/apache/hudi/pull/9462#issuecomment-1681431587
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9ba2ca5c4f4bd4a2af5c621e72592baa77b2262d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ba2ca5c4f4bd4a2af5c621e72592baa77b2262d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ba2ca5c4f4bd4a2af5c621e72592baa77b2262d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-3625] Update RFC-60 [hudi]
Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on code in PR #9462:
URL: https://github.com/apache/hudi/pull/9462#discussion_r1510428362
##########
rfc/rfc-60/rfc-60.md:
##########
@@ -97,22 +97,21 @@ public interface HoodieStorageStrategy extends Serializable {
}
```
-### Generating file paths for object store optimized layout
+### Generating File Paths for Object Store Optimized Layout
We want to distribute files evenly across multiple random prefixes, instead of following the traditional Hive storage
layout of keeping them under a common table path/prefix. In addition to the `Table Path`, for this new layout user will
configure another `Table Storage Path` under which the actual data files will be distributed. The original `Table Path` will
Review Comment:
Yes, users may want to have multiple buckets and have different access controls/encryptions on different buckets. It's not a common use case but it would be nice to enable that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9462: [HUDI-3625] Update RFC-60
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9462:
URL: https://github.com/apache/hudi/pull/9462#issuecomment-1681472947
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9ba2ca5c4f4bd4a2af5c621e72592baa77b2262d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19328",
"triggerID" : "9ba2ca5c4f4bd4a2af5c621e72592baa77b2262d",
"triggerType" : "PUSH"
}, {
"hash" : "6891d53be9579bc698e6266781fc69c43d08db27",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6891d53be9579bc698e6266781fc69c43d08db27",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ba2ca5c4f4bd4a2af5c621e72592baa77b2262d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19328)
* 6891d53be9579bc698e6266781fc69c43d08db27 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-3625] Update RFC-60 [hudi]
Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on code in PR #9462:
URL: https://github.com/apache/hudi/pull/9462#discussion_r1510428461
##########
rfc/rfc-60/rfc-60.md:
##########
@@ -196,13 +195,75 @@ for metadata table to be populated.
4. If there is an error reading from Metadata table, we will not fall back listing from file system.
-5. In case of metadata table getting corrupted or lost, we need to have a solution here to reconstruct metadata table
-from the files which distributed using federated storage. We will likely have to implement a file system listing
-logic, that can get all the partition to files mapping by listing all the prefixes under the `Table Storage Path`.
-Following the folder structure of adding table name/partitions under the prefix will help in getting the listing and
-identifying the table/partition they belong to.
+### Integration
+This section mainly describes how storage strategy is integrated with other components and how read/write
+would look like from Hudi side with object storage layout.
+
+We propose integrating the storage strategy at the filesystem level, specifically within `HoodieWrapperFileSystem`.
+This way, only file read/write operations undergo path conversion and we can limit the usage of
+storage strategy to only filesystem level so other upper-level components don't need to be aware of physical paths.
+
+This also mandates that `HoodieWrapperFileSystem` is the filesystem of choice for all upper-level Hudi components.
+Getting filesystem from `Path` or such won't be allowed anymore as using raw filesystem may not reach
+to physical locations without storage strategy. Hudi components can simply call `HoodieMetaClient#getFs`
+to get `HoodieWrapperFileSystem`, and this needs to be the only allowed way for any filesystem-related operation.
+The only exception is when we need to interact with metadata that's still stored under the original table path,
+and we should call `HoodieMetaClient#getRawFs` in this case so `HoodieMetaClient` can still be the single entry
+for getting filesystem.
+
+
+
+When conducting a read operation, Hudi would:
+1. Access filesystem view, `HoodieMetadataFileSystemView` specifically
+2. Scan metadata table via filesystem view to compose `HoodieMetadataPayload`
+3. Call `HoodieMetadataPayload#getFileStatuses` and employ `HoodieWrapperFileSystem` to get
Review Comment:
Excatly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-3625] Update RFC-60 [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9462:
URL: https://github.com/apache/hudi/pull/9462#issuecomment-1975413474
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9ba2ca5c4f4bd4a2af5c621e72592baa77b2262d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19328",
"triggerID" : "9ba2ca5c4f4bd4a2af5c621e72592baa77b2262d",
"triggerType" : "PUSH"
}, {
"hash" : "6891d53be9579bc698e6266781fc69c43d08db27",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19331",
"triggerID" : "6891d53be9579bc698e6266781fc69c43d08db27",
"triggerType" : "PUSH"
}, {
"hash" : "5543ce244909d9e8aef49411325469a3b07c78dc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5543ce244909d9e8aef49411325469a3b07c78dc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6891d53be9579bc698e6266781fc69c43d08db27 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19331)
* 5543ce244909d9e8aef49411325469a3b07c78dc UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on a diff in pull request #9462: [HUDI-3625] Update RFC-60
Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on code in PR #9462:
URL: https://github.com/apache/hudi/pull/9462#discussion_r1309471028
##########
rfc/rfc-60/rfc-60.md:
##########
@@ -196,13 +195,75 @@ for metadata table to be populated.
4. If there is an error reading from Metadata table, we will not fall back listing from file system.
-5. In case of metadata table getting corrupted or lost, we need to have a solution here to reconstruct metadata table
-from the files which distributed using federated storage. We will likely have to implement a file system listing
-logic, that can get all the partition to files mapping by listing all the prefixes under the `Table Storage Path`.
-Following the folder structure of adding table name/partitions under the prefix will help in getting the listing and
-identifying the table/partition they belong to.
+### Integration
+This section mainly describes how storage strategy is integrated with other components and how read/write
+would look like from Hudi side with object storage layout.
+
+We propose integrating the storage strategy at the filesystem level, specifically within `HoodieWrapperFileSystem`.
+This way, only file read/write operations undergo path conversion and we can limit the usage of
+storage strategy to only filesystem level so other upper-level components don't need to be aware of physical paths.
+
+This also mandates that `HoodieWrapperFileSystem` is the filesystem of choice for all upper-level Hudi components.
+Getting filesystem from `Path` or such won't be allowed anymore as using raw filesystem may not reach
+to physical locations without storage strategy. Hudi components can simply call `HoodieMetaClient#getFs`
+to get `HoodieWrapperFileSystem`, and this needs to be the only allowed way for any filesystem-related operation.
+The only exception is when we need to interact with metadata that's still stored under the original table path,
+and we should call `HoodieMetaClient#getRawFs` in this case so `HoodieMetaClient` can still be the single entry
+for getting filesystem.
+
+
+
+When conducting a read operation, Hudi would:
Review Comment:
can we detail how any write operations will work? I guess it does the hashing and writes the object someplace else?
##########
rfc/rfc-60/rfc-60.md:
##########
@@ -196,13 +195,75 @@ for metadata table to be populated.
4. If there is an error reading from Metadata table, we will not fall back listing from file system.
-5. In case of metadata table getting corrupted or lost, we need to have a solution here to reconstruct metadata table
-from the files which distributed using federated storage. We will likely have to implement a file system listing
-logic, that can get all the partition to files mapping by listing all the prefixes under the `Table Storage Path`.
-Following the folder structure of adding table name/partitions under the prefix will help in getting the listing and
-identifying the table/partition they belong to.
+### Integration
+This section mainly describes how storage strategy is integrated with other components and how read/write
+would look like from Hudi side with object storage layout.
+
+We propose integrating the storage strategy at the filesystem level, specifically within `HoodieWrapperFileSystem`.
+This way, only file read/write operations undergo path conversion and we can limit the usage of
+storage strategy to only filesystem level so other upper-level components don't need to be aware of physical paths.
+
+This also mandates that `HoodieWrapperFileSystem` is the filesystem of choice for all upper-level Hudi components.
+Getting filesystem from `Path` or such won't be allowed anymore as using raw filesystem may not reach
+to physical locations without storage strategy. Hudi components can simply call `HoodieMetaClient#getFs`
+to get `HoodieWrapperFileSystem`, and this needs to be the only allowed way for any filesystem-related operation.
+The only exception is when we need to interact with metadata that's still stored under the original table path,
+and we should call `HoodieMetaClient#getRawFs` in this case so `HoodieMetaClient` can still be the single entry
+for getting filesystem.
+
+
+
+When conducting a read operation, Hudi would:
+1. Access filesystem view, `HoodieMetadataFileSystemView` specifically
+2. Scan metadata table via filesystem view to compose `HoodieMetadataPayload`
+3. Call `HoodieMetadataPayload#getFileStatuses` and employ `HoodieWrapperFileSystem` to get
Review Comment:
So there are no changes to the metadata table, as you see it, correct? i.e it continues to store the same "hive style" view of the table's layout
##########
rfc/rfc-60/rfc-60.md:
##########
@@ -196,13 +195,75 @@ for metadata table to be populated.
4. If there is an error reading from Metadata table, we will not fall back listing from file system.
-5. In case of metadata table getting corrupted or lost, we need to have a solution here to reconstruct metadata table
-from the files which distributed using federated storage. We will likely have to implement a file system listing
-logic, that can get all the partition to files mapping by listing all the prefixes under the `Table Storage Path`.
-Following the folder structure of adding table name/partitions under the prefix will help in getting the listing and
-identifying the table/partition they belong to.
+### Integration
+This section mainly describes how storage strategy is integrated with other components and how read/write
+would look like from Hudi side with object storage layout.
+
+We propose integrating the storage strategy at the filesystem level, specifically within `HoodieWrapperFileSystem`.
+This way, only file read/write operations undergo path conversion and we can limit the usage of
+storage strategy to only filesystem level so other upper-level components don't need to be aware of physical paths.
+
+This also mandates that `HoodieWrapperFileSystem` is the filesystem of choice for all upper-level Hudi components.
+Getting filesystem from `Path` or such won't be allowed anymore as using raw filesystem may not reach
+to physical locations without storage strategy. Hudi components can simply call `HoodieMetaClient#getFs`
+to get `HoodieWrapperFileSystem`, and this needs to be the only allowed way for any filesystem-related operation.
+The only exception is when we need to interact with metadata that's still stored under the original table path,
+and we should call `HoodieMetaClient#getRawFs` in this case so `HoodieMetaClient` can still be the single entry
+for getting filesystem.
+
+
+
+When conducting a read operation, Hudi would:
+1. Access filesystem view, `HoodieMetadataFileSystemView` specifically
+2. Scan metadata table via filesystem view to compose `HoodieMetadataPayload`
+3. Call `HoodieMetadataPayload#getFileStatuses` and employ `HoodieWrapperFileSystem` to get
+file statuses with physical locations
+
+This flow can be concluded in the chart below.
+
+
+
+#### Considerations
+- Path conversion happens on the fly when reading/writing files. This saves Hudi from storing physical locations
Review Comment:
I'd be suprised if cost of hashing once per object becomes noticeable in anyway.
##########
rfc/rfc-60/rfc-60.md:
##########
@@ -196,13 +195,75 @@ for metadata table to be populated.
4. If there is an error reading from Metadata table, we will not fall back listing from file system.
-5. In case of metadata table getting corrupted or lost, we need to have a solution here to reconstruct metadata table
-from the files which distributed using federated storage. We will likely have to implement a file system listing
-logic, that can get all the partition to files mapping by listing all the prefixes under the `Table Storage Path`.
-Following the folder structure of adding table name/partitions under the prefix will help in getting the listing and
-identifying the table/partition they belong to.
+### Integration
+This section mainly describes how storage strategy is integrated with other components and how read/write
+would look like from Hudi side with object storage layout.
+
+We propose integrating the storage strategy at the filesystem level, specifically within `HoodieWrapperFileSystem`.
+This way, only file read/write operations undergo path conversion and we can limit the usage of
+storage strategy to only filesystem level so other upper-level components don't need to be aware of physical paths.
+
+This also mandates that `HoodieWrapperFileSystem` is the filesystem of choice for all upper-level Hudi components.
+Getting filesystem from `Path` or such won't be allowed anymore as using raw filesystem may not reach
+to physical locations without storage strategy. Hudi components can simply call `HoodieMetaClient#getFs`
+to get `HoodieWrapperFileSystem`, and this needs to be the only allowed way for any filesystem-related operation.
+The only exception is when we need to interact with metadata that's still stored under the original table path,
+and we should call `HoodieMetaClient#getRawFs` in this case so `HoodieMetaClient` can still be the single entry
+for getting filesystem.
+
+
+
+When conducting a read operation, Hudi would:
+1. Access filesystem view, `HoodieMetadataFileSystemView` specifically
+2. Scan metadata table via filesystem view to compose `HoodieMetadataPayload`
+3. Call `HoodieMetadataPayload#getFileStatuses` and employ `HoodieWrapperFileSystem` to get
+file statuses with physical locations
+
+This flow can be concluded in the chart below.
+
+
+
+#### Considerations
+- Path conversion happens on the fly when reading/writing files. This saves Hudi from storing physical locations
+but it also means extra performance burden, even though it may be negligible.
+- Since table path and data path will most likely have different top-level folders/authorities,
+`HoodieWrapperFileSystem` should maintain at least two `FileSystem` objects: one to access table path and another
+to access storage path. `HoodieWrapperFileSystem` should intelligently tell if it needs
+to convert the path by checking the path on the fly.
+- When using Hudi file reader/writer implementation, we will need to pass `HoodieWrapperFileSystem` down
+to parent reader. For instance, when using `HoodieAvroHFileReader`, we will need to pass `HoodieWrapperFileSystem`
+to `HFile.Reader` so it can have access to storage strategy. If reader/writer doesn't take filesystem
+directly (e.g. `ParquetFileReader` only takes `Configuration` and `Path` for reading), then we will
+need to register `HoodieWrapperFileSystem` to `Configuration` so it can be initialized/used later.
+
+### Repair Tool
Review Comment:
it'd be great if this tool can work the pluggable interface. i.e can it repair even custom placement strategies?
##########
rfc/rfc-60/rfc-60.md:
##########
@@ -148,7 +147,7 @@ s3://<table_storage_bucket>/0bfb3d6e/<hudi_table_name>/.075f3295-def8-4a42-a927-
...
```
-Note: Storage strategy would only return a storage location instead of a full path. In the above example,
+Storage strategy would only return a storage location instead of a full path. In the above example,
the storage location is `s3://<table_storage_bucket>/0bfb3d6e/`, and the lower-level folder structure would be appended
later automatically to get the actual file path. In another word,
users would only be able to customize upper-level folder structure (storage location).
Review Comment:
note: if we don't have any partitioning on the table, then this will achieve random distribution of files across prefixes. partition path is totally optional.
##########
rfc/rfc-60/rfc-60.md:
##########
@@ -196,13 +195,75 @@ for metadata table to be populated.
4. If there is an error reading from Metadata table, we will not fall back listing from file system.
-5. In case of metadata table getting corrupted or lost, we need to have a solution here to reconstruct metadata table
-from the files which distributed using federated storage. We will likely have to implement a file system listing
-logic, that can get all the partition to files mapping by listing all the prefixes under the `Table Storage Path`.
-Following the folder structure of adding table name/partitions under the prefix will help in getting the listing and
-identifying the table/partition they belong to.
+### Integration
+This section mainly describes how storage strategy is integrated with other components and how read/write
+would look like from Hudi side with object storage layout.
+
+We propose integrating the storage strategy at the filesystem level, specifically within `HoodieWrapperFileSystem`.
Review Comment:
We may have to introduce an abstraction that is not based on Hadoop FileSystem API. Just FYI.
##########
rfc/rfc-60/rfc-60.md:
##########
@@ -97,22 +97,21 @@ public interface HoodieStorageStrategy extends Serializable {
}
```
-### Generating file paths for object store optimized layout
+### Generating File Paths for Object Store Optimized Layout
We want to distribute files evenly across multiple random prefixes, instead of following the traditional Hive storage
layout of keeping them under a common table path/prefix. In addition to the `Table Path`, for this new layout user will
configure another `Table Storage Path` under which the actual data files will be distributed. The original `Table Path` will
Review Comment:
do we see a need to support multiple storage paths, to distribute data across buckets?
##########
rfc/rfc-60/rfc-60.md:
##########
@@ -196,13 +195,75 @@ for metadata table to be populated.
4. If there is an error reading from Metadata table, we will not fall back listing from file system.
-5. In case of metadata table getting corrupted or lost, we need to have a solution here to reconstruct metadata table
-from the files which distributed using federated storage. We will likely have to implement a file system listing
-logic, that can get all the partition to files mapping by listing all the prefixes under the `Table Storage Path`.
-Following the folder structure of adding table name/partitions under the prefix will help in getting the listing and
-identifying the table/partition they belong to.
+### Integration
+This section mainly describes how storage strategy is integrated with other components and how read/write
+would look like from Hudi side with object storage layout.
+
+We propose integrating the storage strategy at the filesystem level, specifically within `HoodieWrapperFileSystem`.
+This way, only file read/write operations undergo path conversion and we can limit the usage of
+storage strategy to only filesystem level so other upper-level components don't need to be aware of physical paths.
+
+This also mandates that `HoodieWrapperFileSystem` is the filesystem of choice for all upper-level Hudi components.
+Getting filesystem from `Path` or such won't be allowed anymore as using raw filesystem may not reach
+to physical locations without storage strategy. Hudi components can simply call `HoodieMetaClient#getFs`
+to get `HoodieWrapperFileSystem`, and this needs to be the only allowed way for any filesystem-related operation.
+The only exception is when we need to interact with metadata that's still stored under the original table path,
+and we should call `HoodieMetaClient#getRawFs` in this case so `HoodieMetaClient` can still be the single entry
+for getting filesystem.
+
+
+
+When conducting a read operation, Hudi would:
+1. Access filesystem view, `HoodieMetadataFileSystemView` specifically
+2. Scan metadata table via filesystem view to compose `HoodieMetadataPayload`
+3. Call `HoodieMetadataPayload#getFileStatuses` and employ `HoodieWrapperFileSystem` to get
+file statuses with physical locations
+
+This flow can be concluded in the chart below.
+
+
+
+#### Considerations
+- Path conversion happens on the fly when reading/writing files. This saves Hudi from storing physical locations
+but it also means extra performance burden, even though it may be negligible.
+- Since table path and data path will most likely have different top-level folders/authorities,
+`HoodieWrapperFileSystem` should maintain at least two `FileSystem` objects: one to access table path and another
+to access storage path. `HoodieWrapperFileSystem` should intelligently tell if it needs
+to convert the path by checking the path on the fly.
+- When using Hudi file reader/writer implementation, we will need to pass `HoodieWrapperFileSystem` down
+to parent reader. For instance, when using `HoodieAvroHFileReader`, we will need to pass `HoodieWrapperFileSystem`
+to `HFile.Reader` so it can have access to storage strategy. If reader/writer doesn't take filesystem
+directly (e.g. `ParquetFileReader` only takes `Configuration` and `Path` for reading), then we will
+need to register `HoodieWrapperFileSystem` to `Configuration` so it can be initialized/used later.
+
+### Repair Tool
+In case of metadata table getting corrupted or lost, we need to have a solution here to reconstruct metadata table
+from the files that are distributed using federated storage. We will need a repair tool
+to get all the partition to files mapping by listing all the prefixes under the `Table Storage Path`
+and then reconstruct metadata table.
+
+In Hudi we already have `HoodieBackedTableMetadataWriter` to list existing data files to initialize/construct
Review Comment:
not sure if this is the final direction. but we can resolve this down the line, on the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9462: [HUDI-3625] Update RFC-60
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9462:
URL: https://github.com/apache/hudi/pull/9462#issuecomment-1681478322
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9ba2ca5c4f4bd4a2af5c621e72592baa77b2262d",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19328",
"triggerID" : "9ba2ca5c4f4bd4a2af5c621e72592baa77b2262d",
"triggerType" : "PUSH"
}, {
"hash" : "6891d53be9579bc698e6266781fc69c43d08db27",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19331",
"triggerID" : "6891d53be9579bc698e6266781fc69c43d08db27",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ba2ca5c4f4bd4a2af5c621e72592baa77b2262d Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19328)
* 6891d53be9579bc698e6266781fc69c43d08db27 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19331)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9462: [HUDI-3625] Update RFC-60
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9462:
URL: https://github.com/apache/hudi/pull/9462#issuecomment-1681436546
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9ba2ca5c4f4bd4a2af5c621e72592baa77b2262d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19328",
"triggerID" : "9ba2ca5c4f4bd4a2af5c621e72592baa77b2262d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ba2ca5c4f4bd4a2af5c621e72592baa77b2262d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19328)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-3625] Update RFC-60 [hudi]
Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on code in PR #9462:
URL: https://github.com/apache/hudi/pull/9462#discussion_r1510428841
##########
rfc/rfc-60/rfc-60.md:
##########
@@ -196,13 +195,75 @@ for metadata table to be populated.
4. If there is an error reading from Metadata table, we will not fall back listing from file system.
-5. In case of metadata table getting corrupted or lost, we need to have a solution here to reconstruct metadata table
-from the files which distributed using federated storage. We will likely have to implement a file system listing
-logic, that can get all the partition to files mapping by listing all the prefixes under the `Table Storage Path`.
-Following the folder structure of adding table name/partitions under the prefix will help in getting the listing and
-identifying the table/partition they belong to.
+### Integration
+This section mainly describes how storage strategy is integrated with other components and how read/write
+would look like from Hudi side with object storage layout.
+
+We propose integrating the storage strategy at the filesystem level, specifically within `HoodieWrapperFileSystem`.
+This way, only file read/write operations undergo path conversion and we can limit the usage of
+storage strategy to only filesystem level so other upper-level components don't need to be aware of physical paths.
+
+This also mandates that `HoodieWrapperFileSystem` is the filesystem of choice for all upper-level Hudi components.
+Getting filesystem from `Path` or such won't be allowed anymore as using raw filesystem may not reach
+to physical locations without storage strategy. Hudi components can simply call `HoodieMetaClient#getFs`
+to get `HoodieWrapperFileSystem`, and this needs to be the only allowed way for any filesystem-related operation.
+The only exception is when we need to interact with metadata that's still stored under the original table path,
+and we should call `HoodieMetaClient#getRawFs` in this case so `HoodieMetaClient` can still be the single entry
+for getting filesystem.
+
+
+
+When conducting a read operation, Hudi would:
+1. Access filesystem view, `HoodieMetadataFileSystemView` specifically
+2. Scan metadata table via filesystem view to compose `HoodieMetadataPayload`
+3. Call `HoodieMetadataPayload#getFileStatuses` and employ `HoodieWrapperFileSystem` to get
+file statuses with physical locations
+
+This flow can be concluded in the chart below.
+
+
+
+#### Considerations
+- Path conversion happens on the fly when reading/writing files. This saves Hudi from storing physical locations
+but it also means extra performance burden, even though it may be negligible.
+- Since table path and data path will most likely have different top-level folders/authorities,
+`HoodieWrapperFileSystem` should maintain at least two `FileSystem` objects: one to access table path and another
+to access storage path. `HoodieWrapperFileSystem` should intelligently tell if it needs
+to convert the path by checking the path on the fly.
+- When using Hudi file reader/writer implementation, we will need to pass `HoodieWrapperFileSystem` down
+to parent reader. For instance, when using `HoodieAvroHFileReader`, we will need to pass `HoodieWrapperFileSystem`
+to `HFile.Reader` so it can have access to storage strategy. If reader/writer doesn't take filesystem
+directly (e.g. `ParquetFileReader` only takes `Configuration` and `Path` for reading), then we will
+need to register `HoodieWrapperFileSystem` to `Configuration` so it can be initialized/used later.
+
+### Repair Tool
Review Comment:
Yes, it should work with custom strategies. I'll add that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-3625] Update RFC-60 [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #9462:
URL: https://github.com/apache/hudi/pull/9462
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org