You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "yihua (via GitHub)" <gi...@apache.org> on 2025/12/10 19:26:49 UTC
[I] Add a HoodieStorageFactory to provide HoodieStorage instance [hudi]
yihua opened a new issue, #17549:
URL: https://github.com/apache/hudi/issues/17549
### Task Description
**What needs to be done:**
Avoid and disallow using the constrcutor of `HoodieStorage` implementation classes directly.
**Why this task is needed:**
Streamline the `HoodieStorage` instantiation
### Task Type
Code improvement/refactoring
### Related Issues
**Parent feature issue:** (if applicable )
**Related issues:**
NOTE: Use `Relationships` button to add parent/blocking issues after issue is created.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add a HoodieStorageFactory to provide HoodieStorage instance [hudi]
Posted by "KiteSoar (via GitHub)" <gi...@apache.org>.
KiteSoar commented on issue #17549:
URL: https://github.com/apache/hudi/issues/17549#issuecomment-3663901631
@yihua Hi,May I ask if this task still needs to be continued? If it does, I am a new contributor to Hudi and would like to take over this work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add a HoodieStorageFactory to provide HoodieStorage instance [hudi]
Posted by "KiteSoar (via GitHub)" <gi...@apache.org>.
KiteSoar commented on issue #17549:
URL: https://github.com/apache/hudi/issues/17549#issuecomment-3668205818
@CTTY Thanks for the guidance! I'd like to work on this. I plan to place the HoodieStorageFactory inside the hudi-io module (under org.apache.hudi.storage) , while strictly defaulting to HoodieHadoopStorage for backward compatibility, and I will proceed to refactor the codebase to replace the direct constructor calls with this factory method. How does this plan sound to you?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add a HoodieStorageFactory to provide HoodieStorage instance [hudi]
Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on issue #17549:
URL: https://github.com/apache/hudi/issues/17549#issuecomment-3667540950
Hi @KiteSoar , thanks for jumping on this! We do need some help on this issue. I don't have exact steps in mind right now but here is what we want to achieve:
- Forbid using the constructor of concrete Storage implementation(e.g. `HoodieHadoopStorage`) to create storage instances across Hudi code base.
- As a replacement, we can add `HoodieStorageFactory` to help users/developers construct a Storage instance when needed based on `StoragePath`, `StorageConfiguration`, and `HOODIE_STORAGE_CLASS` config.
The motive is to avoid usages like this: https://github.com/CTTY/hudi/blob/0c0c402a9a03061155b5c43f1f4cdd37726eacfd/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DefaultSource.scala#L236
### Why?
- `HoodieStorage` is supposed to be an interface that allows downstream users to inject customized storage, so they have more control over the IO operations.
- Using `HoodieHadoopStorage` directly across the code base will cause Hudi to step over downstream customized storage implementations
Please let me know if you want to work on this or have further questions, I'd be available on Slack as well. Thanks again!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add a HoodieStorageFactory to provide HoodieStorage instance [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on issue #17549:
URL: https://github.com/apache/hudi/issues/17549#issuecomment-3638633686
@CTTY could you add your thoughts here too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add a HoodieStorageFactory to provide HoodieStorage instance [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on issue #17549:
URL: https://github.com/apache/hudi/issues/17549#issuecomment-3667456813
@KiteSoar Thanks for your interest! Yes, there should be work to be done on this improvement story. @CTTY could you provide more concrete tasks that @KiteSoar can take up?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add a HoodieStorageFactory to provide HoodieStorage instance [hudi]
Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY closed issue #17549: Add a HoodieStorageFactory to provide HoodieStorage instance
URL: https://github.com/apache/hudi/issues/17549
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add a HoodieStorageFactory to provide HoodieStorage instance [hudi]
Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on issue #17549:
URL: https://github.com/apache/hudi/issues/17549#issuecomment-3668287213
> I plan to place the HoodieStorageFactory inside the hudi-io module (under org.apache.hudi.storage) , while strictly defaulting to HoodieHadoopStorage for backward compatibility,
StorageFactory should not be put in `hudi-io` because you need `HOODIE_STORAGE_CLASS`(in `hudi-common`) to determine the storage type. `HOODIE_STORAGE_CLASS` is set to `HoodieHadoopStorage` by default, so no need to worry about the backward compatibility here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add a HoodieStorageFactory to provide HoodieStorage instance [hudi]
Posted by "KiteSoar (via GitHub)" <gi...@apache.org>.
KiteSoar commented on issue #17549:
URL: https://github.com/apache/hudi/issues/17549#issuecomment-3668529104
> > I plan to place the HoodieStorageFactory inside the hudi-io module (under org.apache.hudi.storage) , while strictly defaulting to HoodieHadoopStorage for backward compatibility,
>
> StorageFactory should not be put in `hudi-io` because you need `HOODIE_STORAGE_CLASS`(in `hudi-common`) to determine the storage type. `HOODIE_STORAGE_CLASS` is set to `HoodieHadoopStorage` by default, so no need to worry about the backward compatibility here
Ah, good catch! I missed that the config is defined in hudi-common. I also checked `HoodieStorageConfig` and saw that `HOODIE_STORAGE_CLASS` already has `HoodieHadoopStorage` as the default value, so backward compatibility is indeed automatically handled.I'll place the factory in hudi-common then and start working on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org