You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "yunchipang (via GitHub)" <gi...@apache.org> on 2025/09/02 08:27:38 UTC
[PR] feat: export `HudiTableConfig` and `HudiReadConfig` as python enums [hudi-rs]
yunchipang opened a new pull request, #433:
URL: https://github.com/apache/hudi-rs/pull/433
## Description
<!--- Describe your changes in detail -->
Introduce type-safe Python enums for Hudi configuration options, mirroring the existing Rust enums.
<!--- If it fixes an open issue, please link to the issue here. -->
closes #404
<!--- Please link any related issues and PRs as well. -->
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [ ] Manual tests
- [ ] Details are described below
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: export `HudiTableConfig` and `HudiReadConfig` as python enums [hudi-rs]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #433:
URL: https://github.com/apache/hudi-rs/pull/433#discussion_r2315959517
##########
python/hudi/_internal.pyi:
##########
@@ -15,12 +15,45 @@
# specific language governing permissions and limitations
# under the License.
from dataclasses import dataclass
+from enum import Enum
from typing import Dict, List, Optional, Tuple
import pyarrow # type: ignore
__version__: str
+class HudiTableConfig(Enum):
+ """Configurations for Hudi tables, most of them are persisted in `hoodie.properties`."""
+
+ BASE_FILE_FORMAT = "hoodie.table.base.file.format"
+ BASE_PATH = "hoodie.base.path"
+ CHECKSUM = "hoodie.table.checksum"
+ CREATE_SCHEMA = "hoodie.table.create.schema"
+ DATABASE_NAME = "hoodie.database.name"
+ DROPS_PARTITION_FIELDS = "hoodie.datasource.write.drop.partition.columns"
+ IS_HIVE_STYLE_PARTITIONING = "hoodie.datasource.write.hive_style_partitioning"
+ IS_PARTITION_PATH_URLENCODED = "hoodie.datasource.write.partitionpath.urlencode"
+ KEY_GENERATOR_CLASS = "hoodie.table.keygenerator.class"
+ PARTITION_FIELDS = "hoodie.table.partition.fields"
+ PRECOMBINE_FIELD = "hoodie.table.precombine.field"
+ POPULATES_META_FIELDS = "hoodie.populate.meta.fields"
+ RECORD_KEY_FIELDS = "hoodie.table.recordkey.fields"
+ RECORD_MERGE_STRATEGY = "hoodie.table.record.merge.strategy"
+ TABLE_NAME = "hoodie.table.name"
+ TABLE_TYPE = "hoodie.table.type"
+ TABLE_VERSION = "hoodie.table.version"
+ TIMELINE_LAYOUT_VERSION = "hoodie.timeline.layout.version"
+ TIMELINE_TIMEZONE = "hoodie.table.timeline.timezone"
+
+class HudiReadConfig(Enum):
Review Comment:
the goal is to auto expose the config enums so we don't have to copy the config strings for another language binding. try to research if this is doable with pyo3 to bind python enum to a rust enum without duplicating every config entry.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: export `HudiTableConfig` and `HudiReadConfig` as python enums [hudi-rs]
Posted by "yunchipang (via GitHub)" <gi...@apache.org>.
yunchipang commented on code in PR #433:
URL: https://github.com/apache/hudi-rs/pull/433#discussion_r2324071534
##########
python/hudi/_internal.pyi:
##########
@@ -15,12 +15,45 @@
# specific language governing permissions and limitations
# under the License.
from dataclasses import dataclass
+from enum import Enum
from typing import Dict, List, Optional, Tuple
import pyarrow # type: ignore
__version__: str
+class HudiTableConfig(Enum):
+ """Configurations for Hudi tables, most of them are persisted in `hoodie.properties`."""
+
+ BASE_FILE_FORMAT = "hoodie.table.base.file.format"
+ BASE_PATH = "hoodie.base.path"
+ CHECKSUM = "hoodie.table.checksum"
+ CREATE_SCHEMA = "hoodie.table.create.schema"
+ DATABASE_NAME = "hoodie.database.name"
+ DROPS_PARTITION_FIELDS = "hoodie.datasource.write.drop.partition.columns"
+ IS_HIVE_STYLE_PARTITIONING = "hoodie.datasource.write.hive_style_partitioning"
+ IS_PARTITION_PATH_URLENCODED = "hoodie.datasource.write.partitionpath.urlencode"
+ KEY_GENERATOR_CLASS = "hoodie.table.keygenerator.class"
+ PARTITION_FIELDS = "hoodie.table.partition.fields"
+ PRECOMBINE_FIELD = "hoodie.table.precombine.field"
+ POPULATES_META_FIELDS = "hoodie.populate.meta.fields"
+ RECORD_KEY_FIELDS = "hoodie.table.recordkey.fields"
+ RECORD_MERGE_STRATEGY = "hoodie.table.record.merge.strategy"
+ TABLE_NAME = "hoodie.table.name"
+ TABLE_TYPE = "hoodie.table.type"
+ TABLE_VERSION = "hoodie.table.version"
+ TIMELINE_LAYOUT_VERSION = "hoodie.timeline.layout.version"
+ TIMELINE_TIMEZONE = "hoodie.table.timeline.timezone"
+
+class HudiReadConfig(Enum):
Review Comment:
@xushiyan i added a `macros` crate to auto-gen enums for python using pyo3.. PTAL if this aligns better with the goal. thanks!
##########
python/hudi/_internal.pyi:
##########
@@ -15,12 +15,45 @@
# specific language governing permissions and limitations
# under the License.
from dataclasses import dataclass
+from enum import Enum
from typing import Dict, List, Optional, Tuple
import pyarrow # type: ignore
__version__: str
+class HudiTableConfig(Enum):
+ """Configurations for Hudi tables, most of them are persisted in `hoodie.properties`."""
+
+ BASE_FILE_FORMAT = "hoodie.table.base.file.format"
+ BASE_PATH = "hoodie.base.path"
+ CHECKSUM = "hoodie.table.checksum"
+ CREATE_SCHEMA = "hoodie.table.create.schema"
+ DATABASE_NAME = "hoodie.database.name"
+ DROPS_PARTITION_FIELDS = "hoodie.datasource.write.drop.partition.columns"
+ IS_HIVE_STYLE_PARTITIONING = "hoodie.datasource.write.hive_style_partitioning"
+ IS_PARTITION_PATH_URLENCODED = "hoodie.datasource.write.partitionpath.urlencode"
+ KEY_GENERATOR_CLASS = "hoodie.table.keygenerator.class"
+ PARTITION_FIELDS = "hoodie.table.partition.fields"
+ PRECOMBINE_FIELD = "hoodie.table.precombine.field"
+ POPULATES_META_FIELDS = "hoodie.populate.meta.fields"
+ RECORD_KEY_FIELDS = "hoodie.table.recordkey.fields"
+ RECORD_MERGE_STRATEGY = "hoodie.table.record.merge.strategy"
+ TABLE_NAME = "hoodie.table.name"
+ TABLE_TYPE = "hoodie.table.type"
+ TABLE_VERSION = "hoodie.table.version"
+ TIMELINE_LAYOUT_VERSION = "hoodie.timeline.layout.version"
+ TIMELINE_TIMEZONE = "hoodie.table.timeline.timezone"
+
+class HudiReadConfig(Enum):
Review Comment:
@xushiyan i added a `macros` crate to auto-gen enums for python using pyo3. PTAL if this aligns better with the goal. thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: export `HudiTableConfig` and `HudiReadConfig` as python enums [hudi-rs]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #433:
URL: https://github.com/apache/hudi-rs/pull/433#issuecomment-4348175147
## [Codecov](https://app.codecov.io/gh/apache/hudi-rs/pull/433?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 82.10%. Comparing base ([`41f313c`](https://app.codecov.io/gh/apache/hudi-rs/commit/41f313c86506e5ca8e7d78d9fad5d88400e42bf3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`264cb50`](https://app.codecov.io/gh/apache/hudi-rs/commit/264cb503ca7bd345bcf2ba00f0acbef194ffa249?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 2 commits behind head on main.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #433 +/- ##
==========================================
+ Coverage 82.06% 82.10% +0.03%
==========================================
Files 77 77
Lines 5202 5213 +11
==========================================
+ Hits 4269 4280 +11
Misses 933 933
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/hudi-rs/pull/433?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: export `HudiTableConfig` and `HudiReadConfig` as python enums [hudi-rs]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan merged PR #433:
URL: https://github.com/apache/hudi-rs/pull/433
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org