You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "brayanjuls (via GitHub)" <gi...@apache.org> on 2025/08/04 02:47:49 UTC
[PR] feat: expose datafusion custom table provider to python [hudi-rs]
brayanjuls opened a new pull request, #412:
URL: https://github.com/apache/hudi-rs/pull/412
## Description
This work seeks to expose the existing datafusion datasource to the python api, with the goal of being able to use dafusion in python hudi-rs. To implement this we leverage datafusion FFI custom table provider as mentioned in this [doc](https://github.com/apache/datafusion-python/blob/main/docs/source/user-guide/io/table_provider.rst).
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [ ] Manual tests
- [ ] Details are described below
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: expose datafusion custom table provider to python [hudi-rs]
Posted by "brayanjuls (via GitHub)" <gi...@apache.org>.
brayanjuls commented on code in PR #412:
URL: https://github.com/apache/hudi-rs/pull/412#discussion_r2271948810
##########
python/src/internal.rs:
##########
@@ -239,6 +251,48 @@ impl From<&Instant> for HudiInstant {
}
}
+#[cfg(not(tarpaulin_include))]
+#[pyclass(name = "HudiDataSource")]
+#[derive(Clone)]
+pub struct HudiDataSource {
+ table: InternalHudiDataSource,
Review Comment:
I renamed HudiDataSource to HudiDataFusionDataSource, also make datafusion dependency optional in the pyproject.toml so people can use `pip install hudi[datafusion]`.
Additionally I extracted HudiDataFusionDataSource(datafusion_internal.rs) to a different module so we can enable or disable it with rust feature flags.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: expose datafusion custom table provider to python [hudi-rs]
Posted by "brayanjuls (via GitHub)" <gi...@apache.org>.
brayanjuls commented on code in PR #412:
URL: https://github.com/apache/hudi-rs/pull/412#discussion_r2271951498
##########
python/hudi/_internal.pyi:
##########
@@ -349,3 +349,18 @@ def build_hudi_table(
HudiTable: An instance of hudi table.
"""
...
+@dataclass(init=False)
+class HudiDataSource:
Review Comment:
I tried to do this but got some errors, so I think I will try to do it in a different PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: expose datafusion custom table provider to python [hudi-rs]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on PR #412:
URL: https://github.com/apache/hudi-rs/pull/412#issuecomment-3214703327
> @xushiyan do you need any additional changes from me in this PR?
being away for some work stuff. LGTM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: expose datafusion custom table provider to python [hudi-rs]
Posted by "brayanjuls (via GitHub)" <gi...@apache.org>.
brayanjuls commented on PR #412:
URL: https://github.com/apache/hudi-rs/pull/412#issuecomment-3148975201
hi @xushiyan, I don't have an issue associated for this PR if you need it I can go ahead and create one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: expose datafusion custom table provider to python [hudi-rs]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #412:
URL: https://github.com/apache/hudi-rs/pull/412#issuecomment-3151810817
## [Codecov](https://app.codecov.io/gh/apache/hudi-rs/pull/412?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `4 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 88.79%. Comparing base ([`0eea341`](https://app.codecov.io/gh/apache/hudi-rs/commit/0eea3411cd374f1b1e1c0f5a5a9597ed39749424?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`7e30e1e`](https://app.codecov.io/gh/apache/hudi-rs/commit/7e30e1e83bd88e106d20b8ffee702df6da8f683c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi-rs/pull/412?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [python/src/internal.rs](https://app.codecov.io/gh/apache/hudi-rs/pull/412?src=pr&el=tree&filepath=python%2Fsrc%2Finternal.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHl0aG9uL3NyYy9pbnRlcm5hbC5ycw==) | 0.00% | [4 Missing :warning: ](https://app.codecov.io/gh/apache/hudi-rs/pull/412?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #412 +/- ##
==========================================
- Coverage 88.89% 88.79% -0.10%
==========================================
Files 49 49
Lines 2711 2714 +3
==========================================
Hits 2410 2410
- Misses 301 304 +3
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/hudi-rs/pull/412?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: expose datafusion custom table provider to python [hudi-rs]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on PR #412:
URL: https://github.com/apache/hudi-rs/pull/412#issuecomment-3151791447
> hi @xushiyan, I don't have an issue associated for this PR if you need it I can go ahead and create one.
@brayanjuls thanks for the PR! It's alright we don't need to have an issue for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: expose datafusion custom table provider to python [hudi-rs]
Posted by "brayanjuls (via GitHub)" <gi...@apache.org>.
brayanjuls commented on PR #412:
URL: https://github.com/apache/hudi-rs/pull/412#issuecomment-3152824593
I see the overall coverage decreases because of my changes on internal.rs, do I need to add unit testing there or it is ok to only have unit testing in python?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: expose datafusion custom table provider to python [hudi-rs]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on PR #412:
URL: https://github.com/apache/hudi-rs/pull/412#issuecomment-3164838889
> I see the overall coverage decreases because of my changes on internal.rs, do I need to add unit testing there or it is ok to only have unit testing in python?
@brayanjuls we evaluate the coverage case by case. if it's non-critical, and the CI shows green, then we're good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: expose datafusion custom table provider to python [hudi-rs]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #412:
URL: https://github.com/apache/hudi-rs/pull/412#discussion_r2264340257
##########
python/src/internal.rs:
##########
@@ -239,6 +251,48 @@ impl From<&Instant> for HudiInstant {
}
}
+#[cfg(not(tarpaulin_include))]
+#[pyclass(name = "HudiDataSource")]
+#[derive(Clone)]
+pub struct HudiDataSource {
+ table: InternalHudiDataSource,
Review Comment:
in rust, we have hudi-datafusion crate, so HudiDataSource is meant for datafusion.
In python, we need to make the name explicit for datafusion so people don't confuse it with datasource for other integrations (in future).
Also can we make all related functionalities in this PR available only when the python `datafusion` feature is enabled? like people use `pip install hudi[datafusion]` to access the related features.
##########
python/hudi/_internal.pyi:
##########
@@ -349,3 +349,18 @@ def build_hudi_table(
HudiTable: An instance of hudi table.
"""
...
+@dataclass(init=False)
+class HudiDataSource:
Review Comment:
would love to see this refactored into another `.pyi` for better code organization. it'll be optional here and you can do it separately if you like.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: expose datafusion custom table provider to python [hudi-rs]
Posted by "brayanjuls (via GitHub)" <gi...@apache.org>.
brayanjuls commented on PR #412:
URL: https://github.com/apache/hudi-rs/pull/412#issuecomment-3214587004
@xushiyan do you need any additional changes from me in this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: expose datafusion custom table provider to python [hudi-rs]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan merged PR #412:
URL: https://github.com/apache/hudi-rs/pull/412
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org