You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "abyssnlp (via GitHub)" <gi...@apache.org> on 2024/07/31 21:10:09 UTC
[PR] fix:register object store with datafusion [hudi-rs]
abyssnlp opened a new pull request, #107:
URL: https://github.com/apache/hudi-rs/pull/107
## Description
Reading Hudi tables from an object store, s3 in this case, was failing due to the object store not being registered with Datafusion.
Before the changes in this PR, the following error was encountered:
```
Error: Internal("No suitable object store found for s3://test-hudi-rs/. See `RuntimeEnv::register_object_store`")
```
Datafusion requires the object store to be registered. Refer [here](https://github.com/apache/datafusion/blob/fa50636c6aec0f321212ed8678c77de25b71a8f9/datafusion/core/src/execution/context/mod.rs#L375).
## Note
- `Storage` and `FileSystemView` have public struct fields `object_store` and `storage`. There should be a better way to expose these to the `datafusion` crate to be able to register them.
## How are the changes test-covered
- [x] N/A
- [ ] Automated tests (unit and/or integration tests)
- [ ] Manual tests
- [ ] Details are described below
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: register object store with datafusion [hudi-rs]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan merged PR #107:
URL: https://github.com/apache/hudi-rs/pull/107
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: register object store with datafusion [hudi-rs]
Posted by "abyssnlp (via GitHub)" <gi...@apache.org>.
abyssnlp commented on PR #107:
URL: https://github.com/apache/hudi-rs/pull/107#issuecomment-2262167780
This change should make it okay to proceed with #81
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: register object store with datafusion [hudi-rs]
Posted by "abyssnlp (via GitHub)" <gi...@apache.org>.
abyssnlp commented on PR #107:
URL: https://github.com/apache/hudi-rs/pull/107#issuecomment-2264626163
Possible fix for #108
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: register object store with datafusion [hudi-rs]
Posted by "abyssnlp (via GitHub)" <gi...@apache.org>.
abyssnlp commented on PR #107:
URL: https://github.com/apache/hudi-rs/pull/107#issuecomment-2283195868
@xushiyan I verified it for 3 scenarios and it works as expected.
- `HudiDataSource` with AWS S3 (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` , `AWS_DEFAULT_REGION` in the env vars)
- `HudiDataSource` with local FS
- `HudiDataSource` with minio (requires the AWS env var `AWS_ENDPOINT` and `ALLOW_HTTP`)
```rust
let hudi = HudiDataSource::new_with_options(
"s3://test-bucket/v6_nonpartitioned",
vec![
("allow_http", "true"),
],
)
.await?;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: register object store with datafusion [hudi-rs]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #107:
URL: https://github.com/apache/hudi-rs/pull/107#issuecomment-2282340858
## [Codecov](https://app.codecov.io/gh/apache/hudi-rs/pull/107?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 87.39%. Comparing base [(`7566337`)](https://app.codecov.io/gh/apache/hudi-rs/commit/75663379bf3cbf794af2fcf3823045f07851754c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`f744c34`)](https://app.codecov.io/gh/apache/hudi-rs/commit/f744c343e458357835f9730225fbab7b127c5186?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> Report is 2 commits behind head on main.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #107 +/- ##
==========================================
+ Coverage 87.24% 87.39% +0.14%
==========================================
Files 14 14
Lines 690 698 +8
==========================================
+ Hits 602 610 +8
Misses 88 88
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/hudi-rs/pull/107?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: register object store with datafusion [hudi-rs]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on PR #107:
URL: https://github.com/apache/hudi-rs/pull/107#issuecomment-2282362964
@abyssnlp thanks for the fix. refactored a bit. can you verify this change e2e in your env please? would love to unblock the minio integ test!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org