You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "crrow (via GitHub)" <gi...@apache.org> on 2024/07/24 15:51:29 UTC
[PR] Feat/run ci on windows [hudi-rs]
crrow opened a new pull request, #92:
URL: https://github.com/apache/hudi-rs/pull/92
## Description
<!--- Describe your changes in detail -->
Use cargo-binstall to install tarpaulin.
<!--- If it fixes an open issue, please link to the issue here. -->
https://github.com/apache/hudi-rs/issues/90
<!--- Please link any related issues and PRs as well. -->
## How are the changes test-covered
- [x] N/A
- [ ] Automated tests (unit and/or integration tests)
- [ ] Manual tests
- [ ] Details are described below
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ci: run rust-tests on windows [hudi-rs]
Posted by "crrow (via GitHub)" <gi...@apache.org>.
crrow commented on code in PR #92:
URL: https://github.com/apache/hudi-rs/pull/92#discussion_r1691583734
##########
.github/workflows/ci.yml:
##########
@@ -44,14 +44,16 @@ jobs:
strategy:
fail-fast: false
matrix:
- # TODO: add windows which does not support container
- os: [ ubuntu-22.04 ]
+ os:
+ - ubuntu-22.04
+ - windows-latest
Review Comment:
Pinned the Windows version to 2022, let's test it out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ci: run rust-tests on windows [hudi-rs]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #92:
URL: https://github.com/apache/hudi-rs/pull/92#discussion_r1712920994
##########
.github/workflows/ci.yml:
##########
@@ -53,14 +53,15 @@ jobs:
strategy:
fail-fast: false
matrix:
- # TODO: add windows which does not support container
- os: [ ubuntu-22.04 ]
+ os:
+ - ubuntu-22.04
+ - windows-2022
runs-on: ${{ matrix.os }}
- container:
- image: xd009642/tarpaulin:0.30.0
- options: --security-opt seccomp=unconfined
steps:
- uses: actions/checkout@v4
+ - uses: cargo-bins/cargo-binstall@main
+ - name: Install tarpaulin
+ run: cargo binstall -y cargo-tarpaulin
Review Comment:
@crrow would you do a bit of research on how windows paths are usually handled in consideration of scheme? we want to be flexible in supporting paths - it can start with `C:\`, `file:///`, `/`, `s3://` as these are valid paths anyway.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ci: run rust-tests on windows [hudi-rs]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #92:
URL: https://github.com/apache/hudi-rs/pull/92#discussion_r1690708670
##########
.github/workflows/ci.yml:
##########
@@ -44,14 +44,16 @@ jobs:
strategy:
fail-fast: false
matrix:
- # TODO: add windows which does not support container
- os: [ ubuntu-22.04 ]
+ os:
+ - ubuntu-22.04
+ - windows-latest
Review Comment:
should the windows os version be fixed a certain version?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ci: run rust-tests on windows [hudi-rs]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #92:
URL: https://github.com/apache/hudi-rs/pull/92#discussion_r1695691238
##########
.github/workflows/ci.yml:
##########
@@ -53,14 +53,15 @@ jobs:
strategy:
fail-fast: false
matrix:
- # TODO: add windows which does not support container
- os: [ ubuntu-22.04 ]
+ os:
+ - ubuntu-22.04
+ - windows-2022
runs-on: ${{ matrix.os }}
- container:
- image: xd009642/tarpaulin:0.30.0
- options: --security-opt seccomp=unconfined
steps:
- uses: actions/checkout@v4
+ - uses: cargo-bins/cargo-binstall@main
+ - name: Install tarpaulin
+ run: cargo binstall -y cargo-tarpaulin
Review Comment:
> @xushiyan Do we have to handle this [case?](https://github.com/apache/hudi-rs/blob/main/crates/core/src/storage/utils.rs#L94) , its tricky for windows
@crrow that path is a unix path which should not be applicable to windows os. Can we split that case to test case which does not run on windows, and also add a windows specific test case with an example windows path ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ci: run rust-tests on windows [hudi-rs]
Posted by "crrow (via GitHub)" <gi...@apache.org>.
crrow commented on PR #92:
URL: https://github.com/apache/hudi-rs/pull/92#issuecomment-2254097080
> > @crrow i suspect your previous approach using binstall could work but the github action was not whitelisted by asf so the whole workflow is not triggered. if you can verify that binstall approach in your own fork and get CI to pass, then i can reach out to have that white listed.
>
> ok, i'll check it out later.
Hello @xushiyan , I setup the workflow, have a check when you are free.
https://github.com/crrow/hudi-rs/actions/runs/10122608881/job/27994952767
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ci: run rust-tests on windows [hudi-rs]
Posted by "crrow (via GitHub)" <gi...@apache.org>.
crrow commented on code in PR #92:
URL: https://github.com/apache/hudi-rs/pull/92#discussion_r1698417775
##########
.github/workflows/ci.yml:
##########
@@ -53,14 +53,15 @@ jobs:
strategy:
fail-fast: false
matrix:
- # TODO: add windows which does not support container
- os: [ ubuntu-22.04 ]
+ os:
+ - ubuntu-22.04
+ - windows-2022
runs-on: ${{ matrix.os }}
- container:
- image: xd009642/tarpaulin:0.30.0
- options: --security-opt seccomp=unconfined
steps:
- uses: actions/checkout@v4
+ - uses: cargo-bins/cargo-binstall@main
+ - name: Install tarpaulin
+ run: cargo binstall -y cargo-tarpaulin
Review Comment:
Oh...I found that its still tricky, like "C:\\foo", `url::parse` will success, it thoughts 'c' is the schema, rather than a filepath; for "\\foo", `url::parse` and `url::from_filepath` will all fail; We can handle both cases by adding additional check like if current os is windows and the given str has ":\\".
Can we be stricter about what qualifies as a 'valid URL'? Like a valid URL must have a schema.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ci: run rust-tests on windows [hudi-rs]
Posted by "crrow (via GitHub)" <gi...@apache.org>.
crrow commented on code in PR #92:
URL: https://github.com/apache/hudi-rs/pull/92#discussion_r1698298633
##########
.github/workflows/ci.yml:
##########
@@ -53,14 +53,15 @@ jobs:
strategy:
fail-fast: false
matrix:
- # TODO: add windows which does not support container
- os: [ ubuntu-22.04 ]
+ os:
+ - ubuntu-22.04
+ - windows-2022
runs-on: ${{ matrix.os }}
- container:
- image: xd009642/tarpaulin:0.30.0
- options: --security-opt seccomp=unconfined
steps:
- uses: actions/checkout@v4
+ - uses: cargo-bins/cargo-binstall@main
+ - name: Install tarpaulin
+ run: cargo binstall -y cargo-tarpaulin
Review Comment:
Sure, I'd love to
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ci: run rust-tests on windows [hudi-rs]
Posted by "crrow (via GitHub)" <gi...@apache.org>.
crrow commented on PR #92:
URL: https://github.com/apache/hudi-rs/pull/92#issuecomment-2254003623
> @crrow i suspect your previous approach using binstall could work but the github action was not whitelisted by asf so the whole workflow is not triggered. if you can verify that binstall approach in your own fork and get CI to pass, then i can reach out to have that white listed.
ok, i'll check it out later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ci: run rust-tests on windows [hudi-rs]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan merged PR #92:
URL: https://github.com/apache/hudi-rs/pull/92
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ci: run rust-tests on windows [hudi-rs]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on PR #92:
URL: https://github.com/apache/hudi-rs/pull/92#issuecomment-2252127160
@crrow i suspect your previous approach using binstall could work but the github action was not whitelisted by asf so the whole workflow is not triggered. if you can verify that binstall approach in your own fork and get CI to pass, then i can reach out to have that white listed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ci: run rust-tests on windows [hudi-rs]
Posted by "crrow (via GitHub)" <gi...@apache.org>.
crrow commented on code in PR #92:
URL: https://github.com/apache/hudi-rs/pull/92#discussion_r1694049605
##########
.github/workflows/ci.yml:
##########
@@ -53,14 +53,15 @@ jobs:
strategy:
fail-fast: false
matrix:
- # TODO: add windows which does not support container
- os: [ ubuntu-22.04 ]
+ os:
+ - ubuntu-22.04
+ - windows-2022
runs-on: ${{ matrix.os }}
- container:
- image: xd009642/tarpaulin:0.30.0
- options: --security-opt seccomp=unconfined
steps:
- uses: actions/checkout@v4
+ - uses: cargo-bins/cargo-binstall@main
+ - name: Install tarpaulin
+ run: cargo binstall -y cargo-tarpaulin
Review Comment:
What's your thought on separator and test case? Some test failed because of rust's automatic normalize, and URL parse failed since some test cases are Linux path, and it cannot be parsed on Windows.
By the way, I think some test cases can be rewritten with https://crates.io/crates/test-case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ci: run rust-tests on windows [hudi-rs]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #92:
URL: https://github.com/apache/hudi-rs/pull/92#discussion_r1693990015
##########
.github/workflows/ci.yml:
##########
@@ -53,14 +53,15 @@ jobs:
strategy:
fail-fast: false
matrix:
- # TODO: add windows which does not support container
- os: [ ubuntu-22.04 ]
+ os:
+ - ubuntu-22.04
+ - windows-2022
runs-on: ${{ matrix.os }}
- container:
- image: xd009642/tarpaulin:0.30.0
- options: --security-opt seccomp=unconfined
steps:
- uses: actions/checkout@v4
+ - uses: cargo-bins/cargo-binstall@main
+ - name: Install tarpaulin
+ run: cargo binstall -y cargo-tarpaulin
Review Comment:
@crrow using `cargo-bins/cargo-binstall` works! The job fails due to some test failure - can you look into and fix any code if needed? meanwhile i'll get binstall whitelist so it can be triggered here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ci: run rust-tests on windows [hudi-rs]
Posted by "crrow (via GitHub)" <gi...@apache.org>.
crrow commented on code in PR #92:
URL: https://github.com/apache/hudi-rs/pull/92#discussion_r1694252699
##########
.github/workflows/ci.yml:
##########
@@ -53,14 +53,15 @@ jobs:
strategy:
fail-fast: false
matrix:
- # TODO: add windows which does not support container
- os: [ ubuntu-22.04 ]
+ os:
+ - ubuntu-22.04
+ - windows-2022
runs-on: ${{ matrix.os }}
- container:
- image: xd009642/tarpaulin:0.30.0
- options: --security-opt seccomp=unconfined
steps:
- uses: actions/checkout@v4
+ - uses: cargo-bins/cargo-binstall@main
+ - name: Install tarpaulin
+ run: cargo binstall -y cargo-tarpaulin
Review Comment:
@xushiyan Do we have to handle this [case?](https://github.com/apache/hudi-rs/blob/main/crates/core/src/storage/utils.rs#L94) , its tricky for windows
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ci: run rust-tests on windows [hudi-rs]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #92:
URL: https://github.com/apache/hudi-rs/pull/92#discussion_r1694051699
##########
.github/workflows/ci.yml:
##########
@@ -53,14 +53,15 @@ jobs:
strategy:
fail-fast: false
matrix:
- # TODO: add windows which does not support container
- os: [ ubuntu-22.04 ]
+ os:
+ - ubuntu-22.04
+ - windows-2022
runs-on: ${{ matrix.os }}
- container:
- image: xd009642/tarpaulin:0.30.0
- options: --security-opt seccomp=unconfined
steps:
- uses: actions/checkout@v4
+ - uses: cargo-bins/cargo-binstall@main
+ - name: Install tarpaulin
+ run: cargo binstall -y cargo-tarpaulin
Review Comment:
@crrow absolutely! we should separate test code fix from CI setup change. Feel free to create another PR for the test fix wrt url parsing. You may also make the code change here so we know the test fix can pass with the CI, then separate it out. It's up to you.
I've thought about using some test helper lib but not sure if https://github.com/frondeus/test-case is the best candidate. there are many crates maintained by community like https://github.com/la10736/rstest seems more popular with more recent releases. I'll create an issue about this for future refactoring work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ci: run rust-tests on windows [hudi-rs]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #92:
URL: https://github.com/apache/hudi-rs/pull/92#issuecomment-3367481697
## [Codecov](https://app.codecov.io/gh/apache/hudi-rs/pull/92?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 88.94%. Comparing base ([`8b8451a`](https://app.codecov.io/gh/apache/hudi-rs/commit/8b8451a57aff308a071bd06e05dc77377c879f77?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`e10aef1`](https://app.codecov.io/gh/apache/hudi-rs/commit/e10aef1c05c2961c32c3cb5ec74355bc1726fc2c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #92 +/- ##
=======================================
Coverage 88.94% 88.94%
=======================================
Files 49 49
Lines 2704 2704
=======================================
Hits 2405 2405
Misses 299 299
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/hudi-rs/pull/92?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] ci: run rust-tests on windows [hudi-rs]
Posted by "crrow (via GitHub)" <gi...@apache.org>.
crrow commented on code in PR #92:
URL: https://github.com/apache/hudi-rs/pull/92#discussion_r1694049059
##########
.github/workflows/ci.yml:
##########
@@ -53,14 +53,15 @@ jobs:
strategy:
fail-fast: false
matrix:
- # TODO: add windows which does not support container
- os: [ ubuntu-22.04 ]
+ os:
+ - ubuntu-22.04
+ - windows-2022
runs-on: ${{ matrix.os }}
- container:
- image: xd009642/tarpaulin:0.30.0
- options: --security-opt seccomp=unconfined
steps:
- uses: actions/checkout@v4
+ - uses: cargo-bins/cargo-binstall@main
+ - name: Install tarpaulin
+ run: cargo binstall -y cargo-tarpaulin
Review Comment:
@xushiyan I'd love to, but I'd prefer to address the Windows code in a separate pull request.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org