You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "yihua (via GitHub)" <gi...@apache.org> on 2026/09/04 03:21:43 UTC
[PR] feat(benchmark): add AWS EC2 setup for the TPC-H harness [hudi-rs]
yihua opened a new pull request, #718:
URL: https://github.com/apache/hudi-rs/pull/718
## Description
The TPC-H harness could only be driven from a GCP VM against GCS. This adds the AWS counterpart, so either cloud is a documented path: `bootstrap.sh` and `sync.sh` under `benchmark/tpch/infra/aws` mirroring the GCP pair, an `s3://` filesystem mapping for the Spark leg (Hadoop 3 dropped that scheme, so it needs routing to S3A), and a README section alongside the GCP one.
Two details that are easy to get wrong on AWS and are handled here: `object_store` takes no region from instance metadata and falls back to `us-east-1`, so bootstrap persists `AWS_REGION`; and the instance store is wiped on stop/start, so the mount step sits outside the script's sentinel guard while the package installs stay guarded.
Also creates `$SPARK_HOME/conf` before copying configs into it. A pip-installed PySpark ships no `conf` directory, so `create-tables` failed there, which is how the missing directory surfaced.
## How are the changes test-covered
- [ ] N/A
- [ ] Automated tests (unit and/or integration tests)
- [x] Manual tests
- [x] Details are described below
Ran the full SF1 pipeline (`generate`, `create-tables`, both engines, `compare`) on macOS with Corretto 17 and a pip-installed PySpark 3.5.8, which is the configuration that exposed the missing `conf` directory: datafusion+hudi-rs 1840.9 ms total / 74.9 ms geomean vs spark+hudi 32708.3 ms / 1254.5 ms across the 22 queries. Re-ran both engines afterwards starting from a deleted `$SPARK_HOME/conf` to confirm the fix, and checked both new scripts with `bash -n` and shellcheck.
The AWS scripts themselves are not exercised by this run; they are scheduled to be used for an SF100 comparison on EC2 next.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(benchmark): add AWS EC2 setup for the TPC-H harness [hudi-rs]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #718:
URL: https://github.com/apache/hudi-rs/pull/718#issuecomment-5535363944
## [Codecov](https://app.codecov.io/gh/apache/hudi-rs/pull/718?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 79.53%. Comparing base ([`922a62c`](https://app.codecov.io/gh/apache/hudi-rs/commit/922a62cd0ae9af6d0737dd70c4c786ac0de542ff?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`f3c0926`](https://app.codecov.io/gh/apache/hudi-rs/commit/f3c092607292b2374e4f13301ff69e23dfb0c422?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #718 +/- ##
=======================================
Coverage 79.53% 79.53%
=======================================
Files 126 126
Lines 12301 12301
=======================================
Hits 9783 9783
Misses 2518 2518
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/hudi-rs/pull/718?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(benchmark): add AWS EC2 setup for the TPC-H harness [hudi-rs]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #718:
URL: https://github.com/apache/hudi-rs/pull/718
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org