You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@sedona.apache.org by "prantogg (via GitHub)" <gi...@apache.org> on 2026/01/03 08:06:48 UTC
[PR] [WIP] feat: add s3 write support [sedona-spatialbench]
prantogg opened a new pull request, #71:
URL: https://github.com/apache/sedona-spatialbench/pull/71
This PR enables direct S3 streaming for generated data, eliminating local storage requirements for large-scale dataset generation. S3 URIs in `--output-dir` are automatically detected and routed to a streaming multipart upload implementation.
```bash
spatialbench --scale-factor 1000 --output-dir s3://my-bucket/spatialbench/sf1000
```
- Implements streaming S3 writer with `Write` trait compatibility.
- Uses S3 multipart upload API with 32MB parts for efficient streaming. Small files (<5MB) use direct PUT requests.
- Multipart completion handled asynchronously via Tokio runtime. Built on `object_store` crate's `AmazonS3Builder`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add S3 write support [sedona-spatialbench]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on PR #71:
URL: https://github.com/apache/sedona-spatialbench/pull/71#issuecomment-3892922252
@prantogg Does this work with the `--mb-per-file 256` config?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add S3 write support [sedona-spatialbench]
Posted by "prantogg (via GitHub)" <gi...@apache.org>.
prantogg commented on PR #71:
URL: https://github.com/apache/sedona-spatialbench/pull/71#issuecomment-3893178420
> @prantogg Does this work with the `--mb-per-file 256` config?
@jiayuasu Yes, it works with all CLI options including `--mb-per-file`. Data is just streamed to S3 via multipart upload instead of written to disk.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add S3 write support [sedona-spatialbench]
Posted by "prantogg (via GitHub)" <gi...@apache.org>.
prantogg commented on PR #71:
URL: https://github.com/apache/sedona-spatialbench/pull/71#issuecomment-3893048841
> Thank you!
>
> I don't think it is necessary for this PR, but the `ObjectStore` may be a good future choice (here it would let you unify the local and s3 code path and make it easier to add future gcs or azure support).
>
> It may be worth setting up a test for this in CI using MinIO or another S3 alternative as otherwise I don't think there are any tests here to validate the behaviour.
Thanks @paleolimbot! Agreed on `ObjectStore` for local path. The current implementation for local path is an exact copy of tpchgen-rs project. Happy to tackle that as a follow-up!
On the MinIO CI test - my thinking was that since `S3Writer` implements `std::io::Write`, the datagen pipeline is agnostic to the destination. The unit tests validate the writer logic (buffering, multipart, finalization) against an InMemory object store, which covers our code. Testing against an actual S3 endpoint would mostly be validating `object_store::aws::AmazonS3Builder`'s backend rather than ours?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add S3 write support [sedona-spatialbench]
Posted by "prantogg (via GitHub)" <gi...@apache.org>.
prantogg commented on PR #71:
URL: https://github.com/apache/sedona-spatialbench/pull/71#issuecomment-3893161600
Adding a small fix for S3 error propagation for things like token expiry.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add S3 write support [sedona-spatialbench]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on PR #71:
URL: https://github.com/apache/sedona-spatialbench/pull/71#issuecomment-3893511163
> Testing against an actual S3 endpoint would mostly be validating not our code
There's nothing in this PR to indicate that `spatialbench-cli -s 1 --format=parquet --output-dir s3://my-bucket/sf1-parquet` actually runs (and nothing to ensure that future LLMs or contributors won't break it). I call this an "is it plugged in check" and I while I think future you will be glad you added it now, the consequences of this not working are very low (and not my problem 🙂 ) so feel free to punt!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add S3 write support [sedona-spatialbench]
Posted by "prantogg (via GitHub)" <gi...@apache.org>.
prantogg commented on PR #71:
URL: https://github.com/apache/sedona-spatialbench/pull/71#issuecomment-3893916401
> There's nothing in this PR to indicate that `spatialbench-cli -s 1 --format=parquet --output-dir s3://my-bucket/sf1-parquet` actually runs (and nothing to ensure that future LLMs or contributors won't break it). I call this an "is it plugged in check" and I while I think future you will be glad you added it now, the consequences of this not working are very low (and not my problem 🙂 ) so feel free to punt!
I see your point @paleolimbot! Thanks for that. Just added MinIO tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add S3 write support [sedona-spatialbench]
Posted by "prantogg (via GitHub)" <gi...@apache.org>.
prantogg merged PR #71:
URL: https://github.com/apache/sedona-spatialbench/pull/71
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org