[I] feat(storage): write pipeline robustness and edge-case handling for object_store backend [iceberg-rust]

["Sruhvx-jpg (via GitHub)" <gi...@apache.org>]

Sruhvx-jpg opened a new issue, #3166:
URL: https://github.com/apache/iceberg-rust/issues/3166

   ### Problem / Context
   
   In #3165 (reviving #2257 for #2259), the foundational `ObjectStoreStorage::S3` backend is established. To ensure production robustness and performance in the write pipeline, several edge cases and optimizations should be addressed as follow-up work:
   
   ### Proposed Improvements
   
   1. **Zero-Copy Writes in `ObjectStoreWriter`**:
      - `FileWrite::write` receives owned `bytes::Bytes`.
      - `object_store::WriteMultipart` provides a `put(Bytes)` method that buffers without slice copying, avoiding redundant `extend_from_slice` allocations.
   
   2. **S3 URL Boundary Validation**:
      - Empty bucket strings (e.g. `s3:///path/to/file` or `s3://`) currently pass initial URL host parsing as `""`. These should explicitly return `ErrorKind::DataInvalid`.
      - Ensure paths without trailing slashes (`s3://bucket`) extract `bucket` and empty relative path cleanly.
   
   3. **Stream Handling & State Machine Guards**:
      - Validate `delete_stream` on empty streams (`futures::stream::empty()`) and ensure immediate error propagation on invalid URLs.
      - Add unit test coverage for `ObjectStoreWriter` state guards (e.g., verifying `close()` and `write()` error handling after writer has been closed).
   
   4. **Backpressure & Memory Throttling**:
      - For high-throughput streaming, evaluate `WriteMultipart::wait_for_capacity` to throttle producers and prevent unbounded 5MB chunk allocations in memory ahead of S3 network uploads.
   
   ### Related
   - #2258 (Umbrella: Implement Storage using object_store)
   - #2259 (Implement ObjectStoreStorage::S3)
   - #3165 (feat(storage): Implement ObjectStoreStorage::S3)
   
   ## AI Disclosure
   Drafted with AI assistance; reviewed and verified by contributor.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org