[I] GCSFileIO: use single-shot upload for small objects below a configurable threshold [iceberg]

["manideep4597 (via GitHub)" <gi...@apache.org>]

manideep4597 opened a new issue, #17524:
URL: https://github.com/apache/iceberg/issues/17524

   ### Feature Request / Improvement
   
   ### Feature Request / Improvement
   
   #### Summary
   `GCSFileIO` / `GCSOutputStream` always uploads via GCS `WriteChannel` (`storage.writer(...)`), which uses **resumable uploads**. For small objects this adds unnecessary round-trips and latency compared to a single-shot / simple upload (`storage.create(...)`).
   
   `S3FileIO` already switches between single `PutObject` and multipart based on size (`s3.multipart.part-size-bytes`, `s3.multipart.threshold`). GCS has no equivalent.
   
   #### Motivation / Use case
   Iceberg writes many relatively small objects (manifests, metadata JSON, delete files, etc.). For these, resumable-upload setup cost often dominates transfer time.
   
   Google Cloud Storage Java guidance also recommends:
   - small content → `storage.create(blobInfo, bytes)` (single request)
   - large / streaming content → `storage.writer(...)` (resumable)
   
   Today Iceberg always takes the resumable path, which is suboptimal for small files and creates a feature gap vs S3.
   
   #### Current behavior
   In `GCSOutputStream`, uploads always open a write channel:
   
   ```java
   WriteChannel channel =
       storage.writer(
           BlobInfo.newBuilder(blobId).build(), writeOptions.toArray(new BlobWriteOption[0]));
   gcpProperties.channelWriteChunkSize().ifPresent(channel::setChunkSize);
   stream = Channels.newOutputStream(channel);
   
   ### Query engine
   
   _No response_
   
   ### Willingness to contribute
   
   - [ ] I can contribute this improvement/feature independently
   - [x] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
   - [ ] I cannot contribute this improvement/feature at this time


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [I] GCSFileIO: use single-shot upload for small objects below a configurable threshold [iceberg]

["piyushka-ally (via GitHub)" <gi...@apache.org>]

piyushka-ally commented on issue #17524:
URL: https://github.com/apache/iceberg/issues/17524#issuecomment-5354340867

   Can you kindly assign this to me.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org