[GitHub] [iceberg-rust] Xuanwo commented on issue #34: Writer Design

["Xuanwo (via GitHub)" <gi...@apache.org>]

Xuanwo commented on issue #34:
URL: https://github.com/apache/iceberg-rust/issues/34#issuecomment-1684854751

   Most of the writing tasks are handled by @liurenjie1024 and @ZENOTME, so I may not have sufficient experience on this particular topic. However, I will do my best to provide input from the OpenDAL perspective to assist with the design.
   
   ---
   
   When we talking about `writer`, we are talking about two different things: One for the IO writer, one for the Value writer. 
   
   - IO Writer handles the underlying IO operations, a.k.a, `write(&[u8])` into storage.
   - Value Writer handles the upper value operations, a.k.a, `write(record_batches)`.
   
   For IO Writer, there are three kinds of APIs we can use:
   
   - PutObject: write all content at once. a.k.a: `op.write(path, bs)` or `op.writer_with(path).content_length(size)`
   - MultipartUpload: allow write content in multipart, we will need to buffer some content in-memory, and write to storage. a.k.a: `op.writer(path)`.
   - AppendObject: allow append upon object (few storage support this). a.k.a: `op.writer(path).append(true)`.
   
   ## PutObject
   
   - Good: fast and cheap, only one API call
   - Bad:
     - need all content inside memory, not subiable for large content.
     - need to know the actual size before uploading.
   
   ## MultipartUpload
   
   - Good: require at most 8MiB in memory.
   - Bad: need (size/8MiB) + 2 API Call. slow, and more expensive.
   
   ## AppendObject
   
   - Good: require at most 8MiB in memory, no extra API call.
   - Bad: Only few service support this, and AWS S3 doesn't support.
   
   I suggest we can support two ways of IO Writer: `PutObject` and `MultipartUpload`. Both of them can be constructed as `AsyncWriter`. This IO writer should be injected in to `SpecificWriter` so they and handile those logic internally.
   
   We should not handle them differently based on format. Maybe we can add a internal buffer inside avro writer so that it can accept the same writer as parquet writer does?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org