[I] new feature: Expose API to configure chunk size for java binding [opendal]

["BaurzhanSakhariev (via GitHub)" <gi...@apache.org>]

BaurzhanSakhariev opened a new issue, #5421:
URL: https://github.com/apache/opendal/issues/5421

   ### Feature Description
   
   Ability to configure chunk size for `Operator` 
   
   ### Problem and Solution
   
   `OperatorOutputStream` uses hardcoded chunk size which leads to `BlockCountExceedsLimit` error when uploading a large file using azblob service. Increasing chunk size helps to have fewer blocks and avoid this error. 
   
   Also, upload time decreased significantly with buffer size 10x of the current size.
   
   
   `OperatorOutputStream` uses hardcoded chunk size which leads to `BlockCountExceedsLimit` error when uploading a large file using azblob service.
   
   
   I used a test
   ```
   @Test
   public void testLargeFile() {
     final String path = UUID.randomUUID().toString();
     final Random random = new Random();
     final int size = 16384 * 10; // 10 x OperatorOutputStream.MAX_BYTES (10 flushes per write)
     final byte[] content = new byte[size];
     random.nextBytes(content);
     try (Operator op = op(); OperatorOutputStream operatorOutputStream = op.createOutputStream(path)) {
         for (int i = 0; i < 20000; i++) { // More iterations in case BlockCountExceedsLimit doesn't pop up exactly after 100K blocks.
             operatorOutputStream.write(content);
         }
     } catch (IOException e) {
         throw new RuntimeException(e);
     }
   }
   ```
   against a real Azure container and got an error after exactly 100K iterations. 
   
   Then manually changed OperatorOutputStream.MAX_BYTES to make it 10x and tests passed (and confirmed that file uploaded to the container).
   
   ### Additional Context
   
   https://github.com/apache/opendal/discussions/5420
   
   ### Are you willing to contribute to the development of this feature?
   
   - [ ] Yes, I am willing to contribute to the development of this feature.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] new feature: Expose API to configure chunk size for java binding [opendal]

["Xuanwo (via GitHub)" <gi...@apache.org>]

Xuanwo commented on issue #5421:
URL: https://github.com/apache/opendal/issues/5421#issuecomment-2550223883

   Thank you @tisonkun for fixing this!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] new feature: Expose API to configure chunk size for java binding [opendal]

["BaurzhanSakhariev (via GitHub)" <gi...@apache.org>]

BaurzhanSakhariev commented on issue #5421:
URL: https://github.com/apache/opendal/issues/5421#issuecomment-2551448698

   This will be part of the 0.51.0, right?
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] new feature: Expose API to configure chunk size for java binding [opendal]

["Xuanwo (via GitHub)" <gi...@apache.org>]

Xuanwo closed issue #5421: new feature: Expose API to configure chunk size for java binding
URL: https://github.com/apache/opendal/issues/5421


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org