You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opendal.apache.org by "nagraham (via GitHub)" <gi...@apache.org> on 2025/06/02 22:41:20 UTC
[I] Cloudflare R2 requires "All non-trailing parts must have the same length" on Multipart Uploads [opendal]
nagraham opened a new issue, #6252:
URL: https://github.com/apache/opendal/issues/6252
### Context
I'm working on writing data to R2 Data Catalog (https://developers.cloudflare.com/r2/data-catalog/) with iceberg-rust, which uses OpenDAL.
### Problem
When writing large files into the Iceberg table (on R2), I get the following error:
```
S3Error { code: "InvalidPart", message: "All non-trailing parts must have the same length.", resource: "", request_id: "" }
```
I traced it to the Multi-part upload in OpenDAL. It seems this was fixed a while back (https://github.com/apache/opendal/issues/2904), which a change which introduced a `enable_exact_buf_write` configuration value, which is still referenced [in the docs under Cloudflare R2](https://opendal.apache.org/docs/rust/opendal/services/struct.S3.html#cloudflare-r2). However, that config value doesn't exist anymore. I guess it was deprecated in v0.42 ([change log](https://github.com/apache/opendal/blob/de6f5e4f7fa8da846995d4bfb6c0e3d183732fbf/core/src/docs/upgrade.md?plain=1#L541)), and replaced with `BufferedWriter`.
But multi-part uploads are failing with that issue (see some code below which can reproduce the error). Perhaps there has been a regression to the BufferedWriter? I notice the R2 Integ tests are disabled (https://github.com/apache/opendal/issues/4324), so perhaps a regression snuck in?
```
# This is using opendal = { version = "0.53.0", features = ["services-s3"] }
let mut cfg: S3Config = S3Config::default();
// My configuration
let operator = Operator::new(builder).unwrap().finish();
let mut w = operator.writer(&cli.object).await.expect("A writer");
// Does OpenDAL write to a buffer and then write the same sized chunks in a multipart upload?
// Write 10MB
w.write(vec![0; 10 * 1024 * 1024])
.await
.map_err(|e| format!("Error writing first 10MB: {e:?}"))?;
// Write 20MB
w.write(vec![0; 20 * 1024 * 1024])
.await
.map_err(|e| format!("Error writing second 20MB chunk: {e:?}"))?;
// Write 8MB
w.write(vec![0; 8 * 1024 * 1024])
.await
.map_err(|e| format!("Error writing third 8MB chunk: {e:?}"))?;
w.close()
.await
.map_err(|e| format!("Error while closing the writer: {e:?}"))?;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Cloudflare R2 requires "All non-trailing parts must have the same length" on Multipart Uploads [opendal]
Posted by "nagraham (via GitHub)" <gi...@apache.org>.
nagraham commented on issue #6252:
URL: https://github.com/apache/opendal/issues/6252#issuecomment-2936085197
Thank you for your fast response, Xuanwo! I created the issue in the iceberg-rust repo: https://github.com/apache/iceberg-rust/issues/1406.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Cloudflare R2 requires "All non-trailing parts must have the same length" on Multipart Uploads [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #6252:
URL: https://github.com/apache/opendal/issues/6252#issuecomment-2938674250
Closed in favor of https://github.com/apache/iceberg-rust/issues/1406.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Cloudflare R2 requires "All non-trailing parts must have the same length" on Multipart Uploads [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo closed issue #6252: Cloudflare R2 requires "All non-trailing parts must have the same length" on Multipart Uploads
URL: https://github.com/apache/opendal/issues/6252
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Cloudflare R2 requires "All non-trailing parts must have the same length" on Multipart Uploads [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #6252:
URL: https://github.com/apache/opendal/issues/6252#issuecomment-2933230594
Hi, this can be addressed by adding `chunk(8MiB)` to the writer process, ensuring that OpenDAL uses a consistent upload size.
Would you like to raise an issue at iceberg side and ping me there? I'm will to help fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org