You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opendal.apache.org by "jihuayu (via GitHub)" <gi...@apache.org> on 2026/05/13 00:27:40 UTC
[PR] fix(services/gcs): validate batch delete response size [opendal]
jihuayu opened a new pull request, #7525:
URL: https://github.com/apache/opendal/pull/7525
# Which issue does this PR close?
Closes #7524
# Rationale for this change
`GcsDeleter::delete_batch` maps multipart response parts back to request paths by index. Before this PR, it did not validate that the GCS batch delete response contains the same number of parts as the requested paths.
If a server, proxy, or GCS-compatible implementation returns more parts than requested, OpenDAL can panic due to an out-of-bounds index. If fewer parts are returned, the mismatch is reported later by `BatchDeleter` as a generic size mismatch.
# What changes are included in this PR?
This PR extracts the existing response parsing part of `delete_batch` into `parse_batch_delete_response`. This is mainly to keep the unit tests focused on response parsing: the tests can feed malformed multipart responses directly and assert that OpenDAL returns `ErrorKind::Unexpected`.
The extraction is not intended to change behavior by itself; the functional change is the new `paths.len() != parts.len()` validation.
If `paths.len() != parts.len()`, GCS batch delete now returns `ErrorKind::Unexpected` with `expected` and `actual` context.
This PR also adds unit tests for both mismatch cases:
- response contains more parts than requested paths
- response contains fewer parts than requested paths
# Are there any user-facing changes?
No public API changes.
The only behavior change is that invalid GCS batch delete multipart responses now return a structured `Unexpected` error instead of potentially panicking or surfacing a delayed generic size mismatch.
# AI Usage Statement
This PR was developed with assistance from OpenAI Codex.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(services/gcs): validate batch delete response size [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo closed pull request #7525: fix(services/gcs): validate batch delete response size
URL: https://github.com/apache/opendal/pull/7525
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(services/gcs): validate batch delete response size [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on PR #7525:
URL: https://github.com/apache/opendal/pull/7525#issuecomment-4471363384
Thank you for this PR! But I think this change is not meaningful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(services/gcs): validate batch delete response size [opendal]
Posted by "jihuayu (via GitHub)" <gi...@apache.org>.
jihuayu commented on code in PR #7525:
URL: https://github.com/apache/opendal/pull/7525#discussion_r3230669298
##########
core/services/gcs/src/deleter.rs:
##########
@@ -51,48 +51,120 @@ impl oio::BatchDelete for GcsDeleter {
let paths: Vec<String> = batch.into_iter().map(|(p, _)| p).collect();
let resp = self.core.gcs_delete_objects(paths.clone()).await?;
- let status = resp.status();
+ parse_batch_delete_response(paths, resp)
+ }
+}
- // If the overall request isn't formatted correctly and Cloud Storage is unable to parse it into sub-requests, you receive a 400 error.
- // Otherwise, Cloud Storage returns a 200 status code, even if some or all of the sub-requests fail.
- if status != StatusCode::OK {
- return Err(parse_error(resp));
- }
+fn parse_batch_delete_response(
+ paths: Vec<String>,
+ resp: http::Response<Buffer>,
+) -> Result<BatchDeleteResult> {
+ let status = resp.status();
- let boundary = parse_multipart_boundary(resp.headers())?.ok_or_else(|| {
- Error::new(
- ErrorKind::Unexpected,
- "gcs batch delete response content type is empty",
- )
- })?;
- let multipart: Multipart<MixedPart> = Multipart::new()
- .with_boundary(boundary)
- .parse(resp.into_body().to_bytes())?;
- let parts = multipart.into_parts();
-
- let mut batched_result = BatchDeleteResult::default();
-
- for (i, part) in parts.into_iter().enumerate() {
- let resp = part.into_response();
- // TODO: maybe we can take it directly?
- let path = paths[i].clone();
-
- // deleting not existing objects is ok
- if resp.status().is_success() || resp.status() == StatusCode::NOT_FOUND {
- batched_result.succeeded.push((path, OpDelete::default()));
- } else {
- batched_result
- .failed
- .push((path, OpDelete::default(), parse_error(resp)));
- }
+ // If the overall request isn't formatted correctly and Cloud Storage is unable to parse it into sub-requests, you receive a 400 error.
+ // Otherwise, Cloud Storage returns a 200 status code, even if some or all of the sub-requests fail.
+ if status != StatusCode::OK {
+ return Err(parse_error(resp));
+ }
+
+ let boundary = parse_multipart_boundary(resp.headers())?.ok_or_else(|| {
+ Error::new(
+ ErrorKind::Unexpected,
+ "gcs batch delete response content type is empty",
+ )
+ })?;
+ let multipart: Multipart<MixedPart> = Multipart::new()
+ .with_boundary(boundary)
+ .parse(resp.into_body().to_bytes())?;
+ let parts = multipart.into_parts();
+
+ if paths.len() != parts.len() {
+ return Err(Error::new(
+ ErrorKind::Unexpected,
+ "invalid batch response, paths and response parts don't match",
+ )
+ .with_context("expected", paths.len())
+ .with_context("actual", parts.len()));
+ }
Review Comment:
I only add those.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(services/gcs): validate batch delete response size [opendal]
Posted by "jihuayu (via GitHub)" <gi...@apache.org>.
jihuayu commented on code in PR #7525:
URL: https://github.com/apache/opendal/pull/7525#discussion_r3230673778
##########
core/services/gcs/src/deleter.rs:
##########
@@ -51,48 +51,120 @@ impl oio::BatchDelete for GcsDeleter {
let paths: Vec<String> = batch.into_iter().map(|(p, _)| p).collect();
let resp = self.core.gcs_delete_objects(paths.clone()).await?;
- let status = resp.status();
+ parse_batch_delete_response(paths, resp)
+ }
+}
- // If the overall request isn't formatted correctly and Cloud Storage is unable to parse it into sub-requests, you receive a 400 error.
- // Otherwise, Cloud Storage returns a 200 status code, even if some or all of the sub-requests fail.
- if status != StatusCode::OK {
- return Err(parse_error(resp));
- }
+fn parse_batch_delete_response(
+ paths: Vec<String>,
+ resp: http::Response<Buffer>,
+) -> Result<BatchDeleteResult> {
+ let status = resp.status();
- let boundary = parse_multipart_boundary(resp.headers())?.ok_or_else(|| {
- Error::new(
- ErrorKind::Unexpected,
- "gcs batch delete response content type is empty",
- )
- })?;
- let multipart: Multipart<MixedPart> = Multipart::new()
- .with_boundary(boundary)
- .parse(resp.into_body().to_bytes())?;
- let parts = multipart.into_parts();
-
- let mut batched_result = BatchDeleteResult::default();
-
- for (i, part) in parts.into_iter().enumerate() {
- let resp = part.into_response();
- // TODO: maybe we can take it directly?
- let path = paths[i].clone();
-
- // deleting not existing objects is ok
- if resp.status().is_success() || resp.status() == StatusCode::NOT_FOUND {
- batched_result.succeeded.push((path, OpDelete::default()));
- } else {
- batched_result
- .failed
- .push((path, OpDelete::default(), parse_error(resp)));
- }
+ // If the overall request isn't formatted correctly and Cloud Storage is unable to parse it into sub-requests, you receive a 400 error.
+ // Otherwise, Cloud Storage returns a 200 status code, even if some or all of the sub-requests fail.
+ if status != StatusCode::OK {
+ return Err(parse_error(resp));
+ }
+
+ let boundary = parse_multipart_boundary(resp.headers())?.ok_or_else(|| {
+ Error::new(
+ ErrorKind::Unexpected,
+ "gcs batch delete response content type is empty",
+ )
+ })?;
+ let multipart: Multipart<MixedPart> = Multipart::new()
+ .with_boundary(boundary)
+ .parse(resp.into_body().to_bytes())?;
+ let parts = multipart.into_parts();
+
+ if paths.len() != parts.len() {
+ return Err(Error::new(
+ ErrorKind::Unexpected,
+ "invalid batch response, paths and response parts don't match",
+ )
+ .with_context("expected", paths.len())
+ .with_context("actual", parts.len()));
+ }
+
+ let mut batched_result = BatchDeleteResult::default();
+
+ for (i, part) in parts.into_iter().enumerate() {
+ let resp = part.into_response();
+ // TODO: maybe we can take it directly?
+ let path = paths[i].clone();
+
+ // deleting not existing objects is ok
+ if resp.status().is_success() || resp.status() == StatusCode::NOT_FOUND {
+ batched_result.succeeded.push((path, OpDelete::default()));
+ } else {
+ batched_result
+ .failed
+ .push((path, OpDelete::default(), parse_error(resp)));
}
+ }
+
+ // If no object is deleted, return directly.
+ if batched_result.succeeded.is_empty() {
+ let err = batched_result.failed.remove(0).2;
+ return Err(err);
+ }
+
+ Ok(batched_result)
+}
- // If no object is deleted, return directly.
- if batched_result.succeeded.is_empty() {
- let err = batched_result.failed.remove(0).2;
- return Err(err);
+#[cfg(test)]
+mod tests {
Review Comment:
And tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org