You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opendal.apache.org by "hzxa21 (via GitHub)" <gi...@apache.org> on 2024/09/06 06:52:23 UTC

[I] bug: encounter "writer has not been closed or aborted, must be a bug" [opendal]

hzxa21 opened a new issue, #5101:
URL: https://github.com/apache/opendal/issues/5101

   ### Describe the bug
   
   Version: `0.47.2`
   With the following writer configurations, we encounter "writer has not been closed or aborted, must be a bug".
   ```rust
       let writer = op
               .clone()
               .layer(TimeoutLayer::new().with_io_timeout(Duration::from_millis(
                   config.retry.streaming_upload_attempt_timeout_ms,
               )))
               .layer(
                   RetryLayer::new()
                       .with_min_delay(Duration::from_millis(1000))
                       .with_max_delay(Duration::from_millis(10000))
                       .with_max_times(3)
                       .with_factor(2.0)
                       .with_jitter(),
               )
               .writer_with(&path)
               .concurrent(8)
               .executor(Executor::with(monitored_execute))
               .await?;
   ```
   
   It seems that this happens when the opendal retry is triggered on `writer.close()` .
   
   ### Steps to Reproduce
   
   -
   
   ### Expected Behavior
   
   -
   
   ### Additional Context
   
   Logs:
   
   2024-09-04T07:42:02.770667396Z  WARN opendal::layers::retry: will retry after 1.604547739s because: Unexpected (temporary) at Writer::close, context: { timeout: 10 } => io operation timeout reached    
   
   2024-09-04T07:42:04.377279911Z  WARN opendal::services: service=s3 operation=Writer::close path=xxx -> data close failed: NotFound (permanent) at Writer::close, context: { uri: ..., response: Parts { status: 404, version: HTTP/1.1, headers: {"accept-ranges": "bytes", "cache-control": "no-cache", "content-length": "467", "content-security-policy": "block-all-mixed-content", "content-type": "application/xml", "server": "MinIO", "strict-transport-security": "max-age=31536000; includeSubDomains", "vary": "Origin", "vary": "Accept-Encoding", "x-accel-buffering": "no", "x-amz-id-2": "..."} }, service: s3, path: xxx, written: 138426184 } => S3Error { code: "NoSuchUpload", message: "The specified multipart upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed.", resource: "xxx", request_id: "xxx" }    
   
   2024-09-04T07:42:04.377323167Z  WARN opendal::layers::complete: writer has not been closed or aborted, must be a bug    
   
   2024-09-04T07:42:04.37733207Z ERROR risingwave_object_store::object: streaming_upload_finish failed error=NotFound (persistent) at Writer::close, context: { uri:..., response: Parts { status: 404, version: HTTP/1.1, headers: {"accept-ranges": "bytes", "cache-control": "no-cache", "content-length": "467", "content-security-policy": "block-all-mixed-content", "content-type": "application/xml", "server": "MinIO", "strict-transport-security": "max-age=31536000; includeSubDomains", "vary": "Origin", "vary": "Accept-Encoding", "x-accel-buffering": "no", "x-amz-id-2": "..."} }, service: s3, path: ... } => S3Error { code: "NoSuchUpload", message: "The specified multipart upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed.", resource: "...", request_id: "..." }
   
   
   ### Are you willing to submit a PR to fix this bug?
   
   - [ ] Yes, I would like to submit a PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: encounter "writer has not been closed or aborted, must be a bug" [opendal]

Posted by "wcy-fdu (via GitHub)" <gi...@apache.org>.
wcy-fdu commented on issue #5101:
URL: https://github.com/apache/opendal/issues/5101#issuecomment-2337071283

   >Can you reproduce this over AWS S3? Or it's just on minio?
   
   This error occurs in our CI, which is based on minio. The issue has not yet been reproduced on AWS S3, and it is a bit bard to check whether the file was created successfully on minio.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: encounter "writer has not been closed or aborted, must be a bug" [opendal]

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo closed issue #5101: bug: encounter "writer has not been closed or aborted, must be a bug" 
URL: https://github.com/apache/opendal/issues/5101


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: encounter "writer has not been closed or aborted, must be a bug" [opendal]

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #5101:
URL: https://github.com/apache/opendal/issues/5101#issuecomment-2337091701

   > This error occurs in our CI, which is based on minio. The issue has not yet been reproduced on AWS S3, and it is a bit bard to check whether the file was created successfully on minio.
   
   Hi, can you run `op.stat()` on this file if `NotFound` occurs during `w.close()` and ignore it file created? If the error disappears, I believe that's the issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: encounter "writer has not been closed or aborted, must be a bug" [opendal]

Posted by "wcy-fdu (via GitHub)" <gi...@apache.org>.
wcy-fdu commented on issue #5101:
URL: https://github.com/apache/opendal/issues/5101#issuecomment-2337473458

   > > This error occurs in our CI, which is based on minio. The issue has not yet been reproduced on AWS S3, and it is a bit bard to check whether the file was created successfully on minio.
   > 
   > Hi, can you run `op.stat()` on this file if `NotFound` occurs during `w.close()` and ignore it file created? If the error disappears, I believe that's the issue.
   
   The file is not written into minio.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: encounter "writer has not been closed or aborted, must be a bug" [opendal]

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #5101:
URL: https://github.com/apache/opendal/issues/5101#issuecomment-2475590743

   It doesn't seem related to opendal, so let's close it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: encounter "writer has not been closed or aborted, must be a bug" [opendal]

Posted by "wcy-fdu (via GitHub)" <gi...@apache.org>.
wcy-fdu commented on issue #5101:
URL: https://github.com/apache/opendal/issues/5101#issuecomment-2357831862

   Update:
   OpenDAL and the AWS SDK handle multipart uploads differently. 
   - The AWS SDK operates completely in parallel
   - while OpenDAL processes uploads based on concurrency, utilizing an internal task queue. During the final close, it continuously polls this task queue until all tasks are completed.
   
   As a result, in extreme cases, the timeout in OpenDAL may require waiting through two rounds of I/O: the first round waits for `task.execute`, and the second waits for all tasks in the batch to finish. Currently, the timeout is not per operation; it also includes the waiting time for tasks. Therefore, when the number of tasks exceeds the concurrency, the timeout is more likely to be triggered compared to the SDK.
   
   Additionally, there may be a MinIO issue: the `CompleteMultipartUpload` operation is not necessarily idempotent.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: encounter "writer has not been closed or aborted, must be a bug" [opendal]

Posted by "wcy-fdu (via GitHub)" <gi...@apache.org>.
wcy-fdu commented on issue #5101:
URL: https://github.com/apache/opendal/issues/5101#issuecomment-2333574690

   Also reproduce in Version: 0.49.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: encounter "writer has not been closed or aborted, must be a bug" [opendal]

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #5101:
URL: https://github.com/apache/opendal/issues/5101#issuecomment-2335153137

   Hi, could you check if this file has been created successfully? I'm wondering if this is the case:
   
   - A complete multiple is sent, which returns a timeout.
   - We cancel this request and send a new one.
   - But the old request has been processed successfully, leading to the new request returning an error.
   
   Do you think this is possible?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: encounter "writer has not been closed or aborted, must be a bug" [opendal]

Posted by "choucavalier (via GitHub)" <gi...@apache.org>.
choucavalier commented on issue #5101:
URL: https://github.com/apache/opendal/issues/5101#issuecomment-3339667564

   @wcy-fdu did you manage to solve this? i'm not using minio but "garage" and i get the same error
   
   ```
   [2025-09-26T17:02:14Z WARN  opendal::layers::complete] writer has not been closed or aborted, must be a bug
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org