You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opendal.apache.org by "kingsword09 (via GitHub)" <gi...@apache.org> on 2025/06/22 03:03:21 UTC
[PR] fix(service/fs): handle if_not_exists flag to raise ConditionNotMatch error [opendal]
kingsword09 opened a new pull request, #6326:
URL: https://github.com/apache/opendal/pull/6326
# Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
Closes #.
# Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
While implementing the `WriteOptions`#6322 in Node.js, I found that the `if_not_exists` property test fails without throwing the `ConditionNotMatch` exception.
# What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
When `if_not_exists` is true, if the file already exists, a `ConditionNotMatch` exception should be thrown.
https://github.com/apache/opendal/blob/23f01dd545ebd2a7445f08cc5e7de8a19e70fb25/core/tests/behavior/async_write.rs#L773-L778
# Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!---
If there are any breaking changes to public APIs, please add the `breaking-changes` label.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(service/fs): handle if_not_exists flag to raise ConditionNotMatch error [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on PR #6326:
URL: https://github.com/apache/opendal/pull/6326#issuecomment-3003541061
> I tested locally and two tests would fail
Let me take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(service/fs): handle if_not_exists flag to raise ConditionNotMatch error [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on PR #6326:
URL: https://github.com/apache/opendal/pull/6326#issuecomment-2993941311
> Change to check if `if_not_exists` is true before creating `tmp_path`. If true, do not create the `tmp_path` temporary file.
Wow, you're so clever! Could you add some comments to explain it a bit? I spent some time trying to understand how it works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(service/fs): handle if_not_exists flag to raise ConditionNotMatch error [opendal]
Posted by "kingsword09 (via GitHub)" <gi...@apache.org>.
kingsword09 commented on PR #6326:
URL: https://github.com/apache/opendal/pull/6326#issuecomment-2993923045
@Xuanwo At first, I also thought this part was functional, but there's an extra `tmp_path` here. When the `atomic_write_dir` exists, a `tmp_path` gets generated, causing every write operation to use new `tmp_path` instead. As a result, the "ConditionNotMatch" exception won't be thrown.
https://github.com/apache/opendal/blob/577912dfeaa2fcdb0de794607d8c45bf76cb419b/core/src/services/fs/core.rs#L162
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(service/fs): handle if_not_exists flag to raise ConditionNotMatch error [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on PR #6326:
URL: https://github.com/apache/opendal/pull/6326#issuecomment-2993916286
This should have been catched at https://github.com/apache/opendal/blob/577912dfeaa2fcdb0de794607d8c45bf76cb419b/core/src/services/fs/core.rs#L146-L152
Maybe there are other issues?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(service/fs): handle if_not_exists flag to raise ConditionNotMatch error [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #6326:
URL: https://github.com/apache/opendal/pull/6326#discussion_r2160224892
##########
core/src/services/fs/core.rs:
##########
@@ -119,9 +119,12 @@ impl FsCore {
) -> Result<(PathBuf, Option<PathBuf>)> {
if let Some(atomic_write_dir) = &self.atomic_write_dir {
let target_path = self.ensure_write_abs_path(&self.root, path).await?;
- let tmp_path = self
- .ensure_write_abs_path(atomic_write_dir, &build_tmp_path_of(path))
- .await?;
+ let path = if op.if_not_exists() {
+ target_path.to_string_lossy().to_string()
Review Comment:
Hi, this could be logically incorrect since `atomic_write_dir` is meant to ensure our write operation is atomic. Even if the target path is empty, we still need to write to the temporary path first.
I think that in the case of `atomic_write_dir`, it's unavoidable to call `try_exists` first. The logic here is somewhat complex, so perhaps we should consider a major refactor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(service/fs): handle if_not_exists flag to raise ConditionNotMatch error [opendal]
Posted by "kingsword09 (via GitHub)" <gi...@apache.org>.
kingsword09 commented on PR #6326:
URL: https://github.com/apache/opendal/pull/6326#issuecomment-2993940427
> I see. So the real fix should use the correct open options to open the target path. In this way, we can avoid an extra syscall.
Change to check if `if_not_exists` is true before creating `tmp_path`. If true, do not create the `tmp_path` temporary file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(service/fs): handle if_not_exists flag to raise ConditionNotMatch error [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo merged PR #6326:
URL: https://github.com/apache/opendal/pull/6326
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(service/fs): handle if_not_exists flag to raise ConditionNotMatch error [opendal]
Posted by "kingsword09 (via GitHub)" <gi...@apache.org>.
kingsword09 commented on PR #6326:
URL: https://github.com/apache/opendal/pull/6326#issuecomment-2993943118
In the code, the `if_not_exists` flag is used to ensure that the write operation only happens if the target file does not already exist. If `if_not_exists = true`, we want to throw a `ConditionNotMatch` exception when the target file exists. In the previous implementation, a `tmp_path` temporary file was always created, so each write operation was actually performed on a new temporary file. As a result, the existence of the target file was never checked, and the exception was not triggered as expected.
The core of this change is to check whether `if_not_exists` is true before creating the temporary file (`tmp_path`). If it is, we try to write directly to the target path using `OpenOptions::create_new(true)`, which guarantees that the write will only succeed if the file does not already exist. If the target file exists, a `ConditionNotMatch` error is thrown. This avoids the problem of always writing to a temporary file and missing the existence check.
In short:
• The previous logic always created a temporary file, so it couldn’t detect if the target file existed.
• Now, we check `if_not_exists` first. If it’s true, we try to write directly to the target path, so the error can be thrown correctly and the expected behavior is achieved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(service/fs): handle if_not_exists flag to raise ConditionNotMatch error [opendal]
Posted by "kingsword09 (via GitHub)" <gi...@apache.org>.
kingsword09 commented on PR #6326:
URL: https://github.com/apache/opendal/pull/6326#issuecomment-2994260375
After merging #6327, simply adjusting the `should_append` code as shown below can temporarily resolve the issue, removing the need for an additional `try_exists` check since the append operation inherently includes a verification step.
before:
https://github.com/apache/opendal/blob/8f277e92cc49cbe5d0595640cdb0902b8d469130/core/src/services/fs/core.rs#L127-L130
after:
```rust
let should_append = (op.append() || op.if_not_exists())
&& tokio::fs::try_exists(&target_path)
.await
.map_err(new_std_io_error)?;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(service/fs): handle if_not_exists flag to raise ConditionNotMatch error [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on PR #6326:
URL: https://github.com/apache/opendal/pull/6326#issuecomment-2993923573
> @Xuanwo At first, I also thought this part was functional, but there's an extra `tmp_path` here. When the `atomic_write_dir` exists, a `tmp_path` gets generated, causing every write operation to use new `tmp_path` instead. As a result, the `ConditionNotMatch` exception won't be thrown.
>
>
>
> https://github.com/apache/opendal/blob/577912dfeaa2fcdb0de794607d8c45bf76cb419b/core/src/services/fs/core.rs#L162
I see. So the real fix should use the correct open options to open the target path. In this way, we can avoid an extra syscall.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(service/fs): handle if_not_exists flag to raise ConditionNotMatch error [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on PR #6326:
URL: https://github.com/apache/opendal/pull/6326#issuecomment-3003397235
Hi, @kingsword09 I did the refactor on your PR directly. Let see how it works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(service/fs): handle if_not_exists flag to raise ConditionNotMatch error [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #6326:
URL: https://github.com/apache/opendal/pull/6326#discussion_r2160225042
##########
core/src/services/fs/core.rs:
##########
@@ -119,9 +119,12 @@ impl FsCore {
) -> Result<(PathBuf, Option<PathBuf>)> {
if let Some(atomic_write_dir) = &self.atomic_write_dir {
let target_path = self.ensure_write_abs_path(&self.root, path).await?;
- let tmp_path = self
- .ensure_write_abs_path(atomic_write_dir, &build_tmp_path_of(path))
- .await?;
+ let path = if op.if_not_exists() {
+ target_path.to_string_lossy().to_string()
Review Comment:
cc @kingsword09, after giving it some thought, I realize the current behavior isn't what I expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(service/fs): handle if_not_exists flag to raise ConditionNotMatch error [opendal]
Posted by "kingsword09 (via GitHub)" <gi...@apache.org>.
kingsword09 commented on code in PR #6326:
URL: https://github.com/apache/opendal/pull/6326#discussion_r2160225876
##########
core/src/services/fs/core.rs:
##########
@@ -119,9 +119,12 @@ impl FsCore {
) -> Result<(PathBuf, Option<PathBuf>)> {
if let Some(atomic_write_dir) = &self.atomic_write_dir {
let target_path = self.ensure_write_abs_path(&self.root, path).await?;
- let tmp_path = self
- .ensure_write_abs_path(atomic_write_dir, &build_tmp_path_of(path))
- .await?;
+ let path = if op.if_not_exists() {
+ target_path.to_string_lossy().to_string()
Review Comment:
You're right, it might really need to be refactored here to make more sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(service/fs): handle if_not_exists flag to raise ConditionNotMatch error [opendal]
Posted by "kingsword09 (via GitHub)" <gi...@apache.org>.
kingsword09 commented on PR #6326:
URL: https://github.com/apache/opendal/pull/6326#issuecomment-3003482681
I tested locally and two tests would fail

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org