You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opendal.apache.org by "wcy-fdu (via GitHub)" <gi...@apache.org> on 2024/07/08 06:01:13 UTC
[I] bug: fail to connect hdfs if set `atomic_write_dir` [opendal]
wcy-fdu opened a new issue, #4867:
URL: https://github.com/apache/opendal/issues/4867
error log:
```
2024-06-27 19:15:26,794 WARN org.apache.hadoop.hdfs.StateChange: DIR* FSDirectory.unprotectedRenameTo: failed to rename /atomic_write_dir/0.bbca8834-a39b-432c-a0b9-72743ba5f218 to /data/cluster_id/0 because destination exists
```
The audit log shows that the file to be written was created earlier than the first temporary file, causing the temporary file rename to fail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: fail to connect hdfs if set `atomic_write_dir` [opendal]
Posted by "meteorgan (via GitHub)" <gi...@apache.org>.
meteorgan commented on issue #4867:
URL: https://github.com/apache/opendal/issues/4867#issuecomment-2306770771
Oh, Let me try to change it to `hdfs`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: fail to connect hdfs if set `atomic_write_dir` [opendal]
Posted by "meteorgan (via GitHub)" <gi...@apache.org>.
meteorgan commented on issue #4867:
URL: https://github.com/apache/opendal/issues/4867#issuecomment-2306558798
I'm not sure if i missed something.
Should `hdfs-default-with-atomic-write-dir` cover this case ? in this job, `atomic_write_dir` is enabled, and in `Access::write` the `target_path` is always created if not exists
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: fail to connect hdfs if set `atomic_write_dir` [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo closed issue #4867: bug: fail to connect hdfs if set `atomic_write_dir`
URL: https://github.com/apache/opendal/issues/4867
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: fail to connect hdfs if set `atomic_write_dir` [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #4867:
URL: https://github.com/apache/opendal/issues/4867#issuecomment-2305184122
Hi, I believe this issue is not covered by CI so far.
We will need:
- Enable `atomic_write_dir`: this will write data in temp dir first and then rename to dest.
- Have data existing in place.
The hdfs `Access::rename` will check the `dest` first before calling rename, but the `rename` inside `Access:write` doesn't handle it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: fail to connect hdfs if set `atomic_write_dir` [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #4867:
URL: https://github.com/apache/opendal/issues/4867#issuecomment-2306610129
> Should `hdfs-default-with-atomic-write-dir` cover this case ? in this job, `atomic_write_dir` is enabled, and in `Access::write` the `target_path` is always created if not exists
The problematic case is as follows. Users attempted to write `x.pdf`, and with `atomic_write_dir` enabled, we first write to `tmp/x.pdf` and then rename it to `x.pdf`. However, in some instances, `x.pdf` already exists, causing this renaming to fail.
https://github.com/apache/opendal/blob/0cec8ba8991edf226997df2b429f3fdb0285620d/core/src/services/hdfs/writer.rs#L104-L109
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: fail to connect hdfs if set `atomic_write_dir` [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #4867:
URL: https://github.com/apache/opendal/issues/4867#issuecomment-2306690985
Oh, I see. `hdfs-default` will use `local fs` instead the `hdfs` cluster. I believe this is the difference.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: fail to connect hdfs if set `atomic_write_dir` [opendal]
Posted by "meteorgan (via GitHub)" <gi...@apache.org>.
meteorgan commented on issue #4867:
URL: https://github.com/apache/opendal/issues/4867#issuecomment-2305157627
I encountered the same issue when i tried to reproduce this issue (https://github.com/apache/opendal/actions/runs/10486150908/job/29043772467) on my local machine.
The reason should be the destination file is created before rename. the code is here:
https://github.com/apache/opendal/blob/0cec8ba8991edf226997df2b429f3fdb0285620d/core/src/services/hdfs/backend.rs#L356
According to https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/filesystem/filesystem.html#boolean_rename.28Path_src.2C_Path_d.29, `There must not be an existing file at the end of the destination path`
and this problem caused many HDFS test cases to fail on my machine. However when I comment out the above code, all the tests pass. Strangely, they all succeed in the Github Action environment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: fail to connect hdfs if set `atomic_write_dir` [opendal]
Posted by "meteorgan (via GitHub)" <gi...@apache.org>.
meteorgan commented on issue #4867:
URL: https://github.com/apache/opendal/issues/4867#issuecomment-2306681259
yes. I've done some trials on my laptop, when I set
```
OPENDAL_HDFS_ATOMIC_WRITE_DIR=/tmp/atomic_write_dir/opendal/
OPENDAL_HDFS_ENABLE_APPEND=false
```
these setting are the same to `hdfs-default-with-atomic-write-dir` in https://github.com/apache/opendal/blob/0cec8ba8991edf226997df2b429f3fdb0285620d/.github/workflows/service_test_hdfs.yml
and run `OPENDAL_TEST=hdfs cargo test behavior --features tests,services-hdfs`. many tests all failed:
```
failures:
behavior::test_list_dir
behavior::test_list_prefix
behavior::test_remove_one_file
behavior::test_list_dir_with_file_path
behavior::test_list_dir_with_metakey
behavior::test_delete_file
behavior::test_list_dir_with_metakey_complete
behavior::test_delete_with_special_chars
behavior::test_list_rich_dir
behavior::test_delete_stream
behavior::test_list_nested_dir
behavior::test_list_dir_with_recursive
behavior::test_read_full
behavior::test_list_file_with_recursive
behavior::test_remove_all
behavior::test_list_dir_with_recursive_no_trailing_slash
behavior::test_rename_file
behavior::test_read_with_special_chars
behavior::test_read_range
behavior::test_reader
behavior::test_rename_target_dir
behavior::test_rename_self
behavior::test_rename_nested
behavior::test_rename_overwrite
behavior::test_stat_not_cleaned_path
behavior::test_stat_file
behavior::test_stat_nested_parent_dir
behavior::test_stat_with_special_chars
behavior::test_write_only
behavior::test_blocking_remove_one_file
behavior::test_blocking_read_full
behavior::test_blocking_delete_file
behavior::test_write_with_special_chars
behavior::test_blocking_stat_file
behavior::test_blocking_read_range
behavior::test_writer_write_with_overwrite
behavior::test_blocking_stat_with_special_chars
behavior::test_blocking_write_file
behavior::test_blocking_write_with_special_chars
```
so I'm wondering why the `hdfs-default-with-atomic-write-dir` is successful. Do I misunderstand something ?
since we set `atomic_write_dir` and don't append, every write operation will use this feature. Additionally, `Access:write` will create the target file, making this issue easy to reproduce
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org