You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2020/04/30 11:17:52 UTC
[GitHub] [hadoop-ozone] aryangupta1998 opened a new pull request #887: HDDS-3492. With hdfs-fuse file implementation 'cp' command is not working - fixed.
aryangupta1998 opened a new pull request #887:
URL: https://github.com/apache/hadoop-ozone/pull/887
## What changes were proposed in this pull request?
'cp' (copy) command doesn’t work with hdfs-fuse file system. It works by passing 'cp' command two times for the same file. It mainly happens when destination file doesn’t exists. It happens because when 'cp' command is performed for the first time it creates the file at the destination path, but as per ozone file system protocols any file has no access till its committed. So, 'getattr' function is not able to locate the file(though it’s present at the destination path). Hence, 'cp' command fails in the first go. Second time when 'cp' command is executed, file is already present in the destination path so 'cp' command simply override the data in the file.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-3492
## How was this patch tested?
Tested manually
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on pull request #887: HDDS-3492. With hdfs-fuse file implementation 'cp' command is not working - fixed.
Posted by GitBox <gi...@apache.org>.
elek commented on pull request #887:
URL: https://github.com/apache/hadoop-ozone/pull/887#issuecomment-626616058
Thanks the answer @aryangupta1998, we also discussed it offline, and I try to summarize it:
* We have more serious problems, as this is just a workaround for NFS
* Even this workaround doesn't solve all the problems. (It creates a 0 size file first, and add content later. This behavior may or may not be accepted by any other tools).
* One example: when some configuration is copied to this file system, other service can found it as a zero size file and start with wrong parameters as real content is added later
* HBase/Impala/... and other projects have the same problems with Ozone (Should be fixed it in our side) AND with other Object Stores (s3a)
* Related Impala bug for exmaple: https://issues.apache.org/jira/browse/IMPALA-9437
* But fixing is not easy as there could be multiple parallel file creation (no lease).
TLDR; This seems to be a dangerous workaround. I don't think that it's good solution for production-ready Fuse file system. But it can be a workaround to make possible the further testing of hdfs-fuse with Ozone. As hdfs-fuse is in alpha stage, I am fine to commit it.
But we need to continue to search for a generic, long-term fix (IMHO)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on pull request #887: HDDS-3492. With hdfs-fuse file implementation 'cp' command is not working - fixed.
Posted by GitBox <gi...@apache.org>.
elek commented on pull request #887:
URL: https://github.com/apache/hadoop-ozone/pull/887#issuecomment-624661879
Thanks to create this patch @aryangupta1998. Don't see any problem with this approach, but neither understand it fully. Is `getattr` supposed to be working according to POSIX before closing the file?
Do we need to fix it in our side? (Checking the open key tables when we do getFileStatus)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek edited a comment on pull request #887: HDDS-3492. With hdfs-fuse file implementation 'cp' command is not working - fixed.
Posted by GitBox <gi...@apache.org>.
elek edited a comment on pull request #887:
URL: https://github.com/apache/hadoop-ozone/pull/887#issuecomment-626616058
Thanks the answer @aryangupta1998, we also discussed it offline, and I try to summarize it:
* We have more serious problems, as this is just a workaround for NFS
* Even this workaround doesn't solve all the problems. (It creates a 0 size file first, and add content later. This behavior may or may not be accepted by any other tools).
* One example: when some configuration is copied to this file system, other service can found it as a zero size file and start with wrong parameters as real content is added later
* HBase/Impala/... and other projects have the same problems with Ozone (Should be fixed it in our side) AND with other Object Stores (s3a)
* Related Impala bug for exmaple: https://issues.apache.org/jira/browse/IMPALA-9437
* But fixing is not easy as there could be multiple parallel file creation (no lease).
TLDR; This seems to be a dangerous workaround. I don't think that it's good solution for production-ready Fuse file system. But it can be a workaround to make possible the further testing of hdfs-fuse with Ozone. As hdfs-fuse is in alpha stage, **I am fine with merge it**
But we need to continue to search for a generic, long-term fix (IMHO)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] nandakumar131 commented on pull request #887: HDDS-3492. With hdfs-fuse file implementation 'cp' command is not working - fixed.
Posted by GitBox <gi...@apache.org>.
nandakumar131 commented on pull request #887:
URL: https://github.com/apache/hadoop-ozone/pull/887#issuecomment-626599985
Thanks @aryangupta1998 for the PR and @elek for review. I will merge this shorty.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] aryangupta1998 commented on pull request #887: HDDS-3492. With hdfs-fuse file implementation 'cp' command is not working - fixed.
Posted by GitBox <gi...@apache.org>.
aryangupta1998 commented on pull request #887:
URL: https://github.com/apache/hadoop-ozone/pull/887#issuecomment-625325133
> Thanks to create this patch @aryangupta1998. Don't see any problem with this approach, but neither understand it fully. Is `getattr` supposed to be working according to POSIX before closing the file?
>
> Do we need to fix it in our side? (Checking the open key tables when we do getFileStatus)
@elek Yes 'getattr' is working according to POSIX before closing the file, but first time 'getattr' works before closing of file is to check whether the file exists at destination path or not. But second time 'getattr' doesn’t works before closing of file is when system creates empty file at destination path and opens it to write something but according to ozone filesystem, file is not visible till it’s committed so 'getattr' function fails this time before even closing of file. To address this issue we have raised this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org