You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Koji Noguchi (JIRA)" <ji...@apache.org> on 2009/04/21 20:04:47 UTC
[jira] Created: (HADOOP-5712) Namenode slowed down when many files
with same filename were moved to Trash
Namenode slowed down when many files with same filename were moved to Trash
---------------------------------------------------------------------------
Key: HADOOP-5712
URL: https://issues.apache.org/jira/browse/HADOOP-5712
Project: Hadoop Core
Issue Type: Bug
Components: dfs
Affects Versions: 0.18.3
Reporter: Koji Noguchi
Priority: Minor
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5712) Namenode slowed down when many
files with same filename were moved to Trash
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12701284#action_12701284 ]
Hairong Kuang commented on HADOOP-5712:
---------------------------------------
Wow! After Koji figured out the cause of the problem, he talked about the same timestamp solution as what Doug suggested. This reminded me of a Chinese saying: all heroes have the same great idea! :-)
One minor suggestion... This solution assumes that all client machines have synchronized clocks. We could eliminate possible conflicts by appending client's machine name as well.
> Namenode slowed down when many files with same filename were moved to Trash
> ---------------------------------------------------------------------------
>
> Key: HADOOP-5712
> URL: https://issues.apache.org/jira/browse/HADOOP-5712
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.18.3
> Reporter: Koji Noguchi
> Priority: Minor
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5712) Namenode slowed down when many
files with same filename were moved to Trash
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708441#action_12708441 ]
dhruba borthakur commented on HADOOP-5712:
------------------------------------------
> In short, I want the Trash feature to stay as simple as it is now without involving the Namenode much.
Ok, sounds good to me.
> Namenode slowed down when many files with same filename were moved to Trash
> ---------------------------------------------------------------------------
>
> Key: HADOOP-5712
> URL: https://issues.apache.org/jira/browse/HADOOP-5712
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.18.3
> Reporter: Koji Noguchi
> Priority: Minor
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5712) Namenode slowed down when many
files with same filename were moved to Trash
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708718#action_12708718 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-5712:
------------------------------------------------
> For me, the contract is we try to move the files to Trash but if that fails, we simply delete them.
I agree. I sometimes create a file named .Trash in my hdfs home directory for disabling trash in my account.
> Namenode slowed down when many files with same filename were moved to Trash
> ---------------------------------------------------------------------------
>
> Key: HADOOP-5712
> URL: https://issues.apache.org/jira/browse/HADOOP-5712
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.18.3
> Reporter: Koji Noguchi
> Priority: Minor
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5712) Namenode slowed down when many
files with same filename were moved to Trash
Posted by "He Yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708727#action_12708727 ]
He Yongqiang commented on HADOOP-5712:
--------------------------------------
Then what about adding the file's create time?
> Namenode slowed down when many files with same filename were moved to Trash
> ---------------------------------------------------------------------------
>
> Key: HADOOP-5712
> URL: https://issues.apache.org/jira/browse/HADOOP-5712
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.18.3
> Reporter: Koji Noguchi
> Priority: Minor
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5712) Namenode slowed down when many
files with same filename were moved to Trash
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12701227#action_12701227 ]
Doug Cutting commented on HADOOP-5712:
--------------------------------------
Maybe this should be changed to something like:
{code}
while (fs.exists(trashPath)) {
trashPath = new Path(orig + "." + System.currentTimeMillis());
}
{code}
> Namenode slowed down when many files with same filename were moved to Trash
> ---------------------------------------------------------------------------
>
> Key: HADOOP-5712
> URL: https://issues.apache.org/jira/browse/HADOOP-5712
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.18.3
> Reporter: Koji Noguchi
> Priority: Minor
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5712) Namenode slowed down when many
files with same filename were moved to Trash
Posted by "Koji Noguchi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12701226#action_12701226 ]
Koji Noguchi commented on HADOOP-5712:
--------------------------------------
>From 0.20, HADOOP-4970 changed to preserve the full path for Trash and this should reduce the chances of name
clashing. However, we still have this problem if a user decides to repeat create&delete on some temp files.
> Namenode slowed down when many files with same filename were moved to Trash
> ---------------------------------------------------------------------------
>
> Key: HADOOP-5712
> URL: https://issues.apache.org/jira/browse/HADOOP-5712
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.18.3
> Reporter: Koji Noguchi
> Priority: Minor
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5712) Namenode slowed down when many
files with same filename were moved to Trash
Posted by "Koji Noguchi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708720#action_12708720 ]
Koji Noguchi commented on HADOOP-5712:
--------------------------------------
bq. Maybe we can also add the file's path info as well as the timestamp when it is moved to trash.
HADOOP-4970 from 0.20.
bq. I sometimes create a file named .Trash in my hdfs home directory for disabling trash in my account.
Never thought of that. Interesting.
> Namenode slowed down when many files with same filename were moved to Trash
> ---------------------------------------------------------------------------
>
> Key: HADOOP-5712
> URL: https://issues.apache.org/jira/browse/HADOOP-5712
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.18.3
> Reporter: Koji Noguchi
> Priority: Minor
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5712) Namenode slowed down when many
files with same filename were moved to Trash
Posted by "Koji Noguchi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12701222#action_12701222 ]
Koji Noguchi commented on HADOOP-5712:
--------------------------------------
We observed Namenode was spending more time in GC than usual.
No metrics stood out.
Audit showed many rename operations like the following.
{noformat}
2009-04-17 23:00:03,957 INFO org.apache.hadoop.fs.FSNamesystem.audit: ugi=user1,users,
ip=/99.111.222.111 cmd=rename
src=/user/user1/agg_results/20090417/part-00489 dst=/user/user1/.Trash/Current/part-00489.507
perm=user1:users:rw-r--r--
{noformat}
It turns out that this user had large number of small jobs and was calling
{noformat}
hadoop dfs -rmr $outdir/\*
{noformat}
after each job.
> Namenode slowed down when many files with same filename were moved to Trash
> ---------------------------------------------------------------------------
>
> Key: HADOOP-5712
> URL: https://issues.apache.org/jira/browse/HADOOP-5712
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.18.3
> Reporter: Koji Noguchi
> Priority: Minor
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5712) Namenode slowed down when many
files with same filename were moved to Trash
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707774#action_12707774 ]
dhruba borthakur commented on HADOOP-5712:
------------------------------------------
The proposal of using the timestamp instead of a monotonically increasing number solves the issue. However, it is not very user-friendly. What do other filesystems that have a Trash feature do in this case... the only other FS that I know of implements this feature uses the monotonically increasing number instead of appending the timestamp. At present, it is very easy to find out the last version of the file in Trash.
Another option would be to make the Trash client retrieve the contents of the Trash directory and then scan what files pre-exist in the list. Also, this code is not fool-proof because there is no atomicity between the exists and the rename.
> Namenode slowed down when many files with same filename were moved to Trash
> ---------------------------------------------------------------------------
>
> Key: HADOOP-5712
> URL: https://issues.apache.org/jira/browse/HADOOP-5712
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.18.3
> Reporter: Koji Noguchi
> Priority: Minor
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5712) Namenode slowed down when many
files with same filename were moved to Trash
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12701370#action_12701370 ]
Doug Cutting commented on HADOOP-5712:
--------------------------------------
> This solution assumes that all client machines have synchronized clocks.
We don't require that these are strictly ascending, just that they're unique. There's very little chance that two nodes, whether they have synchronized clocks or not, will collide to the millisecond, and, if they do, one will have to try twice. But the chances of ever having to try more than twice are even smaller.
> Namenode slowed down when many files with same filename were moved to Trash
> ---------------------------------------------------------------------------
>
> Key: HADOOP-5712
> URL: https://issues.apache.org/jira/browse/HADOOP-5712
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.18.3
> Reporter: Koji Noguchi
> Priority: Minor
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5712) Namenode slowed down when many
files with same filename were moved to Trash
Posted by "Koji Noguchi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12701229#action_12701229 ]
Koji Noguchi commented on HADOOP-5712:
--------------------------------------
Thanks Doug. +1 for the change.
Created HADOOP-5714 for a new metric (for tracking number of getFileInfo calls).
> Namenode slowed down when many files with same filename were moved to Trash
> ---------------------------------------------------------------------------
>
> Key: HADOOP-5712
> URL: https://issues.apache.org/jira/browse/HADOOP-5712
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.18.3
> Reporter: Koji Noguchi
> Priority: Minor
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5712) Namenode slowed down when many
files with same filename were moved to Trash
Posted by "He Yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708713#action_12708713 ]
He Yongqiang commented on HADOOP-5712:
--------------------------------------
Maybe we can also add the file's path info as well as the timestamp when it is moved to trash.
> Namenode slowed down when many files with same filename were moved to Trash
> ---------------------------------------------------------------------------
>
> Key: HADOOP-5712
> URL: https://issues.apache.org/jira/browse/HADOOP-5712
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.18.3
> Reporter: Koji Noguchi
> Priority: Minor
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5712) Namenode slowed down when many
files with same filename were moved to Trash
Posted by "Koji Noguchi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708134#action_12708134 ]
Koji Noguchi commented on HADOOP-5712:
--------------------------------------
bq. At present, it is very easy to find out the last version of the file in Trash.
If all the nodes in the cluster are in the same timezone, timestamp would (almost) serve this purpose?
bq. Another option would be to make the Trash client retrieve the contents of the Trash directory and then scan what files pre-exist in the list.
listStatus is one of the most expensive call to Namenode right now.
I really want to avoid an extra overhead to the namenode with this common command.
bq. Also, this code is not fool-proof because there is no atomicity between the exists and the rename.
True. But I haven't seen this become a problem yet.
For me, the contract is we *try* to move the files to Trash but if that fails, we simply delete them.
We completely delete the files if rename fails twice in a row anyway.
In short, I want the Trash feature to stay as simple as it is now without involving the Namenode much.
> Namenode slowed down when many files with same filename were moved to Trash
> ---------------------------------------------------------------------------
>
> Key: HADOOP-5712
> URL: https://issues.apache.org/jira/browse/HADOOP-5712
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.18.3
> Reporter: Koji Noguchi
> Priority: Minor
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5712) Namenode slowed down when many
files with same filename were moved to Trash
Posted by "Koji Noguchi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12701224#action_12701224 ]
Koji Noguchi commented on HADOOP-5712:
--------------------------------------
Looking at the Trash.java,
{noformat}
118 String orig = trashPath.toString();
119 for (int j = 1; fs.exists(trashPath); j++) {
120 trashPath = new Path(orig + "." + j);
121 }
122 if (fs.rename(path, trashPath)) // move to current trash
123 return true;
{noformat}
it seems like fs.exists were called 506 times before the single rename in the original description.
When user changed his script to simply call dfs -rmr $outdir, namenode came back to normal.
> Namenode slowed down when many files with same filename were moved to Trash
> ---------------------------------------------------------------------------
>
> Key: HADOOP-5712
> URL: https://issues.apache.org/jira/browse/HADOOP-5712
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.18.3
> Reporter: Koji Noguchi
> Priority: Minor
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.