You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Christian Kunz (JIRA)" <ji...@apache.org> on 2008/01/31 18:17:07 UTC
[jira] Created: (HADOOP-2755) dfs fsck extremely slow, dfs ls times
out
dfs fsck extremely slow, dfs ls times out
-----------------------------------------
Key: HADOOP-2755
URL: https://issues.apache.org/jira/browse/HADOOP-2755
Project: Hadoop Core
Issue Type: Bug
Affects Versions: 0.16.0
Environment:
Jan 28 nightly build
With patches 2095, 2119, and 2723
Reporter: Christian Kunz
Fix For: 0.16.0
I upgraded a Hadoop installation to the Jan 28 nightly build.
DFS contains 2.4+ M files.
Upgrade finished but not finalized.
Before finalizing I wanted to run fsck on the DFS. It hardly progressed after 6 hours (not finished yet). With the '-files' option turned on, it lists about 300 entries in 10 minutes.
And when I tried to list a subdirectory with 100,000 files, it repeatedly (about 20 attempts) timed out.
Changing timeout from 1 to 10 minutes did not help.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2755) dfs fsck extremely slow, dfs ls
times out
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12564470#action_12564470 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-2755:
------------------------------------------------
> I think this problem takes it root from HADOOP-2683. Also explained in HADOOP-1298 ...
Since HADOOP-2633, fsck does not use Server.getUserInfo() anymore. So I guess this issue is related to other problems. I will perform an investigation.
> dfs fsck extremely slow, dfs ls times out
> -----------------------------------------
>
> Key: HADOOP-2755
> URL: https://issues.apache.org/jira/browse/HADOOP-2755
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.16.0
> Environment: Jan 28 nightly build
> With patches 2095, 2119, and 2723
> Reporter: Christian Kunz
> Assignee: dhruba borthakur
> Priority: Blocker
> Fix For: 0.16.0
>
>
> I upgraded a Hadoop installation to the Jan 28 nightly build.
> DFS contains 2.4+ M files.
> Upgrade finished but not finalized.
> Before finalizing I wanted to run fsck on the DFS. It hardly progressed after 6 hours (not finished yet). With the '-files' option turned on, it lists about 300 entries in 10 minutes.
> And when I tried to list a subdirectory with 100,000 files, it repeatedly (about 20 attempts) timed out.
> Changing timeout from 1 to 10 minutes did not help.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2755) dfs fsck extremely slow, dfs ls
times out
Posted by "Christian Kunz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12564661#action_12564661 ]
Christian Kunz commented on HADOOP-2755:
----------------------------------------
Just wanted to comment to that effect.
I will open a different jira and transfer the replication timeout issue. The fsck issue seems to be resolved. Thanks.
> dfs fsck extremely slow, dfs ls times out
> -----------------------------------------
>
> Key: HADOOP-2755
> URL: https://issues.apache.org/jira/browse/HADOOP-2755
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.16.0
> Environment: Jan 28 nightly build
> With patches 2095, 2119, and 2723
> Reporter: Christian Kunz
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: 2755_20080131.patch, 2755_20080131b.patch
>
>
> I upgraded a Hadoop installation to the Jan 28 nightly build.
> DFS contains 2.4+ M files.
> Upgrade finished but not finalized.
> Before finalizing I wanted to run fsck on the DFS. It hardly progressed after 6 hours (not finished yet). With the '-files' option turned on, it lists about 300 entries in 10 minutes.
> And when I tried to list a subdirectory with 100,000 files, it repeatedly (about 20 attempts) timed out.
> Changing timeout from 1 to 10 minutes did not help.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2755) dfs fsck extremely slow, dfs ls times
out
Posted by "Sameer Paranjpye (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sameer Paranjpye updated HADOOP-2755:
-------------------------------------
Assignee: dhruba borthakur
Priority: Blocker (was: Major)
Environment:
Jan 28 nightly build
With patches 2095, 2119, and 2723
was:
Jan 28 nightly build
With patches 2095, 2119, and 2723
> dfs fsck extremely slow, dfs ls times out
> -----------------------------------------
>
> Key: HADOOP-2755
> URL: https://issues.apache.org/jira/browse/HADOOP-2755
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.16.0
> Environment: Jan 28 nightly build
> With patches 2095, 2119, and 2723
> Reporter: Christian Kunz
> Assignee: dhruba borthakur
> Priority: Blocker
> Fix For: 0.16.0
>
>
> I upgraded a Hadoop installation to the Jan 28 nightly build.
> DFS contains 2.4+ M files.
> Upgrade finished but not finalized.
> Before finalizing I wanted to run fsck on the DFS. It hardly progressed after 6 hours (not finished yet). With the '-files' option turned on, it lists about 300 entries in 10 minutes.
> And when I tried to list a subdirectory with 100,000 files, it repeatedly (about 20 attempts) timed out.
> Changing timeout from 1 to 10 minutes did not help.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2755) dfs fsck extremely slow, dfs ls
times out
Posted by "Christian Kunz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12564542#action_12564542 ]
Christian Kunz commented on HADOOP-2755:
----------------------------------------
At the time of the upgrade a single node mounted the home directories from the wrong adminhost and did not get restarted with the correct version. This has been fixed, but I suspect, all the blocks of this node needed to get replicated. But this is less than 20,000 blocks. This should be handled easily by dfs.
Also the namenode came out of safemode automatically, i.e. there were no missing blocks, which you would expect from a massive failure of joining datanodes.
60% cpu out of 400% cpu (there are 4 cpu's) is not much in my opinion. What is strange compared to other namenodes is that the system usage of the CPU is relatively high.
> dfs fsck extremely slow, dfs ls times out
> -----------------------------------------
>
> Key: HADOOP-2755
> URL: https://issues.apache.org/jira/browse/HADOOP-2755
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.16.0
> Environment: Jan 28 nightly build
> With patches 2095, 2119, and 2723
> Reporter: Christian Kunz
> Assignee: dhruba borthakur
> Priority: Blocker
> Fix For: 0.16.0
>
>
> I upgraded a Hadoop installation to the Jan 28 nightly build.
> DFS contains 2.4+ M files.
> Upgrade finished but not finalized.
> Before finalizing I wanted to run fsck on the DFS. It hardly progressed after 6 hours (not finished yet). With the '-files' option turned on, it lists about 300 entries in 10 minutes.
> And when I tried to list a subdirectory with 100,000 files, it repeatedly (about 20 attempts) timed out.
> Changing timeout from 1 to 10 minutes did not help.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2755) dfs fsck extremely slow, dfs ls
times out
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12565069#action_12565069 ]
Hudson commented on HADOOP-2755:
--------------------------------
Integrated in Hadoop-trunk #387 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/387/])
> dfs fsck extremely slow, dfs ls times out
> -----------------------------------------
>
> Key: HADOOP-2755
> URL: https://issues.apache.org/jira/browse/HADOOP-2755
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.16.0
> Environment: Jan 28 nightly build
> With patches 2095, 2119, and 2723
> Reporter: Christian Kunz
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: 2755_20080131.patch, 2755_20080131b.patch
>
>
> I upgraded a Hadoop installation to the Jan 28 nightly build.
> DFS contains 2.4+ M files.
> Upgrade finished but not finalized.
> Before finalizing I wanted to run fsck on the DFS. It hardly progressed after 6 hours (not finished yet). With the '-files' option turned on, it lists about 300 entries in 10 minutes.
> And when I tried to list a subdirectory with 100,000 files, it repeatedly (about 20 attempts) timed out.
> Changing timeout from 1 to 10 minutes did not help.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2755) dfs fsck extremely slow, dfs ls
times out
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12564638#action_12564638 ]
Raghu Angadi commented on HADOOP-2755:
--------------------------------------
Christian, replication problem is mostly not related to fsck. Could you grep for this block on the destination datanodes (one of datanode{1-4})?
> dfs fsck extremely slow, dfs ls times out
> -----------------------------------------
>
> Key: HADOOP-2755
> URL: https://issues.apache.org/jira/browse/HADOOP-2755
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.16.0
> Environment: Jan 28 nightly build
> With patches 2095, 2119, and 2723
> Reporter: Christian Kunz
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: 2755_20080131.patch, 2755_20080131b.patch
>
>
> I upgraded a Hadoop installation to the Jan 28 nightly build.
> DFS contains 2.4+ M files.
> Upgrade finished but not finalized.
> Before finalizing I wanted to run fsck on the DFS. It hardly progressed after 6 hours (not finished yet). With the '-files' option turned on, it lists about 300 entries in 10 minutes.
> And when I tried to list a subdirectory with 100,000 files, it repeatedly (about 20 attempts) timed out.
> Changing timeout from 1 to 10 minutes did not help.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2755) dfs fsck extremely slow, dfs ls
times out
Posted by "Christian Kunz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12564630#action_12564630 ]
Christian Kunz commented on HADOOP-2755:
----------------------------------------
In previous comment I accidentally copied the times for the datanode. The last two log messages were at
2008-02-01 03:52:06,238
2008-02-01 03:52:09,470
> dfs fsck extremely slow, dfs ls times out
> -----------------------------------------
>
> Key: HADOOP-2755
> URL: https://issues.apache.org/jira/browse/HADOOP-2755
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.16.0
> Environment: Jan 28 nightly build
> With patches 2095, 2119, and 2723
> Reporter: Christian Kunz
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: 2755_20080131.patch, 2755_20080131b.patch
>
>
> I upgraded a Hadoop installation to the Jan 28 nightly build.
> DFS contains 2.4+ M files.
> Upgrade finished but not finalized.
> Before finalizing I wanted to run fsck on the DFS. It hardly progressed after 6 hours (not finished yet). With the '-files' option turned on, it lists about 300 entries in 10 minutes.
> And when I tried to list a subdirectory with 100,000 files, it repeatedly (about 20 attempts) timed out.
> Changing timeout from 1 to 10 minutes did not help.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2755) dfs fsck extremely slow, dfs ls
times out
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12564533#action_12564533 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-2755:
------------------------------------------------
Christian, from your NameNode log files, there are a lot warnings saying "WARN org.apache.hadoop.ipc.Server: Incorrect header or version mismatch ..." I suspect the NameNode and DataNode were running with different versions of codes. Could you check it?
> dfs fsck extremely slow, dfs ls times out
> -----------------------------------------
>
> Key: HADOOP-2755
> URL: https://issues.apache.org/jira/browse/HADOOP-2755
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.16.0
> Environment: Jan 28 nightly build
> With patches 2095, 2119, and 2723
> Reporter: Christian Kunz
> Assignee: dhruba borthakur
> Priority: Blocker
> Fix For: 0.16.0
>
>
> I upgraded a Hadoop installation to the Jan 28 nightly build.
> DFS contains 2.4+ M files.
> Upgrade finished but not finalized.
> Before finalizing I wanted to run fsck on the DFS. It hardly progressed after 6 hours (not finished yet). With the '-files' option turned on, it lists about 300 entries in 10 minutes.
> And when I tried to list a subdirectory with 100,000 files, it repeatedly (about 20 attempts) timed out.
> Changing timeout from 1 to 10 minutes did not help.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2755) dfs fsck extremely slow, dfs ls times
out
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-2755:
-------------------------------------------
Attachment: 2755_20080131b.patch
2755_20080131b.patch: renamed getDatanodeSize() to getNumberOfDatanodes().
> dfs fsck extremely slow, dfs ls times out
> -----------------------------------------
>
> Key: HADOOP-2755
> URL: https://issues.apache.org/jira/browse/HADOOP-2755
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.16.0
> Environment: Jan 28 nightly build
> With patches 2095, 2119, and 2723
> Reporter: Christian Kunz
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: 2755_20080131.patch, 2755_20080131b.patch
>
>
> I upgraded a Hadoop installation to the Jan 28 nightly build.
> DFS contains 2.4+ M files.
> Upgrade finished but not finalized.
> Before finalizing I wanted to run fsck on the DFS. It hardly progressed after 6 hours (not finished yet). With the '-files' option turned on, it lists about 300 entries in 10 minutes.
> And when I tried to list a subdirectory with 100,000 files, it repeatedly (about 20 attempts) timed out.
> Changing timeout from 1 to 10 minutes did not help.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2755) dfs fsck extremely slow, dfs ls
times out
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12564659#action_12564659 ]
dhruba borthakur commented on HADOOP-2755:
------------------------------------------
Hi Christian, my vote would be to commit this patch as a "fix" to the fsck problem. Nicholas can continue to investigate why a particular block is timing out. We can track that problem by a different jira.
> dfs fsck extremely slow, dfs ls times out
> -----------------------------------------
>
> Key: HADOOP-2755
> URL: https://issues.apache.org/jira/browse/HADOOP-2755
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.16.0
> Environment: Jan 28 nightly build
> With patches 2095, 2119, and 2723
> Reporter: Christian Kunz
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: 2755_20080131.patch, 2755_20080131b.patch
>
>
> I upgraded a Hadoop installation to the Jan 28 nightly build.
> DFS contains 2.4+ M files.
> Upgrade finished but not finalized.
> Before finalizing I wanted to run fsck on the DFS. It hardly progressed after 6 hours (not finished yet). With the '-files' option turned on, it lists about 300 entries in 10 minutes.
> And when I tried to list a subdirectory with 100,000 files, it repeatedly (about 20 attempts) timed out.
> Changing timeout from 1 to 10 minutes did not help.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2755) dfs fsck extremely slow, dfs ls times
out
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-2755:
-------------------------------------------
Attachment: 2755_20080131.patch
2755_20080131.patch: Fixed two reasons that fsck is slow:
- totalRacks and totalDatanodes, which are global system properties, are computed for each file.
- When computing totalDatanodes, it calls NameNode.getDatanodeReport(DatanodeReportType.LIVE) which requires super user privilege.
> dfs fsck extremely slow, dfs ls times out
> -----------------------------------------
>
> Key: HADOOP-2755
> URL: https://issues.apache.org/jira/browse/HADOOP-2755
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.16.0
> Environment: Jan 28 nightly build
> With patches 2095, 2119, and 2723
> Reporter: Christian Kunz
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: 2755_20080131.patch
>
>
> I upgraded a Hadoop installation to the Jan 28 nightly build.
> DFS contains 2.4+ M files.
> Upgrade finished but not finalized.
> Before finalizing I wanted to run fsck on the DFS. It hardly progressed after 6 hours (not finished yet). With the '-files' option turned on, it lists about 300 entries in 10 minutes.
> And when I tried to list a subdirectory with 100,000 files, it repeatedly (about 20 attempts) timed out.
> Changing timeout from 1 to 10 minutes did not help.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2755) dfs fsck extremely slow, dfs ls times
out
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-2755:
-------------------------------------------
Status: Patch Available (was: Open)
> dfs fsck extremely slow, dfs ls times out
> -----------------------------------------
>
> Key: HADOOP-2755
> URL: https://issues.apache.org/jira/browse/HADOOP-2755
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.16.0
> Environment: Jan 28 nightly build
> With patches 2095, 2119, and 2723
> Reporter: Christian Kunz
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: 2755_20080131.patch, 2755_20080131b.patch
>
>
> I upgraded a Hadoop installation to the Jan 28 nightly build.
> DFS contains 2.4+ M files.
> Upgrade finished but not finalized.
> Before finalizing I wanted to run fsck on the DFS. It hardly progressed after 6 hours (not finished yet). With the '-files' option turned on, it lists about 300 entries in 10 minutes.
> And when I tried to list a subdirectory with 100,000 files, it repeatedly (about 20 attempts) timed out.
> Changing timeout from 1 to 10 minutes did not help.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2755) dfs fsck extremely slow, dfs ls
times out
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12564646#action_12564646 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-2755:
------------------------------------------------
> I noticed that before the patch I needed to be the owner, now any user can run fsck.
In both cases, fsck should be able to run by any user since the codes are in NameNode. We will change it so that only superuser can run fsck later.
> dfs fsck extremely slow, dfs ls times out
> -----------------------------------------
>
> Key: HADOOP-2755
> URL: https://issues.apache.org/jira/browse/HADOOP-2755
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.16.0
> Environment: Jan 28 nightly build
> With patches 2095, 2119, and 2723
> Reporter: Christian Kunz
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: 2755_20080131.patch, 2755_20080131b.patch
>
>
> I upgraded a Hadoop installation to the Jan 28 nightly build.
> DFS contains 2.4+ M files.
> Upgrade finished but not finalized.
> Before finalizing I wanted to run fsck on the DFS. It hardly progressed after 6 hours (not finished yet). With the '-files' option turned on, it lists about 300 entries in 10 minutes.
> And when I tried to list a subdirectory with 100,000 files, it repeatedly (about 20 attempts) timed out.
> Changing timeout from 1 to 10 minutes did not help.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2755) dfs fsck extremely slow, dfs ls
times out
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12564578#action_12564578 ]
dhruba borthakur commented on HADOOP-2755:
------------------------------------------
+1. code looks good. You might want to rename the new function from getDatanodeSize() into something like getNumberOfDatanodes().
> dfs fsck extremely slow, dfs ls times out
> -----------------------------------------
>
> Key: HADOOP-2755
> URL: https://issues.apache.org/jira/browse/HADOOP-2755
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.16.0
> Environment: Jan 28 nightly build
> With patches 2095, 2119, and 2723
> Reporter: Christian Kunz
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: 2755_20080131.patch
>
>
> I upgraded a Hadoop installation to the Jan 28 nightly build.
> DFS contains 2.4+ M files.
> Upgrade finished but not finalized.
> Before finalizing I wanted to run fsck on the DFS. It hardly progressed after 6 hours (not finished yet). With the '-files' option turned on, it lists about 300 entries in 10 minutes.
> And when I tried to list a subdirectory with 100,000 files, it repeatedly (about 20 attempts) timed out.
> Changing timeout from 1 to 10 minutes did not help.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2755) dfs fsck extremely slow, dfs ls
times out
Posted by "Christian Kunz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12564628#action_12564628 ]
Christian Kunz commented on HADOOP-2755:
----------------------------------------
The patch helps for fsck -- seems to run as fast as in previous releases.
Thank you, Nicholas.
I noticed that before the patch I needed to be the owner, now any user can run fsck.
Als, the namenode is now up more than for 1 hour, and fsck shows that there are still 25 blocks replicated with a single copy, and the replication monitor consistently times out:
2008-02-01 03:41:24,184 INFO org.apache.hadoop.dfs.StateChange: BLOCK* NameSystem.pendingTransfer: ask datanode to replicate blk_2984271423661664080 to datanode(s) datanode1 datanode2
2008-02-01 03:51:14,104 WARN org.apache.hadoop.fs.FSNamesystem: PendingReplicationMonitor timed out block blk_2984271423661664080
2008-02-01 03:51:22,303 INFO org.apache.hadoop.dfs.StateChange: BLOCK* NameSystem.pendingTransfer: ask datanode to replicate blk_2984271423661664080 to datanode(s) datanode3 datanode4
2008-02-01 04:01:14,150 WARN org.apache.hadoop.fs.FSNamesystem: PendingReplicationMonitor timed out block blk_2984271423661664080
2008-02-01 04:01:19,344 INFO org.apache.hadoop.dfs.StateChange: BLOCK* NameSystem.pendingTransfer: ask datanode to replicate blk_2984271423661664080 to datanode(s) datanode5 datanode6
...
The datanode seems to be successfully transmitting the blocks:
2008-02-01 03:42:06,284 INFO org.apache.hadoop.dfs.DataNode: datanode Starting thread to transfer block blk_2984271423661664080 to datanode1, datannode2
2008-02-01 03:42:09,535 INFO org.apache.hadoop.dfs.DataNode: datanode:Transmitted block blk_2984271423661664080 to /datanode1
2008-02-01 03:42:06,284 INFO org.apache.hadoop.dfs.DataNode: datanode Starting thread to transfer block blk_2984271423661664080 to datanode3,datanode4
2008-02-01 03:42:09,535 INFO org.apache.hadoop.dfs.DataNode: datanode:Transmitted block blk_2984271423661664080 to /datanode3
Is this related?
> dfs fsck extremely slow, dfs ls times out
> -----------------------------------------
>
> Key: HADOOP-2755
> URL: https://issues.apache.org/jira/browse/HADOOP-2755
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.16.0
> Environment: Jan 28 nightly build
> With patches 2095, 2119, and 2723
> Reporter: Christian Kunz
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: 2755_20080131.patch, 2755_20080131b.patch
>
>
> I upgraded a Hadoop installation to the Jan 28 nightly build.
> DFS contains 2.4+ M files.
> Upgrade finished but not finalized.
> Before finalizing I wanted to run fsck on the DFS. It hardly progressed after 6 hours (not finished yet). With the '-files' option turned on, it lists about 300 entries in 10 minutes.
> And when I tried to list a subdirectory with 100,000 files, it repeatedly (about 20 attempts) timed out.
> Changing timeout from 1 to 10 minutes did not help.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2755) dfs fsck extremely slow, dfs ls
times out
Posted by "Christian Kunz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12564511#action_12564511 ]
Christian Kunz commented on HADOOP-2755:
----------------------------------------
Are you sure about this?
Assuming that all replication requests are logged as:
org.apache.hadoop.dfs.StateChange: BLOCK* NameSystem.pendingTransfer: ask <node> to replicate blk_id ...
then there were only about 1000 such requests in the last 10 hours, but most of them timed out with a log message:
org.apache.hadoop.fs.FSNamesystem: PendingReplicationMonitor timed out block blk_id
And the namenode is not really busy, as you would expect during high replication periods.
> dfs fsck extremely slow, dfs ls times out
> -----------------------------------------
>
> Key: HADOOP-2755
> URL: https://issues.apache.org/jira/browse/HADOOP-2755
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.16.0
> Environment: Jan 28 nightly build
> With patches 2095, 2119, and 2723
> Reporter: Christian Kunz
> Assignee: dhruba borthakur
> Priority: Blocker
> Fix For: 0.16.0
>
>
> I upgraded a Hadoop installation to the Jan 28 nightly build.
> DFS contains 2.4+ M files.
> Upgrade finished but not finalized.
> Before finalizing I wanted to run fsck on the DFS. It hardly progressed after 6 hours (not finished yet). With the '-files' option turned on, it lists about 300 entries in 10 minutes.
> And when I tried to list a subdirectory with 100,000 files, it repeatedly (about 20 attempts) timed out.
> Changing timeout from 1 to 10 minutes did not help.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2755) dfs fsck extremely slow, dfs ls
times out
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12564455#action_12564455 ]
Konstantin Shvachko commented on HADOOP-2755:
---------------------------------------------
I think this problem takes it root from HADOOP-2683. Also explained in [HADOOP-1298|http://issues.apache.org/jira/browse/HADOOP-1298#action_12554874]
Server.getUserInfo() is inefficient when somebody/something tries to create a file
calling a name-node method directly without issuing an RPC request.
This makes fsck about a 100 times slower.
> dfs fsck extremely slow, dfs ls times out
> -----------------------------------------
>
> Key: HADOOP-2755
> URL: https://issues.apache.org/jira/browse/HADOOP-2755
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.16.0
> Environment: Jan 28 nightly build
> With patches 2095, 2119, and 2723
> Reporter: Christian Kunz
> Assignee: dhruba borthakur
> Priority: Blocker
> Fix For: 0.16.0
>
>
> I upgraded a Hadoop installation to the Jan 28 nightly build.
> DFS contains 2.4+ M files.
> Upgrade finished but not finalized.
> Before finalizing I wanted to run fsck on the DFS. It hardly progressed after 6 hours (not finished yet). With the '-files' option turned on, it lists about 300 entries in 10 minutes.
> And when I tried to list a subdirectory with 100,000 files, it repeatedly (about 20 attempts) timed out.
> Changing timeout from 1 to 10 minutes did not help.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-2755) dfs fsck extremely slow, dfs ls
times out
Posted by "Robert Chansler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Chansler reassigned HADOOP-2755:
---------------------------------------
Assignee: Tsz Wo (Nicholas), SZE (was: dhruba borthakur)
> dfs fsck extremely slow, dfs ls times out
> -----------------------------------------
>
> Key: HADOOP-2755
> URL: https://issues.apache.org/jira/browse/HADOOP-2755
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.16.0
> Environment: Jan 28 nightly build
> With patches 2095, 2119, and 2723
> Reporter: Christian Kunz
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.0
>
>
> I upgraded a Hadoop installation to the Jan 28 nightly build.
> DFS contains 2.4+ M files.
> Upgrade finished but not finalized.
> Before finalizing I wanted to run fsck on the DFS. It hardly progressed after 6 hours (not finished yet). With the '-files' option turned on, it lists about 300 entries in 10 minutes.
> And when I tried to list a subdirectory with 100,000 files, it repeatedly (about 20 attempts) timed out.
> Changing timeout from 1 to 10 minutes did not help.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2755) dfs fsck extremely slow, dfs ls times
out
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-2755:
-------------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thanks Nicholas.
> dfs fsck extremely slow, dfs ls times out
> -----------------------------------------
>
> Key: HADOOP-2755
> URL: https://issues.apache.org/jira/browse/HADOOP-2755
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.16.0
> Environment: Jan 28 nightly build
> With patches 2095, 2119, and 2723
> Reporter: Christian Kunz
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: 2755_20080131.patch, 2755_20080131b.patch
>
>
> I upgraded a Hadoop installation to the Jan 28 nightly build.
> DFS contains 2.4+ M files.
> Upgrade finished but not finalized.
> Before finalizing I wanted to run fsck on the DFS. It hardly progressed after 6 hours (not finished yet). With the '-files' option turned on, it lists about 300 entries in 10 minutes.
> And when I tried to list a subdirectory with 100,000 files, it repeatedly (about 20 attempts) timed out.
> Changing timeout from 1 to 10 minutes did not help.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2755) dfs fsck extremely slow, dfs ls times
out
Posted by "Robert Chansler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Chansler updated HADOOP-2755:
------------------------------------
Component/s: dfs
> dfs fsck extremely slow, dfs ls times out
> -----------------------------------------
>
> Key: HADOOP-2755
> URL: https://issues.apache.org/jira/browse/HADOOP-2755
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.16.0
> Environment: Jan 28 nightly build
> With patches 2095, 2119, and 2723
> Reporter: Christian Kunz
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.0
>
>
> I upgraded a Hadoop installation to the Jan 28 nightly build.
> DFS contains 2.4+ M files.
> Upgrade finished but not finalized.
> Before finalizing I wanted to run fsck on the DFS. It hardly progressed after 6 hours (not finished yet). With the '-files' option turned on, it lists about 300 entries in 10 minutes.
> And when I tried to list a subdirectory with 100,000 files, it repeatedly (about 20 attempts) timed out.
> Changing timeout from 1 to 10 minutes did not help.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2755) dfs fsck extremely slow, dfs ls
times out
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12564959#action_12564959 ]
dhruba borthakur commented on HADOOP-2755:
------------------------------------------
This patch does not merge cleanly on the 0.16 branch. Can you pl post a patch for the 0.16 branch?
> dfs fsck extremely slow, dfs ls times out
> -----------------------------------------
>
> Key: HADOOP-2755
> URL: https://issues.apache.org/jira/browse/HADOOP-2755
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.16.0
> Environment: Jan 28 nightly build
> With patches 2095, 2119, and 2723
> Reporter: Christian Kunz
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: 2755_20080131.patch, 2755_20080131b.patch
>
>
> I upgraded a Hadoop installation to the Jan 28 nightly build.
> DFS contains 2.4+ M files.
> Upgrade finished but not finalized.
> Before finalizing I wanted to run fsck on the DFS. It hardly progressed after 6 hours (not finished yet). With the '-files' option turned on, it lists about 300 entries in 10 minutes.
> And when I tried to list a subdirectory with 100,000 files, it repeatedly (about 20 attempts) timed out.
> Changing timeout from 1 to 10 minutes did not help.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2755) dfs fsck extremely slow, dfs ls
times out
Posted by "Christian Kunz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12564544#action_12564544 ]
Christian Kunz commented on HADOOP-2755:
----------------------------------------
This is a 1300 active node cluster with blockreports every hour each (they seem to take a couple of seconds each) and heartbeat interval every minute. This configuration did not change, but did anything change lately regarding heartbeat / blockreport?
> dfs fsck extremely slow, dfs ls times out
> -----------------------------------------
>
> Key: HADOOP-2755
> URL: https://issues.apache.org/jira/browse/HADOOP-2755
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.16.0
> Environment: Jan 28 nightly build
> With patches 2095, 2119, and 2723
> Reporter: Christian Kunz
> Assignee: dhruba borthakur
> Priority: Blocker
> Fix For: 0.16.0
>
>
> I upgraded a Hadoop installation to the Jan 28 nightly build.
> DFS contains 2.4+ M files.
> Upgrade finished but not finalized.
> Before finalizing I wanted to run fsck on the DFS. It hardly progressed after 6 hours (not finished yet). With the '-files' option turned on, it lists about 300 entries in 10 minutes.
> And when I tried to list a subdirectory with 100,000 files, it repeatedly (about 20 attempts) timed out.
> Changing timeout from 1 to 10 minutes did not help.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2755) dfs fsck extremely slow, dfs ls
times out
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12564528#action_12564528 ]
dhruba borthakur commented on HADOOP-2755:
------------------------------------------
The namenode machine is about 60% CPU busy. Also, HADOOP-2606 describes why the namenode consumes plenty of CPU when there a big bunch of blocks to be replicated.
That brings us to the next question: why are there so many blocks to be replicated? Maybe you have some datanodes that have not joined the cluster successfully?
> dfs fsck extremely slow, dfs ls times out
> -----------------------------------------
>
> Key: HADOOP-2755
> URL: https://issues.apache.org/jira/browse/HADOOP-2755
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.16.0
> Environment: Jan 28 nightly build
> With patches 2095, 2119, and 2723
> Reporter: Christian Kunz
> Assignee: dhruba borthakur
> Priority: Blocker
> Fix For: 0.16.0
>
>
> I upgraded a Hadoop installation to the Jan 28 nightly build.
> DFS contains 2.4+ M files.
> Upgrade finished but not finalized.
> Before finalizing I wanted to run fsck on the DFS. It hardly progressed after 6 hours (not finished yet). With the '-files' option turned on, it lists about 300 entries in 10 minutes.
> And when I tried to list a subdirectory with 100,000 files, it repeatedly (about 20 attempts) timed out.
> Changing timeout from 1 to 10 minutes did not help.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2755) dfs fsck extremely slow, dfs ls
times out
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12564627#action_12564627 ]
Hadoop QA commented on HADOOP-2755:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12374517/2755_20080131b.patch
against trunk revision 616796.
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new javac compiler warnings.
release audit +1. The applied patch does not generate any new release audit warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1719/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1719/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1719/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1719/console
This message is automatically generated.
> dfs fsck extremely slow, dfs ls times out
> -----------------------------------------
>
> Key: HADOOP-2755
> URL: https://issues.apache.org/jira/browse/HADOOP-2755
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.16.0
> Environment: Jan 28 nightly build
> With patches 2095, 2119, and 2723
> Reporter: Christian Kunz
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.16.0
>
> Attachments: 2755_20080131.patch, 2755_20080131b.patch
>
>
> I upgraded a Hadoop installation to the Jan 28 nightly build.
> DFS contains 2.4+ M files.
> Upgrade finished but not finalized.
> Before finalizing I wanted to run fsck on the DFS. It hardly progressed after 6 hours (not finished yet). With the '-files' option turned on, it lists about 300 entries in 10 minutes.
> And when I tried to list a subdirectory with 100,000 files, it repeatedly (about 20 attempts) timed out.
> Changing timeout from 1 to 10 minutes did not help.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.