You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Robert J Berger (Created) (JIRA)" <ji...@apache.org> on 2012/02/17 11:16:00 UTC
[jira] [Created] (HDFS-2964) No notice in any logs if
dfs.datanode.du.reserved is greater than available disk space
No notice in any logs if dfs.datanode.du.reserved is greater than available disk space
--------------------------------------------------------------------------------------
Key: HDFS-2964
URL: https://issues.apache.org/jira/browse/HDFS-2964
Project: Hadoop HDFS
Issue Type: Bug
Components: data-node, name-node
Affects Versions: 0.20.2
Reporter: Robert J Berger
Priority: Minor
We spent a long time tracking down why a test hdfs cluster seemed to be running fine, but would not allow the mapred system to come up complaining that "could only be replicated to 0 nodes, instead of 1".
There were no namenode or datanode errors in any of the logs. hadoop fsck said everything was good. At first glance dfsadmin -report looked good. It wasn't until I realized that there was 0 Capacity available that we poked around and found https://groups.google.com/a/cloudera.org/group/scm-users/msg/a4252d6623adbc2d which mentioned that the "reserverd space" might be greater than the disk space available. And we did find that our dfs.datanode.du.reserved was indeed higher than our actual since we were only testing a small cluster.
It seems that there should be some warning or error in the logs that say that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2964) No notice in any logs if
dfs.datanode.du.reserved is greater than available disk space
Posted by "Harsh J (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13246193#comment-13246193 ]
Harsh J commented on HDFS-2964:
-------------------------------
Hello Madhukara,
Please attach a proper diff that carries only the changes you have done. Your current diff is a total rewrite of file. Ensure you use unix newlines in your editor/save-actions/etc., in case that is what led to this.
> No notice in any logs if dfs.datanode.du.reserved is greater than available disk space
> --------------------------------------------------------------------------------------
>
> Key: HDFS-2964
> URL: https://issues.apache.org/jira/browse/HDFS-2964
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Affects Versions: 0.20.2
> Reporter: Robert J Berger
> Priority: Minor
> Attachments: HDFS-2964.patch
>
>
> We spent a long time tracking down why a test hdfs cluster seemed to be running fine, but would not allow the mapred system to come up complaining that "could only be replicated to 0 nodes, instead of 1".
> There were no namenode or datanode errors in any of the logs. hadoop fsck said everything was good. At first glance dfsadmin -report looked good. It wasn't until I realized that there was 0 Capacity available that we poked around and found https://groups.google.com/a/cloudera.org/group/scm-users/msg/a4252d6623adbc2d which mentioned that the "reserverd space" might be greater than the disk space available. And we did find that our dfs.datanode.du.reserved was indeed higher than our actual since we were only testing a small cluster.
> It seems that there should be some warning or error in the logs that say that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2964) No notice in any logs if
dfs.datanode.du.reserved is greater than available disk space
Posted by "madhukara phatak (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
madhukara phatak updated HDFS-2964:
-----------------------------------
Attachment: HDFS-2964.patch
Added a warning message in FsVolumeImpl.java where it reads the latest capacity. I m creating the patch first time so not sure its the correct place to add the warning message.
> No notice in any logs if dfs.datanode.du.reserved is greater than available disk space
> --------------------------------------------------------------------------------------
>
> Key: HDFS-2964
> URL: https://issues.apache.org/jira/browse/HDFS-2964
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Affects Versions: 0.20.2
> Reporter: Robert J Berger
> Priority: Minor
> Attachments: HDFS-2964.patch
>
>
> We spent a long time tracking down why a test hdfs cluster seemed to be running fine, but would not allow the mapred system to come up complaining that "could only be replicated to 0 nodes, instead of 1".
> There were no namenode or datanode errors in any of the logs. hadoop fsck said everything was good. At first glance dfsadmin -report looked good. It wasn't until I realized that there was 0 Capacity available that we poked around and found https://groups.google.com/a/cloudera.org/group/scm-users/msg/a4252d6623adbc2d which mentioned that the "reserverd space" might be greater than the disk space available. And we did find that our dfs.datanode.du.reserved was indeed higher than our actual since we were only testing a small cluster.
> It seems that there should be some warning or error in the logs that say that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2964) No notice in any logs if
dfs.datanode.du.reserved is greater than available disk space
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13284789#comment-13284789 ]
Harsh J commented on HDFS-2964:
-------------------------------
Thanks for the patch! Can you make sure to follow all code guidelines?
For instance, the if-else setup looks a little odd compared to the rest of the source base. For inline comments use // not /*. You can also place comments within the else-block rather than outside.
Otherwise looks good.
> No notice in any logs if dfs.datanode.du.reserved is greater than available disk space
> --------------------------------------------------------------------------------------
>
> Key: HDFS-2964
> URL: https://issues.apache.org/jira/browse/HDFS-2964
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Affects Versions: 0.20.2
> Reporter: Robert J Berger
> Priority: Minor
> Attachments: HDFS-2964-1.patch, HDFS-2964.patch
>
>
> We spent a long time tracking down why a test hdfs cluster seemed to be running fine, but would not allow the mapred system to come up complaining that "could only be replicated to 0 nodes, instead of 1".
> There were no namenode or datanode errors in any of the logs. hadoop fsck said everything was good. At first glance dfsadmin -report looked good. It wasn't until I realized that there was 0 Capacity available that we poked around and found https://groups.google.com/a/cloudera.org/group/scm-users/msg/a4252d6623adbc2d which mentioned that the "reserverd space" might be greater than the disk space available. And we did find that our dfs.datanode.du.reserved was indeed higher than our actual since we were only testing a small cluster.
> It seems that there should be some warning or error in the logs that say that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2964) No notice in any logs if
dfs.datanode.du.reserved is greater than available disk space
Posted by "madhukara phatak (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247109#comment-13247109 ]
madhukara phatak commented on HDFS-2964:
----------------------------------------
This is just addition for log message so no need of test case.
> No notice in any logs if dfs.datanode.du.reserved is greater than available disk space
> --------------------------------------------------------------------------------------
>
> Key: HDFS-2964
> URL: https://issues.apache.org/jira/browse/HDFS-2964
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Affects Versions: 0.20.2
> Reporter: Robert J Berger
> Priority: Minor
> Attachments: HDFS-2964-1.patch, HDFS-2964.patch
>
>
> We spent a long time tracking down why a test hdfs cluster seemed to be running fine, but would not allow the mapred system to come up complaining that "could only be replicated to 0 nodes, instead of 1".
> There were no namenode or datanode errors in any of the logs. hadoop fsck said everything was good. At first glance dfsadmin -report looked good. It wasn't until I realized that there was 0 Capacity available that we poked around and found https://groups.google.com/a/cloudera.org/group/scm-users/msg/a4252d6623adbc2d which mentioned that the "reserverd space" might be greater than the disk space available. And we did find that our dfs.datanode.du.reserved was indeed higher than our actual since we were only testing a small cluster.
> It seems that there should be some warning or error in the logs that say that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2964) No notice in any logs if
dfs.datanode.du.reserved is greater than available disk space
Posted by "Robert J Berger (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13211280#comment-13211280 ]
Robert J Berger commented on HDFS-2964:
---------------------------------------
I haven't really hacked on the hadoop source code and have almost no java fu, so it would be a pretty steep learning curve. If someone pointed me to the right place to look I could take a stab at it.
> No notice in any logs if dfs.datanode.du.reserved is greater than available disk space
> --------------------------------------------------------------------------------------
>
> Key: HDFS-2964
> URL: https://issues.apache.org/jira/browse/HDFS-2964
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Affects Versions: 0.20.2
> Reporter: Robert J Berger
> Priority: Minor
>
> We spent a long time tracking down why a test hdfs cluster seemed to be running fine, but would not allow the mapred system to come up complaining that "could only be replicated to 0 nodes, instead of 1".
> There were no namenode or datanode errors in any of the logs. hadoop fsck said everything was good. At first glance dfsadmin -report looked good. It wasn't until I realized that there was 0 Capacity available that we poked around and found https://groups.google.com/a/cloudera.org/group/scm-users/msg/a4252d6623adbc2d which mentioned that the "reserverd space" might be greater than the disk space available. And we did find that our dfs.datanode.du.reserved was indeed higher than our actual since we were only testing a small cluster.
> It seems that there should be some warning or error in the logs that say that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2964) No notice in any logs if
dfs.datanode.du.reserved is greater than available disk space
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247061#comment-13247061 ]
Hadoop QA commented on HDFS-2964:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12521306/HDFS-2964-1.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2193//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2193//console
This message is automatically generated.
> No notice in any logs if dfs.datanode.du.reserved is greater than available disk space
> --------------------------------------------------------------------------------------
>
> Key: HDFS-2964
> URL: https://issues.apache.org/jira/browse/HDFS-2964
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Affects Versions: 0.20.2
> Reporter: Robert J Berger
> Priority: Minor
> Attachments: HDFS-2964-1.patch, HDFS-2964.patch
>
>
> We spent a long time tracking down why a test hdfs cluster seemed to be running fine, but would not allow the mapred system to come up complaining that "could only be replicated to 0 nodes, instead of 1".
> There were no namenode or datanode errors in any of the logs. hadoop fsck said everything was good. At first glance dfsadmin -report looked good. It wasn't until I realized that there was 0 Capacity available that we poked around and found https://groups.google.com/a/cloudera.org/group/scm-users/msg/a4252d6623adbc2d which mentioned that the "reserverd space" might be greater than the disk space available. And we did find that our dfs.datanode.du.reserved was indeed higher than our actual since we were only testing a small cluster.
> It seems that there should be some warning or error in the logs that say that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2964) No notice in any logs if
dfs.datanode.du.reserved is greater than available disk space
Posted by "Harsh J (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13210265#comment-13210265 ]
Harsh J commented on HDFS-2964:
-------------------------------
I guess currently the number is loaded first and the disk-space check is done later but we should be able to easily add a WARN when computing the after-reserve values. Would you like to contribute a patch for this Robert, against trunk?
> No notice in any logs if dfs.datanode.du.reserved is greater than available disk space
> --------------------------------------------------------------------------------------
>
> Key: HDFS-2964
> URL: https://issues.apache.org/jira/browse/HDFS-2964
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Affects Versions: 0.20.2
> Reporter: Robert J Berger
> Priority: Minor
>
> We spent a long time tracking down why a test hdfs cluster seemed to be running fine, but would not allow the mapred system to come up complaining that "could only be replicated to 0 nodes, instead of 1".
> There were no namenode or datanode errors in any of the logs. hadoop fsck said everything was good. At first glance dfsadmin -report looked good. It wasn't until I realized that there was 0 Capacity available that we poked around and found https://groups.google.com/a/cloudera.org/group/scm-users/msg/a4252d6623adbc2d which mentioned that the "reserverd space" might be greater than the disk space available. And we did find that our dfs.datanode.du.reserved was indeed higher than our actual since we were only testing a small cluster.
> It seems that there should be some warning or error in the logs that say that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2964) No notice in any logs if
dfs.datanode.du.reserved is greater than available disk space
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13284792#comment-13284792 ]
Harsh J commented on HDFS-2964:
-------------------------------
Perhaps message can be better too:
> "Disk capacity is lower than reserved capacity"
Instead we may perhaps say: "Available disk capacity on <disk path> is lower than configured reserved space per volume (dfs.datanode.du.reserved)."
Thoughts?
> No notice in any logs if dfs.datanode.du.reserved is greater than available disk space
> --------------------------------------------------------------------------------------
>
> Key: HDFS-2964
> URL: https://issues.apache.org/jira/browse/HDFS-2964
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Affects Versions: 0.20.2
> Reporter: Robert J Berger
> Priority: Minor
> Attachments: HDFS-2964-1.patch, HDFS-2964.patch
>
>
> We spent a long time tracking down why a test hdfs cluster seemed to be running fine, but would not allow the mapred system to come up complaining that "could only be replicated to 0 nodes, instead of 1".
> There were no namenode or datanode errors in any of the logs. hadoop fsck said everything was good. At first glance dfsadmin -report looked good. It wasn't until I realized that there was 0 Capacity available that we poked around and found https://groups.google.com/a/cloudera.org/group/scm-users/msg/a4252d6623adbc2d which mentioned that the "reserverd space" might be greater than the disk space available. And we did find that our dfs.datanode.du.reserved was indeed higher than our actual since we were only testing a small cluster.
> It seems that there should be some warning or error in the logs that say that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2964) No notice in any logs if
dfs.datanode.du.reserved is greater than available disk space
Posted by "Steve Loughran (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13213712#comment-13213712 ]
Steve Loughran commented on HDFS-2964:
--------------------------------------
made sure this is highlighted in the wiki entry http://wiki.apache.org/hadoop/CouldOnlyBeReplicatedTo ; it was there, but is now in its own bullet point.
Presumably this test cluster only had one or two DNs, as the JT only wants one location to save its data. Perhaps the NN status should highlight the "no space left" situation more, it's certainly something you want an alert on in production.
> No notice in any logs if dfs.datanode.du.reserved is greater than available disk space
> --------------------------------------------------------------------------------------
>
> Key: HDFS-2964
> URL: https://issues.apache.org/jira/browse/HDFS-2964
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Affects Versions: 0.20.2
> Reporter: Robert J Berger
> Priority: Minor
>
> We spent a long time tracking down why a test hdfs cluster seemed to be running fine, but would not allow the mapred system to come up complaining that "could only be replicated to 0 nodes, instead of 1".
> There were no namenode or datanode errors in any of the logs. hadoop fsck said everything was good. At first glance dfsadmin -report looked good. It wasn't until I realized that there was 0 Capacity available that we poked around and found https://groups.google.com/a/cloudera.org/group/scm-users/msg/a4252d6623adbc2d which mentioned that the "reserverd space" might be greater than the disk space available. And we did find that our dfs.datanode.du.reserved was indeed higher than our actual since we were only testing a small cluster.
> It seems that there should be some warning or error in the logs that say that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2964) No notice in any logs if
dfs.datanode.du.reserved is greater than available disk space
Posted by "madhukara phatak (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
madhukara phatak updated HDFS-2964:
-----------------------------------
Attachment: HDFS-2964-1.patch
Fixed the patch file to have only changed code and converted new lines into UNIX
> No notice in any logs if dfs.datanode.du.reserved is greater than available disk space
> --------------------------------------------------------------------------------------
>
> Key: HDFS-2964
> URL: https://issues.apache.org/jira/browse/HDFS-2964
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Affects Versions: 0.20.2
> Reporter: Robert J Berger
> Priority: Minor
> Attachments: HDFS-2964-1.patch, HDFS-2964.patch
>
>
> We spent a long time tracking down why a test hdfs cluster seemed to be running fine, but would not allow the mapred system to come up complaining that "could only be replicated to 0 nodes, instead of 1".
> There were no namenode or datanode errors in any of the logs. hadoop fsck said everything was good. At first glance dfsadmin -report looked good. It wasn't until I realized that there was 0 Capacity available that we poked around and found https://groups.google.com/a/cloudera.org/group/scm-users/msg/a4252d6623adbc2d which mentioned that the "reserverd space" might be greater than the disk space available. And we did find that our dfs.datanode.du.reserved was indeed higher than our actual since we were only testing a small cluster.
> It seems that there should be some warning or error in the logs that say that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2964) No notice in any logs if
dfs.datanode.du.reserved is greater than available disk space
Posted by "madhukara phatak (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
madhukara phatak updated HDFS-2964:
-----------------------------------
Status: Patch Available (was: Open)
> No notice in any logs if dfs.datanode.du.reserved is greater than available disk space
> --------------------------------------------------------------------------------------
>
> Key: HDFS-2964
> URL: https://issues.apache.org/jira/browse/HDFS-2964
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, name-node
> Affects Versions: 0.20.2
> Reporter: Robert J Berger
> Priority: Minor
> Attachments: HDFS-2964-1.patch, HDFS-2964.patch
>
>
> We spent a long time tracking down why a test hdfs cluster seemed to be running fine, but would not allow the mapred system to come up complaining that "could only be replicated to 0 nodes, instead of 1".
> There were no namenode or datanode errors in any of the logs. hadoop fsck said everything was good. At first glance dfsadmin -report looked good. It wasn't until I realized that there was 0 Capacity available that we poked around and found https://groups.google.com/a/cloudera.org/group/scm-users/msg/a4252d6623adbc2d which mentioned that the "reserverd space" might be greater than the disk space available. And we did find that our dfs.datanode.du.reserved was indeed higher than our actual since we were only testing a small cluster.
> It seems that there should be some warning or error in the logs that say that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira