You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Steve Loughran (JIRA)" <ji...@apache.org> on 2009/11/17 12:38:39 UTC
[jira] Created: (HDFS-775) FSDataset calls getCapacity() twice
-bug?
FSDataset calls getCapacity() twice -bug?
-----------------------------------------
Key: HDFS-775
URL: https://issues.apache.org/jira/browse/HDFS-775
Project: Hadoop HDFS
Issue Type: Bug
Components: data-node
Affects Versions: 0.22.0
Reporter: Steve Loughran
I'm not sure this is a bug or "as intended", but I thought I'd mention it.
FSDataset.getCapacity() calls DF.getCapacity() twice, when evaluating its capacity. Although there is caching to stop the shell being exec'd twice in a row, there is a risk that the first call doesn't run the shell, and the second does -so the value changes during the method.
If that is not intended, it is better to cache the first value for the whole method
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-775) FSDataset calls getCapacity() twice
-bug?
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12794615#action_12794615 ]
Hudson commented on HDFS-775:
-----------------------------
Integrated in Hadoop-Hdfs-trunk #182 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/182/])
> FSDataset calls getCapacity() twice -bug?
> -----------------------------------------
>
> Key: HDFS-775
> URL: https://issues.apache.org/jira/browse/HDFS-775
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 0.22.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: HDFS-775-1.patch, HDFS-775-2.patch
>
>
> I'm not sure this is a bug or "as intended", but I thought I'd mention it.
> FSDataset.getCapacity() calls DF.getCapacity() twice, when evaluating its capacity. Although there is caching to stop the shell being exec'd twice in a row, there is a risk that the first call doesn't run the shell, and the second does -so the value changes during the method.
> If that is not intended, it is better to cache the first value for the whole method
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-775) FSDataset calls getCapacity() twice
-bug?
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12794248#action_12794248 ]
Tsz Wo (Nicholas), SZE commented on HDFS-775:
---------------------------------------------
Hi Steve, forgot to update CHANGE.txt?
> FSDataset calls getCapacity() twice -bug?
> -----------------------------------------
>
> Key: HDFS-775
> URL: https://issues.apache.org/jira/browse/HDFS-775
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 0.22.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: HDFS-775-1.patch, HDFS-775-2.patch
>
>
> I'm not sure this is a bug or "as intended", but I thought I'd mention it.
> FSDataset.getCapacity() calls DF.getCapacity() twice, when evaluating its capacity. Although there is caching to stop the shell being exec'd twice in a row, there is a risk that the first call doesn't run the shell, and the second does -so the value changes during the method.
> If that is not intended, it is better to cache the first value for the whole method
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-775) FSDataset calls getCapacity() twice
-bug?
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12788152#action_12788152 ]
Hadoop QA commented on HDFS-775:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12427479/HDFS-775-2.patch
against trunk revision 888538.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/139/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/139/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/139/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/139/console
This message is automatically generated.
> FSDataset calls getCapacity() twice -bug?
> -----------------------------------------
>
> Key: HDFS-775
> URL: https://issues.apache.org/jira/browse/HDFS-775
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 0.22.0
> Reporter: Steve Loughran
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: HDFS-775-1.patch, HDFS-775-2.patch
>
>
> I'm not sure this is a bug or "as intended", but I thought I'd mention it.
> FSDataset.getCapacity() calls DF.getCapacity() twice, when evaluating its capacity. Although there is caching to stop the shell being exec'd twice in a row, there is a risk that the first call doesn't run the shell, and the second does -so the value changes during the method.
> If that is not intended, it is better to cache the first value for the whole method
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-775) FSDataset calls getCapacity() twice
-bug?
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated HDFS-775:
--------------------------------
Resolution: Fixed
Assignee: Steve Loughran
Status: Resolved (was: Patch Available)
> FSDataset calls getCapacity() twice -bug?
> -----------------------------------------
>
> Key: HDFS-775
> URL: https://issues.apache.org/jira/browse/HDFS-775
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 0.22.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: HDFS-775-1.patch, HDFS-775-2.patch
>
>
> I'm not sure this is a bug or "as intended", but I thought I'd mention it.
> FSDataset.getCapacity() calls DF.getCapacity() twice, when evaluating its capacity. Although there is caching to stop the shell being exec'd twice in a row, there is a risk that the first call doesn't run the shell, and the second does -so the value changes during the method.
> If that is not intended, it is better to cache the first value for the whole method
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-775) FSDataset calls getCapacity() twice
-bug?
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12794236#action_12794236 ]
Hudson commented on HDFS-775:
-----------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #156 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/156/])
FSDataset calls getCapacity() twice
> FSDataset calls getCapacity() twice -bug?
> -----------------------------------------
>
> Key: HDFS-775
> URL: https://issues.apache.org/jira/browse/HDFS-775
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 0.22.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: HDFS-775-1.patch, HDFS-775-2.patch
>
>
> I'm not sure this is a bug or "as intended", but I thought I'd mention it.
> FSDataset.getCapacity() calls DF.getCapacity() twice, when evaluating its capacity. Although there is caching to stop the shell being exec'd twice in a row, there is a risk that the first call doesn't run the shell, and the second does -so the value changes during the method.
> If that is not intended, it is better to cache the first value for the whole method
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-775) FSDataset calls getCapacity() twice
-bug?
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated HDFS-775:
--------------------------------
Attachment: HDFS-775-2.patch
patch with corrected path
> FSDataset calls getCapacity() twice -bug?
> -----------------------------------------
>
> Key: HDFS-775
> URL: https://issues.apache.org/jira/browse/HDFS-775
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 0.22.0
> Reporter: Steve Loughran
> Priority: Minor
> Attachments: HDFS-775-1.patch, HDFS-775-2.patch
>
>
> I'm not sure this is a bug or "as intended", but I thought I'd mention it.
> FSDataset.getCapacity() calls DF.getCapacity() twice, when evaluating its capacity. Although there is caching to stop the shell being exec'd twice in a row, there is a risk that the first call doesn't run the shell, and the second does -so the value changes during the method.
> If that is not intended, it is better to cache the first value for the whole method
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-775) FSDataset calls getCapacity() twice
-bug?
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12788354#action_12788354 ]
Todd Lipcon commented on HDFS-775:
----------------------------------
+1. Patch looks good to me.
> FSDataset calls getCapacity() twice -bug?
> -----------------------------------------
>
> Key: HDFS-775
> URL: https://issues.apache.org/jira/browse/HDFS-775
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 0.22.0
> Reporter: Steve Loughran
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: HDFS-775-1.patch, HDFS-775-2.patch
>
>
> I'm not sure this is a bug or "as intended", but I thought I'd mention it.
> FSDataset.getCapacity() calls DF.getCapacity() twice, when evaluating its capacity. Although there is caching to stop the shell being exec'd twice in a row, there is a risk that the first call doesn't run the shell, and the second does -so the value changes during the method.
> If that is not intended, it is better to cache the first value for the whole method
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-775) FSDataset calls getCapacity() twice
-bug?
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12778828#action_12778828 ]
Steve Loughran commented on HDFS-775:
-------------------------------------
Here's the code
{code}
long getCapacity() throws IOException {
if (reserved > usage.getCapacity()) { //FIRST CALL
return 0;
}
return usage.getCapacity()-reserved; //SECOND CALL
}
{code}
It looks like the method intends to return capacity as a number >=0, but if the second invocation triggers a shell exec the capacity could decrease and the return value could then be negative, which could have implications elsewhere.
Looking at the usages, FSVolumeSet can get confused by this, as it adds the capacities of all volumes together, no checks for being below zero.
{code}
long getCapacity() throws IOException {
long capacity = 0L;
for (int idx = 0; idx < volumes.length; idx++) {
capacity += volumes[idx].getCapacity();
}
return capacity;
}
{code}
A negative capacity from one volume would make the entire datanode capacity appear smaller than it is.
> FSDataset calls getCapacity() twice -bug?
> -----------------------------------------
>
> Key: HDFS-775
> URL: https://issues.apache.org/jira/browse/HDFS-775
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 0.22.0
> Reporter: Steve Loughran
>
> I'm not sure this is a bug or "as intended", but I thought I'd mention it.
> FSDataset.getCapacity() calls DF.getCapacity() twice, when evaluating its capacity. Although there is caching to stop the shell being exec'd twice in a row, there is a risk that the first call doesn't run the shell, and the second does -so the value changes during the method.
> If that is not intended, it is better to cache the first value for the whole method
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-775) FSDataset calls getCapacity() twice
-bug?
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12794352#action_12794352 ]
Hudson commented on HDFS-775:
-----------------------------
Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #159 (See [http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/159/])
FSDataset calls getCapacity() twice
> FSDataset calls getCapacity() twice -bug?
> -----------------------------------------
>
> Key: HDFS-775
> URL: https://issues.apache.org/jira/browse/HDFS-775
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 0.22.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: HDFS-775-1.patch, HDFS-775-2.patch
>
>
> I'm not sure this is a bug or "as intended", but I thought I'd mention it.
> FSDataset.getCapacity() calls DF.getCapacity() twice, when evaluating its capacity. Although there is caching to stop the shell being exec'd twice in a row, there is a risk that the first call doesn't run the shell, and the second does -so the value changes during the method.
> If that is not intended, it is better to cache the first value for the whole method
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-775) FSDataset calls getCapacity() twice
-bug?
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12799006#action_12799006 ]
Hudson commented on HDFS-775:
-----------------------------
Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #94 (See [http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/94/])
> FSDataset calls getCapacity() twice -bug?
> -----------------------------------------
>
> Key: HDFS-775
> URL: https://issues.apache.org/jira/browse/HDFS-775
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 0.22.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: HDFS-775-1.patch, HDFS-775-2.patch
>
>
> I'm not sure this is a bug or "as intended", but I thought I'd mention it.
> FSDataset.getCapacity() calls DF.getCapacity() twice, when evaluating its capacity. Although there is caching to stop the shell being exec'd twice in a row, there is a risk that the first call doesn't run the shell, and the second does -so the value changes during the method.
> If that is not intended, it is better to cache the first value for the whole method
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-775) FSDataset calls getCapacity() twice
-bug?
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12794220#action_12794220 ]
Steve Loughran commented on HDFS-775:
-------------------------------------
committed.
> FSDataset calls getCapacity() twice -bug?
> -----------------------------------------
>
> Key: HDFS-775
> URL: https://issues.apache.org/jira/browse/HDFS-775
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 0.22.0
> Reporter: Steve Loughran
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: HDFS-775-1.patch, HDFS-775-2.patch
>
>
> I'm not sure this is a bug or "as intended", but I thought I'd mention it.
> FSDataset.getCapacity() calls DF.getCapacity() twice, when evaluating its capacity. Although there is caching to stop the shell being exec'd twice in a row, there is a risk that the first call doesn't run the shell, and the second does -so the value changes during the method.
> If that is not intended, it is better to cache the first value for the whole method
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-775) FSDataset calls getCapacity() twice
-bug?
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated HDFS-775:
--------------------------------
Fix Version/s: 0.22.0
Status: Patch Available (was: Open)
No tests; this is a race condition being eliminated here. Proofs of correctness only are sufficient, and I don't do those.
> FSDataset calls getCapacity() twice -bug?
> -----------------------------------------
>
> Key: HDFS-775
> URL: https://issues.apache.org/jira/browse/HDFS-775
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 0.22.0
> Reporter: Steve Loughran
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: HDFS-775-1.patch, HDFS-775-2.patch
>
>
> I'm not sure this is a bug or "as intended", but I thought I'd mention it.
> FSDataset.getCapacity() calls DF.getCapacity() twice, when evaluating its capacity. Although there is caching to stop the shell being exec'd twice in a row, there is a risk that the first call doesn't run the shell, and the second does -so the value changes during the method.
> If that is not intended, it is better to cache the first value for the whole method
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-775) FSDataset calls getCapacity() twice
-bug?
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated HDFS-775:
--------------------------------
Attachment: HDFS-775-1.patch
This is a patch for this; I am using this defect to understand how to work with git/github for patches, so apologies if it does not apply properly - the path looks odd and may need a -p1 to take
> FSDataset calls getCapacity() twice -bug?
> -----------------------------------------
>
> Key: HDFS-775
> URL: https://issues.apache.org/jira/browse/HDFS-775
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 0.22.0
> Reporter: Steve Loughran
> Priority: Minor
> Attachments: HDFS-775-1.patch
>
>
> I'm not sure this is a bug or "as intended", but I thought I'd mention it.
> FSDataset.getCapacity() calls DF.getCapacity() twice, when evaluating its capacity. Although there is caching to stop the shell being exec'd twice in a row, there is a risk that the first call doesn't run the shell, and the second does -so the value changes during the method.
> If that is not intended, it is better to cache the first value for the whole method
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-775) FSDataset calls getCapacity() twice
-bug?
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated HDFS-775:
--------------------------------
Priority: Minor (was: Major)
minor defect; its transient and nothing will actually break.
> FSDataset calls getCapacity() twice -bug?
> -----------------------------------------
>
> Key: HDFS-775
> URL: https://issues.apache.org/jira/browse/HDFS-775
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 0.22.0
> Reporter: Steve Loughran
> Priority: Minor
>
> I'm not sure this is a bug or "as intended", but I thought I'd mention it.
> FSDataset.getCapacity() calls DF.getCapacity() twice, when evaluating its capacity. Although there is caching to stop the shell being exec'd twice in a row, there is a risk that the first call doesn't run the shell, and the second does -so the value changes during the method.
> If that is not intended, it is better to cache the first value for the whole method
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.