You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Pritam Damania (JIRA)" <ji...@apache.org> on 2011/08/04 00:23:27 UTC
[jira] [Created] (HBASE-4161) Incorrect use of listStatus() in
HBase region initialization.
Incorrect use of listStatus() in HBase region initialization.
-------------------------------------------------------------
Key: HBASE-4161
URL: https://issues.apache.org/jira/browse/HBASE-4161
Project: HBase
Issue Type: Bug
Components: regionserver
Reporter: Pritam Damania
While opening a region, the HBase regionserver tries to list all the children in a "recovered.edits" directory. This directory may not exist and depending on the version of HDFS listStatus() might return null or an exception. If it does throw an exception the entire process of opening the region is aborted, just because the recovered.edits directory is not present.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4161) Incorrect use of listStatus() in
HBase region initialization.
Posted by "Pritam Damania (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13079081#comment-13079081 ]
Pritam Damania commented on HBASE-4161:
---------------------------------------
Attached a patch that fixes this issue.
> Incorrect use of listStatus() in HBase region initialization.
> -------------------------------------------------------------
>
> Key: HBASE-4161
> URL: https://issues.apache.org/jira/browse/HBASE-4161
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Reporter: Pritam Damania
> Attachments: 0001-Fix-FileNotFoundException-in-HLog.java.patch
>
>
> While opening a region, the HBase regionserver tries to list all the children in a "recovered.edits" directory. This directory may not exist and depending on the version of HDFS listStatus() might return null or an exception. If it does throw an exception the entire process of opening the region is aborted, just because the recovered.edits directory is not present.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4161) Incorrect use of listStatus() in
HBase region initialization.
Posted by "Pritam Damania (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pritam Damania updated HBASE-4161:
----------------------------------
Attachment: 0001-Fix-FileNotFoundException-in-HLog.java.patch
> Incorrect use of listStatus() in HBase region initialization.
> -------------------------------------------------------------
>
> Key: HBASE-4161
> URL: https://issues.apache.org/jira/browse/HBASE-4161
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Reporter: Pritam Damania
> Attachments: 0001-Fix-FileNotFoundException-in-HLog.java.patch
>
>
> While opening a region, the HBase regionserver tries to list all the children in a "recovered.edits" directory. This directory may not exist and depending on the version of HDFS listStatus() might return null or an exception. If it does throw an exception the entire process of opening the region is aborted, just because the recovered.edits directory is not present.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4161) Incorrect use of listStatus() in
HBase region initialization.
Posted by "Pritam Damania (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pritam Damania updated HBASE-4161:
----------------------------------
Attachment: 0001-Fix-FileNotFoundException-in-HLog.java.patch
> Incorrect use of listStatus() in HBase region initialization.
> -------------------------------------------------------------
>
> Key: HBASE-4161
> URL: https://issues.apache.org/jira/browse/HBASE-4161
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Reporter: Pritam Damania
> Attachments: 0001-Fix-FileNotFoundException-in-HLog.java.patch
>
>
> While opening a region, the HBase regionserver tries to list all the children in a "recovered.edits" directory. This directory may not exist and depending on the version of HDFS listStatus() might return null or an exception. If it does throw an exception the entire process of opening the region is aborted, just because the recovered.edits directory is not present.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4161) Incorrect use of listStatus() in
HBase region initialization.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081863#comment-13081863 ]
stack commented on HBASE-4161:
------------------------------
Is this patch for hbase TRUNK? We seem to have a check for dir existence already in TRUNK.
> Incorrect use of listStatus() in HBase region initialization.
> -------------------------------------------------------------
>
> Key: HBASE-4161
> URL: https://issues.apache.org/jira/browse/HBASE-4161
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Reporter: Pritam Damania
> Attachments: 0001-Fix-FileNotFoundException-in-HLog.java.patch
>
>
> While opening a region, the HBase regionserver tries to list all the children in a "recovered.edits" directory. This directory may not exist and depending on the version of HDFS listStatus() might return null or an exception. If it does throw an exception the entire process of opening the region is aborted, just because the recovered.edits directory is not present.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4161) Incorrect use of listStatus() in
HBase region initialization.
Posted by "Pritam Damania (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pritam Damania updated HBASE-4161:
----------------------------------
Attachment: (was: 0001-Fix-FileNotFoundException-in-HLog.java.patch)
> Incorrect use of listStatus() in HBase region initialization.
> -------------------------------------------------------------
>
> Key: HBASE-4161
> URL: https://issues.apache.org/jira/browse/HBASE-4161
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Reporter: Pritam Damania
>
> While opening a region, the HBase regionserver tries to list all the children in a "recovered.edits" directory. This directory may not exist and depending on the version of HDFS listStatus() might return null or an exception. If it does throw an exception the entire process of opening the region is aborted, just because the recovered.edits directory is not present.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4161) Incorrect use of listStatus() in
HBase region initialization.
Posted by "Pritam Damania (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13079805#comment-13079805 ]
Pritam Damania commented on HBASE-4161:
---------------------------------------
in FileSystem.java :
{code}
public boolean exists(Path f) throws IOException {
try {
return getFileStatus(f) != null;
} catch (FileNotFoundException e) {
return false;
}
}
{code}
and DistributedFileSystem.java does not override the method exists(), so we will not have the FileNotFoundException, we would just return with false.
> Incorrect use of listStatus() in HBase region initialization.
> -------------------------------------------------------------
>
> Key: HBASE-4161
> URL: https://issues.apache.org/jira/browse/HBASE-4161
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Reporter: Pritam Damania
> Attachments: 0001-Fix-FileNotFoundException-in-HLog.java.patch
>
>
> While opening a region, the HBase regionserver tries to list all the children in a "recovered.edits" directory. This directory may not exist and depending on the version of HDFS listStatus() might return null or an exception. If it does throw an exception the entire process of opening the region is aborted, just because the recovered.edits directory is not present.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4161) Incorrect use of listStatus() in
HBase region initialization.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085943#comment-13085943 ]
stack commented on HBASE-4161:
------------------------------
No worries Pritam. You want this applied to 0.90 then?
> Incorrect use of listStatus() in HBase region initialization.
> -------------------------------------------------------------
>
> Key: HBASE-4161
> URL: https://issues.apache.org/jira/browse/HBASE-4161
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Reporter: Pritam Damania
> Attachments: 0001-Fix-FileNotFoundException-in-HLog.java.patch
>
>
> While opening a region, the HBase regionserver tries to list all the children in a "recovered.edits" directory. This directory may not exist and depending on the version of HDFS listStatus() might return null or an exception. If it does throw an exception the entire process of opening the region is aborted, just because the recovered.edits directory is not present.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4161) Incorrect use of listStatus() in
HBase region initialization.
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13079955#comment-13079955 ]
Ted Yu commented on HBASE-4161:
-------------------------------
Thanks for the update.
Looking at processDeleteOnExit in FileSystem.java:
{code}
catch (IOException e) {
LOG.info("Ignoring failure to deleteOnExit for path " + path);
}
{code}
exists() throws IOException.
I wonder what other IOException may come out of exists(). Should we catch such exception ?
> Incorrect use of listStatus() in HBase region initialization.
> -------------------------------------------------------------
>
> Key: HBASE-4161
> URL: https://issues.apache.org/jira/browse/HBASE-4161
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Reporter: Pritam Damania
> Attachments: 0001-Fix-FileNotFoundException-in-HLog.java.patch
>
>
> While opening a region, the HBase regionserver tries to list all the children in a "recovered.edits" directory. This directory may not exist and depending on the version of HDFS listStatus() might return null or an exception. If it does throw an exception the entire process of opening the region is aborted, just because the recovered.edits directory is not present.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4161) Incorrect use of listStatus() in
HBase region initialization.
Posted by "Pritam Damania (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13080146#comment-13080146 ]
Pritam Damania commented on HBASE-4161:
---------------------------------------
This doesn't seem related to the exists() codepath ? The other exceptions that may come out of exists() are when the DFSClient is down or the namenode is down, we don't want to catch these exceptions and instead just throw them.
> Incorrect use of listStatus() in HBase region initialization.
> -------------------------------------------------------------
>
> Key: HBASE-4161
> URL: https://issues.apache.org/jira/browse/HBASE-4161
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Reporter: Pritam Damania
> Attachments: 0001-Fix-FileNotFoundException-in-HLog.java.patch
>
>
> While opening a region, the HBase regionserver tries to list all the children in a "recovered.edits" directory. This directory may not exist and depending on the version of HDFS listStatus() might return null or an exception. If it does throw an exception the entire process of opening the region is aborted, just because the recovered.edits directory is not present.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4161) Incorrect use of listStatus() in
HBase region initialization.
Posted by "Pritam Damania (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086007#comment-13086007 ]
Pritam Damania commented on HBASE-4161:
---------------------------------------
Yes if it does apply to 90.
> Incorrect use of listStatus() in HBase region initialization.
> -------------------------------------------------------------
>
> Key: HBASE-4161
> URL: https://issues.apache.org/jira/browse/HBASE-4161
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Reporter: Pritam Damania
> Attachments: 0001-Fix-FileNotFoundException-in-HLog.java.patch
>
>
> While opening a region, the HBase regionserver tries to list all the children in a "recovered.edits" directory. This directory may not exist and depending on the version of HDFS listStatus() might return null or an exception. If it does throw an exception the entire process of opening the region is aborted, just because the recovered.edits directory is not present.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4161) Incorrect use of listStatus() in
HBase region initialization.
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13080150#comment-13080150 ]
Ted Yu commented on HBASE-4161:
-------------------------------
That's fine.
> Incorrect use of listStatus() in HBase region initialization.
> -------------------------------------------------------------
>
> Key: HBASE-4161
> URL: https://issues.apache.org/jira/browse/HBASE-4161
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Reporter: Pritam Damania
> Attachments: 0001-Fix-FileNotFoundException-in-HLog.java.patch
>
>
> While opening a region, the HBase regionserver tries to list all the children in a "recovered.edits" directory. This directory may not exist and depending on the version of HDFS listStatus() might return null or an exception. If it does throw an exception the entire process of opening the region is aborted, just because the recovered.edits directory is not present.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4161) Incorrect use of listStatus() in
HBase region initialization.
Posted by "Pritam Damania (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085931#comment-13085931 ]
Pritam Damania commented on HBASE-4161:
---------------------------------------
@Stack : Yes you are right, sorry I didn't look at the right branch.
> Incorrect use of listStatus() in HBase region initialization.
> -------------------------------------------------------------
>
> Key: HBASE-4161
> URL: https://issues.apache.org/jira/browse/HBASE-4161
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Reporter: Pritam Damania
> Attachments: 0001-Fix-FileNotFoundException-in-HLog.java.patch
>
>
> While opening a region, the HBase regionserver tries to list all the children in a "recovered.edits" directory. This directory may not exist and depending on the version of HDFS listStatus() might return null or an exception. If it does throw an exception the entire process of opening the region is aborted, just because the recovered.edits directory is not present.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4161) Incorrect use of listStatus() in
HBase region initialization.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-4161.
--------------------------
Resolution: Fixed
Fix Version/s: 0.90.5
Hadoop Flags: [Reviewed]
Committed to the branch. Thanks for the patch Pritam.
> Incorrect use of listStatus() in HBase region initialization.
> -------------------------------------------------------------
>
> Key: HBASE-4161
> URL: https://issues.apache.org/jira/browse/HBASE-4161
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Reporter: Pritam Damania
> Fix For: 0.90.5
>
> Attachments: 0001-Fix-FileNotFoundException-in-HLog.java.patch
>
>
> While opening a region, the HBase regionserver tries to list all the children in a "recovered.edits" directory. This directory may not exist and depending on the version of HDFS listStatus() might return null or an exception. If it does throw an exception the entire process of opening the region is aborted, just because the recovered.edits directory is not present.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4161) Incorrect use of listStatus() in
HBase region initialization.
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13079113#comment-13079113 ]
Ted Yu commented on HBASE-4161:
-------------------------------
Looking at your patch:
{code}
+ if (fs.exists(editsdir)) {
{code}
The above would in turn call (src/hdfs/org/apache/hadoop/hdfs/DistributedFileSystem.java):
{code}
public FileStatus getFileStatus(Path f) throws IOException {
FileStatus fi = dfs.getFileInfo(getPathName(f));
if (fi != null) {
return makeQualified(fi);
} else {
throw new FileNotFoundException("File does not exist: " + f);
}
}
{code}
So ?
> Incorrect use of listStatus() in HBase region initialization.
> -------------------------------------------------------------
>
> Key: HBASE-4161
> URL: https://issues.apache.org/jira/browse/HBASE-4161
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Reporter: Pritam Damania
> Attachments: 0001-Fix-FileNotFoundException-in-HLog.java.patch
>
>
> While opening a region, the HBase regionserver tries to list all the children in a "recovered.edits" directory. This directory may not exist and depending on the version of HDFS listStatus() might return null or an exception. If it does throw an exception the entire process of opening the region is aborted, just because the recovered.edits directory is not present.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira