You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Lars George (JIRA)" <ji...@apache.org> on 2011/07/05 11:43:22 UTC
[jira] [Created] (HBASE-4061) getTableDirs is missing directories
to skip
getTableDirs is missing directories to skip
-------------------------------------------
Key: HBASE-4061
URL: https://issues.apache.org/jira/browse/HBASE-4061
Project: HBase
Issue Type: Bug
Components: util
Affects Versions: 0.92.0
Reporter: Lars George
Fix For: 0.92.0
The getTableDirs() is missing extra checks:
{code}
public static List<Path> getTableDirs(final FileSystem fs, final Path rootdir)
throws IOException {
// presumes any directory under hbase.rootdir is a table
FileStatus [] dirs = fs.listStatus(rootdir, new DirFilter(fs));
List<Path> tabledirs = new ArrayList<Path>(dirs.length);
for (FileStatus dir: dirs) {
Path p = dir.getPath();
String tableName = p.getName();
if (tableName.equals(HConstants.HREGION_LOGDIR_NAME) ||
tableName.equals(Bytes.toString(HConstants.ROOT_TABLE_NAME)) ||
tableName.equals(Bytes.toString(HConstants.META_TABLE_NAME)) ||
tableName.equals(HConstants.HREGION_OLDLOGDIR_NAME) ) {
continue;
}
tabledirs.add(p);
}
return tabledirs;
}
{code}
It needs to also skip
* .tmp
* .corrupt
* splitlog
A broader check should be performed to make sure it is all covered.
The missing .corrupt check causes for example:
{noformat}
2011-07-05 11:34:33,364 WARN org.apache.hadoop.hbase.master.HMaster: Failed getting all descriptors
java.io.FileNotFoundException: No status for hdfs://localhost:8020/hbase/.corrupt
at org.apache.hadoop.hbase.util.FSUtils.getTableInfoModtime(FSUtils.java:888)
at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:122)
at org.apache.hadoop.hbase.util.FSTableDescriptors.getAll(FSTableDescriptors.java:149)
at org.apache.hadoop.hbase.master.HMaster.getHTableDescriptors(HMaster.java:1429)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:312)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1065)
{noformat}
Not sure yet why others do not have this issue, could be me being on trunk and fiddling?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4061) getTableDirs is missing directories
to skip
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-4061:
--------------------------
Attachment: 4061.txt
This patch changes the default dir for log splitting to .splitlog HConstants.HBASE_NON_USER_TABLE_DIRS is expanded to include ".splitlog"
> getTableDirs is missing directories to skip
> -------------------------------------------
>
> Key: HBASE-4061
> URL: https://issues.apache.org/jira/browse/HBASE-4061
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.92.0
> Reporter: Lars George
> Assignee: Ted Yu
> Fix For: 0.92.0
>
> Attachments: 4061.txt
>
>
> The getTableDirs() is missing extra checks:
> {code}
> public static List<Path> getTableDirs(final FileSystem fs, final Path rootdir)
> throws IOException {
> // presumes any directory under hbase.rootdir is a table
> FileStatus [] dirs = fs.listStatus(rootdir, new DirFilter(fs));
> List<Path> tabledirs = new ArrayList<Path>(dirs.length);
> for (FileStatus dir: dirs) {
> Path p = dir.getPath();
> String tableName = p.getName();
> if (tableName.equals(HConstants.HREGION_LOGDIR_NAME) ||
> tableName.equals(Bytes.toString(HConstants.ROOT_TABLE_NAME)) ||
> tableName.equals(Bytes.toString(HConstants.META_TABLE_NAME)) ||
> tableName.equals(HConstants.HREGION_OLDLOGDIR_NAME) ) {
> continue;
> }
> tabledirs.add(p);
> }
> return tabledirs;
> }
> {code}
> It needs to also skip
> * .tmp
> * .corrupt
> * splitlog
> A broader check should be performed to make sure it is all covered.
> The missing .corrupt check causes for example:
> {noformat}
> 2011-07-05 11:34:33,364 WARN org.apache.hadoop.hbase.master.HMaster: Failed getting all descriptors
> java.io.FileNotFoundException: No status for hdfs://localhost:8020/hbase/.corrupt
> at org.apache.hadoop.hbase.util.FSUtils.getTableInfoModtime(FSUtils.java:888)
> at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:122)
> at org.apache.hadoop.hbase.util.FSTableDescriptors.getAll(FSTableDescriptors.java:149)
> at org.apache.hadoop.hbase.master.HMaster.getHTableDescriptors(HMaster.java:1429)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:312)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1065)
> {noformat}
> Not sure yet why others do not have this issue, could be me being on trunk and fiddling?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4061) getTableDirs is missing directories
to skip
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu resolved HBASE-4061.
---------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
> getTableDirs is missing directories to skip
> -------------------------------------------
>
> Key: HBASE-4061
> URL: https://issues.apache.org/jira/browse/HBASE-4061
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.92.0
> Reporter: Lars George
> Assignee: Ted Yu
> Fix For: 0.92.0
>
> Attachments: 4061.txt
>
>
> The getTableDirs() is missing extra checks:
> {code}
> public static List<Path> getTableDirs(final FileSystem fs, final Path rootdir)
> throws IOException {
> // presumes any directory under hbase.rootdir is a table
> FileStatus [] dirs = fs.listStatus(rootdir, new DirFilter(fs));
> List<Path> tabledirs = new ArrayList<Path>(dirs.length);
> for (FileStatus dir: dirs) {
> Path p = dir.getPath();
> String tableName = p.getName();
> if (tableName.equals(HConstants.HREGION_LOGDIR_NAME) ||
> tableName.equals(Bytes.toString(HConstants.ROOT_TABLE_NAME)) ||
> tableName.equals(Bytes.toString(HConstants.META_TABLE_NAME)) ||
> tableName.equals(HConstants.HREGION_OLDLOGDIR_NAME) ) {
> continue;
> }
> tabledirs.add(p);
> }
> return tabledirs;
> }
> {code}
> It needs to also skip
> * .tmp
> * .corrupt
> * splitlog
> A broader check should be performed to make sure it is all covered.
> The missing .corrupt check causes for example:
> {noformat}
> 2011-07-05 11:34:33,364 WARN org.apache.hadoop.hbase.master.HMaster: Failed getting all descriptors
> java.io.FileNotFoundException: No status for hdfs://localhost:8020/hbase/.corrupt
> at org.apache.hadoop.hbase.util.FSUtils.getTableInfoModtime(FSUtils.java:888)
> at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:122)
> at org.apache.hadoop.hbase.util.FSTableDescriptors.getAll(FSTableDescriptors.java:149)
> at org.apache.hadoop.hbase.master.HMaster.getHTableDescriptors(HMaster.java:1429)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:312)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1065)
> {noformat}
> Not sure yet why others do not have this issue, could be me being on trunk and fiddling?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4061) getTableDirs is missing directories
to skip
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-4061:
--------------------------
Attachment: 4061.txt
Updated patch to reference strings defined in HConstants
> getTableDirs is missing directories to skip
> -------------------------------------------
>
> Key: HBASE-4061
> URL: https://issues.apache.org/jira/browse/HBASE-4061
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.92.0
> Reporter: Lars George
> Assignee: Ted Yu
> Fix For: 0.92.0
>
> Attachments: 4061.txt
>
>
> The getTableDirs() is missing extra checks:
> {code}
> public static List<Path> getTableDirs(final FileSystem fs, final Path rootdir)
> throws IOException {
> // presumes any directory under hbase.rootdir is a table
> FileStatus [] dirs = fs.listStatus(rootdir, new DirFilter(fs));
> List<Path> tabledirs = new ArrayList<Path>(dirs.length);
> for (FileStatus dir: dirs) {
> Path p = dir.getPath();
> String tableName = p.getName();
> if (tableName.equals(HConstants.HREGION_LOGDIR_NAME) ||
> tableName.equals(Bytes.toString(HConstants.ROOT_TABLE_NAME)) ||
> tableName.equals(Bytes.toString(HConstants.META_TABLE_NAME)) ||
> tableName.equals(HConstants.HREGION_OLDLOGDIR_NAME) ) {
> continue;
> }
> tabledirs.add(p);
> }
> return tabledirs;
> }
> {code}
> It needs to also skip
> * .tmp
> * .corrupt
> * splitlog
> A broader check should be performed to make sure it is all covered.
> The missing .corrupt check causes for example:
> {noformat}
> 2011-07-05 11:34:33,364 WARN org.apache.hadoop.hbase.master.HMaster: Failed getting all descriptors
> java.io.FileNotFoundException: No status for hdfs://localhost:8020/hbase/.corrupt
> at org.apache.hadoop.hbase.util.FSUtils.getTableInfoModtime(FSUtils.java:888)
> at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:122)
> at org.apache.hadoop.hbase.util.FSTableDescriptors.getAll(FSTableDescriptors.java:149)
> at org.apache.hadoop.hbase.master.HMaster.getHTableDescriptors(HMaster.java:1429)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:312)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1065)
> {noformat}
> Not sure yet why others do not have this issue, could be me being on trunk and fiddling?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4061) getTableDirs is missing directories
to skip
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13060796#comment-13060796 ]
jiraposter@reviews.apache.org commented on HBASE-4061:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1018/
-----------------------------------------------------------
Review request for hbase.
Summary
-------
getTableDirs() should check for directories such as "splitlog"
The patch reuses HConstants.HBASE_NON_USER_TABLE_DIRS
This addresses bug HBASE-4061.
https://issues.apache.org/jira/browse/HBASE-4061
Diffs
-----
/src/main/java/org/apache/hadoop/hbase/HConstants.java 1143201
/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 1143201
/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKSplitLog.java 1143201
/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 1143201
Diff: https://reviews.apache.org/r/1018/diff
Testing
-------
Ran test suite.
Thanks,
Ted
> getTableDirs is missing directories to skip
> -------------------------------------------
>
> Key: HBASE-4061
> URL: https://issues.apache.org/jira/browse/HBASE-4061
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.92.0
> Reporter: Lars George
> Assignee: Ted Yu
> Fix For: 0.92.0
>
> Attachments: 4061.txt
>
>
> The getTableDirs() is missing extra checks:
> {code}
> public static List<Path> getTableDirs(final FileSystem fs, final Path rootdir)
> throws IOException {
> // presumes any directory under hbase.rootdir is a table
> FileStatus [] dirs = fs.listStatus(rootdir, new DirFilter(fs));
> List<Path> tabledirs = new ArrayList<Path>(dirs.length);
> for (FileStatus dir: dirs) {
> Path p = dir.getPath();
> String tableName = p.getName();
> if (tableName.equals(HConstants.HREGION_LOGDIR_NAME) ||
> tableName.equals(Bytes.toString(HConstants.ROOT_TABLE_NAME)) ||
> tableName.equals(Bytes.toString(HConstants.META_TABLE_NAME)) ||
> tableName.equals(HConstants.HREGION_OLDLOGDIR_NAME) ) {
> continue;
> }
> tabledirs.add(p);
> }
> return tabledirs;
> }
> {code}
> It needs to also skip
> * .tmp
> * .corrupt
> * splitlog
> A broader check should be performed to make sure it is all covered.
> The missing .corrupt check causes for example:
> {noformat}
> 2011-07-05 11:34:33,364 WARN org.apache.hadoop.hbase.master.HMaster: Failed getting all descriptors
> java.io.FileNotFoundException: No status for hdfs://localhost:8020/hbase/.corrupt
> at org.apache.hadoop.hbase.util.FSUtils.getTableInfoModtime(FSUtils.java:888)
> at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:122)
> at org.apache.hadoop.hbase.util.FSTableDescriptors.getAll(FSTableDescriptors.java:149)
> at org.apache.hadoop.hbase.master.HMaster.getHTableDescriptors(HMaster.java:1429)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:312)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1065)
> {noformat}
> Not sure yet why others do not have this issue, could be me being on trunk and fiddling?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4061) getTableDirs is missing directories
to skip
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13060799#comment-13060799 ]
jiraposter@reviews.apache.org commented on HBASE-4061:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1018/
-----------------------------------------------------------
(Updated 2011-07-06 20:10:44.859515)
Review request for hbase.
Changes
-------
Removed white spaces.
Summary
-------
getTableDirs() should check for directories such as "splitlog"
The patch reuses HConstants.HBASE_NON_USER_TABLE_DIRS
This addresses bug HBASE-4061.
https://issues.apache.org/jira/browse/HBASE-4061
Diffs (updated)
-----
/src/main/java/org/apache/hadoop/hbase/HConstants.java 1143201
/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 1143201
/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKSplitLog.java 1143201
/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 1143201
Diff: https://reviews.apache.org/r/1018/diff
Testing
-------
Ran test suite.
Thanks,
Ted
> getTableDirs is missing directories to skip
> -------------------------------------------
>
> Key: HBASE-4061
> URL: https://issues.apache.org/jira/browse/HBASE-4061
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.92.0
> Reporter: Lars George
> Assignee: Ted Yu
> Fix For: 0.92.0
>
> Attachments: 4061.txt
>
>
> The getTableDirs() is missing extra checks:
> {code}
> public static List<Path> getTableDirs(final FileSystem fs, final Path rootdir)
> throws IOException {
> // presumes any directory under hbase.rootdir is a table
> FileStatus [] dirs = fs.listStatus(rootdir, new DirFilter(fs));
> List<Path> tabledirs = new ArrayList<Path>(dirs.length);
> for (FileStatus dir: dirs) {
> Path p = dir.getPath();
> String tableName = p.getName();
> if (tableName.equals(HConstants.HREGION_LOGDIR_NAME) ||
> tableName.equals(Bytes.toString(HConstants.ROOT_TABLE_NAME)) ||
> tableName.equals(Bytes.toString(HConstants.META_TABLE_NAME)) ||
> tableName.equals(HConstants.HREGION_OLDLOGDIR_NAME) ) {
> continue;
> }
> tabledirs.add(p);
> }
> return tabledirs;
> }
> {code}
> It needs to also skip
> * .tmp
> * .corrupt
> * splitlog
> A broader check should be performed to make sure it is all covered.
> The missing .corrupt check causes for example:
> {noformat}
> 2011-07-05 11:34:33,364 WARN org.apache.hadoop.hbase.master.HMaster: Failed getting all descriptors
> java.io.FileNotFoundException: No status for hdfs://localhost:8020/hbase/.corrupt
> at org.apache.hadoop.hbase.util.FSUtils.getTableInfoModtime(FSUtils.java:888)
> at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:122)
> at org.apache.hadoop.hbase.util.FSTableDescriptors.getAll(FSTableDescriptors.java:149)
> at org.apache.hadoop.hbase.master.HMaster.getHTableDescriptors(HMaster.java:1429)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:312)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1065)
> {noformat}
> Not sure yet why others do not have this issue, could be me being on trunk and fiddling?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4061) getTableDirs is missing directories
to skip
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13060821#comment-13060821 ]
jiraposter@reviews.apache.org commented on HBASE-4061:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1018/#review977
-----------------------------------------------------------
Ship it!
- Andrew
On 2011-07-06 20:39:39, Ted Yu wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1018/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-07-06 20:39:39)
bq.
bq.
bq. Review request for hbase.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. getTableDirs() should check for directories such as "splitlog"
bq.
bq. The patch reuses HConstants.HBASE_NON_USER_TABLE_DIRS
bq.
bq.
bq. This addresses bug HBASE-4061.
bq. https://issues.apache.org/jira/browse/HBASE-4061
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. /src/main/java/org/apache/hadoop/hbase/HConstants.java 1143201
bq. /src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 1143525
bq. /src/main/java/org/apache/hadoop/hbase/zookeeper/ZKSplitLog.java 1143201
bq. /src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 1143201
bq.
bq. Diff: https://reviews.apache.org/r/1018/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran test suite.
bq.
bq.
bq. Thanks,
bq.
bq. Ted
bq.
bq.
> getTableDirs is missing directories to skip
> -------------------------------------------
>
> Key: HBASE-4061
> URL: https://issues.apache.org/jira/browse/HBASE-4061
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.92.0
> Reporter: Lars George
> Assignee: Ted Yu
> Fix For: 0.92.0
>
> Attachments: 4061.txt
>
>
> The getTableDirs() is missing extra checks:
> {code}
> public static List<Path> getTableDirs(final FileSystem fs, final Path rootdir)
> throws IOException {
> // presumes any directory under hbase.rootdir is a table
> FileStatus [] dirs = fs.listStatus(rootdir, new DirFilter(fs));
> List<Path> tabledirs = new ArrayList<Path>(dirs.length);
> for (FileStatus dir: dirs) {
> Path p = dir.getPath();
> String tableName = p.getName();
> if (tableName.equals(HConstants.HREGION_LOGDIR_NAME) ||
> tableName.equals(Bytes.toString(HConstants.ROOT_TABLE_NAME)) ||
> tableName.equals(Bytes.toString(HConstants.META_TABLE_NAME)) ||
> tableName.equals(HConstants.HREGION_OLDLOGDIR_NAME) ) {
> continue;
> }
> tabledirs.add(p);
> }
> return tabledirs;
> }
> {code}
> It needs to also skip
> * .tmp
> * .corrupt
> * splitlog
> A broader check should be performed to make sure it is all covered.
> The missing .corrupt check causes for example:
> {noformat}
> 2011-07-05 11:34:33,364 WARN org.apache.hadoop.hbase.master.HMaster: Failed getting all descriptors
> java.io.FileNotFoundException: No status for hdfs://localhost:8020/hbase/.corrupt
> at org.apache.hadoop.hbase.util.FSUtils.getTableInfoModtime(FSUtils.java:888)
> at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:122)
> at org.apache.hadoop.hbase.util.FSTableDescriptors.getAll(FSTableDescriptors.java:149)
> at org.apache.hadoop.hbase.master.HMaster.getHTableDescriptors(HMaster.java:1429)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:312)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1065)
> {noformat}
> Not sure yet why others do not have this issue, could be me being on trunk and fiddling?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4061) getTableDirs is missing directories
to skip
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13060825#comment-13060825 ]
Ted Yu commented on HBASE-4061:
-------------------------------
Integrated to TRUNK.
Thanks for the review Andrew.
> getTableDirs is missing directories to skip
> -------------------------------------------
>
> Key: HBASE-4061
> URL: https://issues.apache.org/jira/browse/HBASE-4061
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.92.0
> Reporter: Lars George
> Assignee: Ted Yu
> Fix For: 0.92.0
>
> Attachments: 4061.txt
>
>
> The getTableDirs() is missing extra checks:
> {code}
> public static List<Path> getTableDirs(final FileSystem fs, final Path rootdir)
> throws IOException {
> // presumes any directory under hbase.rootdir is a table
> FileStatus [] dirs = fs.listStatus(rootdir, new DirFilter(fs));
> List<Path> tabledirs = new ArrayList<Path>(dirs.length);
> for (FileStatus dir: dirs) {
> Path p = dir.getPath();
> String tableName = p.getName();
> if (tableName.equals(HConstants.HREGION_LOGDIR_NAME) ||
> tableName.equals(Bytes.toString(HConstants.ROOT_TABLE_NAME)) ||
> tableName.equals(Bytes.toString(HConstants.META_TABLE_NAME)) ||
> tableName.equals(HConstants.HREGION_OLDLOGDIR_NAME) ) {
> continue;
> }
> tabledirs.add(p);
> }
> return tabledirs;
> }
> {code}
> It needs to also skip
> * .tmp
> * .corrupt
> * splitlog
> A broader check should be performed to make sure it is all covered.
> The missing .corrupt check causes for example:
> {noformat}
> 2011-07-05 11:34:33,364 WARN org.apache.hadoop.hbase.master.HMaster: Failed getting all descriptors
> java.io.FileNotFoundException: No status for hdfs://localhost:8020/hbase/.corrupt
> at org.apache.hadoop.hbase.util.FSUtils.getTableInfoModtime(FSUtils.java:888)
> at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:122)
> at org.apache.hadoop.hbase.util.FSTableDescriptors.getAll(FSTableDescriptors.java:149)
> at org.apache.hadoop.hbase.master.HMaster.getHTableDescriptors(HMaster.java:1429)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:312)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1065)
> {noformat}
> Not sure yet why others do not have this issue, could be me being on trunk and fiddling?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-4061) getTableDirs is missing directories
to skip
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu reassigned HBASE-4061:
-----------------------------
Assignee: Ted Yu
> getTableDirs is missing directories to skip
> -------------------------------------------
>
> Key: HBASE-4061
> URL: https://issues.apache.org/jira/browse/HBASE-4061
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.92.0
> Reporter: Lars George
> Assignee: Ted Yu
> Fix For: 0.92.0
>
>
> The getTableDirs() is missing extra checks:
> {code}
> public static List<Path> getTableDirs(final FileSystem fs, final Path rootdir)
> throws IOException {
> // presumes any directory under hbase.rootdir is a table
> FileStatus [] dirs = fs.listStatus(rootdir, new DirFilter(fs));
> List<Path> tabledirs = new ArrayList<Path>(dirs.length);
> for (FileStatus dir: dirs) {
> Path p = dir.getPath();
> String tableName = p.getName();
> if (tableName.equals(HConstants.HREGION_LOGDIR_NAME) ||
> tableName.equals(Bytes.toString(HConstants.ROOT_TABLE_NAME)) ||
> tableName.equals(Bytes.toString(HConstants.META_TABLE_NAME)) ||
> tableName.equals(HConstants.HREGION_OLDLOGDIR_NAME) ) {
> continue;
> }
> tabledirs.add(p);
> }
> return tabledirs;
> }
> {code}
> It needs to also skip
> * .tmp
> * .corrupt
> * splitlog
> A broader check should be performed to make sure it is all covered.
> The missing .corrupt check causes for example:
> {noformat}
> 2011-07-05 11:34:33,364 WARN org.apache.hadoop.hbase.master.HMaster: Failed getting all descriptors
> java.io.FileNotFoundException: No status for hdfs://localhost:8020/hbase/.corrupt
> at org.apache.hadoop.hbase.util.FSUtils.getTableInfoModtime(FSUtils.java:888)
> at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:122)
> at org.apache.hadoop.hbase.util.FSTableDescriptors.getAll(FSTableDescriptors.java:149)
> at org.apache.hadoop.hbase.master.HMaster.getHTableDescriptors(HMaster.java:1429)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:312)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1065)
> {noformat}
> Not sure yet why others do not have this issue, could be me being on trunk and fiddling?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4061) getTableDirs is missing directories
to skip
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13061000#comment-13061000 ]
Hudson commented on HBASE-4061:
-------------------------------
Integrated in HBase-TRUNK #2008 (See [https://builds.apache.org/job/HBase-TRUNK/2008/])
HBASE-4061 getTableDirs is missing directories to skip
tedyu :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKSplitLog.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/HConstants.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
> getTableDirs is missing directories to skip
> -------------------------------------------
>
> Key: HBASE-4061
> URL: https://issues.apache.org/jira/browse/HBASE-4061
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.92.0
> Reporter: Lars George
> Assignee: Ted Yu
> Fix For: 0.92.0
>
> Attachments: 4061.txt
>
>
> The getTableDirs() is missing extra checks:
> {code}
> public static List<Path> getTableDirs(final FileSystem fs, final Path rootdir)
> throws IOException {
> // presumes any directory under hbase.rootdir is a table
> FileStatus [] dirs = fs.listStatus(rootdir, new DirFilter(fs));
> List<Path> tabledirs = new ArrayList<Path>(dirs.length);
> for (FileStatus dir: dirs) {
> Path p = dir.getPath();
> String tableName = p.getName();
> if (tableName.equals(HConstants.HREGION_LOGDIR_NAME) ||
> tableName.equals(Bytes.toString(HConstants.ROOT_TABLE_NAME)) ||
> tableName.equals(Bytes.toString(HConstants.META_TABLE_NAME)) ||
> tableName.equals(HConstants.HREGION_OLDLOGDIR_NAME) ) {
> continue;
> }
> tabledirs.add(p);
> }
> return tabledirs;
> }
> {code}
> It needs to also skip
> * .tmp
> * .corrupt
> * splitlog
> A broader check should be performed to make sure it is all covered.
> The missing .corrupt check causes for example:
> {noformat}
> 2011-07-05 11:34:33,364 WARN org.apache.hadoop.hbase.master.HMaster: Failed getting all descriptors
> java.io.FileNotFoundException: No status for hdfs://localhost:8020/hbase/.corrupt
> at org.apache.hadoop.hbase.util.FSUtils.getTableInfoModtime(FSUtils.java:888)
> at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:122)
> at org.apache.hadoop.hbase.util.FSTableDescriptors.getAll(FSTableDescriptors.java:149)
> at org.apache.hadoop.hbase.master.HMaster.getHTableDescriptors(HMaster.java:1429)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:312)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1065)
> {noformat}
> Not sure yet why others do not have this issue, could be me being on trunk and fiddling?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4061) getTableDirs is missing directories
to skip
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-4061:
--------------------------
Attachment: (was: 4061.txt)
> getTableDirs is missing directories to skip
> -------------------------------------------
>
> Key: HBASE-4061
> URL: https://issues.apache.org/jira/browse/HBASE-4061
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.92.0
> Reporter: Lars George
> Assignee: Ted Yu
> Fix For: 0.92.0
>
> Attachments: 4061.txt
>
>
> The getTableDirs() is missing extra checks:
> {code}
> public static List<Path> getTableDirs(final FileSystem fs, final Path rootdir)
> throws IOException {
> // presumes any directory under hbase.rootdir is a table
> FileStatus [] dirs = fs.listStatus(rootdir, new DirFilter(fs));
> List<Path> tabledirs = new ArrayList<Path>(dirs.length);
> for (FileStatus dir: dirs) {
> Path p = dir.getPath();
> String tableName = p.getName();
> if (tableName.equals(HConstants.HREGION_LOGDIR_NAME) ||
> tableName.equals(Bytes.toString(HConstants.ROOT_TABLE_NAME)) ||
> tableName.equals(Bytes.toString(HConstants.META_TABLE_NAME)) ||
> tableName.equals(HConstants.HREGION_OLDLOGDIR_NAME) ) {
> continue;
> }
> tabledirs.add(p);
> }
> return tabledirs;
> }
> {code}
> It needs to also skip
> * .tmp
> * .corrupt
> * splitlog
> A broader check should be performed to make sure it is all covered.
> The missing .corrupt check causes for example:
> {noformat}
> 2011-07-05 11:34:33,364 WARN org.apache.hadoop.hbase.master.HMaster: Failed getting all descriptors
> java.io.FileNotFoundException: No status for hdfs://localhost:8020/hbase/.corrupt
> at org.apache.hadoop.hbase.util.FSUtils.getTableInfoModtime(FSUtils.java:888)
> at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:122)
> at org.apache.hadoop.hbase.util.FSTableDescriptors.getAll(FSTableDescriptors.java:149)
> at org.apache.hadoop.hbase.master.HMaster.getHTableDescriptors(HMaster.java:1429)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:312)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1065)
> {noformat}
> Not sure yet why others do not have this issue, could be me being on trunk and fiddling?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4061) getTableDirs is missing directories
to skip
Posted by "Lars George (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13059788#comment-13059788 ]
Lars George commented on HBASE-4061:
------------------------------------
I am wondering if "splitlog" shouldn't be ".splitlog" and then have a filter that skips all dot prefixed dirs and files.
> getTableDirs is missing directories to skip
> -------------------------------------------
>
> Key: HBASE-4061
> URL: https://issues.apache.org/jira/browse/HBASE-4061
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.92.0
> Reporter: Lars George
> Fix For: 0.92.0
>
>
> The getTableDirs() is missing extra checks:
> {code}
> public static List<Path> getTableDirs(final FileSystem fs, final Path rootdir)
> throws IOException {
> // presumes any directory under hbase.rootdir is a table
> FileStatus [] dirs = fs.listStatus(rootdir, new DirFilter(fs));
> List<Path> tabledirs = new ArrayList<Path>(dirs.length);
> for (FileStatus dir: dirs) {
> Path p = dir.getPath();
> String tableName = p.getName();
> if (tableName.equals(HConstants.HREGION_LOGDIR_NAME) ||
> tableName.equals(Bytes.toString(HConstants.ROOT_TABLE_NAME)) ||
> tableName.equals(Bytes.toString(HConstants.META_TABLE_NAME)) ||
> tableName.equals(HConstants.HREGION_OLDLOGDIR_NAME) ) {
> continue;
> }
> tabledirs.add(p);
> }
> return tabledirs;
> }
> {code}
> It needs to also skip
> * .tmp
> * .corrupt
> * splitlog
> A broader check should be performed to make sure it is all covered.
> The missing .corrupt check causes for example:
> {noformat}
> 2011-07-05 11:34:33,364 WARN org.apache.hadoop.hbase.master.HMaster: Failed getting all descriptors
> java.io.FileNotFoundException: No status for hdfs://localhost:8020/hbase/.corrupt
> at org.apache.hadoop.hbase.util.FSUtils.getTableInfoModtime(FSUtils.java:888)
> at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:122)
> at org.apache.hadoop.hbase.util.FSTableDescriptors.getAll(FSTableDescriptors.java:149)
> at org.apache.hadoop.hbase.master.HMaster.getHTableDescriptors(HMaster.java:1429)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:312)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1065)
> {noformat}
> Not sure yet why others do not have this issue, could be me being on trunk and fiddling?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4061) getTableDirs is missing directories
to skip
Posted by "Lars George (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13059803#comment-13059803 ]
Lars George commented on HBASE-4061:
------------------------------------
Here is a patch that fixes it, but has to hardcode (*gasp*) the "splitlog" name, courtesy of ZKSplitLog.getSplitLogDir(). :) This should be in HConstants.java - but I would still prefer to change this to ".splitlog" and make this a rule that all non-table dirs/files are dot prefixed.
{noformat}
--- src/main/java/org/apache/hadoop/hbase/util/FSUtils.java (revision 1139502)
+++ src/main/java/org/apache/hadoop/hbase/util/FSUtils.java (working copy)
@@ -712,7 +712,8 @@
public boolean accept(Path p) {
boolean isdir = false;
try {
- isdir = this.fs.getFileStatus(p).isDir();
+ isdir = this.fs.getFileStatus(p).isDir() &&
+ !p.getName().startsWith(".");
} catch (IOException e) {
e.printStackTrace();
}
@@ -854,6 +855,7 @@
if (tableName.equals(HConstants.HREGION_LOGDIR_NAME) ||
tableName.equals(Bytes.toString(HConstants.ROOT_TABLE_NAME)) ||
tableName.equals(Bytes.toString(HConstants.META_TABLE_NAME)) ||
+ tableName.equals("splitlog") ||
tableName.equals(HConstants.HREGION_OLDLOGDIR_NAME) ) {
continue;
}
{noformat}
> getTableDirs is missing directories to skip
> -------------------------------------------
>
> Key: HBASE-4061
> URL: https://issues.apache.org/jira/browse/HBASE-4061
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.92.0
> Reporter: Lars George
> Fix For: 0.92.0
>
>
> The getTableDirs() is missing extra checks:
> {code}
> public static List<Path> getTableDirs(final FileSystem fs, final Path rootdir)
> throws IOException {
> // presumes any directory under hbase.rootdir is a table
> FileStatus [] dirs = fs.listStatus(rootdir, new DirFilter(fs));
> List<Path> tabledirs = new ArrayList<Path>(dirs.length);
> for (FileStatus dir: dirs) {
> Path p = dir.getPath();
> String tableName = p.getName();
> if (tableName.equals(HConstants.HREGION_LOGDIR_NAME) ||
> tableName.equals(Bytes.toString(HConstants.ROOT_TABLE_NAME)) ||
> tableName.equals(Bytes.toString(HConstants.META_TABLE_NAME)) ||
> tableName.equals(HConstants.HREGION_OLDLOGDIR_NAME) ) {
> continue;
> }
> tabledirs.add(p);
> }
> return tabledirs;
> }
> {code}
> It needs to also skip
> * .tmp
> * .corrupt
> * splitlog
> A broader check should be performed to make sure it is all covered.
> The missing .corrupt check causes for example:
> {noformat}
> 2011-07-05 11:34:33,364 WARN org.apache.hadoop.hbase.master.HMaster: Failed getting all descriptors
> java.io.FileNotFoundException: No status for hdfs://localhost:8020/hbase/.corrupt
> at org.apache.hadoop.hbase.util.FSUtils.getTableInfoModtime(FSUtils.java:888)
> at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:122)
> at org.apache.hadoop.hbase.util.FSTableDescriptors.getAll(FSTableDescriptors.java:149)
> at org.apache.hadoop.hbase.master.HMaster.getHTableDescriptors(HMaster.java:1429)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:312)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1065)
> {noformat}
> Not sure yet why others do not have this issue, could be me being on trunk and fiddling?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4061) getTableDirs is missing directories
to skip
Posted by "Lars George (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13059792#comment-13059792 ]
Lars George commented on HBASE-4061:
------------------------------------
Come to think of it, in addition to the above, I would also find it better for the getHTableDescriptor not to fail like this, when the directory has no .tableinfo. Right now the master UI simply reports no user tables at all because of the error. This should be handled more gracefully.
> getTableDirs is missing directories to skip
> -------------------------------------------
>
> Key: HBASE-4061
> URL: https://issues.apache.org/jira/browse/HBASE-4061
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.92.0
> Reporter: Lars George
> Fix For: 0.92.0
>
>
> The getTableDirs() is missing extra checks:
> {code}
> public static List<Path> getTableDirs(final FileSystem fs, final Path rootdir)
> throws IOException {
> // presumes any directory under hbase.rootdir is a table
> FileStatus [] dirs = fs.listStatus(rootdir, new DirFilter(fs));
> List<Path> tabledirs = new ArrayList<Path>(dirs.length);
> for (FileStatus dir: dirs) {
> Path p = dir.getPath();
> String tableName = p.getName();
> if (tableName.equals(HConstants.HREGION_LOGDIR_NAME) ||
> tableName.equals(Bytes.toString(HConstants.ROOT_TABLE_NAME)) ||
> tableName.equals(Bytes.toString(HConstants.META_TABLE_NAME)) ||
> tableName.equals(HConstants.HREGION_OLDLOGDIR_NAME) ) {
> continue;
> }
> tabledirs.add(p);
> }
> return tabledirs;
> }
> {code}
> It needs to also skip
> * .tmp
> * .corrupt
> * splitlog
> A broader check should be performed to make sure it is all covered.
> The missing .corrupt check causes for example:
> {noformat}
> 2011-07-05 11:34:33,364 WARN org.apache.hadoop.hbase.master.HMaster: Failed getting all descriptors
> java.io.FileNotFoundException: No status for hdfs://localhost:8020/hbase/.corrupt
> at org.apache.hadoop.hbase.util.FSUtils.getTableInfoModtime(FSUtils.java:888)
> at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:122)
> at org.apache.hadoop.hbase.util.FSTableDescriptors.getAll(FSTableDescriptors.java:149)
> at org.apache.hadoop.hbase.master.HMaster.getHTableDescriptors(HMaster.java:1429)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:312)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1065)
> {noformat}
> Not sure yet why others do not have this issue, could be me being on trunk and fiddling?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4061) getTableDirs is missing directories
to skip
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-4061:
--------------------------
Attachment: (was: 4061.txt)
> getTableDirs is missing directories to skip
> -------------------------------------------
>
> Key: HBASE-4061
> URL: https://issues.apache.org/jira/browse/HBASE-4061
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.92.0
> Reporter: Lars George
> Assignee: Ted Yu
> Fix For: 0.92.0
>
> Attachments: 4061.txt
>
>
> The getTableDirs() is missing extra checks:
> {code}
> public static List<Path> getTableDirs(final FileSystem fs, final Path rootdir)
> throws IOException {
> // presumes any directory under hbase.rootdir is a table
> FileStatus [] dirs = fs.listStatus(rootdir, new DirFilter(fs));
> List<Path> tabledirs = new ArrayList<Path>(dirs.length);
> for (FileStatus dir: dirs) {
> Path p = dir.getPath();
> String tableName = p.getName();
> if (tableName.equals(HConstants.HREGION_LOGDIR_NAME) ||
> tableName.equals(Bytes.toString(HConstants.ROOT_TABLE_NAME)) ||
> tableName.equals(Bytes.toString(HConstants.META_TABLE_NAME)) ||
> tableName.equals(HConstants.HREGION_OLDLOGDIR_NAME) ) {
> continue;
> }
> tabledirs.add(p);
> }
> return tabledirs;
> }
> {code}
> It needs to also skip
> * .tmp
> * .corrupt
> * splitlog
> A broader check should be performed to make sure it is all covered.
> The missing .corrupt check causes for example:
> {noformat}
> 2011-07-05 11:34:33,364 WARN org.apache.hadoop.hbase.master.HMaster: Failed getting all descriptors
> java.io.FileNotFoundException: No status for hdfs://localhost:8020/hbase/.corrupt
> at org.apache.hadoop.hbase.util.FSUtils.getTableInfoModtime(FSUtils.java:888)
> at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:122)
> at org.apache.hadoop.hbase.util.FSTableDescriptors.getAll(FSTableDescriptors.java:149)
> at org.apache.hadoop.hbase.master.HMaster.getHTableDescriptors(HMaster.java:1429)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:312)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1065)
> {noformat}
> Not sure yet why others do not have this issue, could be me being on trunk and fiddling?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4061) getTableDirs is missing directories
to skip
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-4061:
--------------------------
Attachment: 4061.txt
> getTableDirs is missing directories to skip
> -------------------------------------------
>
> Key: HBASE-4061
> URL: https://issues.apache.org/jira/browse/HBASE-4061
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.92.0
> Reporter: Lars George
> Assignee: Ted Yu
> Fix For: 0.92.0
>
> Attachments: 4061.txt
>
>
> The getTableDirs() is missing extra checks:
> {code}
> public static List<Path> getTableDirs(final FileSystem fs, final Path rootdir)
> throws IOException {
> // presumes any directory under hbase.rootdir is a table
> FileStatus [] dirs = fs.listStatus(rootdir, new DirFilter(fs));
> List<Path> tabledirs = new ArrayList<Path>(dirs.length);
> for (FileStatus dir: dirs) {
> Path p = dir.getPath();
> String tableName = p.getName();
> if (tableName.equals(HConstants.HREGION_LOGDIR_NAME) ||
> tableName.equals(Bytes.toString(HConstants.ROOT_TABLE_NAME)) ||
> tableName.equals(Bytes.toString(HConstants.META_TABLE_NAME)) ||
> tableName.equals(HConstants.HREGION_OLDLOGDIR_NAME) ) {
> continue;
> }
> tabledirs.add(p);
> }
> return tabledirs;
> }
> {code}
> It needs to also skip
> * .tmp
> * .corrupt
> * splitlog
> A broader check should be performed to make sure it is all covered.
> The missing .corrupt check causes for example:
> {noformat}
> 2011-07-05 11:34:33,364 WARN org.apache.hadoop.hbase.master.HMaster: Failed getting all descriptors
> java.io.FileNotFoundException: No status for hdfs://localhost:8020/hbase/.corrupt
> at org.apache.hadoop.hbase.util.FSUtils.getTableInfoModtime(FSUtils.java:888)
> at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:122)
> at org.apache.hadoop.hbase.util.FSTableDescriptors.getAll(FSTableDescriptors.java:149)
> at org.apache.hadoop.hbase.master.HMaster.getHTableDescriptors(HMaster.java:1429)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:312)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1065)
> {noformat}
> Not sure yet why others do not have this issue, could be me being on trunk and fiddling?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4061) getTableDirs is missing directories
to skip
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13060817#comment-13060817 ]
jiraposter@reviews.apache.org commented on HBASE-4061:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1018/#review976
-----------------------------------------------------------
/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
<https://reviews.apache.org/r/1018/#comment2043>
This works but a non user directory is still a directory. Maybe rename 'isdir' to 'isValid' or similar?
/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
<https://reviews.apache.org/r/1018/#comment2042>
ASF style problem, put else conditional in braces.
- Andrew
On 2011-07-06 20:10:44, Ted Yu wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1018/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-07-06 20:10:44)
bq.
bq.
bq. Review request for hbase.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. getTableDirs() should check for directories such as "splitlog"
bq.
bq. The patch reuses HConstants.HBASE_NON_USER_TABLE_DIRS
bq.
bq.
bq. This addresses bug HBASE-4061.
bq. https://issues.apache.org/jira/browse/HBASE-4061
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. /src/main/java/org/apache/hadoop/hbase/HConstants.java 1143201
bq. /src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 1143201
bq. /src/main/java/org/apache/hadoop/hbase/zookeeper/ZKSplitLog.java 1143201
bq. /src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 1143201
bq.
bq. Diff: https://reviews.apache.org/r/1018/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran test suite.
bq.
bq.
bq. Thanks,
bq.
bq. Ted
bq.
bq.
> getTableDirs is missing directories to skip
> -------------------------------------------
>
> Key: HBASE-4061
> URL: https://issues.apache.org/jira/browse/HBASE-4061
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.92.0
> Reporter: Lars George
> Assignee: Ted Yu
> Fix For: 0.92.0
>
> Attachments: 4061.txt
>
>
> The getTableDirs() is missing extra checks:
> {code}
> public static List<Path> getTableDirs(final FileSystem fs, final Path rootdir)
> throws IOException {
> // presumes any directory under hbase.rootdir is a table
> FileStatus [] dirs = fs.listStatus(rootdir, new DirFilter(fs));
> List<Path> tabledirs = new ArrayList<Path>(dirs.length);
> for (FileStatus dir: dirs) {
> Path p = dir.getPath();
> String tableName = p.getName();
> if (tableName.equals(HConstants.HREGION_LOGDIR_NAME) ||
> tableName.equals(Bytes.toString(HConstants.ROOT_TABLE_NAME)) ||
> tableName.equals(Bytes.toString(HConstants.META_TABLE_NAME)) ||
> tableName.equals(HConstants.HREGION_OLDLOGDIR_NAME) ) {
> continue;
> }
> tabledirs.add(p);
> }
> return tabledirs;
> }
> {code}
> It needs to also skip
> * .tmp
> * .corrupt
> * splitlog
> A broader check should be performed to make sure it is all covered.
> The missing .corrupt check causes for example:
> {noformat}
> 2011-07-05 11:34:33,364 WARN org.apache.hadoop.hbase.master.HMaster: Failed getting all descriptors
> java.io.FileNotFoundException: No status for hdfs://localhost:8020/hbase/.corrupt
> at org.apache.hadoop.hbase.util.FSUtils.getTableInfoModtime(FSUtils.java:888)
> at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:122)
> at org.apache.hadoop.hbase.util.FSTableDescriptors.getAll(FSTableDescriptors.java:149)
> at org.apache.hadoop.hbase.master.HMaster.getHTableDescriptors(HMaster.java:1429)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:312)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1065)
> {noformat}
> Not sure yet why others do not have this issue, could be me being on trunk and fiddling?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4061) getTableDirs is missing directories
to skip
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13060820#comment-13060820 ]
jiraposter@reviews.apache.org commented on HBASE-4061:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1018/
-----------------------------------------------------------
(Updated 2011-07-06 20:39:39.975522)
Review request for hbase.
Changes
-------
Updated patch according to review comments.
Summary
-------
getTableDirs() should check for directories such as "splitlog"
The patch reuses HConstants.HBASE_NON_USER_TABLE_DIRS
This addresses bug HBASE-4061.
https://issues.apache.org/jira/browse/HBASE-4061
Diffs (updated)
-----
/src/main/java/org/apache/hadoop/hbase/HConstants.java 1143201
/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 1143525
/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKSplitLog.java 1143201
/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 1143201
Diff: https://reviews.apache.org/r/1018/diff
Testing
-------
Ran test suite.
Thanks,
Ted
> getTableDirs is missing directories to skip
> -------------------------------------------
>
> Key: HBASE-4061
> URL: https://issues.apache.org/jira/browse/HBASE-4061
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.92.0
> Reporter: Lars George
> Assignee: Ted Yu
> Fix For: 0.92.0
>
> Attachments: 4061.txt
>
>
> The getTableDirs() is missing extra checks:
> {code}
> public static List<Path> getTableDirs(final FileSystem fs, final Path rootdir)
> throws IOException {
> // presumes any directory under hbase.rootdir is a table
> FileStatus [] dirs = fs.listStatus(rootdir, new DirFilter(fs));
> List<Path> tabledirs = new ArrayList<Path>(dirs.length);
> for (FileStatus dir: dirs) {
> Path p = dir.getPath();
> String tableName = p.getName();
> if (tableName.equals(HConstants.HREGION_LOGDIR_NAME) ||
> tableName.equals(Bytes.toString(HConstants.ROOT_TABLE_NAME)) ||
> tableName.equals(Bytes.toString(HConstants.META_TABLE_NAME)) ||
> tableName.equals(HConstants.HREGION_OLDLOGDIR_NAME) ) {
> continue;
> }
> tabledirs.add(p);
> }
> return tabledirs;
> }
> {code}
> It needs to also skip
> * .tmp
> * .corrupt
> * splitlog
> A broader check should be performed to make sure it is all covered.
> The missing .corrupt check causes for example:
> {noformat}
> 2011-07-05 11:34:33,364 WARN org.apache.hadoop.hbase.master.HMaster: Failed getting all descriptors
> java.io.FileNotFoundException: No status for hdfs://localhost:8020/hbase/.corrupt
> at org.apache.hadoop.hbase.util.FSUtils.getTableInfoModtime(FSUtils.java:888)
> at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:122)
> at org.apache.hadoop.hbase.util.FSTableDescriptors.getAll(FSTableDescriptors.java:149)
> at org.apache.hadoop.hbase.master.HMaster.getHTableDescriptors(HMaster.java:1429)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:312)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1065)
> {noformat}
> Not sure yet why others do not have this issue, could be me being on trunk and fiddling?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira