You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Andrew Purtell (JIRA)" <ji...@apache.org> on 2014/11/19 00:09:34 UTC
[jira] [Reopened] (HBASE-10378) Divide HLog interface into User and
Implementor specific interfaces
[ https://issues.apache.org/jira/browse/HBASE-10378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell reopened HBASE-10378:
------------------------------------
Pardon if I'm implicating the wrong issue.
Working on a Macbook dev box.
I have found that some TestHRegion unit tests fail now because the can't get a WAL writer:
{noformat}
$ mvn -DskipTests clean install
$ mvn test -Dtest=TestHRegion
Tests in error:
TestHRegion.testDelete_multiDeleteColumn:1836->initHRegion:4883->initHRegion:4904->initHRegion:4923 » IOException cannot get log writer
TestHRegion.testRegionReplicaSecondary:4587 » IOException cannot get log writer
TestHRegion.testScanner_DeleteOneFamilyNotAnother:2096->initHRegion:4883->initHRegion:4904->initHRegion:4923 » IOException cannot get log writer
TestHRegion.testScanner_JoinedScannersWithLimits:3156->initHRegion:4883->initHRegion:4904->initHRegion:4923 » IOException cannot get log writer
{noformat}
For example:
{noformat}
testRegionReplicaSecondary(org.apache.hadoop.hbase.regionserver.TestHRegion) Time elapsed: 0.063 sec <<< ERROR!
java.io.IOException: cannot get log writer
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:791)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:774)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:646)
at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:293)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:451)
at org.apache.hadoop.fs.ChecksumFileSystem.createNonRecursive(ChecksumFileSystem.java:469)
at org.apache.hadoop.fs.FileSystem.createNonRecursive(FileSystem.java:1086)
at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.init(ProtobufLogWriter.java:90)
at org.apache.hadoop.hbase.wal.DefaultWALProvider.createWriter(DefaultWALProvider.java:361)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.createWriterInstance(FSHLog.java:736)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:701)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:612)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.<init>(FSHLog.java:549)
at org.apache.hadoop.hbase.wal.DefaultWALProvider.init(DefaultWALProvider.java:97)
at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:139)
at org.apache.hadoop.hbase.wal.WALFactory.<init>(WALFactory.java:171)
at org.apache.hadoop.hbase.regionserver.HRegion.createHRegion(HRegion.java:4566)
at org.apache.hadoop.hbase.regionserver.HRegion.createHRegion(HRegion.java:4533)
at org.apache.hadoop.hbase.regionserver.HRegion.createHRegion(HRegion.java:4508)
at org.apache.hadoop.hbase.regionserver.HRegion.createHRegion(HRegion.java:4585)
at org.apache.hadoop.hbase.regionserver.HRegion.createHRegion(HRegion.java:4469)
at org.apache.hadoop.hbase.regionserver.TestHRegion.testRegionReplicaSecondary(TestHRegion.java:4587)
{noformat}
Then, I can't clean:
{noformat}
$ mvn clean
...
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-clean-plugin:2.5:clean (default-clean) on project hbase-server: Failed to clean project: Failed to delete /Users/apurtell/src/hbase/hbase-server/target/test-data/c8c12cda-613a-488d-9a9c-2463c8f884d5/WALs -> [Help 1]
{noformat}
I need to sudo rm -rf .
It wasn't like this last week.
> Divide HLog interface into User and Implementor specific interfaces
> -------------------------------------------------------------------
>
> Key: HBASE-10378
> URL: https://issues.apache.org/jira/browse/HBASE-10378
> Project: HBase
> Issue Type: Sub-task
> Components: wal
> Reporter: Himanshu Vashishtha
> Assignee: Sean Busbey
> Fix For: 2.0.0
>
> Attachments: 10378-1.patch, 10378-2.patch, HBASE-10378.3.patch.txt, HBASE-10378.4.patch.txt, HBASE-10378.5.patch.txt
>
>
> HBASE-5937 introduces the HLog interface as a first step to support multiple WAL implementations. This interface is a good start, but has some limitations/drawbacks in its current state, such as:
> 1) There is no clear distinction b/w User and Implementor APIs, and it provides APIs both for WAL users (append, sync, etc) and also WAL implementors (Reader/Writer interfaces, etc). There are APIs which are very much implementation specific (getFileNum, etc) and a user such as a RegionServer shouldn't know about it.
> 2) There are about 14 methods in FSHLog which are not present in HLog interface but are used at several places in the unit test code. These tests typecast HLog to FSHLog, which makes it very difficult to test multiple WAL implementations without doing some ugly checks.
> I'd like to propose some changes in HLog interface that would ease the multi WAL story:
> 1) Have two interfaces WAL and WALService. WAL provides APIs for implementors. WALService provides APIs for users (such as RegionServer).
> 2) A skeleton implementation of the above two interface as the base class for other WAL implementations (AbstractWAL). It provides required fields for all subclasses (fs, conf, log dir, etc). Make a minimal set of test only methods and add this set in AbstractWAL.
> 3) HLogFactory returns a WALService reference when creating a WAL instance; if a user need to access impl specific APIs (there are unit tests which get WAL from a HRegionServer and then call impl specific APIs), use AbstractWAL type casting,
> 4) Make TestHLog abstract and let all implementors provide their respective test class which extends TestHLog (TestFSHLog, for example).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)