You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Myungbo Kim <mb...@vcnc.co.kr> on 2013/10/10 07:55:08 UTC
Haeinsa: linear scalable multi-row transaction library for HBase
Hello everyone in the mailing list.
I want to introduce an open-source library Haeinsa which supports multi-row, multi-table transaction on HBase.
Haeinsa is client-only library that support serializability and linear scalability.
Here is the github repository and presentation that describes mechanism of it.
github : https://github.com/VCNC/haeinsa
presentation : https://speakerdeck.com/vcnc/haeinsa-overview
It was inspired by Google's percolator, but implementation detail is different.
It use two-phase commit protocol and optimistic concurrency control to implement, and Haeinsa now processes more than 300M+ transactions per day in single cluster without any consistency problem for more than 2 month.
I know that there has been lots of libraries and papers for HBase transaction, but within my knowledge, this is the only open-source library which support serializability and linear scalability.
There is no theoretical limit of transaction throughput.
It was tested against cluster on AWS until 40,000 transaction/sec. (Still testing on bigger cluster)
If you find it interesting, please leave me comment.
Thanks,
Andrew Kim
Re: Haeinsa: linear scalable multi-row transaction library for HBase
Posted by Ted Yu <yu...@gmail.com>.
This optimization is feasible.
Cheers
On Nov 8, 2013, at 4:53 AM, James Lee <ro...@gmail.com> wrote:
> Thanks for letting us know the good feature.
>
> Maybe I can find the way to optimize Haeinsa transaction which mutations
> are in single HBase region with MultiRowMutationEndpoint.
>
>
> On Thu, Nov 7, 2013 at 11:11 AM, Ted Yu <yu...@gmail.com> wrote:
>
>> Thanks for the update.
>>
>> Have you taken a look at the following endpoint in 0.94 ?
>>
>> src/main/java/org/apache/hadoop/hbase/coprocessor/MultiRowMutationEndpoint.java
>>
>>
>> On Wed, Nov 6, 2013 at 5:40 PM, James Lee <ro...@gmail.com> wrote:
>>
>>> That's for optimization of performance in Haeinsa.
>>> We can use fewer HBase operation in single row transaction.
>>> It is related to algorithm design of Haeinsa.
>>>
>>> For example, consider about following raw-HBase code. (2get/2put to 2
>> rows)
>>>
>>> get(row1, col1)
>>> get(row2, col2)
>>> put(row1, col1, value1)
>>> put(row2, col2, value2)
>>>
>>> If you use Haeinsa, this will be executed by following order.
>>> (2get/5checkAndPut)
>>>
>>> get(row1, col1)
>>> get(row2, col2)
>>> // commit() here
>>> checkAndPut(row1, prevLock1, {col1:value1})
>>> checkAndPut(row2, prevLock2, {col2:value2})
>>> checkAndPut(row1, prevLock1, COMMITED)
>>> checkAndPut(row2, prevLock2, STABLE)
>>> checkAndPut(row1, prevLock1, STABLE)
>>>
>>> Now, consider single row transaction. (2get/2put to single row)
>>>
>>> get(row1, col1)
>>> get(row1, col2)
>>> put(row1, col1, value1)
>>> put(row1, col2, value2)
>>>
>>> If you use Haeinsa, this will be executed by following order.
>>> (2get/1checkAndPut)
>>>
>>> get(row1, col1)
>>> get(row1, col2)
>>> // commit() here
>>> checkAndPut(row1, prevLock1, {col1:value1, col2:value2, lock:STABLE})
>>>
>>> Single row transaction needs much fewer operation to achieve ACID
>>> properties in Haeinsa
>>>
>>> Thanks.
>>>
>>>
>>> On Thu, Nov 7, 2013 at 7:40 AM, Ted Yu <yu...@gmail.com> wrote:
>>>
>>>> Jung-Haeng:
>>>> In HaeinsaTransaction.java :
>>>>
>>>> /**
>>>> * If there is only one rowTx and type of its mutation is
>>>> HaeinsaPut.
>>>> */
>>>> SINGLE_ROW_PUT_ONLY,
>>>>
>>>> Can you elaborate a bit on why the single row put gets special
>> treatment
>>> ?
>>>>
>>>> Cheers
>>>>
>>>>
>>>> On Fri, Oct 11, 2013 at 6:09 PM, Ted Yu <yu...@gmail.com> wrote:
>>>>
>>>>> I am now able to build on Linux.
>>>>>
>>>>> Running test suite.
>>>>>
>>>>> Will look deeper into the code over the weekend.
>>>>>
>>>>>
>>>>> On Thu, Oct 10, 2013 at 10:16 PM, Jung-Haeng Lee <roth2520@gmail.com
>>>>> wrote:
>>>>>
>>>>>> Thanks for your advice.
>>>>>> I fixed to Hbase 0.94.3 on pom.xml.
>>>>>> (To ensure consistency, HBASE-7051 should be solved.)
>>>>>>
>>>>>> I also fixed pom.xml of Haeinsa:
>>>>>> Now maven-thrift-plugin is activate on linux using default profile.
>>>>>> Please ensure the "thrift" executable is in your PATH.
>>>>>>
>>>>>> And you should build Haeinsa with jdk 1.7.
>>>>>> I think System.lineSeparator() is new method of Java7.
>>>>>>
>>>>>> - James Lee
>>>>>>
>>>>>>
>>>>>> On Fri, Oct 11, 2013 at 3:18 AM, Ted Yu <yu...@gmail.com>
>> wrote:
>>>>>>
>>>>>>> Using the 'mac' profile, I got the following:
>>>>>>>
>>>>>>> Number of foreign imports: 1
>>>>>>> import: Entry[import from realm ClassRealm[maven.api, parent:
>>> null]]
>>>>>>>
>>>>>>> -----------------------------------------------------
>>>>>>>
>>>>>>> at
>> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
>>>>>>> ... 20 more
>>>>>>> Caused by: java.lang.NoSuchMethodError:
>>>>>>> java.lang.System.lineSeparator()Ljava/lang/String;
>>>>>>> at
>>>> io.netty.build.checkstyle.NewlineCheck.<clinit>(NewlineCheck.java:43)
>>>>>>> at java.lang.Class.forName0(Native Method)
>>>>>>> at java.lang.Class.forName(Class.java:249)
>>>>>>> at
>> com.puppycrawl.tools.checkstyle.PackageObjectFactory.createObject(PackageObjectFactory.java:111)
>>>>>>> at
>> com.puppycrawl.tools.checkstyle.PackageObjectFactory.doMakeObject(PackageObjectFactory.java:80)
>>>>>>> at
>> com.puppycrawl.tools.checkstyle.PackageObjectFactory.createModule(PackageObjectFactory.java:147)
>>>>>>> at
>>>> com.puppycrawl.tools.checkstyle.Checker.setupChild(Checker.java:153)
>>>>>>> at
>> com.puppycrawl.tools.checkstyle.api.AutomaticBean.configure(AutomaticBean.java:184)
>>>>>>> at
>> org.apache.maven.plugin.checkstyle.DefaultCheckstyleExecutor.executeCheckstyle(DefaultCheckstyleExecutor.java:172)
>>>>>>> at
>> org.apache.maven.plugin.checkstyle.CheckstyleViolationCheckMojo.execute(CheckstyleViolationCheckMojo.java:365)
>>>>>>> at
>> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
>>>>>>> ... 20 more
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Oct 10, 2013 at 11:06 AM, Ted Yu <yu...@gmail.com>
>>> wrote:
>>>>>>>
>>>>>>>> Looks like HBase 0.92.1 is used. How about upgrading to 0.94.12
>> ?
>>>>>>>>
>>>>>>>> I cloned the git repo. But I got some compilation errors:
>>>>>>>>
>>>>>>>> [ERROR]
>> /grid/0/dev/ty/haeinsa/src/main/java/kr/co/vcnc/haeinsa/HaeinsaTransaction.java:[324,20]
>>>>>>>> cannot find symbol
>>>>>>>> [ERROR] symbol: class TRowKey
>>>>>>>> [ERROR] location: class kr.co.vcnc.haeinsa.HaeinsaTransaction
>>>>>>>> [ERROR]
>> /grid/0/dev/ty/haeinsa/src/main/java/kr/co/vcnc/haeinsa/HaeinsaTransaction.java:[325,13]
>>>>>>>> cannot find symbol
>>>>>>>> [ERROR] symbol: class TRowKey
>>>>>>>> [ERROR] location: class kr.co.vcnc.haeinsa.HaeinsaTransaction
>>>>>>>>
>>>>>>>> How do I activate maven-thrift-plugin on Linux ?
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>>
>>>>>>>> On Thu, Oct 10, 2013 at 9:41 AM, Michael Segel <
>>>>>>> msegel_hadoop@hotmail.com>wrote:
>>>>>>>>
>>>>>>>>> Which level of isolation do you support?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Oct 10, 2013, at 12:55 AM, Myungbo Kim <mb...@vcnc.co.kr>
>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hello everyone in the mailing list.
>>>>>>>>>>
>>>>>>>>>> I want to introduce an open-source library Haeinsa which
>>> supports
>>>>>>>>> multi-row, multi-table transaction on HBase.
>>>>>>>>>> Haeinsa is client-only library that support serializability
>> and
>>>>>> linear
>>>>>>>>> scalability.
>>>>>>>>>> Here is the github repository and presentation that describes
>>>>>>> mechanism
>>>>>>>>> of it.
>>>>>>>>>>
>>>>>>>>>> github : https://github.com/VCNC/haeinsa
>>>>>>>>>> presentation : https://speakerdeck.com/vcnc/haeinsa-overview
>>>>>>>>>>
>>>>>>>>>> It was inspired by Google's percolator, but implementation
>>> detail
>>>>>> is
>>>>>>>>> different.
>>>>>>>>>> It use two-phase commit protocol and optimistic concurrency
>>>>>> control to
>>>>>>>>> implement, and Haeinsa now processes more than 300M+
>> transactions
>>>> per
>>>>>>> day
>>>>>>>>> in single cluster without any consistency problem for more
>> than 2
>>>>>> month.
>>>>>>>>>> I know that there has been lots of libraries and papers for
>>> HBase
>>>>>>>>> transaction, but within my knowledge, this is the only
>>> open-source
>>>>>>> library
>>>>>>>>> which support serializability and linear scalability.
>>>>>>>>>> There is no theoretical limit of transaction throughput.
>>>>>>>>>> It was tested against cluster on AWS until 40,000
>>>> transaction/sec.
>>>>>>>>> (Still testing on bigger cluster)
>>>>>>>>>> If you find it interesting, please leave me comment.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Andrew Kim
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> *이 정 행 / Jung-Haeng Lee*
>>>>>> *
>>>>>> *
>>>>>> Blog: http://eincs.net
>>>>>> Facebook: http://www.facebook.com/eincs
>>>>>> Twitter: http://twitter.com/eincs
>>>>>> LinkedIn: http://linkedin.com/in/eincs
>>>
>>>
>>>
>>> --
>>> *이 정 행 / Jung-Haeng Lee*
>>>
>>> Blog: http://eincs.net
>>> Facebook: http://www.facebook.com/eincs
>>> Twitter: http://twitter.com/eincs
>>> LinkedIn: http://linkedin.com/in/eincs
>
>
>
> --
> *이 정 행 / Jung-Haeng Lee*
>
> Blog: http://eincs.net
> Facebook: http://www.facebook.com/eincs
> Twitter: http://twitter.com/eincs
> LinkedIn: http://linkedin.com/in/eincs
Re: Haeinsa: linear scalable multi-row transaction library for HBase
Posted by James Lee <ro...@gmail.com>.
Thanks for letting us know the good feature.
Maybe I can find the way to optimize Haeinsa transaction which mutations
are in single HBase region with MultiRowMutationEndpoint.
On Thu, Nov 7, 2013 at 11:11 AM, Ted Yu <yu...@gmail.com> wrote:
> Thanks for the update.
>
> Have you taken a look at the following endpoint in 0.94 ?
>
> src/main/java/org/apache/hadoop/hbase/coprocessor/MultiRowMutationEndpoint.java
>
>
> On Wed, Nov 6, 2013 at 5:40 PM, James Lee <ro...@gmail.com> wrote:
>
> > That's for optimization of performance in Haeinsa.
> > We can use fewer HBase operation in single row transaction.
> > It is related to algorithm design of Haeinsa.
> >
> > For example, consider about following raw-HBase code. (2get/2put to 2
> rows)
> >
> > get(row1, col1)
> > get(row2, col2)
> > put(row1, col1, value1)
> > put(row2, col2, value2)
> >
> > If you use Haeinsa, this will be executed by following order.
> > (2get/5checkAndPut)
> >
> > get(row1, col1)
> > get(row2, col2)
> > // commit() here
> > checkAndPut(row1, prevLock1, {col1:value1})
> > checkAndPut(row2, prevLock2, {col2:value2})
> > checkAndPut(row1, prevLock1, COMMITED)
> > checkAndPut(row2, prevLock2, STABLE)
> > checkAndPut(row1, prevLock1, STABLE)
> >
> > Now, consider single row transaction. (2get/2put to single row)
> >
> > get(row1, col1)
> > get(row1, col2)
> > put(row1, col1, value1)
> > put(row1, col2, value2)
> >
> > If you use Haeinsa, this will be executed by following order.
> > (2get/1checkAndPut)
> >
> > get(row1, col1)
> > get(row1, col2)
> > // commit() here
> > checkAndPut(row1, prevLock1, {col1:value1, col2:value2, lock:STABLE})
> >
> > Single row transaction needs much fewer operation to achieve ACID
> > properties in Haeinsa
> >
> > Thanks.
> >
> >
> > On Thu, Nov 7, 2013 at 7:40 AM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > Jung-Haeng:
> > > In HaeinsaTransaction.java :
> > >
> > > /**
> > > * If there is only one rowTx and type of its mutation is
> > > HaeinsaPut.
> > > */
> > > SINGLE_ROW_PUT_ONLY,
> > >
> > > Can you elaborate a bit on why the single row put gets special
> treatment
> > ?
> > >
> > > Cheers
> > >
> > >
> > > On Fri, Oct 11, 2013 at 6:09 PM, Ted Yu <yu...@gmail.com> wrote:
> > >
> > > > I am now able to build on Linux.
> > > >
> > > > Running test suite.
> > > >
> > > > Will look deeper into the code over the weekend.
> > > >
> > > >
> > > > On Thu, Oct 10, 2013 at 10:16 PM, Jung-Haeng Lee <roth2520@gmail.com
> > > >wrote:
> > > >
> > > >> Thanks for your advice.
> > > >> I fixed to Hbase 0.94.3 on pom.xml.
> > > >> (To ensure consistency, HBASE-7051 should be solved.)
> > > >>
> > > >> I also fixed pom.xml of Haeinsa:
> > > >> Now maven-thrift-plugin is activate on linux using default profile.
> > > >> Please ensure the "thrift" executable is in your PATH.
> > > >>
> > > >> And you should build Haeinsa with jdk 1.7.
> > > >> I think System.lineSeparator() is new method of Java7.
> > > >>
> > > >> - James Lee
> > > >>
> > > >>
> > > >> On Fri, Oct 11, 2013 at 3:18 AM, Ted Yu <yu...@gmail.com>
> wrote:
> > > >>
> > > >> > Using the 'mac' profile, I got the following:
> > > >> >
> > > >> > Number of foreign imports: 1
> > > >> > import: Entry[import from realm ClassRealm[maven.api, parent:
> > null]]
> > > >> >
> > > >> > -----------------------------------------------------
> > > >> >
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
> > > >> > ... 20 more
> > > >> > Caused by: java.lang.NoSuchMethodError:
> > > >> > java.lang.System.lineSeparator()Ljava/lang/String;
> > > >> > at
> > > io.netty.build.checkstyle.NewlineCheck.<clinit>(NewlineCheck.java:43)
> > > >> > at java.lang.Class.forName0(Native Method)
> > > >> > at java.lang.Class.forName(Class.java:249)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> com.puppycrawl.tools.checkstyle.PackageObjectFactory.createObject(PackageObjectFactory.java:111)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> com.puppycrawl.tools.checkstyle.PackageObjectFactory.doMakeObject(PackageObjectFactory.java:80)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> com.puppycrawl.tools.checkstyle.PackageObjectFactory.createModule(PackageObjectFactory.java:147)
> > > >> > at
> > > com.puppycrawl.tools.checkstyle.Checker.setupChild(Checker.java:153)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> com.puppycrawl.tools.checkstyle.api.AutomaticBean.configure(AutomaticBean.java:184)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.maven.plugin.checkstyle.DefaultCheckstyleExecutor.executeCheckstyle(DefaultCheckstyleExecutor.java:172)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.maven.plugin.checkstyle.CheckstyleViolationCheckMojo.execute(CheckstyleViolationCheckMojo.java:365)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
> > > >> > ... 20 more
> > > >> >
> > > >> >
> > > >> > On Thu, Oct 10, 2013 at 11:06 AM, Ted Yu <yu...@gmail.com>
> > wrote:
> > > >> >
> > > >> > > Looks like HBase 0.92.1 is used. How about upgrading to 0.94.12
> ?
> > > >> > >
> > > >> > > I cloned the git repo. But I got some compilation errors:
> > > >> > >
> > > >> > > [ERROR]
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> /grid/0/dev/ty/haeinsa/src/main/java/kr/co/vcnc/haeinsa/HaeinsaTransaction.java:[324,20]
> > > >> > > cannot find symbol
> > > >> > > [ERROR] symbol: class TRowKey
> > > >> > > [ERROR] location: class kr.co.vcnc.haeinsa.HaeinsaTransaction
> > > >> > > [ERROR]
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> /grid/0/dev/ty/haeinsa/src/main/java/kr/co/vcnc/haeinsa/HaeinsaTransaction.java:[325,13]
> > > >> > > cannot find symbol
> > > >> > > [ERROR] symbol: class TRowKey
> > > >> > > [ERROR] location: class kr.co.vcnc.haeinsa.HaeinsaTransaction
> > > >> > >
> > > >> > > How do I activate maven-thrift-plugin on Linux ?
> > > >> > >
> > > >> > > Thanks
> > > >> > >
> > > >> > > On Thu, Oct 10, 2013 at 9:41 AM, Michael Segel <
> > > >> > msegel_hadoop@hotmail.com>wrote:
> > > >> > >
> > > >> > >> Which level of isolation do you support?
> > > >> > >>
> > > >> > >>
> > > >> > >> On Oct 10, 2013, at 12:55 AM, Myungbo Kim <mb...@vcnc.co.kr>
> > > wrote:
> > > >> > >>
> > > >> > >> > Hello everyone in the mailing list.
> > > >> > >> >
> > > >> > >> > I want to introduce an open-source library Haeinsa which
> > supports
> > > >> > >> multi-row, multi-table transaction on HBase.
> > > >> > >> > Haeinsa is client-only library that support serializability
> and
> > > >> linear
> > > >> > >> scalability.
> > > >> > >> > Here is the github repository and presentation that describes
> > > >> > mechanism
> > > >> > >> of it.
> > > >> > >> >
> > > >> > >> > github : https://github.com/VCNC/haeinsa
> > > >> > >> > presentation : https://speakerdeck.com/vcnc/haeinsa-overview
> > > >> > >> >
> > > >> > >> > It was inspired by Google's percolator, but implementation
> > detail
> > > >> is
> > > >> > >> different.
> > > >> > >> > It use two-phase commit protocol and optimistic concurrency
> > > >> control to
> > > >> > >> implement, and Haeinsa now processes more than 300M+
> transactions
> > > per
> > > >> > day
> > > >> > >> in single cluster without any consistency problem for more
> than 2
> > > >> month.
> > > >> > >> > I know that there has been lots of libraries and papers for
> > HBase
> > > >> > >> transaction, but within my knowledge, this is the only
> > open-source
> > > >> > library
> > > >> > >> which support serializability and linear scalability.
> > > >> > >> > There is no theoretical limit of transaction throughput.
> > > >> > >> > It was tested against cluster on AWS until 40,000
> > > transaction/sec.
> > > >> > >> (Still testing on bigger cluster)
> > > >> > >> > If you find it interesting, please leave me comment.
> > > >> > >> >
> > > >> > >> > Thanks,
> > > >> > >> > Andrew Kim
> > > >> > >>
> > > >> > >>
> > > >> > >
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> *이 정 행 / Jung-Haeng Lee*
> > > >> *
> > > >> *
> > > >> Blog: http://eincs.net
> > > >> Facebook: http://www.facebook.com/eincs
> > > >> Twitter: http://twitter.com/eincs
> > > >> LinkedIn: http://linkedin.com/in/eincs
> > > >>
> > > >
> > > >
> > >
> >
> >
> >
> > --
> > *이 정 행 / Jung-Haeng Lee*
> >
> > Blog: http://eincs.net
> > Facebook: http://www.facebook.com/eincs
> > Twitter: http://twitter.com/eincs
> > LinkedIn: http://linkedin.com/in/eincs
> >
>
--
*이 정 행 / Jung-Haeng Lee*
Blog: http://eincs.net
Facebook: http://www.facebook.com/eincs
Twitter: http://twitter.com/eincs
LinkedIn: http://linkedin.com/in/eincs
Re: Haeinsa: linear scalable multi-row transaction library for HBase
Posted by Ted Yu <yu...@gmail.com>.
Thanks for the update.
Have you taken a look at the following endpoint in 0.94 ?
src/main/java/org/apache/hadoop/hbase/coprocessor/MultiRowMutationEndpoint.java
On Wed, Nov 6, 2013 at 5:40 PM, James Lee <ro...@gmail.com> wrote:
> That's for optimization of performance in Haeinsa.
> We can use fewer HBase operation in single row transaction.
> It is related to algorithm design of Haeinsa.
>
> For example, consider about following raw-HBase code. (2get/2put to 2 rows)
>
> get(row1, col1)
> get(row2, col2)
> put(row1, col1, value1)
> put(row2, col2, value2)
>
> If you use Haeinsa, this will be executed by following order.
> (2get/5checkAndPut)
>
> get(row1, col1)
> get(row2, col2)
> // commit() here
> checkAndPut(row1, prevLock1, {col1:value1})
> checkAndPut(row2, prevLock2, {col2:value2})
> checkAndPut(row1, prevLock1, COMMITED)
> checkAndPut(row2, prevLock2, STABLE)
> checkAndPut(row1, prevLock1, STABLE)
>
> Now, consider single row transaction. (2get/2put to single row)
>
> get(row1, col1)
> get(row1, col2)
> put(row1, col1, value1)
> put(row1, col2, value2)
>
> If you use Haeinsa, this will be executed by following order.
> (2get/1checkAndPut)
>
> get(row1, col1)
> get(row1, col2)
> // commit() here
> checkAndPut(row1, prevLock1, {col1:value1, col2:value2, lock:STABLE})
>
> Single row transaction needs much fewer operation to achieve ACID
> properties in Haeinsa
>
> Thanks.
>
>
> On Thu, Nov 7, 2013 at 7:40 AM, Ted Yu <yu...@gmail.com> wrote:
>
> > Jung-Haeng:
> > In HaeinsaTransaction.java :
> >
> > /**
> > * If there is only one rowTx and type of its mutation is
> > HaeinsaPut.
> > */
> > SINGLE_ROW_PUT_ONLY,
> >
> > Can you elaborate a bit on why the single row put gets special treatment
> ?
> >
> > Cheers
> >
> >
> > On Fri, Oct 11, 2013 at 6:09 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > I am now able to build on Linux.
> > >
> > > Running test suite.
> > >
> > > Will look deeper into the code over the weekend.
> > >
> > >
> > > On Thu, Oct 10, 2013 at 10:16 PM, Jung-Haeng Lee <roth2520@gmail.com
> > >wrote:
> > >
> > >> Thanks for your advice.
> > >> I fixed to Hbase 0.94.3 on pom.xml.
> > >> (To ensure consistency, HBASE-7051 should be solved.)
> > >>
> > >> I also fixed pom.xml of Haeinsa:
> > >> Now maven-thrift-plugin is activate on linux using default profile.
> > >> Please ensure the "thrift" executable is in your PATH.
> > >>
> > >> And you should build Haeinsa with jdk 1.7.
> > >> I think System.lineSeparator() is new method of Java7.
> > >>
> > >> - James Lee
> > >>
> > >>
> > >> On Fri, Oct 11, 2013 at 3:18 AM, Ted Yu <yu...@gmail.com> wrote:
> > >>
> > >> > Using the 'mac' profile, I got the following:
> > >> >
> > >> > Number of foreign imports: 1
> > >> > import: Entry[import from realm ClassRealm[maven.api, parent:
> null]]
> > >> >
> > >> > -----------------------------------------------------
> > >> >
> > >> > at
> > >> >
> > >> >
> > >>
> >
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
> > >> > ... 20 more
> > >> > Caused by: java.lang.NoSuchMethodError:
> > >> > java.lang.System.lineSeparator()Ljava/lang/String;
> > >> > at
> > io.netty.build.checkstyle.NewlineCheck.<clinit>(NewlineCheck.java:43)
> > >> > at java.lang.Class.forName0(Native Method)
> > >> > at java.lang.Class.forName(Class.java:249)
> > >> > at
> > >> >
> > >> >
> > >>
> >
> com.puppycrawl.tools.checkstyle.PackageObjectFactory.createObject(PackageObjectFactory.java:111)
> > >> > at
> > >> >
> > >> >
> > >>
> >
> com.puppycrawl.tools.checkstyle.PackageObjectFactory.doMakeObject(PackageObjectFactory.java:80)
> > >> > at
> > >> >
> > >> >
> > >>
> >
> com.puppycrawl.tools.checkstyle.PackageObjectFactory.createModule(PackageObjectFactory.java:147)
> > >> > at
> > com.puppycrawl.tools.checkstyle.Checker.setupChild(Checker.java:153)
> > >> > at
> > >> >
> > >> >
> > >>
> >
> com.puppycrawl.tools.checkstyle.api.AutomaticBean.configure(AutomaticBean.java:184)
> > >> > at
> > >> >
> > >> >
> > >>
> >
> org.apache.maven.plugin.checkstyle.DefaultCheckstyleExecutor.executeCheckstyle(DefaultCheckstyleExecutor.java:172)
> > >> > at
> > >> >
> > >> >
> > >>
> >
> org.apache.maven.plugin.checkstyle.CheckstyleViolationCheckMojo.execute(CheckstyleViolationCheckMojo.java:365)
> > >> > at
> > >> >
> > >> >
> > >>
> >
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
> > >> > ... 20 more
> > >> >
> > >> >
> > >> > On Thu, Oct 10, 2013 at 11:06 AM, Ted Yu <yu...@gmail.com>
> wrote:
> > >> >
> > >> > > Looks like HBase 0.92.1 is used. How about upgrading to 0.94.12 ?
> > >> > >
> > >> > > I cloned the git repo. But I got some compilation errors:
> > >> > >
> > >> > > [ERROR]
> > >> > >
> > >> >
> > >>
> >
> /grid/0/dev/ty/haeinsa/src/main/java/kr/co/vcnc/haeinsa/HaeinsaTransaction.java:[324,20]
> > >> > > cannot find symbol
> > >> > > [ERROR] symbol: class TRowKey
> > >> > > [ERROR] location: class kr.co.vcnc.haeinsa.HaeinsaTransaction
> > >> > > [ERROR]
> > >> > >
> > >> >
> > >>
> >
> /grid/0/dev/ty/haeinsa/src/main/java/kr/co/vcnc/haeinsa/HaeinsaTransaction.java:[325,13]
> > >> > > cannot find symbol
> > >> > > [ERROR] symbol: class TRowKey
> > >> > > [ERROR] location: class kr.co.vcnc.haeinsa.HaeinsaTransaction
> > >> > >
> > >> > > How do I activate maven-thrift-plugin on Linux ?
> > >> > >
> > >> > > Thanks
> > >> > >
> > >> > > On Thu, Oct 10, 2013 at 9:41 AM, Michael Segel <
> > >> > msegel_hadoop@hotmail.com>wrote:
> > >> > >
> > >> > >> Which level of isolation do you support?
> > >> > >>
> > >> > >>
> > >> > >> On Oct 10, 2013, at 12:55 AM, Myungbo Kim <mb...@vcnc.co.kr>
> > wrote:
> > >> > >>
> > >> > >> > Hello everyone in the mailing list.
> > >> > >> >
> > >> > >> > I want to introduce an open-source library Haeinsa which
> supports
> > >> > >> multi-row, multi-table transaction on HBase.
> > >> > >> > Haeinsa is client-only library that support serializability and
> > >> linear
> > >> > >> scalability.
> > >> > >> > Here is the github repository and presentation that describes
> > >> > mechanism
> > >> > >> of it.
> > >> > >> >
> > >> > >> > github : https://github.com/VCNC/haeinsa
> > >> > >> > presentation : https://speakerdeck.com/vcnc/haeinsa-overview
> > >> > >> >
> > >> > >> > It was inspired by Google's percolator, but implementation
> detail
> > >> is
> > >> > >> different.
> > >> > >> > It use two-phase commit protocol and optimistic concurrency
> > >> control to
> > >> > >> implement, and Haeinsa now processes more than 300M+ transactions
> > per
> > >> > day
> > >> > >> in single cluster without any consistency problem for more than 2
> > >> month.
> > >> > >> > I know that there has been lots of libraries and papers for
> HBase
> > >> > >> transaction, but within my knowledge, this is the only
> open-source
> > >> > library
> > >> > >> which support serializability and linear scalability.
> > >> > >> > There is no theoretical limit of transaction throughput.
> > >> > >> > It was tested against cluster on AWS until 40,000
> > transaction/sec.
> > >> > >> (Still testing on bigger cluster)
> > >> > >> > If you find it interesting, please leave me comment.
> > >> > >> >
> > >> > >> > Thanks,
> > >> > >> > Andrew Kim
> > >> > >>
> > >> > >>
> > >> > >
> > >> >
> > >>
> > >>
> > >>
> > >> --
> > >> *이 정 행 / Jung-Haeng Lee*
> > >> *
> > >> *
> > >> Blog: http://eincs.net
> > >> Facebook: http://www.facebook.com/eincs
> > >> Twitter: http://twitter.com/eincs
> > >> LinkedIn: http://linkedin.com/in/eincs
> > >>
> > >
> > >
> >
>
>
>
> --
> *이 정 행 / Jung-Haeng Lee*
>
> Blog: http://eincs.net
> Facebook: http://www.facebook.com/eincs
> Twitter: http://twitter.com/eincs
> LinkedIn: http://linkedin.com/in/eincs
>
Re: Haeinsa: linear scalable multi-row transaction library for HBase
Posted by James Lee <ro...@gmail.com>.
That's for optimization of performance in Haeinsa.
We can use fewer HBase operation in single row transaction.
It is related to algorithm design of Haeinsa.
For example, consider about following raw-HBase code. (2get/2put to 2 rows)
get(row1, col1)
get(row2, col2)
put(row1, col1, value1)
put(row2, col2, value2)
If you use Haeinsa, this will be executed by following order.
(2get/5checkAndPut)
get(row1, col1)
get(row2, col2)
// commit() here
checkAndPut(row1, prevLock1, {col1:value1})
checkAndPut(row2, prevLock2, {col2:value2})
checkAndPut(row1, prevLock1, COMMITED)
checkAndPut(row2, prevLock2, STABLE)
checkAndPut(row1, prevLock1, STABLE)
Now, consider single row transaction. (2get/2put to single row)
get(row1, col1)
get(row1, col2)
put(row1, col1, value1)
put(row1, col2, value2)
If you use Haeinsa, this will be executed by following order.
(2get/1checkAndPut)
get(row1, col1)
get(row1, col2)
// commit() here
checkAndPut(row1, prevLock1, {col1:value1, col2:value2, lock:STABLE})
Single row transaction needs much fewer operation to achieve ACID
properties in Haeinsa
Thanks.
On Thu, Nov 7, 2013 at 7:40 AM, Ted Yu <yu...@gmail.com> wrote:
> Jung-Haeng:
> In HaeinsaTransaction.java :
>
> /**
> * If there is only one rowTx and type of its mutation is
> HaeinsaPut.
> */
> SINGLE_ROW_PUT_ONLY,
>
> Can you elaborate a bit on why the single row put gets special treatment ?
>
> Cheers
>
>
> On Fri, Oct 11, 2013 at 6:09 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > I am now able to build on Linux.
> >
> > Running test suite.
> >
> > Will look deeper into the code over the weekend.
> >
> >
> > On Thu, Oct 10, 2013 at 10:16 PM, Jung-Haeng Lee <roth2520@gmail.com
> >wrote:
> >
> >> Thanks for your advice.
> >> I fixed to Hbase 0.94.3 on pom.xml.
> >> (To ensure consistency, HBASE-7051 should be solved.)
> >>
> >> I also fixed pom.xml of Haeinsa:
> >> Now maven-thrift-plugin is activate on linux using default profile.
> >> Please ensure the "thrift" executable is in your PATH.
> >>
> >> And you should build Haeinsa with jdk 1.7.
> >> I think System.lineSeparator() is new method of Java7.
> >>
> >> - James Lee
> >>
> >>
> >> On Fri, Oct 11, 2013 at 3:18 AM, Ted Yu <yu...@gmail.com> wrote:
> >>
> >> > Using the 'mac' profile, I got the following:
> >> >
> >> > Number of foreign imports: 1
> >> > import: Entry[import from realm ClassRealm[maven.api, parent: null]]
> >> >
> >> > -----------------------------------------------------
> >> >
> >> > at
> >> >
> >> >
> >>
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
> >> > ... 20 more
> >> > Caused by: java.lang.NoSuchMethodError:
> >> > java.lang.System.lineSeparator()Ljava/lang/String;
> >> > at
> io.netty.build.checkstyle.NewlineCheck.<clinit>(NewlineCheck.java:43)
> >> > at java.lang.Class.forName0(Native Method)
> >> > at java.lang.Class.forName(Class.java:249)
> >> > at
> >> >
> >> >
> >>
> com.puppycrawl.tools.checkstyle.PackageObjectFactory.createObject(PackageObjectFactory.java:111)
> >> > at
> >> >
> >> >
> >>
> com.puppycrawl.tools.checkstyle.PackageObjectFactory.doMakeObject(PackageObjectFactory.java:80)
> >> > at
> >> >
> >> >
> >>
> com.puppycrawl.tools.checkstyle.PackageObjectFactory.createModule(PackageObjectFactory.java:147)
> >> > at
> com.puppycrawl.tools.checkstyle.Checker.setupChild(Checker.java:153)
> >> > at
> >> >
> >> >
> >>
> com.puppycrawl.tools.checkstyle.api.AutomaticBean.configure(AutomaticBean.java:184)
> >> > at
> >> >
> >> >
> >>
> org.apache.maven.plugin.checkstyle.DefaultCheckstyleExecutor.executeCheckstyle(DefaultCheckstyleExecutor.java:172)
> >> > at
> >> >
> >> >
> >>
> org.apache.maven.plugin.checkstyle.CheckstyleViolationCheckMojo.execute(CheckstyleViolationCheckMojo.java:365)
> >> > at
> >> >
> >> >
> >>
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
> >> > ... 20 more
> >> >
> >> >
> >> > On Thu, Oct 10, 2013 at 11:06 AM, Ted Yu <yu...@gmail.com> wrote:
> >> >
> >> > > Looks like HBase 0.92.1 is used. How about upgrading to 0.94.12 ?
> >> > >
> >> > > I cloned the git repo. But I got some compilation errors:
> >> > >
> >> > > [ERROR]
> >> > >
> >> >
> >>
> /grid/0/dev/ty/haeinsa/src/main/java/kr/co/vcnc/haeinsa/HaeinsaTransaction.java:[324,20]
> >> > > cannot find symbol
> >> > > [ERROR] symbol: class TRowKey
> >> > > [ERROR] location: class kr.co.vcnc.haeinsa.HaeinsaTransaction
> >> > > [ERROR]
> >> > >
> >> >
> >>
> /grid/0/dev/ty/haeinsa/src/main/java/kr/co/vcnc/haeinsa/HaeinsaTransaction.java:[325,13]
> >> > > cannot find symbol
> >> > > [ERROR] symbol: class TRowKey
> >> > > [ERROR] location: class kr.co.vcnc.haeinsa.HaeinsaTransaction
> >> > >
> >> > > How do I activate maven-thrift-plugin on Linux ?
> >> > >
> >> > > Thanks
> >> > >
> >> > > On Thu, Oct 10, 2013 at 9:41 AM, Michael Segel <
> >> > msegel_hadoop@hotmail.com>wrote:
> >> > >
> >> > >> Which level of isolation do you support?
> >> > >>
> >> > >>
> >> > >> On Oct 10, 2013, at 12:55 AM, Myungbo Kim <mb...@vcnc.co.kr>
> wrote:
> >> > >>
> >> > >> > Hello everyone in the mailing list.
> >> > >> >
> >> > >> > I want to introduce an open-source library Haeinsa which supports
> >> > >> multi-row, multi-table transaction on HBase.
> >> > >> > Haeinsa is client-only library that support serializability and
> >> linear
> >> > >> scalability.
> >> > >> > Here is the github repository and presentation that describes
> >> > mechanism
> >> > >> of it.
> >> > >> >
> >> > >> > github : https://github.com/VCNC/haeinsa
> >> > >> > presentation : https://speakerdeck.com/vcnc/haeinsa-overview
> >> > >> >
> >> > >> > It was inspired by Google's percolator, but implementation detail
> >> is
> >> > >> different.
> >> > >> > It use two-phase commit protocol and optimistic concurrency
> >> control to
> >> > >> implement, and Haeinsa now processes more than 300M+ transactions
> per
> >> > day
> >> > >> in single cluster without any consistency problem for more than 2
> >> month.
> >> > >> > I know that there has been lots of libraries and papers for HBase
> >> > >> transaction, but within my knowledge, this is the only open-source
> >> > library
> >> > >> which support serializability and linear scalability.
> >> > >> > There is no theoretical limit of transaction throughput.
> >> > >> > It was tested against cluster on AWS until 40,000
> transaction/sec.
> >> > >> (Still testing on bigger cluster)
> >> > >> > If you find it interesting, please leave me comment.
> >> > >> >
> >> > >> > Thanks,
> >> > >> > Andrew Kim
> >> > >>
> >> > >>
> >> > >
> >> >
> >>
> >>
> >>
> >> --
> >> *이 정 행 / Jung-Haeng Lee*
> >> *
> >> *
> >> Blog: http://eincs.net
> >> Facebook: http://www.facebook.com/eincs
> >> Twitter: http://twitter.com/eincs
> >> LinkedIn: http://linkedin.com/in/eincs
> >>
> >
> >
>
--
*이 정 행 / Jung-Haeng Lee*
Blog: http://eincs.net
Facebook: http://www.facebook.com/eincs
Twitter: http://twitter.com/eincs
LinkedIn: http://linkedin.com/in/eincs
Re: Haeinsa: linear scalable multi-row transaction library for HBase
Posted by Ted Yu <yu...@gmail.com>.
Jung-Haeng:
In HaeinsaTransaction.java :
/**
* If there is only one rowTx and type of its mutation is
HaeinsaPut.
*/
SINGLE_ROW_PUT_ONLY,
Can you elaborate a bit on why the single row put gets special treatment ?
Cheers
On Fri, Oct 11, 2013 at 6:09 PM, Ted Yu <yu...@gmail.com> wrote:
> I am now able to build on Linux.
>
> Running test suite.
>
> Will look deeper into the code over the weekend.
>
>
> On Thu, Oct 10, 2013 at 10:16 PM, Jung-Haeng Lee <ro...@gmail.com>wrote:
>
>> Thanks for your advice.
>> I fixed to Hbase 0.94.3 on pom.xml.
>> (To ensure consistency, HBASE-7051 should be solved.)
>>
>> I also fixed pom.xml of Haeinsa:
>> Now maven-thrift-plugin is activate on linux using default profile.
>> Please ensure the "thrift" executable is in your PATH.
>>
>> And you should build Haeinsa with jdk 1.7.
>> I think System.lineSeparator() is new method of Java7.
>>
>> - James Lee
>>
>>
>> On Fri, Oct 11, 2013 at 3:18 AM, Ted Yu <yu...@gmail.com> wrote:
>>
>> > Using the 'mac' profile, I got the following:
>> >
>> > Number of foreign imports: 1
>> > import: Entry[import from realm ClassRealm[maven.api, parent: null]]
>> >
>> > -----------------------------------------------------
>> >
>> > at
>> >
>> >
>> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
>> > ... 20 more
>> > Caused by: java.lang.NoSuchMethodError:
>> > java.lang.System.lineSeparator()Ljava/lang/String;
>> > at io.netty.build.checkstyle.NewlineCheck.<clinit>(NewlineCheck.java:43)
>> > at java.lang.Class.forName0(Native Method)
>> > at java.lang.Class.forName(Class.java:249)
>> > at
>> >
>> >
>> com.puppycrawl.tools.checkstyle.PackageObjectFactory.createObject(PackageObjectFactory.java:111)
>> > at
>> >
>> >
>> com.puppycrawl.tools.checkstyle.PackageObjectFactory.doMakeObject(PackageObjectFactory.java:80)
>> > at
>> >
>> >
>> com.puppycrawl.tools.checkstyle.PackageObjectFactory.createModule(PackageObjectFactory.java:147)
>> > at com.puppycrawl.tools.checkstyle.Checker.setupChild(Checker.java:153)
>> > at
>> >
>> >
>> com.puppycrawl.tools.checkstyle.api.AutomaticBean.configure(AutomaticBean.java:184)
>> > at
>> >
>> >
>> org.apache.maven.plugin.checkstyle.DefaultCheckstyleExecutor.executeCheckstyle(DefaultCheckstyleExecutor.java:172)
>> > at
>> >
>> >
>> org.apache.maven.plugin.checkstyle.CheckstyleViolationCheckMojo.execute(CheckstyleViolationCheckMojo.java:365)
>> > at
>> >
>> >
>> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
>> > ... 20 more
>> >
>> >
>> > On Thu, Oct 10, 2013 at 11:06 AM, Ted Yu <yu...@gmail.com> wrote:
>> >
>> > > Looks like HBase 0.92.1 is used. How about upgrading to 0.94.12 ?
>> > >
>> > > I cloned the git repo. But I got some compilation errors:
>> > >
>> > > [ERROR]
>> > >
>> >
>> /grid/0/dev/ty/haeinsa/src/main/java/kr/co/vcnc/haeinsa/HaeinsaTransaction.java:[324,20]
>> > > cannot find symbol
>> > > [ERROR] symbol: class TRowKey
>> > > [ERROR] location: class kr.co.vcnc.haeinsa.HaeinsaTransaction
>> > > [ERROR]
>> > >
>> >
>> /grid/0/dev/ty/haeinsa/src/main/java/kr/co/vcnc/haeinsa/HaeinsaTransaction.java:[325,13]
>> > > cannot find symbol
>> > > [ERROR] symbol: class TRowKey
>> > > [ERROR] location: class kr.co.vcnc.haeinsa.HaeinsaTransaction
>> > >
>> > > How do I activate maven-thrift-plugin on Linux ?
>> > >
>> > > Thanks
>> > >
>> > > On Thu, Oct 10, 2013 at 9:41 AM, Michael Segel <
>> > msegel_hadoop@hotmail.com>wrote:
>> > >
>> > >> Which level of isolation do you support?
>> > >>
>> > >>
>> > >> On Oct 10, 2013, at 12:55 AM, Myungbo Kim <mb...@vcnc.co.kr> wrote:
>> > >>
>> > >> > Hello everyone in the mailing list.
>> > >> >
>> > >> > I want to introduce an open-source library Haeinsa which supports
>> > >> multi-row, multi-table transaction on HBase.
>> > >> > Haeinsa is client-only library that support serializability and
>> linear
>> > >> scalability.
>> > >> > Here is the github repository and presentation that describes
>> > mechanism
>> > >> of it.
>> > >> >
>> > >> > github : https://github.com/VCNC/haeinsa
>> > >> > presentation : https://speakerdeck.com/vcnc/haeinsa-overview
>> > >> >
>> > >> > It was inspired by Google's percolator, but implementation detail
>> is
>> > >> different.
>> > >> > It use two-phase commit protocol and optimistic concurrency
>> control to
>> > >> implement, and Haeinsa now processes more than 300M+ transactions per
>> > day
>> > >> in single cluster without any consistency problem for more than 2
>> month.
>> > >> > I know that there has been lots of libraries and papers for HBase
>> > >> transaction, but within my knowledge, this is the only open-source
>> > library
>> > >> which support serializability and linear scalability.
>> > >> > There is no theoretical limit of transaction throughput.
>> > >> > It was tested against cluster on AWS until 40,000 transaction/sec.
>> > >> (Still testing on bigger cluster)
>> > >> > If you find it interesting, please leave me comment.
>> > >> >
>> > >> > Thanks,
>> > >> > Andrew Kim
>> > >>
>> > >>
>> > >
>> >
>>
>>
>>
>> --
>> *이 정 행 / Jung-Haeng Lee*
>> *
>> *
>> Blog: http://eincs.net
>> Facebook: http://www.facebook.com/eincs
>> Twitter: http://twitter.com/eincs
>> LinkedIn: http://linkedin.com/in/eincs
>>
>
>
Re: Haeinsa: linear scalable multi-row transaction library for HBase
Posted by Ted Yu <yu...@gmail.com>.
I am now able to build on Linux.
Running test suite.
Will look deeper into the code over the weekend.
On Thu, Oct 10, 2013 at 10:16 PM, Jung-Haeng Lee <ro...@gmail.com> wrote:
> Thanks for your advice.
> I fixed to Hbase 0.94.3 on pom.xml.
> (To ensure consistency, HBASE-7051 should be solved.)
>
> I also fixed pom.xml of Haeinsa:
> Now maven-thrift-plugin is activate on linux using default profile.
> Please ensure the "thrift" executable is in your PATH.
>
> And you should build Haeinsa with jdk 1.7.
> I think System.lineSeparator() is new method of Java7.
>
> - James Lee
>
>
> On Fri, Oct 11, 2013 at 3:18 AM, Ted Yu <yu...@gmail.com> wrote:
>
> > Using the 'mac' profile, I got the following:
> >
> > Number of foreign imports: 1
> > import: Entry[import from realm ClassRealm[maven.api, parent: null]]
> >
> > -----------------------------------------------------
> >
> > at
> >
> >
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
> > ... 20 more
> > Caused by: java.lang.NoSuchMethodError:
> > java.lang.System.lineSeparator()Ljava/lang/String;
> > at io.netty.build.checkstyle.NewlineCheck.<clinit>(NewlineCheck.java:43)
> > at java.lang.Class.forName0(Native Method)
> > at java.lang.Class.forName(Class.java:249)
> > at
> >
> >
> com.puppycrawl.tools.checkstyle.PackageObjectFactory.createObject(PackageObjectFactory.java:111)
> > at
> >
> >
> com.puppycrawl.tools.checkstyle.PackageObjectFactory.doMakeObject(PackageObjectFactory.java:80)
> > at
> >
> >
> com.puppycrawl.tools.checkstyle.PackageObjectFactory.createModule(PackageObjectFactory.java:147)
> > at com.puppycrawl.tools.checkstyle.Checker.setupChild(Checker.java:153)
> > at
> >
> >
> com.puppycrawl.tools.checkstyle.api.AutomaticBean.configure(AutomaticBean.java:184)
> > at
> >
> >
> org.apache.maven.plugin.checkstyle.DefaultCheckstyleExecutor.executeCheckstyle(DefaultCheckstyleExecutor.java:172)
> > at
> >
> >
> org.apache.maven.plugin.checkstyle.CheckstyleViolationCheckMojo.execute(CheckstyleViolationCheckMojo.java:365)
> > at
> >
> >
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
> > ... 20 more
> >
> >
> > On Thu, Oct 10, 2013 at 11:06 AM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > Looks like HBase 0.92.1 is used. How about upgrading to 0.94.12 ?
> > >
> > > I cloned the git repo. But I got some compilation errors:
> > >
> > > [ERROR]
> > >
> >
> /grid/0/dev/ty/haeinsa/src/main/java/kr/co/vcnc/haeinsa/HaeinsaTransaction.java:[324,20]
> > > cannot find symbol
> > > [ERROR] symbol: class TRowKey
> > > [ERROR] location: class kr.co.vcnc.haeinsa.HaeinsaTransaction
> > > [ERROR]
> > >
> >
> /grid/0/dev/ty/haeinsa/src/main/java/kr/co/vcnc/haeinsa/HaeinsaTransaction.java:[325,13]
> > > cannot find symbol
> > > [ERROR] symbol: class TRowKey
> > > [ERROR] location: class kr.co.vcnc.haeinsa.HaeinsaTransaction
> > >
> > > How do I activate maven-thrift-plugin on Linux ?
> > >
> > > Thanks
> > >
> > > On Thu, Oct 10, 2013 at 9:41 AM, Michael Segel <
> > msegel_hadoop@hotmail.com>wrote:
> > >
> > >> Which level of isolation do you support?
> > >>
> > >>
> > >> On Oct 10, 2013, at 12:55 AM, Myungbo Kim <mb...@vcnc.co.kr> wrote:
> > >>
> > >> > Hello everyone in the mailing list.
> > >> >
> > >> > I want to introduce an open-source library Haeinsa which supports
> > >> multi-row, multi-table transaction on HBase.
> > >> > Haeinsa is client-only library that support serializability and
> linear
> > >> scalability.
> > >> > Here is the github repository and presentation that describes
> > mechanism
> > >> of it.
> > >> >
> > >> > github : https://github.com/VCNC/haeinsa
> > >> > presentation : https://speakerdeck.com/vcnc/haeinsa-overview
> > >> >
> > >> > It was inspired by Google's percolator, but implementation detail is
> > >> different.
> > >> > It use two-phase commit protocol and optimistic concurrency control
> to
> > >> implement, and Haeinsa now processes more than 300M+ transactions per
> > day
> > >> in single cluster without any consistency problem for more than 2
> month.
> > >> > I know that there has been lots of libraries and papers for HBase
> > >> transaction, but within my knowledge, this is the only open-source
> > library
> > >> which support serializability and linear scalability.
> > >> > There is no theoretical limit of transaction throughput.
> > >> > It was tested against cluster on AWS until 40,000 transaction/sec.
> > >> (Still testing on bigger cluster)
> > >> > If you find it interesting, please leave me comment.
> > >> >
> > >> > Thanks,
> > >> > Andrew Kim
> > >>
> > >>
> > >
> >
>
>
>
> --
> *이 정 행 / Jung-Haeng Lee*
> *
> *
> Blog: http://eincs.net
> Facebook: http://www.facebook.com/eincs
> Twitter: http://twitter.com/eincs
> LinkedIn: http://linkedin.com/in/eincs
>
Re: Haeinsa: linear scalable multi-row transaction library for HBase
Posted by Jung-Haeng Lee <ro...@gmail.com>.
Thanks for your advice.
I fixed to Hbase 0.94.3 on pom.xml.
(To ensure consistency, HBASE-7051 should be solved.)
I also fixed pom.xml of Haeinsa:
Now maven-thrift-plugin is activate on linux using default profile.
Please ensure the "thrift" executable is in your PATH.
And you should build Haeinsa with jdk 1.7.
I think System.lineSeparator() is new method of Java7.
- James Lee
On Fri, Oct 11, 2013 at 3:18 AM, Ted Yu <yu...@gmail.com> wrote:
> Using the 'mac' profile, I got the following:
>
> Number of foreign imports: 1
> import: Entry[import from realm ClassRealm[maven.api, parent: null]]
>
> -----------------------------------------------------
>
> at
>
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
> ... 20 more
> Caused by: java.lang.NoSuchMethodError:
> java.lang.System.lineSeparator()Ljava/lang/String;
> at io.netty.build.checkstyle.NewlineCheck.<clinit>(NewlineCheck.java:43)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:249)
> at
>
> com.puppycrawl.tools.checkstyle.PackageObjectFactory.createObject(PackageObjectFactory.java:111)
> at
>
> com.puppycrawl.tools.checkstyle.PackageObjectFactory.doMakeObject(PackageObjectFactory.java:80)
> at
>
> com.puppycrawl.tools.checkstyle.PackageObjectFactory.createModule(PackageObjectFactory.java:147)
> at com.puppycrawl.tools.checkstyle.Checker.setupChild(Checker.java:153)
> at
>
> com.puppycrawl.tools.checkstyle.api.AutomaticBean.configure(AutomaticBean.java:184)
> at
>
> org.apache.maven.plugin.checkstyle.DefaultCheckstyleExecutor.executeCheckstyle(DefaultCheckstyleExecutor.java:172)
> at
>
> org.apache.maven.plugin.checkstyle.CheckstyleViolationCheckMojo.execute(CheckstyleViolationCheckMojo.java:365)
> at
>
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
> ... 20 more
>
>
> On Thu, Oct 10, 2013 at 11:06 AM, Ted Yu <yu...@gmail.com> wrote:
>
> > Looks like HBase 0.92.1 is used. How about upgrading to 0.94.12 ?
> >
> > I cloned the git repo. But I got some compilation errors:
> >
> > [ERROR]
> >
> /grid/0/dev/ty/haeinsa/src/main/java/kr/co/vcnc/haeinsa/HaeinsaTransaction.java:[324,20]
> > cannot find symbol
> > [ERROR] symbol: class TRowKey
> > [ERROR] location: class kr.co.vcnc.haeinsa.HaeinsaTransaction
> > [ERROR]
> >
> /grid/0/dev/ty/haeinsa/src/main/java/kr/co/vcnc/haeinsa/HaeinsaTransaction.java:[325,13]
> > cannot find symbol
> > [ERROR] symbol: class TRowKey
> > [ERROR] location: class kr.co.vcnc.haeinsa.HaeinsaTransaction
> >
> > How do I activate maven-thrift-plugin on Linux ?
> >
> > Thanks
> >
> > On Thu, Oct 10, 2013 at 9:41 AM, Michael Segel <
> msegel_hadoop@hotmail.com>wrote:
> >
> >> Which level of isolation do you support?
> >>
> >>
> >> On Oct 10, 2013, at 12:55 AM, Myungbo Kim <mb...@vcnc.co.kr> wrote:
> >>
> >> > Hello everyone in the mailing list.
> >> >
> >> > I want to introduce an open-source library Haeinsa which supports
> >> multi-row, multi-table transaction on HBase.
> >> > Haeinsa is client-only library that support serializability and linear
> >> scalability.
> >> > Here is the github repository and presentation that describes
> mechanism
> >> of it.
> >> >
> >> > github : https://github.com/VCNC/haeinsa
> >> > presentation : https://speakerdeck.com/vcnc/haeinsa-overview
> >> >
> >> > It was inspired by Google's percolator, but implementation detail is
> >> different.
> >> > It use two-phase commit protocol and optimistic concurrency control to
> >> implement, and Haeinsa now processes more than 300M+ transactions per
> day
> >> in single cluster without any consistency problem for more than 2 month.
> >> > I know that there has been lots of libraries and papers for HBase
> >> transaction, but within my knowledge, this is the only open-source
> library
> >> which support serializability and linear scalability.
> >> > There is no theoretical limit of transaction throughput.
> >> > It was tested against cluster on AWS until 40,000 transaction/sec.
> >> (Still testing on bigger cluster)
> >> > If you find it interesting, please leave me comment.
> >> >
> >> > Thanks,
> >> > Andrew Kim
> >>
> >>
> >
>
--
*이 정 행 / Jung-Haeng Lee*
*
*
Blog: http://eincs.net
Facebook: http://www.facebook.com/eincs
Twitter: http://twitter.com/eincs
LinkedIn: http://linkedin.com/in/eincs
Re: Haeinsa: linear scalable multi-row transaction library for HBase
Posted by Ted Yu <yu...@gmail.com>.
Using the 'mac' profile, I got the following:
Number of foreign imports: 1
import: Entry[import from realm ClassRealm[maven.api, parent: null]]
-----------------------------------------------------
at
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
... 20 more
Caused by: java.lang.NoSuchMethodError:
java.lang.System.lineSeparator()Ljava/lang/String;
at io.netty.build.checkstyle.NewlineCheck.<clinit>(NewlineCheck.java:43)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:249)
at
com.puppycrawl.tools.checkstyle.PackageObjectFactory.createObject(PackageObjectFactory.java:111)
at
com.puppycrawl.tools.checkstyle.PackageObjectFactory.doMakeObject(PackageObjectFactory.java:80)
at
com.puppycrawl.tools.checkstyle.PackageObjectFactory.createModule(PackageObjectFactory.java:147)
at com.puppycrawl.tools.checkstyle.Checker.setupChild(Checker.java:153)
at
com.puppycrawl.tools.checkstyle.api.AutomaticBean.configure(AutomaticBean.java:184)
at
org.apache.maven.plugin.checkstyle.DefaultCheckstyleExecutor.executeCheckstyle(DefaultCheckstyleExecutor.java:172)
at
org.apache.maven.plugin.checkstyle.CheckstyleViolationCheckMojo.execute(CheckstyleViolationCheckMojo.java:365)
at
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
... 20 more
On Thu, Oct 10, 2013 at 11:06 AM, Ted Yu <yu...@gmail.com> wrote:
> Looks like HBase 0.92.1 is used. How about upgrading to 0.94.12 ?
>
> I cloned the git repo. But I got some compilation errors:
>
> [ERROR]
> /grid/0/dev/ty/haeinsa/src/main/java/kr/co/vcnc/haeinsa/HaeinsaTransaction.java:[324,20]
> cannot find symbol
> [ERROR] symbol: class TRowKey
> [ERROR] location: class kr.co.vcnc.haeinsa.HaeinsaTransaction
> [ERROR]
> /grid/0/dev/ty/haeinsa/src/main/java/kr/co/vcnc/haeinsa/HaeinsaTransaction.java:[325,13]
> cannot find symbol
> [ERROR] symbol: class TRowKey
> [ERROR] location: class kr.co.vcnc.haeinsa.HaeinsaTransaction
>
> How do I activate maven-thrift-plugin on Linux ?
>
> Thanks
>
> On Thu, Oct 10, 2013 at 9:41 AM, Michael Segel <ms...@hotmail.com>wrote:
>
>> Which level of isolation do you support?
>>
>>
>> On Oct 10, 2013, at 12:55 AM, Myungbo Kim <mb...@vcnc.co.kr> wrote:
>>
>> > Hello everyone in the mailing list.
>> >
>> > I want to introduce an open-source library Haeinsa which supports
>> multi-row, multi-table transaction on HBase.
>> > Haeinsa is client-only library that support serializability and linear
>> scalability.
>> > Here is the github repository and presentation that describes mechanism
>> of it.
>> >
>> > github : https://github.com/VCNC/haeinsa
>> > presentation : https://speakerdeck.com/vcnc/haeinsa-overview
>> >
>> > It was inspired by Google's percolator, but implementation detail is
>> different.
>> > It use two-phase commit protocol and optimistic concurrency control to
>> implement, and Haeinsa now processes more than 300M+ transactions per day
>> in single cluster without any consistency problem for more than 2 month.
>> > I know that there has been lots of libraries and papers for HBase
>> transaction, but within my knowledge, this is the only open-source library
>> which support serializability and linear scalability.
>> > There is no theoretical limit of transaction throughput.
>> > It was tested against cluster on AWS until 40,000 transaction/sec.
>> (Still testing on bigger cluster)
>> > If you find it interesting, please leave me comment.
>> >
>> > Thanks,
>> > Andrew Kim
>>
>>
>
Re: Haeinsa: linear scalable multi-row transaction library for HBase
Posted by Ted Yu <yu...@gmail.com>.
Looks like HBase 0.92.1 is used. How about upgrading to 0.94.12 ?
I cloned the git repo. But I got some compilation errors:
[ERROR]
/grid/0/dev/ty/haeinsa/src/main/java/kr/co/vcnc/haeinsa/HaeinsaTransaction.java:[324,20]
cannot find symbol
[ERROR] symbol: class TRowKey
[ERROR] location: class kr.co.vcnc.haeinsa.HaeinsaTransaction
[ERROR]
/grid/0/dev/ty/haeinsa/src/main/java/kr/co/vcnc/haeinsa/HaeinsaTransaction.java:[325,13]
cannot find symbol
[ERROR] symbol: class TRowKey
[ERROR] location: class kr.co.vcnc.haeinsa.HaeinsaTransaction
How do I activate maven-thrift-plugin on Linux ?
Thanks
On Thu, Oct 10, 2013 at 9:41 AM, Michael Segel <ms...@hotmail.com>wrote:
> Which level of isolation do you support?
>
> On Oct 10, 2013, at 12:55 AM, Myungbo Kim <mb...@vcnc.co.kr> wrote:
>
> > Hello everyone in the mailing list.
> >
> > I want to introduce an open-source library Haeinsa which supports
> multi-row, multi-table transaction on HBase.
> > Haeinsa is client-only library that support serializability and linear
> scalability.
> > Here is the github repository and presentation that describes mechanism
> of it.
> >
> > github : https://github.com/VCNC/haeinsa
> > presentation : https://speakerdeck.com/vcnc/haeinsa-overview
> >
> > It was inspired by Google's percolator, but implementation detail is
> different.
> > It use two-phase commit protocol and optimistic concurrency control to
> implement, and Haeinsa now processes more than 300M+ transactions per day
> in single cluster without any consistency problem for more than 2 month.
> > I know that there has been lots of libraries and papers for HBase
> transaction, but within my knowledge, this is the only open-source library
> which support serializability and linear scalability.
> > There is no theoretical limit of transaction throughput.
> > It was tested against cluster on AWS until 40,000 transaction/sec.
> (Still testing on bigger cluster)
> > If you find it interesting, please leave me comment.
> >
> > Thanks,
> > Andrew Kim
>
>
Re: Haeinsa: linear scalable multi-row transaction library for HBase
Posted by James Lee <ja...@vcnc.co.kr>.
I'm one of the developer who developed Haeinsa.
Isolation level of Haeinsa is Serializable.
Thanks for having interest in Haeinsa
- James Lee
On Fri, Oct 11, 2013 at 1:41 AM, Michael Segel <ms...@hotmail.com>wrote:
> Which level of isolation do you support?
>
> On Oct 10, 2013, at 12:55 AM, Myungbo Kim <mb...@vcnc.co.kr> wrote:
>
> > Hello everyone in the mailing list.
> >
> > I want to introduce an open-source library Haeinsa which supports
> multi-row, multi-table transaction on HBase.
> > Haeinsa is client-only library that support serializability and linear
> scalability.
> > Here is the github repository and presentation that describes mechanism
> of it.
> >
> > github : https://github.com/VCNC/haeinsa
> > presentation : https://speakerdeck.com/vcnc/haeinsa-overview
> >
> > It was inspired by Google's percolator, but implementation detail is
> different.
> > It use two-phase commit protocol and optimistic concurrency control to
> implement, and Haeinsa now processes more than 300M+ transactions per day
> in single cluster without any consistency problem for more than 2 month.
> > I know that there has been lots of libraries and papers for HBase
> transaction, but within my knowledge, this is the only open-source library
> which support serializability and linear scalability.
> > There is no theoretical limit of transaction throughput.
> > It was tested against cluster on AWS until 40,000 transaction/sec.
> (Still testing on bigger cluster)
> > If you find it interesting, please leave me comment.
> >
> > Thanks,
> > Andrew Kim
>
>
--
*이 정 행 / Jung-Haeng Lee*
Value Developer, VCNC
"연인들을 위한 둘 만의 온라인 비밀 공간, Between"
http://appbetween.us
Mobile: 010-5054-5321
E-mail: jhlee@vcnc.co.kr
Twitter: @eincs <http://twitter.com/eincs>
Facebook: http://www.facebook.com/eincs
Homepage: http://www.vcnc.co.kr
서울시 강남구 역삼2동 719-35 IS빌딩 5층 VCNC
IS Building 5th Fl., 719-35, Yeoksam 2-dong, Gangnam-gu, Seoul, Korea
Re: Haeinsa: linear scalable multi-row transaction library for HBase
Posted by Michael Segel <ms...@hotmail.com>.
Which level of isolation do you support?
On Oct 10, 2013, at 12:55 AM, Myungbo Kim <mb...@vcnc.co.kr> wrote:
> Hello everyone in the mailing list.
>
> I want to introduce an open-source library Haeinsa which supports multi-row, multi-table transaction on HBase.
> Haeinsa is client-only library that support serializability and linear scalability.
> Here is the github repository and presentation that describes mechanism of it.
>
> github : https://github.com/VCNC/haeinsa
> presentation : https://speakerdeck.com/vcnc/haeinsa-overview
>
> It was inspired by Google's percolator, but implementation detail is different.
> It use two-phase commit protocol and optimistic concurrency control to implement, and Haeinsa now processes more than 300M+ transactions per day in single cluster without any consistency problem for more than 2 month.
> I know that there has been lots of libraries and papers for HBase transaction, but within my knowledge, this is the only open-source library which support serializability and linear scalability.
> There is no theoretical limit of transaction throughput.
> It was tested against cluster on AWS until 40,000 transaction/sec. (Still testing on bigger cluster)
> If you find it interesting, please leave me comment.
>
> Thanks,
> Andrew Kim
Re: Haeinsa: linear scalable multi-row transaction library for HBase
Posted by lars hofhansl <la...@apache.org>.
Thanks Andrew.
This sounds very interesting!
Will definitely have a look.
-- Lars
________________________________
From: Myungbo Kim <mb...@vcnc.co.kr>
To: user@hbase.apache.org
Cc: Jung-Haeng Lee <ja...@vcnc.co.kr>; YoungMok Kim <br...@vcnc.co.kr>
Sent: Wednesday, October 9, 2013 10:55 PM
Subject: Haeinsa: linear scalable multi-row transaction library for HBase
Hello everyone in the mailing list.
I want to introduce an open-source library Haeinsa which supports multi-row, multi-table transaction on HBase.
Haeinsa is client-only library that support serializability and linear scalability.
Here is the github repository and presentation that describes mechanism of it.
github : https://github.com/VCNC/haeinsa
presentation : https://speakerdeck.com/vcnc/haeinsa-overview
It was inspired by Google's percolator, but implementation detail is different.
It use two-phase commit protocol and optimistic concurrency control to implement, and Haeinsa now processes more than 300M+ transactions per day in single cluster without any consistency problem for more than 2 month.
I know that there has been lots of libraries and papers for HBase transaction, but within my knowledge, this is the only open-source library which support serializability and linear scalability.
There is no theoretical limit of transaction throughput.
It was tested against cluster on AWS until 40,000 transaction/sec. (Still testing on bigger cluster)
If you find it interesting, please leave me comment.
Thanks,
Andrew Kim