You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by fding hbase <fd...@gmail.com> on 2012/07/06 03:50:24 UTC
secondary index using coprocessors
Hi all,
I've implemented an indexing system. It needs refinement of course
but I have my own work to do, so I decided to open it for everyone to
access, modify and any improvement or ideas would be great.
https://github.com/danix800/hbase-indexed
--
Best Regards!
Fei Ding
fding.church@gmail.com
Re: secondary index using coprocessors
Posted by Sever Fundatureanu <fu...@gmail.com>.
Hi,
I also looked at your implementation. I have some issues with it but maybe
someone can correct me if I'm wrong.
What you're doing is writing the secondary index from the memstore in the
preFlush hook. However there is no mutual exclusive lock (write lock) that
is taken on the whole memstore for both the preFlush hook and the internal
flush operation. The only time a write lock is taken on the whole memstore
is when taking a snapshot and that comes after the preFlush hook.
So after completion of the preFlush hook, a couple more elements can be
added to the memstore until it is actually flushed to disk. This means you
lose consistency between your primary and your secondary index.
Can this issue be solved somehow? I am also interested in building a
secondary index from the preflush hook.
Thanks,
Sever
On Fri, Jul 6, 2012 at 6:04 AM, fding hbase <fd...@gmail.com> wrote:
> Sorry, I don't have much experience on unit test. The performance may not
> be good
> since this is my leisure time work, nor do I have good understanding for
> consistency.
> I really wanna help but for now it's maybe beyond my ability. I'm still
> digging the code
> of hbase too... :-)
>
> HBase is quite HUGE to me...
>
> On Fri, Jul 6, 2012 at 10:32 AM, Ted Yu <yu...@gmail.com> wrote:
>
> > The following action allows me to compile your code:
> >
> > 551 cp ~/hbase-indexed/src/org/apache/hadoop/hbase/regionserver/*.java
> > src/main/java/org/apache/hadoop/hbase/regionserver/
> > 552 mkdir src/main/java/org/apache/hadoop/hbase/regionserver/indexed
> > 553 cp
> > ~/hbase-indexed/src/org/apache/hadoop/hbase/regionserver/indexed/*
> > src/main/java/org/apache/hadoop/hbase/regionserver/indexed/
> >
> > There is no unit test. Can you add some ?
> >
> > Do you have performance data: overhead for writing index, speedup in
> query
> > ?
> >
> > The most recent motion on this subject is HBASE-5723.
> > I still need to dig into your code. If you can comment on the level of
> > consistency guarantees for your design, that would be nice.
> >
> > Cheers
> >
> > On Thu, Jul 5, 2012 at 6:50 PM, fding hbase <fd...@gmail.com>
> wrote:
> >
> > > Hi all,
> > >
> > > I've implemented an indexing system. It needs refinement of course
> > > but I have my own work to do, so I decided to open it for everyone to
> > > access, modify and any improvement or ideas would be great.
> > >
> > > https://github.com/danix800/hbase-indexed
> > >
> > > --
> > >
> > > Best Regards!
> > >
> > > Fei Ding
> > > fding.church@gmail.com
> > >
> >
>
>
>
> --
>
> Best Regards!
>
> Fei Ding
> fding.church@gmail.com
>
--
Sever Fundatureanu
Vrije Universiteit Amsterdam
E-mail: fundatureanu.sever@gmail.com
Re: secondary index using coprocessors
Posted by fding hbase <fd...@gmail.com>.
Sorry, I don't have much experience on unit test. The performance may not
be good
since this is my leisure time work, nor do I have good understanding for
consistency.
I really wanna help but for now it's maybe beyond my ability. I'm still
digging the code
of hbase too... :-)
HBase is quite HUGE to me...
On Fri, Jul 6, 2012 at 10:32 AM, Ted Yu <yu...@gmail.com> wrote:
> The following action allows me to compile your code:
>
> 551 cp ~/hbase-indexed/src/org/apache/hadoop/hbase/regionserver/*.java
> src/main/java/org/apache/hadoop/hbase/regionserver/
> 552 mkdir src/main/java/org/apache/hadoop/hbase/regionserver/indexed
> 553 cp
> ~/hbase-indexed/src/org/apache/hadoop/hbase/regionserver/indexed/*
> src/main/java/org/apache/hadoop/hbase/regionserver/indexed/
>
> There is no unit test. Can you add some ?
>
> Do you have performance data: overhead for writing index, speedup in query
> ?
>
> The most recent motion on this subject is HBASE-5723.
> I still need to dig into your code. If you can comment on the level of
> consistency guarantees for your design, that would be nice.
>
> Cheers
>
> On Thu, Jul 5, 2012 at 6:50 PM, fding hbase <fd...@gmail.com> wrote:
>
> > Hi all,
> >
> > I've implemented an indexing system. It needs refinement of course
> > but I have my own work to do, so I decided to open it for everyone to
> > access, modify and any improvement or ideas would be great.
> >
> > https://github.com/danix800/hbase-indexed
> >
> > --
> >
> > Best Regards!
> >
> > Fei Ding
> > fding.church@gmail.com
> >
>
--
Best Regards!
Fei Ding
fding.church@gmail.com
Re: secondary index using coprocessors
Posted by Ted Yu <yu...@gmail.com>.
The following action allows me to compile your code:
551 cp ~/hbase-indexed/src/org/apache/hadoop/hbase/regionserver/*.java
src/main/java/org/apache/hadoop/hbase/regionserver/
552 mkdir src/main/java/org/apache/hadoop/hbase/regionserver/indexed
553 cp
~/hbase-indexed/src/org/apache/hadoop/hbase/regionserver/indexed/*
src/main/java/org/apache/hadoop/hbase/regionserver/indexed/
There is no unit test. Can you add some ?
Do you have performance data: overhead for writing index, speedup in query ?
The most recent motion on this subject is HBASE-5723.
I still need to dig into your code. If you can comment on the level of
consistency guarantees for your design, that would be nice.
Cheers
On Thu, Jul 5, 2012 at 6:50 PM, fding hbase <fd...@gmail.com> wrote:
> Hi all,
>
> I've implemented an indexing system. It needs refinement of course
> but I have my own work to do, so I decided to open it for everyone to
> access, modify and any improvement or ideas would be great.
>
> https://github.com/danix800/hbase-indexed
>
> --
>
> Best Regards!
>
> Fei Ding
> fding.church@gmail.com
>