You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@phoenix.apache.org by Sean Busbey <bu...@cloudera.com> on 2014/09/12 18:52:14 UTC
Phoenix use of HBase WALs
Hey Folks!
Can someone give me a few pointers on what Phoenix needs from the HBase WAL?
Looks like the docs are all user-facing, so they skip things like how
Phoenix's parts fit into HBase. I'll start looking at code, but I'd rather
hear directly from y'all than just make my own guesses.
A pointer to a design doc or some JIRAs would be great.
Background: I'm in the process of refactoring the HBase wal internally
(HBASE-10378). I've run into two major issues.
1) I'd like to make some changes to our Coprocessor limited APIs around
WALs. I want to figure out what parts of this I can alter without overly
burdening Phoenix. Even if the changes can't land in HBase 0.98.x I'd like
to get an idea in advance of how they'd impact Phoenix moving to an
eventual HBase 1.0+
2) I've already found some changes I'm planning for
InterfaceAudience.Private classes within HBase that will break at least
current Phoenix tests. This second point makes me very nervous. I'd prefer
not to re-scope all of HBase's WAL stuff to be non-Private.
--
Sean
Re: Phoenix use of HBase WALs
Posted by Jesse Yates <je...@gmail.com>.
Hey Sean,
Take a look at
https://github.com/apache/phoenix/tree/master/phoenix-core/src/main/java/org/apache/hadoop/hbase/regionserver/wal
Basically, we use the WAL for storing the mutable index updates. They are
carried along with the primary WAL update (as added key values) and then
restored independently on WAL replay (filtered at replay and written to the
correct table by a phoenix writer).
As long as we can do that, should be ok. I was hoping the phoenix codec
should be enough, but it gets a little complicated as we do need to
manipulate the WAL at Region#batch time too.
-------------------
Jesse Yates
@jesse_yates
jyates.github.com
On Fri, Sep 12, 2014 at 9:52 AM, Sean Busbey <bu...@cloudera.com> wrote:
> Hey Folks!
>
> Can someone give me a few pointers on what Phoenix needs from the HBase
> WAL?
>
> Looks like the docs are all user-facing, so they skip things like how
> Phoenix's parts fit into HBase. I'll start looking at code, but I'd rather
> hear directly from y'all than just make my own guesses.
>
> A pointer to a design doc or some JIRAs would be great.
>
> Background: I'm in the process of refactoring the HBase wal internally
> (HBASE-10378). I've run into two major issues.
>
> 1) I'd like to make some changes to our Coprocessor limited APIs around
> WALs. I want to figure out what parts of this I can alter without overly
> burdening Phoenix. Even if the changes can't land in HBase 0.98.x I'd like
> to get an idea in advance of how they'd impact Phoenix moving to an
> eventual HBase 1.0+
>
> 2) I've already found some changes I'm planning for
> InterfaceAudience.Private classes within HBase that will break at least
> current Phoenix tests. This second point makes me very nervous. I'd prefer
> not to re-scope all of HBase's WAL stuff to be non-Private.
>
> --
> Sean
>