Handling consistency issues with HBASE partition (was Re: HBase partition integration in trunks ?)

[Selcuk AYA <ay...@gmail.com>]

Hi Stefan,
Working on JDBM, I was wondering what kind of cosistency gurantees
does HBASE partition implemenation expose? In particular, do put and
fetch operations on the underlying master table and indices have ACID
properties? How about cursors? Do cursors see future changes as they
move back and forth?

regards,
Selcuk

On Tue, Aug 16, 2011 at 4:49 PM, Kiran Ayyagari <ka...@apache.org> wrote:
> On Tue, Aug 16, 2011 at 7:06 PM, Emmanuel Lecharny <el...@gmail.com> wrote:
>> On 8/16/11 3:22 PM, Kiran Ayyagari wrote:
>>>
>>> On Tue, Aug 16, 2011 at 6:45 PM, Stefan Seelmann<se...@apache.org>
>>>  wrote:
>>>>
>>>> On Tue, Aug 16, 2011 at 2:22 PM, Kiran Ayyagari<ka...@apache.org>
>>>>  wrote:
>>>>>>
>>>>>> One solution would be to store two more elements in the ParentIdAndRdn
>>>>>> data
>>>>>> structure : the number of children directly below the RDN, and the
>>>>>> number of
>>>>>> children and descendant. That would probably solve the issue I'm
>>>>>> mentioning.
>>>>>> Of course, that also means we wil have to update all the RDN hierarchy
>>>>>> from
>>>>>> top to bottom (but affecting only the RDN part of the entry DN) each
>>>>>> time we
>>>>>> add/move/delete an entry. Note that we already do that for the oneLevel
>>>>>> and
>>>>>> Sublevel index.
>>>>>>
>>>>> Just to make a point:
>>>>> I think, in the case of achieving SubLevel index evaluation with RDN
>>>>> index it becomes a costly and complex operation
>>>>> (recursive scanning and updating) where as with the current sublevel
>>>>> index it takes O(1) to fetch all the sublevel children of
>>>>> an entry.
>>>>
>>>> Hm, evalutation can easly be done by using the reverse RDN index table.
>>>>
>>> for one level it is straight forward, but for sublevel we still need
>>> to use recursion, no?
>>
>> No, if we update all the parents when we add/move/delete an entry. That
>> means we have to update more than one RdnAndparent element (in fact, as many
>> as we have RDns but the namingContext in the DN). Nothing more though that
>> what we already do in the sub-level context.
>>
>> Then we have a O(1) operation to get the number of children in the case of a
>> SUB search.
>>
> if we are only talking about 'number of children' then this holds
> good, but not when it comes to getting the
> sublevel child entry ids, in case of existing sublevel index we get
> *all* the child ids in one lookup
>> Now, Stefan is right, fetching entries means we use more than one cursor for
>> that.
>>
> yeap, the above step is where this many cursors will be opened to
> collect the entry ids
>>
>> --
>> Regards,
>> Cordialement,
>> Emmanuel Lécharny
>> www.iktek.com
>>
>>
>
>
>
> --
> Kiran Ayyagari
>

Re: Handling consistency issues with HBASE partition (was Re: HBase partition integration in trunks ?)

[Stefan Seelmann <se...@apache.org>]

Hi Selcuk,

On Mon, Aug 29, 2011 at 4:52 PM, Selcuk AYA <ay...@gmail.com> wrote:
> Working on JDBM, I was wondering what kind of cosistency gurantees
> does HBASE partition implemenation expose? In particular, do put and
> fetch operations on the underlying master table and indices have ACID
> properties? How about cursors? Do cursors see future changes as they
> move back and forth?

Thanks for asking, [1] gives a nice overview.

Write operations:
In HBase a single "put" to a single row of a table is ACID, even if
multiple columns are modified. However on an update within ApacheDS we
have to write to multiple tables and even multiple rows within a
table. Such a modification is not ACID. HBase doesn't provide
transactions.

Read operations:
There is no isolation in HBase so cursors (called scans in HBase) see
future changes.

One idea we had some time back was to integrate MVCC into XDBM so that
transactions and isolation is independent from the underlying store.

Another thing is that scans in HBase only work in forward direction,
not backward. That wasn't a real a problem till now because most times
in ApacheDS only the forward direction is used. There are just rare
cased where a Curosor.previous() is used, for that case I keep a small
amount (10) index entries in memory so the previous() can be handled.
When we add more features that require backward traversal (e.g. VLV)
there is a workaround: in that case we need to create a second index
table with an inverse key.

Kind Regards,
Stefan


[1| http://hbase.apache.org/acid-semantics.html