Record index within a Table

[mi...@playstation.sony.com]

Does Accumulo have the idea of record index within a table?  As an 
example, take a table with the following values:

Row  ColFam  ColQual
1        1            10
1        1            20
1        1            30


As I understand it Accumulo will naturally sort the table in the above 
order (first by row, then by colFam, lastly by colQual).  If I insert 
(1, 1, 25), is there any way for me to get the index of the newly 
inserted value (3 in this case)?

Further more is there anyway to lookup an index by knowing the full 
row:colFam:colQual key (say (1, 1, 30))?

Lastly, is there an easy way to get the total number of rows within a 
table (and the same question for ColFam's within a row and ColQual's 
with a row:colFam pair)?

I've made it through the documentation and scanned through the mailing 
list, but I haven't seen any information on the above (which lends me to 
believe I'm asking for behaviour that Accumulo doesn't natively provide).

Many thanks for any information,

- MT

Re: Record index within a Table

[Christopher <ct...@apache.org>]

It's worse than just merging the tablet sources and iterating to find the
offset... because the underlying sources may contain deleted records, old
versions that are filtered by an iterator, duplicates, and it is further
complicated if you are using combiners in the iterator stack.

Your best bet is probably to perform this sort of indexing within an ingest
framework that understands a bigger picture of how you will use the data
you are ingesting.



--
Christopher L Tubbs II
http://gravatar.com/ctubbsii


On Tue, Dec 4, 2012 at 12:45 PM, Josh Elser <jo...@gmail.com> wrote:

> I was thinking a little more on the subject, and convinced myself that I
> was wrong.
>
> Since many files on disk correspond to a tablet, the best you can get is
> the index of a key-value pair in a given file for a tablet. To get a sorted
> stream of key-value pairs for this tablet (to compute index offset for a
> key in a tablet), a merged read is performed over all of those files. Local
> key offset for a file is meaningless as it does not imply the correct
> offset for a tablet.
>
>
> On 12/3/12 9:30 PM, Josh Elser wrote:
>
>>
>> Accumulo doesn't expose any internal offsets of Key-Value pairs through
>> the API. While it might be able to extrapolate some of this knowledge from
>> the underlying structure of Accumulo, that isn't the intent of what
>> Accumulo is trying to provide.
>>
>>
>

Re: Record index within a Table

[Josh Elser <jo...@gmail.com>]

I was thinking a little more on the subject, and convinced myself that I 
was wrong.

Since many files on disk correspond to a tablet, the best you can get is 
the index of a key-value pair in a given file for a tablet. To get a 
sorted stream of key-value pairs for this tablet (to compute index 
offset for a key in a tablet), a merged read is performed over all of 
those files. Local key offset for a file is meaningless as it does not 
imply the correct offset for a tablet.

On 12/3/12 9:30 PM, Josh Elser wrote:
>
> Accumulo doesn't expose any internal offsets of Key-Value pairs 
> through the API. While it might be able to extrapolate some of this 
> knowledge from the underlying structure of Accumulo, that isn't the 
> intent of what Accumulo is trying to provide.
>

Re: Record index within a Table

[Josh Elser <jo...@gmail.com>]

Michael,

Accumulo doesn't expose any internal offsets of Key-Value pairs through 
the API. While it might be able to extrapolate some of this knowledge 
from the underlying structure of Accumulo, that isn't the intent of what 
Accumulo is trying to provide.

The first idea that comes to mind is that you could write an 
SortedKeyValueIterator which takes some array of indices you wish to 
find in the ColQual ([3], in your case) and perform the heavy-lifting on 
the server. Your SKVI can scan through the data, consuming records which 
do not match your index ([3]). Since you know that Accumulo will not 
split your row "1" across multiple server (a tablet is never split 
across a row), you should be able to fairly quickly get a stream back of 
all key for the relative indices you provided. You can think of this as 
a sequential read over the disk which should be relatively efficient.

The only caveat is that you would need to create a range to scan over 
your entire row+colfam, for example:

Range=(1 1: , 1 1\x00: )

Where the colqual is empty.

- Josh

On 12/3/2012 9:04 PM, michael_taylor@playstation.sony.com wrote:
> Does Accumulo have the idea of record index within a table?  As an
> example, take a table with the following values:
>
> Row  ColFam  ColQual
> 1        1            10
> 1        1            20
> 1        1            30
>
>
> As I understand it Accumulo will naturally sort the table in the above
> order (first by row, then by colFam, lastly by colQual).  If I insert
> (1, 1, 25), is there any way for me to get the index of the newly
> inserted value (3 in this case)?
>
> Further more is there anyway to lookup an index by knowing the full
> row:colFam:colQual key (say (1, 1, 30))?
>
> Lastly, is there an easy way to get the total number of rows within a
> table (and the same question for ColFam's within a row and ColQual's
> with a row:colFam pair)?
>
> I've made it through the documentation and scanned through the mailing
> list, but I haven't seen any information on the above (which lends me to
> believe I'm asking for behaviour that Accumulo doesn't natively provide).
>
> Many thanks for any information,
>
> - MT