You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Mark <st...@gmail.com> on 2011/08/20 03:39:24 UTC
Reverse timestamp
I understand the need but I don't understand how to generate a reverse
timestamp. Can someone please explain how this is accomplished and how I
can test that its working correctly?
Thanks
RE: Reverse timestamp
Posted by "steve.boyle@connexity.com" <st...@connexity.com>.
Well, Long.MAX_VALUE is "A constant holding the maximum value a long can have, (2^63)-1". I don't know much about ruby but I'm guessing that you could use ruby's bignum to hold that value or some value sufficiently large for your timestamp value.
-Steve
-----Original Message-----
From: Mark [mailto:static.void.dev@gmail.com]
Sent: Saturday, August 20, 2011 7:57 AM
To: user@hbase.apache.org
Subject: Re: Reverse timestamp
Thanks all but I'm actually using ruby as my client so there is no
Long.MAX_VALUE.
Apparently there is an alternative method...
"These composite row keys are similar to what RDBMSs offer, yet you can
control the sort order for each field separately. You could do, for
example, *a bitwise inversion of the date expressed as a long value (the
Linux epoch)*. This would then sort the rows descending by date"
I just want to be clear on the above statement. A bitwise inversion is
simply flipping the bits correct? So if the current time represented as
bits is
1001110010011111100101011001110
Then the bitwise inversion is
0110001101100000011010100000110
Is this correct?
On 8/20/11 7:10 AM, steve.boyle@connexity.com wrote:
> I use this bit from the link sent by Sonal:
> reverse_order_stamp is an 8 byte, big endian long with a value of (Long.MAX_VALUE - epoch). This is so the most recent stamp is at the top rather than the bottom.
>
> I use (Long.MAX_VALUE - System.getCurrentTimeMillis()).
>
> -Steve
>
> -----Original Message-----
> From: Mark [mailto:static.void.dev@gmail.com]
> Sent: Saturday, August 20, 2011 6:29 AM
> To: user@hbase.apache.org
> Subject: Re: Reverse timestamp
>
> Thanks but I was more referring to reverse ordered timestamps for
> composite keys to be used for sorting purposes.
>
> On 8/19/11 10:26 PM, lars hofhansl wrote:
>> HBase maps (row-key, column family name, column, timestamp) to a value.
>>
>> The KeyValues are also sorted by the same attributes in reverse timestamp order.
>>
>> The default timestamp is the current time, but you can set any long value (which does not need correlate in any way to the time domain)
>> as the time stamp and hence define your own ordering within multiple version of the same row key.
>>
>> See Put: Put(byte[] row, long ts).
>>
>> Is that what you meant?
>>
>> -- Lars
>>
>>
>>
>> ________________________________
>> From: Mark<st...@gmail.com>
>> To: user@hbase.apache.org
>> Sent: Friday, August 19, 2011 6:39 PM
>> Subject: Reverse timestamp
>>
>> I understand the need but I don't understand how to generate a reverse timestamp. Can someone please explain how this is accomplished and how I can test that its working correctly?
>>
>> Thanks
Re: Reverse timestamp
Posted by Mark <st...@gmail.com>.
Shouldn't be an issue since it will always be used for the current
timestamp. Thanks
On 8/20/11 8:06 AM, Mohamed Ibrahim wrote:
> That's right if you're using *unsigned* integers. If your integers are
> signed (can be negative), then keep the leftmost bit (the highest
> significant bit) zero.
>
> Mohamed
>
>
>
> On Sat, Aug 20, 2011 at 10:56 AM, Mark<st...@gmail.com> wrote:
>
>> Thanks all but I'm actually using ruby as my client so there is no
>> Long.MAX_VALUE.
>>
>> Apparently there is an alternative method...
>>
>> "These composite row keys are similar to what RDBMSs offer, yet you can
>> control the sort order for each field separately. You could do, for example,
>> *a bitwise inversion of the date expressed as a long value (the Linux
>> epoch)*. This would then sort the rows descending by date"
>>
>> I just want to be clear on the above statement. A bitwise inversion is
>> simply flipping the bits correct? So if the current time represented as bits
>> is
>>
>> 100111001001111110010101100111**0
>>
>> Then the bitwise inversion is
>>
>> 011000110110000001101010000011**0
>>
>> Is this correct?
>>
>>
>>
>> On 8/20/11 7:10 AM, steve.boyle@connexity.com wrote:
>>
>>> I use this bit from the link sent by Sonal:
>>> reverse_order_stamp is an 8 byte, big endian long with a value of
>>> (Long.MAX_VALUE - epoch). This is so the most recent stamp is at the top
>>> rather than the bottom.
>>>
>>> I use (Long.MAX_VALUE - System.getCurrentTimeMillis())**.
>>>
>>> -Steve
>>>
>>> -----Original Message-----
>>> From: Mark [mailto:static.void.dev@gmail.**com<st...@gmail.com>
>>> ]
>>> Sent: Saturday, August 20, 2011 6:29 AM
>>> To: user@hbase.apache.org
>>> Subject: Re: Reverse timestamp
>>>
>>> Thanks but I was more referring to reverse ordered timestamps for
>>> composite keys to be used for sorting purposes.
>>>
>>> On 8/19/11 10:26 PM, lars hofhansl wrote:
>>>
>>>> HBase maps (row-key, column family name, column, timestamp) to a value.
>>>>
>>>> The KeyValues are also sorted by the same attributes in reverse timestamp
>>>> order.
>>>>
>>>> The default timestamp is the current time, but you can set any long value
>>>> (which does not need correlate in any way to the time domain)
>>>> as the time stamp and hence define your own ordering within multiple
>>>> version of the same row key.
>>>>
>>>> See Put: Put(byte[] row, long ts).
>>>>
>>>> Is that what you meant?
>>>>
>>>> -- Lars
>>>>
>>>>
>>>>
>>>> ______________________________**__
>>>> From: Mark<static.void.dev@gmail.com**>
>>>> To: user@hbase.apache.org
>>>> Sent: Friday, August 19, 2011 6:39 PM
>>>> Subject: Reverse timestamp
>>>>
>>>> I understand the need but I don't understand how to generate a reverse
>>>> timestamp. Can someone please explain how this is accomplished and how I can
>>>> test that its working correctly?
>>>>
>>>> Thanks
>>>>
Re: Reverse timestamp
Posted by Mohamed Ibrahim <mi...@mibrahim.net>.
That's right if you're using *unsigned* integers. If your integers are
signed (can be negative), then keep the leftmost bit (the highest
significant bit) zero.
Mohamed
On Sat, Aug 20, 2011 at 10:56 AM, Mark <st...@gmail.com> wrote:
> Thanks all but I'm actually using ruby as my client so there is no
> Long.MAX_VALUE.
>
> Apparently there is an alternative method...
>
> "These composite row keys are similar to what RDBMSs offer, yet you can
> control the sort order for each field separately. You could do, for example,
> *a bitwise inversion of the date expressed as a long value (the Linux
> epoch)*. This would then sort the rows descending by date"
>
> I just want to be clear on the above statement. A bitwise inversion is
> simply flipping the bits correct? So if the current time represented as bits
> is
>
> 100111001001111110010101100111**0
>
> Then the bitwise inversion is
>
> 011000110110000001101010000011**0
>
> Is this correct?
>
>
>
> On 8/20/11 7:10 AM, steve.boyle@connexity.com wrote:
>
>> I use this bit from the link sent by Sonal:
>> reverse_order_stamp is an 8 byte, big endian long with a value of
>> (Long.MAX_VALUE - epoch). This is so the most recent stamp is at the top
>> rather than the bottom.
>>
>> I use (Long.MAX_VALUE - System.getCurrentTimeMillis())**.
>>
>> -Steve
>>
>> -----Original Message-----
>> From: Mark [mailto:static.void.dev@gmail.**com<st...@gmail.com>
>> ]
>> Sent: Saturday, August 20, 2011 6:29 AM
>> To: user@hbase.apache.org
>> Subject: Re: Reverse timestamp
>>
>> Thanks but I was more referring to reverse ordered timestamps for
>> composite keys to be used for sorting purposes.
>>
>> On 8/19/11 10:26 PM, lars hofhansl wrote:
>>
>>> HBase maps (row-key, column family name, column, timestamp) to a value.
>>>
>>> The KeyValues are also sorted by the same attributes in reverse timestamp
>>> order.
>>>
>>> The default timestamp is the current time, but you can set any long value
>>> (which does not need correlate in any way to the time domain)
>>> as the time stamp and hence define your own ordering within multiple
>>> version of the same row key.
>>>
>>> See Put: Put(byte[] row, long ts).
>>>
>>> Is that what you meant?
>>>
>>> -- Lars
>>>
>>>
>>>
>>> ______________________________**__
>>> From: Mark<static.void.dev@gmail.com**>
>>> To: user@hbase.apache.org
>>> Sent: Friday, August 19, 2011 6:39 PM
>>> Subject: Reverse timestamp
>>>
>>> I understand the need but I don't understand how to generate a reverse
>>> timestamp. Can someone please explain how this is accomplished and how I can
>>> test that its working correctly?
>>>
>>> Thanks
>>>
>>
Re: Reverse timestamp
Posted by Mark <st...@gmail.com>.
Thanks all but I'm actually using ruby as my client so there is no
Long.MAX_VALUE.
Apparently there is an alternative method...
"These composite row keys are similar to what RDBMSs offer, yet you can
control the sort order for each field separately. You could do, for
example, *a bitwise inversion of the date expressed as a long value (the
Linux epoch)*. This would then sort the rows descending by date"
I just want to be clear on the above statement. A bitwise inversion is
simply flipping the bits correct? So if the current time represented as
bits is
1001110010011111100101011001110
Then the bitwise inversion is
0110001101100000011010100000110
Is this correct?
On 8/20/11 7:10 AM, steve.boyle@connexity.com wrote:
> I use this bit from the link sent by Sonal:
> reverse_order_stamp is an 8 byte, big endian long with a value of (Long.MAX_VALUE - epoch). This is so the most recent stamp is at the top rather than the bottom.
>
> I use (Long.MAX_VALUE - System.getCurrentTimeMillis()).
>
> -Steve
>
> -----Original Message-----
> From: Mark [mailto:static.void.dev@gmail.com]
> Sent: Saturday, August 20, 2011 6:29 AM
> To: user@hbase.apache.org
> Subject: Re: Reverse timestamp
>
> Thanks but I was more referring to reverse ordered timestamps for
> composite keys to be used for sorting purposes.
>
> On 8/19/11 10:26 PM, lars hofhansl wrote:
>> HBase maps (row-key, column family name, column, timestamp) to a value.
>>
>> The KeyValues are also sorted by the same attributes in reverse timestamp order.
>>
>> The default timestamp is the current time, but you can set any long value (which does not need correlate in any way to the time domain)
>> as the time stamp and hence define your own ordering within multiple version of the same row key.
>>
>> See Put: Put(byte[] row, long ts).
>>
>> Is that what you meant?
>>
>> -- Lars
>>
>>
>>
>> ________________________________
>> From: Mark<st...@gmail.com>
>> To: user@hbase.apache.org
>> Sent: Friday, August 19, 2011 6:39 PM
>> Subject: Reverse timestamp
>>
>> I understand the need but I don't understand how to generate a reverse timestamp. Can someone please explain how this is accomplished and how I can test that its working correctly?
>>
>> Thanks
RE: Reverse timestamp
Posted by "steve.boyle@connexity.com" <st...@connexity.com>.
I use this bit from the link sent by Sonal:
reverse_order_stamp is an 8 byte, big endian long with a value of (Long.MAX_VALUE - epoch). This is so the most recent stamp is at the top rather than the bottom.
I use (Long.MAX_VALUE - System.getCurrentTimeMillis()).
-Steve
-----Original Message-----
From: Mark [mailto:static.void.dev@gmail.com]
Sent: Saturday, August 20, 2011 6:29 AM
To: user@hbase.apache.org
Subject: Re: Reverse timestamp
Thanks but I was more referring to reverse ordered timestamps for
composite keys to be used for sorting purposes.
On 8/19/11 10:26 PM, lars hofhansl wrote:
> HBase maps (row-key, column family name, column, timestamp) to a value.
>
> The KeyValues are also sorted by the same attributes in reverse timestamp order.
>
> The default timestamp is the current time, but you can set any long value (which does not need correlate in any way to the time domain)
> as the time stamp and hence define your own ordering within multiple version of the same row key.
>
> See Put: Put(byte[] row, long ts).
>
> Is that what you meant?
>
> -- Lars
>
>
>
> ________________________________
> From: Mark<st...@gmail.com>
> To: user@hbase.apache.org
> Sent: Friday, August 19, 2011 6:39 PM
> Subject: Reverse timestamp
>
> I understand the need but I don't understand how to generate a reverse timestamp. Can someone please explain how this is accomplished and how I can test that its working correctly?
>
> Thanks
Re: Reverse timestamp
Posted by lars hofhansl <lh...@yahoo.com>.
Ah OK... In that case you can take any portion of the key and just invert all the bits.
For example you can have a compound key:
part1|part2, and say part2 is a date (from Date.getTime()) you just invert all the bits and use that as part2.
Inverting the bits is nice, because you need to know nothing about the domain of the key. I.e. it works for strings, longs, ints, etc.
For floating points it's a bit more tricky.
You can look at the HBaseIndex library that comes with Lily for an example.
http://www.lilyproject.org/lily/about/downloads.html, look at the FieldDefinition classes.
-- Lars
________________________________
From: Mark <st...@gmail.com>
To: user@hbase.apache.org
Sent: Saturday, August 20, 2011 6:29 AM
Subject: Re: Reverse timestamp
Thanks but I was more referring to reverse ordered timestamps for
composite keys to be used for sorting purposes.
On 8/19/11 10:26 PM, lars hofhansl wrote:
> HBase maps (row-key, column family name, column, timestamp) to a value.
>
> The KeyValues are also sorted by the same attributes in reverse timestamp order.
>
> The default timestamp is the current time, but you can set any long value (which does not need correlate in any way to the time domain)
> as the time stamp and hence define your own ordering within multiple version of the same row key.
>
> See Put: Put(byte[] row, long ts).
>
> Is that what you meant?
>
> -- Lars
>
>
>
> ________________________________
> From: Mark<st...@gmail.com>
> To: user@hbase.apache.org
> Sent: Friday, August 19, 2011 6:39 PM
> Subject: Reverse timestamp
>
> I understand the need but I don't understand how to generate a reverse timestamp. Can someone please explain how this is accomplished and how I can test that its working correctly?
>
> Thanks
Re: Reverse timestamp
Posted by Mark <st...@gmail.com>.
Thanks but I was more referring to reverse ordered timestamps for
composite keys to be used for sorting purposes.
On 8/19/11 10:26 PM, lars hofhansl wrote:
> HBase maps (row-key, column family name, column, timestamp) to a value.
>
> The KeyValues are also sorted by the same attributes in reverse timestamp order.
>
> The default timestamp is the current time, but you can set any long value (which does not need correlate in any way to the time domain)
> as the time stamp and hence define your own ordering within multiple version of the same row key.
>
> See Put: Put(byte[] row, long ts).
>
> Is that what you meant?
>
> -- Lars
>
>
>
> ________________________________
> From: Mark<st...@gmail.com>
> To: user@hbase.apache.org
> Sent: Friday, August 19, 2011 6:39 PM
> Subject: Reverse timestamp
>
> I understand the need but I don't understand how to generate a reverse timestamp. Can someone please explain how this is accomplished and how I can test that its working correctly?
>
> Thanks
Re: Reverse timestamp
Posted by Sonal Goyal <so...@gmail.com>.
Hi Mark,
Here is an example for generating reverse timestamps. Hope it helps:
http://devblog.streamy.com/2009/04/23/hbase-row-key-design-for-paging-limit-offset-queries/
Best Regards,
Sonal
Crux: Reporting for HBase <https://github.com/sonalgoyal/crux>
Nube Technologies <http://www.nubetech.co>
<http://in.linkedin.com/in/sonalgoyal>
On Sat, Aug 20, 2011 at 10:56 AM, lars hofhansl <lh...@yahoo.com> wrote:
> HBase maps (row-key, column family name, column, timestamp) to a value.
>
> The KeyValues are also sorted by the same attributes in reverse timestamp
> order.
>
> The default timestamp is the current time, but you can set any long value
> (which does not need correlate in any way to the time domain)
> as the time stamp and hence define your own ordering within multiple
> version of the same row key.
>
> See Put: Put(byte[] row, long ts).
>
> Is that what you meant?
>
> -- Lars
>
>
>
> ________________________________
> From: Mark <st...@gmail.com>
> To: user@hbase.apache.org
> Sent: Friday, August 19, 2011 6:39 PM
> Subject: Reverse timestamp
>
> I understand the need but I don't understand how to generate a reverse
> timestamp. Can someone please explain how this is accomplished and how I can
> test that its working correctly?
>
> Thanks
>
Re: Reverse timestamp
Posted by lars hofhansl <lh...@yahoo.com>.
HBase maps (row-key, column family name, column, timestamp) to a value.
The KeyValues are also sorted by the same attributes in reverse timestamp order.
The default timestamp is the current time, but you can set any long value (which does not need correlate in any way to the time domain)
as the time stamp and hence define your own ordering within multiple version of the same row key.
See Put: Put(byte[] row, long ts).
Is that what you meant?
-- Lars
________________________________
From: Mark <st...@gmail.com>
To: user@hbase.apache.org
Sent: Friday, August 19, 2011 6:39 PM
Subject: Reverse timestamp
I understand the need but I don't understand how to generate a reverse timestamp. Can someone please explain how this is accomplished and how I can test that its working correctly?
Thanks