You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Cosmin Lehene <cl...@adobe.com> on 2015/04/10 16:26:08 UTC
Nondeterministic outcome based on cell TTL and major compaction event
order
I've been initially puzzled by this, although I realize how it's likely as designed.
The cell TTL expiration and compactions events can lead to either some (the older) data left or no data at all for a particular (row, family, qualifier, ts) coordinate.
Write (r1, f1, q1, v1, 1)
Write (r1, f1, q1, v1, 2) - TTL=1 minute
Scenario 1:
If a major compaction happens within a minute
it will remove (r1, f1, q1, v1, 1)
then after a minute (r1, f1, q1, v1, 2) will expire
no data left
Scenario 2:
A minute passes
(r1, f1, q1, v1, 2) expires
Compaction runs..
(r1, f1, q1, v1, 1) remains
This seems, by and large expected behavior, but it still seems "uncomfortable" that the (overall) outcome is not decided by me, but by a chance of event ordering.
I wonder we'd want this to behave differently (perhaps it has been discussed already), but if not, it's worth a more detailed documentation in the book.
What do you think?
Cosmin
Re: Nondeterministic outcome based on cell TTL and major compaction event order
Posted by Michael Segel <mi...@hotmail.com>.
Ok,
So we agree that we’re talking about a hypothetical case because no one in their right mind would do something like this….
(Which means that someone will do this… ;-)
If I had to describe HBase columns to a group of people attending a conference, I would probably describe it as the following:
Think of each column to be a stack of finite depth, where there can be no cells or N cells present with a preset max number of cells in that stack. By default the stack has a maximum depth of 3 cells defined.
If a stack has 0 cells, then the column no longer exists for that row and no space is reserved. (This is a major difference from RDBMSs)
On this stack, we add to the top of the stack and once the stack reaches its full capacity, we drop from the bottom.
Would that be a fair description?
Now regardless of the max value (mv), where mv >= 1, the code handling the insertion of a cell is the same.
if your depth is 2, you can have (rk, cf, cd, val1) and (rk, cf, cd, val2).
1: (rk,cf, cd, val2)
0: (rk,cf, cd, val1)
If you add a new value, (rk,cf,cd,val3) the your stack looks like this…
1: (rk,cf, cd, val3)
0: (rk,cf, cd, val2)
So you can see we’re adding to the top, and dropping from the bottom.
Now if the depth of the stack was set to 3 it would look like this:
2: (rk,cf, cd, val3)
1: (rk,cf, cd, val2)
0: (rk,cf, cd, val1)
So lets consider what happens if we set the depth of the stack to 1.
The same thing will happen. We have a row (rk, cf, cd, val1) sitting on our stack.
We insert a new row, (rk, cf, cd, val2) it will replace the tuple (rk,cf,cd,val1) that is sitting on the stack.
In short regardless of the depth of the stack where the depth has to be greater or equal to 1, the actions are the same.
Do you agree that this is a good way to describe the feature of cells / cell versioning ?
There’s more, but I want to just walk you through the problem one step at a time.
-Mike
> On Apr 20, 2015, at 1:20 PM, Anoop John <an...@gmail.com> wrote:
>
> What I mean is this.
> If ur system has the req that u add permanent place for a user and allow to
> add a temp place for him based on his request (with a TTL as per his
> duration of stay there), then while designing that column you should not
> give versions=1 (Which is the def case).
> If max versions was set as (say atleast 2) then the outcome *is
> deterministic* whatever time major compaction happens.
>
> If the versions was left as def value (=1) then yes we will get non
> deterministic results as per the time when major compaction has run. This
> case is similar one with delete and major compaction case which Lars said
> above.(Which is a known problem)
>
>
> -Anoop-
>
>
> On Mon, Apr 20, 2015 at 10:42 PM, Michael Segel <mi...@hotmail.com>
> wrote:
>
>> Yes, how you handle versioning, it will be the same regardless of the
>> number of cell versions. (Pop from the top, drop from the bottom.)
>>
>> What I think would help is to define the expected outcome or contract.
>>
>> The code should be deterministic.
>>
>> That is to say, the action of writing a cell with a TTL that isn’t set to
>> Max Long Value, should be the same each time.
>>
>> Then the issue of compaction should also be deterministic.
>>
>> There should be a small finite number of permutations based on conditional
>> statements within the code and a single entry and exit point to the method
>> within the class.
>> (This gets into a coding practice that unless you’re throwing an
>> exception, you only have one return from the code.) [Note: Its been 20 +
>> years since I read Kernighan and Plauger … ]
>>
>> The idea is that we should define the outcomes and then code to it.
>>
>> So you can define a set of scenarios, and the defined outcome.
>> Then take those defined scenarios / outcomes and consider compactions
>> occuring at times during the ingestion process and then decay.
>> Again its my understanding that the desired outcome is that if the cell
>> decays, only that cell is affected, and will be inert (non returned ) if it
>> exists until a major compaction removes it all together?
>>
>> Is this not the case?
>>
>> -Mike
>>
>>
>>
>>> On Apr 19, 2015, at 10:10 PM, Anoop John <an...@gmail.com> wrote:
>>>
>>> Interested example for cell level TTL Michael.
>>> But one thing I want to say. In the above example, the versions for the
>>> corresponding CF should have been >1. In such case there wont be issue
>>> with major compaction right?
>>> When versions =1 yes, it will give non deterministic results.
>>>
>>> -Anoop-
>>>
>>>
>>> On Sun, Apr 19, 2015 at 6:59 PM, Michael Segel <
>> michael_segel@hotmail.com>
>>> wrote:
>>>
>>>> Actually I just thought of a better example…
>>>>
>>>> Credit Card Fraud detection.
>>>> Imagine you’re being sent to work on a project out of the country.
>>>> So suppose I head over across the pond and invaded Europe. ;-P
>>>>
>>>> I would want the credit card companies to not weigh a foreign
>> transaction
>>>> heavily when determining fraud, so that if they know my location is in
>>>> London, then spending $$ on a dinner in London is not fraud.
>>>>
>>>> So I call ahead and tell my bank I’m going to be in Europe for XXX
>> months..
>>>>
>>>>
>>>>>
>>>>> As to why you would want to TTL on a column that doesn’t always use a
>>>> TTL?
>>>>>
>>>>> I used this example in a different post…
>>>>>
>>>>> Imagine you have a road link which has an attribute of speed.
>>>>>
>>>>> You could have construction, or variable speed limits.
>>>>> So you would want to change the speed limit with a TTL.
>>>>>
>>>>> Or you’re a retailer and you’re offering a 20% discount on a product
>> for
>>>> a limited time only?
>>>>>
>>>>> Sure, these are bad examples because in reality the database is a sync
>>>> and the application would manage these type of issues.
>>>>>
>>>>>
>>>>>> On Apr 18, 2015, at 12:23 AM, lars hofhansl <la...@apache.org> wrote:
>>>>>>
>>>>>> The formatting did not come out right. Lemme try again...
>>>>>>
>>>>>>
>>>>>> Just came here to say that. From our (maybe not clearly enough)
>> defined
>>>> semantics this how it should behave.
>>>>>>
>>>>>> It _is_ confusing, though, since compactions are - in a sense - just
>>>> optimizations that run in the background to prevent the number of
>> HFiles to
>>>> be unbounded.
>>>>>> In this case the schedule of the compactions influences the outcome.
>>>>>>
>>>>>> Note that even tombstone markers can be confusing. Here's another
>>>> confusing example:
>>>>>> 1. delete (r1, f1, q1, T2)
>>>>>> 2. put (r1, f1, q1, v1, T1)
>>>>>>
>>>>>> If a compaction happens after #1 but before #2 the put will remain:
>>>>>> delete
>>>>>> compaction
>>>>>> put (remains visible)
>>>>>>
>>>>>> If the compaction happens after #2 the put will be affected by the
>>>> delete and hence removed:
>>>>>> delete
>>>>>> put
>>>>>> compaction (will remove the put)
>>>>>>
>>>>>> Notice though that both of these examples _are_ a bit weird.
>>>>>> Why would only a newer version of the cell have a TTL?
>>>>>> Why would you date a delete into the future?
>>>>>>
>>>>>> -- Lars
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ________________________________
>>>>>> From: lars hofhansl <la...@apache.org>
>>>>>> To: "dev@hbase.apache.org" <de...@hbase.apache.org>
>>>>>> Sent: Friday, April 17, 2015 10:18 PM
>>>>>> Subject: Re: Nondeterministic outcome based on cell TTL and major
>>>> compaction event order
>>>>>>
>>>>>>
>>>>>> Just came here to say that. From our (maybe not clearly enough)
>> defined
>>>> semantics this how it should behave.
>>>>>>
>>>>>> It _is_ confusing, though, since compactions are - in a sense - just
>>>> optimizations that run in the background to prevent the number of
>> HFiles to
>>>> be unbounded.In this case the schedule of the compactions influences the
>>>> outcome.
>>>>>> Note that even tombstone markers can be confusing. Here's another
>>>> confusing example:1. delete (r1, f1, q1, T2)2. put (r1, f1, q1, v1, T1)
>>>>>> If a compaction happens after #1 but before #2 the put will
>>>> remain:deletecompactionput (remains visible)
>>>>>>
>>>>>> If the compaction happens after #2 the put will be affected by the
>>>> delete and hence removed.deleteputcompaction (will remove the put)
>>>>>>
>>>>>> Notice though that both of these examples _are_ a bit weird.Why would
>>>> only a newer version of the cell have a TTL?Why would you date a delete
>>>> into the future?
>>>>>> -- Lars
>>>>>>
>>>>>> From: Sean Busbey <bu...@cloudera.com>
>>>>>>
>>>>>>
>>>>>>
>>>>>> To: dev <de...@hbase.apache.org>
>>>>>> Sent: Friday, April 17, 2015 4:52 PM
>>>>>> Subject: Re: Nondeterministic outcome based on cell TTL and major
>>>> compaction event order
>>>>>>
>>>>>> If you have max versions set to 1 (the default), then c1 should be
>>>> removed
>>>>>> at compaction time if c2 still exists then.
>>>>>>
>>>>>> --
>>>>>> Sean
>>>>>>
>>>>>>
>>>>>> On Apr 17, 2015 6:41 PM, "Michael Segel" <mi...@hotmail.com>
>>>> wrote:
>>>>>>
>>>>>>> Ok,
>>>>>>> So then if you have a previous cell (c1) and you insert a new cell c2
>>>> that
>>>>>>> has a TTL of lets say 5 mins, then c1 should always exist?
>>>>>>> That is my understanding but from Cosmin’s post, he’s saying its
>>>>>>> different. And that’s why I don’t understand. You couldn’t lose the
>>>> cell
>>>>>>> c1 at all.
>>>>>>> Compaction or no compaction.
>>>>>>>
>>>>>>> That’s why I’m confused. Current behavior doesn’t match the expected
>>>>>>> contract.
>>>>>>>
>>>>>>> -Mike
>>>>>>>
>>>>>>>> On Apr 17, 2015, at 4:37 PM, Andrew Purtell <ap...@apache.org>
>>>> wrote:
>>>>>>>>
>>>>>>>> The way TTLs work today is they define the interval of time a cell
>>>>>>>> exists - exactly as that. There is no tombstone laid like a normal
>>>>>>>> delete. Once the TTL elapses the cell just ceases to exist to normal
>>>>>>>> scanners. The interaction of expired cells, multiple versions,
>> minimum
>>>>>>>> versions, raw scanners, etc. can be confusing. We can absolutely
>>>>>>>> revisit this.
>>>>>>>>
>>>>>>>> A cell with an expired TTL could be treated as the combination of
>>>>>>>> tombstone and the most recent value it lays over. This is not how
>> the
>>>>>>>> implementation works today, but could be changed for an upcoming
>> major
>>>>>>>> version like 2.0 if there's consensus to do it.
>>>>>>>>
>>>>>>>>
>>>>>>>>> On Apr 10, 2015, at 7:26 AM, Cosmin Lehene <cl...@adobe.com>
>>>> wrote:
>>>>>>>>>
>>>>>>>>> I've been initially puzzled by this, although I realize how it's
>>>> likely
>>>>>>> as designed.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> The cell TTL expiration and compactions events can lead to either
>>>> some
>>>>>>> (the older) data left or no data at all for a particular (row,
>> family,
>>>>>>> qualifier, ts) coordinate.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Write (r1, f1, q1, v1, 1)
>>>>>>>>>
>>>>>>>>> Write (r1, f1, q1, v1, 2) - TTL=1 minute
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Scenario 1:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> If a major compaction happens within a minute
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> it will remove (r1, f1, q1, v1, 1)
>>>>>>>>>
>>>>>>>>> then after a minute (r1, f1, q1, v1, 2) will expire
>>>>>>>>>
>>>>>>>>> no data left
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Scenario 2:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> A minute passes
>>>>>>>>>
>>>>>>>>> (r1, f1, q1, v1, 2) expires
>>>>>>>>>
>>>>>>>>> Compaction runs..
>>>>>>>>>
>>>>>>>>> (r1, f1, q1, v1, 1) remains
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> This seems, by and large expected behavior, but it still seems
>>>>>>> "uncomfortable" that the (overall) outcome is not decided by me, but
>>>> by a
>>>>>>> chance of event ordering.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I wonder we'd want this to behave differently (perhaps it has been
>>>>>>> discussed already), but if not, it's worth a more detailed
>>>> documentation in
>>>>>>> the book.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> What do you think?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Cosmin
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Best regards,
>>>>>>>>
>>>>>>>> - Andy
>>>>>>>>
>>>>>>>> Problems worthy of attack prove their worth by hitting back. - Piet
>>>>>>>> Hein (via Tom White)
>>>>>>>>
>>>>>>>
>>>>>>> The opinions expressed here are mine, while they may reflect a
>>>> cognitive
>>>>>>> thought, that is purely accidental.
>>>>>>> Use at your own risk.
>>>>>>> Michael Segel
>>>>>>> michael_segel (AT) hotmail.com
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> The opinions expressed here are mine, while they may reflect a
>> cognitive
>>>> thought, that is purely accidental.
>>>>> Use at your own risk.
>>>>> Michael Segel
>>>>> michael_segel (AT) hotmail.com
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> The opinions expressed here are mine, while they may reflect a cognitive
>>>> thought, that is purely accidental.
>>>> Use at your own risk.
>>>> Michael Segel
>>>> michael_segel (AT) hotmail.com
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>
>> The opinions expressed here are mine, while they may reflect a cognitive
>> thought, that is purely accidental.
>> Use at your own risk.
>> Michael Segel
>> michael_segel (AT) hotmail.com
>>
>>
>>
>>
>>
>>
The opinions expressed here are mine, while they may reflect a cognitive thought, that is purely accidental.
Use at your own risk.
Michael Segel
michael_segel (AT) hotmail.com
Re: Nondeterministic outcome based on cell TTL and major compaction
event order
Posted by Anoop John <an...@gmail.com>.
What I mean is this.
If ur system has the req that u add permanent place for a user and allow to
add a temp place for him based on his request (with a TTL as per his
duration of stay there), then while designing that column you should not
give versions=1 (Which is the def case).
If max versions was set as (say atleast 2) then the outcome *is
deterministic* whatever time major compaction happens.
If the versions was left as def value (=1) then yes we will get non
deterministic results as per the time when major compaction has run. This
case is similar one with delete and major compaction case which Lars said
above.(Which is a known problem)
-Anoop-
On Mon, Apr 20, 2015 at 10:42 PM, Michael Segel <mi...@hotmail.com>
wrote:
> Yes, how you handle versioning, it will be the same regardless of the
> number of cell versions. (Pop from the top, drop from the bottom.)
>
> What I think would help is to define the expected outcome or contract.
>
> The code should be deterministic.
>
> That is to say, the action of writing a cell with a TTL that isn’t set to
> Max Long Value, should be the same each time.
>
> Then the issue of compaction should also be deterministic.
>
> There should be a small finite number of permutations based on conditional
> statements within the code and a single entry and exit point to the method
> within the class.
> (This gets into a coding practice that unless you’re throwing an
> exception, you only have one return from the code.) [Note: Its been 20 +
> years since I read Kernighan and Plauger … ]
>
> The idea is that we should define the outcomes and then code to it.
>
> So you can define a set of scenarios, and the defined outcome.
> Then take those defined scenarios / outcomes and consider compactions
> occuring at times during the ingestion process and then decay.
> Again its my understanding that the desired outcome is that if the cell
> decays, only that cell is affected, and will be inert (non returned ) if it
> exists until a major compaction removes it all together?
>
> Is this not the case?
>
> -Mike
>
>
>
> > On Apr 19, 2015, at 10:10 PM, Anoop John <an...@gmail.com> wrote:
> >
> > Interested example for cell level TTL Michael.
> > But one thing I want to say. In the above example, the versions for the
> > corresponding CF should have been >1. In such case there wont be issue
> > with major compaction right?
> > When versions =1 yes, it will give non deterministic results.
> >
> > -Anoop-
> >
> >
> > On Sun, Apr 19, 2015 at 6:59 PM, Michael Segel <
> michael_segel@hotmail.com>
> > wrote:
> >
> >> Actually I just thought of a better example…
> >>
> >> Credit Card Fraud detection.
> >> Imagine you’re being sent to work on a project out of the country.
> >> So suppose I head over across the pond and invaded Europe. ;-P
> >>
> >> I would want the credit card companies to not weigh a foreign
> transaction
> >> heavily when determining fraud, so that if they know my location is in
> >> London, then spending $$ on a dinner in London is not fraud.
> >>
> >> So I call ahead and tell my bank I’m going to be in Europe for XXX
> months..
> >>
> >>
> >>>
> >>> As to why you would want to TTL on a column that doesn’t always use a
> >> TTL?
> >>>
> >>> I used this example in a different post…
> >>>
> >>> Imagine you have a road link which has an attribute of speed.
> >>>
> >>> You could have construction, or variable speed limits.
> >>> So you would want to change the speed limit with a TTL.
> >>>
> >>> Or you’re a retailer and you’re offering a 20% discount on a product
> for
> >> a limited time only?
> >>>
> >>> Sure, these are bad examples because in reality the database is a sync
> >> and the application would manage these type of issues.
> >>>
> >>>
> >>>> On Apr 18, 2015, at 12:23 AM, lars hofhansl <la...@apache.org> wrote:
> >>>>
> >>>> The formatting did not come out right. Lemme try again...
> >>>>
> >>>>
> >>>> Just came here to say that. From our (maybe not clearly enough)
> defined
> >> semantics this how it should behave.
> >>>>
> >>>> It _is_ confusing, though, since compactions are - in a sense - just
> >> optimizations that run in the background to prevent the number of
> HFiles to
> >> be unbounded.
> >>>> In this case the schedule of the compactions influences the outcome.
> >>>>
> >>>> Note that even tombstone markers can be confusing. Here's another
> >> confusing example:
> >>>> 1. delete (r1, f1, q1, T2)
> >>>> 2. put (r1, f1, q1, v1, T1)
> >>>>
> >>>> If a compaction happens after #1 but before #2 the put will remain:
> >>>> delete
> >>>> compaction
> >>>> put (remains visible)
> >>>>
> >>>> If the compaction happens after #2 the put will be affected by the
> >> delete and hence removed:
> >>>> delete
> >>>> put
> >>>> compaction (will remove the put)
> >>>>
> >>>> Notice though that both of these examples _are_ a bit weird.
> >>>> Why would only a newer version of the cell have a TTL?
> >>>> Why would you date a delete into the future?
> >>>>
> >>>> -- Lars
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> ________________________________
> >>>> From: lars hofhansl <la...@apache.org>
> >>>> To: "dev@hbase.apache.org" <de...@hbase.apache.org>
> >>>> Sent: Friday, April 17, 2015 10:18 PM
> >>>> Subject: Re: Nondeterministic outcome based on cell TTL and major
> >> compaction event order
> >>>>
> >>>>
> >>>> Just came here to say that. From our (maybe not clearly enough)
> defined
> >> semantics this how it should behave.
> >>>>
> >>>> It _is_ confusing, though, since compactions are - in a sense - just
> >> optimizations that run in the background to prevent the number of
> HFiles to
> >> be unbounded.In this case the schedule of the compactions influences the
> >> outcome.
> >>>> Note that even tombstone markers can be confusing. Here's another
> >> confusing example:1. delete (r1, f1, q1, T2)2. put (r1, f1, q1, v1, T1)
> >>>> If a compaction happens after #1 but before #2 the put will
> >> remain:deletecompactionput (remains visible)
> >>>>
> >>>> If the compaction happens after #2 the put will be affected by the
> >> delete and hence removed.deleteputcompaction (will remove the put)
> >>>>
> >>>> Notice though that both of these examples _are_ a bit weird.Why would
> >> only a newer version of the cell have a TTL?Why would you date a delete
> >> into the future?
> >>>> -- Lars
> >>>>
> >>>> From: Sean Busbey <bu...@cloudera.com>
> >>>>
> >>>>
> >>>>
> >>>> To: dev <de...@hbase.apache.org>
> >>>> Sent: Friday, April 17, 2015 4:52 PM
> >>>> Subject: Re: Nondeterministic outcome based on cell TTL and major
> >> compaction event order
> >>>>
> >>>> If you have max versions set to 1 (the default), then c1 should be
> >> removed
> >>>> at compaction time if c2 still exists then.
> >>>>
> >>>> --
> >>>> Sean
> >>>>
> >>>>
> >>>> On Apr 17, 2015 6:41 PM, "Michael Segel" <mi...@hotmail.com>
> >> wrote:
> >>>>
> >>>>> Ok,
> >>>>> So then if you have a previous cell (c1) and you insert a new cell c2
> >> that
> >>>>> has a TTL of lets say 5 mins, then c1 should always exist?
> >>>>> That is my understanding but from Cosmin’s post, he’s saying its
> >>>>> different. And that’s why I don’t understand. You couldn’t lose the
> >> cell
> >>>>> c1 at all.
> >>>>> Compaction or no compaction.
> >>>>>
> >>>>> That’s why I’m confused. Current behavior doesn’t match the expected
> >>>>> contract.
> >>>>>
> >>>>> -Mike
> >>>>>
> >>>>>> On Apr 17, 2015, at 4:37 PM, Andrew Purtell <ap...@apache.org>
> >> wrote:
> >>>>>>
> >>>>>> The way TTLs work today is they define the interval of time a cell
> >>>>>> exists - exactly as that. There is no tombstone laid like a normal
> >>>>>> delete. Once the TTL elapses the cell just ceases to exist to normal
> >>>>>> scanners. The interaction of expired cells, multiple versions,
> minimum
> >>>>>> versions, raw scanners, etc. can be confusing. We can absolutely
> >>>>>> revisit this.
> >>>>>>
> >>>>>> A cell with an expired TTL could be treated as the combination of
> >>>>>> tombstone and the most recent value it lays over. This is not how
> the
> >>>>>> implementation works today, but could be changed for an upcoming
> major
> >>>>>> version like 2.0 if there's consensus to do it.
> >>>>>>
> >>>>>>
> >>>>>>> On Apr 10, 2015, at 7:26 AM, Cosmin Lehene <cl...@adobe.com>
> >> wrote:
> >>>>>>>
> >>>>>>> I've been initially puzzled by this, although I realize how it's
> >> likely
> >>>>> as designed.
> >>>>>>>
> >>>>>>>
> >>>>>>> The cell TTL expiration and compactions events can lead to either
> >> some
> >>>>> (the older) data left or no data at all for a particular (row,
> family,
> >>>>> qualifier, ts) coordinate.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> Write (r1, f1, q1, v1, 1)
> >>>>>>>
> >>>>>>> Write (r1, f1, q1, v1, 2) - TTL=1 minute
> >>>>>>>
> >>>>>>>
> >>>>>>> Scenario 1:
> >>>>>>>
> >>>>>>>
> >>>>>>> If a major compaction happens within a minute
> >>>>>>>
> >>>>>>>
> >>>>>>> it will remove (r1, f1, q1, v1, 1)
> >>>>>>>
> >>>>>>> then after a minute (r1, f1, q1, v1, 2) will expire
> >>>>>>>
> >>>>>>> no data left
> >>>>>>>
> >>>>>>>
> >>>>>>> Scenario 2:
> >>>>>>>
> >>>>>>>
> >>>>>>> A minute passes
> >>>>>>>
> >>>>>>> (r1, f1, q1, v1, 2) expires
> >>>>>>>
> >>>>>>> Compaction runs..
> >>>>>>>
> >>>>>>> (r1, f1, q1, v1, 1) remains
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> This seems, by and large expected behavior, but it still seems
> >>>>> "uncomfortable" that the (overall) outcome is not decided by me, but
> >> by a
> >>>>> chance of event ordering.
> >>>>>>>
> >>>>>>>
> >>>>>>> I wonder we'd want this to behave differently (perhaps it has been
> >>>>> discussed already), but if not, it's worth a more detailed
> >> documentation in
> >>>>> the book.
> >>>>>>>
> >>>>>>>
> >>>>>>> What do you think?
> >>>>>>>
> >>>>>>>
> >>>>>>> Cosmin
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Best regards,
> >>>>>>
> >>>>>> - Andy
> >>>>>>
> >>>>>> Problems worthy of attack prove their worth by hitting back. - Piet
> >>>>>> Hein (via Tom White)
> >>>>>>
> >>>>>
> >>>>> The opinions expressed here are mine, while they may reflect a
> >> cognitive
> >>>>> thought, that is purely accidental.
> >>>>> Use at your own risk.
> >>>>> Michael Segel
> >>>>> michael_segel (AT) hotmail.com
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>> The opinions expressed here are mine, while they may reflect a
> cognitive
> >> thought, that is purely accidental.
> >>> Use at your own risk.
> >>> Michael Segel
> >>> michael_segel (AT) hotmail.com
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>
> >> The opinions expressed here are mine, while they may reflect a cognitive
> >> thought, that is purely accidental.
> >> Use at your own risk.
> >> Michael Segel
> >> michael_segel (AT) hotmail.com
> >>
> >>
> >>
> >>
> >>
> >>
>
> The opinions expressed here are mine, while they may reflect a cognitive
> thought, that is purely accidental.
> Use at your own risk.
> Michael Segel
> michael_segel (AT) hotmail.com
>
>
>
>
>
>
Re: Nondeterministic outcome based on cell TTL and major compaction event order
Posted by Michael Segel <mi...@hotmail.com>.
Yes, how you handle versioning, it will be the same regardless of the number of cell versions. (Pop from the top, drop from the bottom.)
What I think would help is to define the expected outcome or contract.
The code should be deterministic.
That is to say, the action of writing a cell with a TTL that isn’t set to Max Long Value, should be the same each time.
Then the issue of compaction should also be deterministic.
There should be a small finite number of permutations based on conditional statements within the code and a single entry and exit point to the method within the class.
(This gets into a coding practice that unless you’re throwing an exception, you only have one return from the code.) [Note: Its been 20 + years since I read Kernighan and Plauger … ]
The idea is that we should define the outcomes and then code to it.
So you can define a set of scenarios, and the defined outcome.
Then take those defined scenarios / outcomes and consider compactions occuring at times during the ingestion process and then decay.
Again its my understanding that the desired outcome is that if the cell decays, only that cell is affected, and will be inert (non returned ) if it exists until a major compaction removes it all together?
Is this not the case?
-Mike
> On Apr 19, 2015, at 10:10 PM, Anoop John <an...@gmail.com> wrote:
>
> Interested example for cell level TTL Michael.
> But one thing I want to say. In the above example, the versions for the
> corresponding CF should have been >1. In such case there wont be issue
> with major compaction right?
> When versions =1 yes, it will give non deterministic results.
>
> -Anoop-
>
>
> On Sun, Apr 19, 2015 at 6:59 PM, Michael Segel <mi...@hotmail.com>
> wrote:
>
>> Actually I just thought of a better example…
>>
>> Credit Card Fraud detection.
>> Imagine you’re being sent to work on a project out of the country.
>> So suppose I head over across the pond and invaded Europe. ;-P
>>
>> I would want the credit card companies to not weigh a foreign transaction
>> heavily when determining fraud, so that if they know my location is in
>> London, then spending $$ on a dinner in London is not fraud.
>>
>> So I call ahead and tell my bank I’m going to be in Europe for XXX months..
>>
>>
>>>
>>> As to why you would want to TTL on a column that doesn’t always use a
>> TTL?
>>>
>>> I used this example in a different post…
>>>
>>> Imagine you have a road link which has an attribute of speed.
>>>
>>> You could have construction, or variable speed limits.
>>> So you would want to change the speed limit with a TTL.
>>>
>>> Or you’re a retailer and you’re offering a 20% discount on a product for
>> a limited time only?
>>>
>>> Sure, these are bad examples because in reality the database is a sync
>> and the application would manage these type of issues.
>>>
>>>
>>>> On Apr 18, 2015, at 12:23 AM, lars hofhansl <la...@apache.org> wrote:
>>>>
>>>> The formatting did not come out right. Lemme try again...
>>>>
>>>>
>>>> Just came here to say that. From our (maybe not clearly enough) defined
>> semantics this how it should behave.
>>>>
>>>> It _is_ confusing, though, since compactions are - in a sense - just
>> optimizations that run in the background to prevent the number of HFiles to
>> be unbounded.
>>>> In this case the schedule of the compactions influences the outcome.
>>>>
>>>> Note that even tombstone markers can be confusing. Here's another
>> confusing example:
>>>> 1. delete (r1, f1, q1, T2)
>>>> 2. put (r1, f1, q1, v1, T1)
>>>>
>>>> If a compaction happens after #1 but before #2 the put will remain:
>>>> delete
>>>> compaction
>>>> put (remains visible)
>>>>
>>>> If the compaction happens after #2 the put will be affected by the
>> delete and hence removed:
>>>> delete
>>>> put
>>>> compaction (will remove the put)
>>>>
>>>> Notice though that both of these examples _are_ a bit weird.
>>>> Why would only a newer version of the cell have a TTL?
>>>> Why would you date a delete into the future?
>>>>
>>>> -- Lars
>>>>
>>>>
>>>>
>>>>
>>>> ________________________________
>>>> From: lars hofhansl <la...@apache.org>
>>>> To: "dev@hbase.apache.org" <de...@hbase.apache.org>
>>>> Sent: Friday, April 17, 2015 10:18 PM
>>>> Subject: Re: Nondeterministic outcome based on cell TTL and major
>> compaction event order
>>>>
>>>>
>>>> Just came here to say that. From our (maybe not clearly enough) defined
>> semantics this how it should behave.
>>>>
>>>> It _is_ confusing, though, since compactions are - in a sense - just
>> optimizations that run in the background to prevent the number of HFiles to
>> be unbounded.In this case the schedule of the compactions influences the
>> outcome.
>>>> Note that even tombstone markers can be confusing. Here's another
>> confusing example:1. delete (r1, f1, q1, T2)2. put (r1, f1, q1, v1, T1)
>>>> If a compaction happens after #1 but before #2 the put will
>> remain:deletecompactionput (remains visible)
>>>>
>>>> If the compaction happens after #2 the put will be affected by the
>> delete and hence removed.deleteputcompaction (will remove the put)
>>>>
>>>> Notice though that both of these examples _are_ a bit weird.Why would
>> only a newer version of the cell have a TTL?Why would you date a delete
>> into the future?
>>>> -- Lars
>>>>
>>>> From: Sean Busbey <bu...@cloudera.com>
>>>>
>>>>
>>>>
>>>> To: dev <de...@hbase.apache.org>
>>>> Sent: Friday, April 17, 2015 4:52 PM
>>>> Subject: Re: Nondeterministic outcome based on cell TTL and major
>> compaction event order
>>>>
>>>> If you have max versions set to 1 (the default), then c1 should be
>> removed
>>>> at compaction time if c2 still exists then.
>>>>
>>>> --
>>>> Sean
>>>>
>>>>
>>>> On Apr 17, 2015 6:41 PM, "Michael Segel" <mi...@hotmail.com>
>> wrote:
>>>>
>>>>> Ok,
>>>>> So then if you have a previous cell (c1) and you insert a new cell c2
>> that
>>>>> has a TTL of lets say 5 mins, then c1 should always exist?
>>>>> That is my understanding but from Cosmin’s post, he’s saying its
>>>>> different. And that’s why I don’t understand. You couldn’t lose the
>> cell
>>>>> c1 at all.
>>>>> Compaction or no compaction.
>>>>>
>>>>> That’s why I’m confused. Current behavior doesn’t match the expected
>>>>> contract.
>>>>>
>>>>> -Mike
>>>>>
>>>>>> On Apr 17, 2015, at 4:37 PM, Andrew Purtell <ap...@apache.org>
>> wrote:
>>>>>>
>>>>>> The way TTLs work today is they define the interval of time a cell
>>>>>> exists - exactly as that. There is no tombstone laid like a normal
>>>>>> delete. Once the TTL elapses the cell just ceases to exist to normal
>>>>>> scanners. The interaction of expired cells, multiple versions, minimum
>>>>>> versions, raw scanners, etc. can be confusing. We can absolutely
>>>>>> revisit this.
>>>>>>
>>>>>> A cell with an expired TTL could be treated as the combination of
>>>>>> tombstone and the most recent value it lays over. This is not how the
>>>>>> implementation works today, but could be changed for an upcoming major
>>>>>> version like 2.0 if there's consensus to do it.
>>>>>>
>>>>>>
>>>>>>> On Apr 10, 2015, at 7:26 AM, Cosmin Lehene <cl...@adobe.com>
>> wrote:
>>>>>>>
>>>>>>> I've been initially puzzled by this, although I realize how it's
>> likely
>>>>> as designed.
>>>>>>>
>>>>>>>
>>>>>>> The cell TTL expiration and compactions events can lead to either
>> some
>>>>> (the older) data left or no data at all for a particular (row, family,
>>>>> qualifier, ts) coordinate.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Write (r1, f1, q1, v1, 1)
>>>>>>>
>>>>>>> Write (r1, f1, q1, v1, 2) - TTL=1 minute
>>>>>>>
>>>>>>>
>>>>>>> Scenario 1:
>>>>>>>
>>>>>>>
>>>>>>> If a major compaction happens within a minute
>>>>>>>
>>>>>>>
>>>>>>> it will remove (r1, f1, q1, v1, 1)
>>>>>>>
>>>>>>> then after a minute (r1, f1, q1, v1, 2) will expire
>>>>>>>
>>>>>>> no data left
>>>>>>>
>>>>>>>
>>>>>>> Scenario 2:
>>>>>>>
>>>>>>>
>>>>>>> A minute passes
>>>>>>>
>>>>>>> (r1, f1, q1, v1, 2) expires
>>>>>>>
>>>>>>> Compaction runs..
>>>>>>>
>>>>>>> (r1, f1, q1, v1, 1) remains
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> This seems, by and large expected behavior, but it still seems
>>>>> "uncomfortable" that the (overall) outcome is not decided by me, but
>> by a
>>>>> chance of event ordering.
>>>>>>>
>>>>>>>
>>>>>>> I wonder we'd want this to behave differently (perhaps it has been
>>>>> discussed already), but if not, it's worth a more detailed
>> documentation in
>>>>> the book.
>>>>>>>
>>>>>>>
>>>>>>> What do you think?
>>>>>>>
>>>>>>>
>>>>>>> Cosmin
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best regards,
>>>>>>
>>>>>> - Andy
>>>>>>
>>>>>> Problems worthy of attack prove their worth by hitting back. - Piet
>>>>>> Hein (via Tom White)
>>>>>>
>>>>>
>>>>> The opinions expressed here are mine, while they may reflect a
>> cognitive
>>>>> thought, that is purely accidental.
>>>>> Use at your own risk.
>>>>> Michael Segel
>>>>> michael_segel (AT) hotmail.com
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>> The opinions expressed here are mine, while they may reflect a cognitive
>> thought, that is purely accidental.
>>> Use at your own risk.
>>> Michael Segel
>>> michael_segel (AT) hotmail.com
>>>
>>>
>>>
>>>
>>>
>>>
>>
>> The opinions expressed here are mine, while they may reflect a cognitive
>> thought, that is purely accidental.
>> Use at your own risk.
>> Michael Segel
>> michael_segel (AT) hotmail.com
>>
>>
>>
>>
>>
>>
The opinions expressed here are mine, while they may reflect a cognitive thought, that is purely accidental.
Use at your own risk.
Michael Segel
michael_segel (AT) hotmail.com
Re: Nondeterministic outcome based on cell TTL and major compaction
event order
Posted by Anoop John <an...@gmail.com>.
Interested example for cell level TTL Michael.
But one thing I want to say. In the above example, the versions for the
corresponding CF should have been >1. In such case there wont be issue
with major compaction right?
When versions =1 yes, it will give non deterministic results.
-Anoop-
On Sun, Apr 19, 2015 at 6:59 PM, Michael Segel <mi...@hotmail.com>
wrote:
> Actually I just thought of a better example…
>
> Credit Card Fraud detection.
> Imagine you’re being sent to work on a project out of the country.
> So suppose I head over across the pond and invaded Europe. ;-P
>
> I would want the credit card companies to not weigh a foreign transaction
> heavily when determining fraud, so that if they know my location is in
> London, then spending $$ on a dinner in London is not fraud.
>
> So I call ahead and tell my bank I’m going to be in Europe for XXX months..
>
>
> >
> > As to why you would want to TTL on a column that doesn’t always use a
> TTL?
> >
> > I used this example in a different post…
> >
> > Imagine you have a road link which has an attribute of speed.
> >
> > You could have construction, or variable speed limits.
> > So you would want to change the speed limit with a TTL.
> >
> > Or you’re a retailer and you’re offering a 20% discount on a product for
> a limited time only?
> >
> > Sure, these are bad examples because in reality the database is a sync
> and the application would manage these type of issues.
> >
> >
> >> On Apr 18, 2015, at 12:23 AM, lars hofhansl <la...@apache.org> wrote:
> >>
> >> The formatting did not come out right. Lemme try again...
> >>
> >>
> >> Just came here to say that. From our (maybe not clearly enough) defined
> semantics this how it should behave.
> >>
> >> It _is_ confusing, though, since compactions are - in a sense - just
> optimizations that run in the background to prevent the number of HFiles to
> be unbounded.
> >> In this case the schedule of the compactions influences the outcome.
> >>
> >> Note that even tombstone markers can be confusing. Here's another
> confusing example:
> >> 1. delete (r1, f1, q1, T2)
> >> 2. put (r1, f1, q1, v1, T1)
> >>
> >> If a compaction happens after #1 but before #2 the put will remain:
> >> delete
> >> compaction
> >> put (remains visible)
> >>
> >> If the compaction happens after #2 the put will be affected by the
> delete and hence removed:
> >> delete
> >> put
> >> compaction (will remove the put)
> >>
> >> Notice though that both of these examples _are_ a bit weird.
> >> Why would only a newer version of the cell have a TTL?
> >> Why would you date a delete into the future?
> >>
> >> -- Lars
> >>
> >>
> >>
> >>
> >> ________________________________
> >> From: lars hofhansl <la...@apache.org>
> >> To: "dev@hbase.apache.org" <de...@hbase.apache.org>
> >> Sent: Friday, April 17, 2015 10:18 PM
> >> Subject: Re: Nondeterministic outcome based on cell TTL and major
> compaction event order
> >>
> >>
> >> Just came here to say that. From our (maybe not clearly enough) defined
> semantics this how it should behave.
> >>
> >> It _is_ confusing, though, since compactions are - in a sense - just
> optimizations that run in the background to prevent the number of HFiles to
> be unbounded.In this case the schedule of the compactions influences the
> outcome.
> >> Note that even tombstone markers can be confusing. Here's another
> confusing example:1. delete (r1, f1, q1, T2)2. put (r1, f1, q1, v1, T1)
> >> If a compaction happens after #1 but before #2 the put will
> remain:deletecompactionput (remains visible)
> >>
> >> If the compaction happens after #2 the put will be affected by the
> delete and hence removed.deleteputcompaction (will remove the put)
> >>
> >> Notice though that both of these examples _are_ a bit weird.Why would
> only a newer version of the cell have a TTL?Why would you date a delete
> into the future?
> >> -- Lars
> >>
> >> From: Sean Busbey <bu...@cloudera.com>
> >>
> >>
> >>
> >> To: dev <de...@hbase.apache.org>
> >> Sent: Friday, April 17, 2015 4:52 PM
> >> Subject: Re: Nondeterministic outcome based on cell TTL and major
> compaction event order
> >>
> >> If you have max versions set to 1 (the default), then c1 should be
> removed
> >> at compaction time if c2 still exists then.
> >>
> >> --
> >> Sean
> >>
> >>
> >> On Apr 17, 2015 6:41 PM, "Michael Segel" <mi...@hotmail.com>
> wrote:
> >>
> >>> Ok,
> >>> So then if you have a previous cell (c1) and you insert a new cell c2
> that
> >>> has a TTL of lets say 5 mins, then c1 should always exist?
> >>> That is my understanding but from Cosmin’s post, he’s saying its
> >>> different. And that’s why I don’t understand. You couldn’t lose the
> cell
> >>> c1 at all.
> >>> Compaction or no compaction.
> >>>
> >>> That’s why I’m confused. Current behavior doesn’t match the expected
> >>> contract.
> >>>
> >>> -Mike
> >>>
> >>>> On Apr 17, 2015, at 4:37 PM, Andrew Purtell <ap...@apache.org>
> wrote:
> >>>>
> >>>> The way TTLs work today is they define the interval of time a cell
> >>>> exists - exactly as that. There is no tombstone laid like a normal
> >>>> delete. Once the TTL elapses the cell just ceases to exist to normal
> >>>> scanners. The interaction of expired cells, multiple versions, minimum
> >>>> versions, raw scanners, etc. can be confusing. We can absolutely
> >>>> revisit this.
> >>>>
> >>>> A cell with an expired TTL could be treated as the combination of
> >>>> tombstone and the most recent value it lays over. This is not how the
> >>>> implementation works today, but could be changed for an upcoming major
> >>>> version like 2.0 if there's consensus to do it.
> >>>>
> >>>>
> >>>>> On Apr 10, 2015, at 7:26 AM, Cosmin Lehene <cl...@adobe.com>
> wrote:
> >>>>>
> >>>>> I've been initially puzzled by this, although I realize how it's
> likely
> >>> as designed.
> >>>>>
> >>>>>
> >>>>> The cell TTL expiration and compactions events can lead to either
> some
> >>> (the older) data left or no data at all for a particular (row, family,
> >>> qualifier, ts) coordinate.
> >>>>>
> >>>>>
> >>>>>
> >>>>> Write (r1, f1, q1, v1, 1)
> >>>>>
> >>>>> Write (r1, f1, q1, v1, 2) - TTL=1 minute
> >>>>>
> >>>>>
> >>>>> Scenario 1:
> >>>>>
> >>>>>
> >>>>> If a major compaction happens within a minute
> >>>>>
> >>>>>
> >>>>> it will remove (r1, f1, q1, v1, 1)
> >>>>>
> >>>>> then after a minute (r1, f1, q1, v1, 2) will expire
> >>>>>
> >>>>> no data left
> >>>>>
> >>>>>
> >>>>> Scenario 2:
> >>>>>
> >>>>>
> >>>>> A minute passes
> >>>>>
> >>>>> (r1, f1, q1, v1, 2) expires
> >>>>>
> >>>>> Compaction runs..
> >>>>>
> >>>>> (r1, f1, q1, v1, 1) remains
> >>>>>
> >>>>>
> >>>>>
> >>>>> This seems, by and large expected behavior, but it still seems
> >>> "uncomfortable" that the (overall) outcome is not decided by me, but
> by a
> >>> chance of event ordering.
> >>>>>
> >>>>>
> >>>>> I wonder we'd want this to behave differently (perhaps it has been
> >>> discussed already), but if not, it's worth a more detailed
> documentation in
> >>> the book.
> >>>>>
> >>>>>
> >>>>> What do you think?
> >>>>>
> >>>>>
> >>>>> Cosmin
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>> --
> >>>> Best regards,
> >>>>
> >>>> - Andy
> >>>>
> >>>> Problems worthy of attack prove their worth by hitting back. - Piet
> >>>> Hein (via Tom White)
> >>>>
> >>>
> >>> The opinions expressed here are mine, while they may reflect a
> cognitive
> >>> thought, that is purely accidental.
> >>> Use at your own risk.
> >>> Michael Segel
> >>> michael_segel (AT) hotmail.com
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>
> >
> > The opinions expressed here are mine, while they may reflect a cognitive
> thought, that is purely accidental.
> > Use at your own risk.
> > Michael Segel
> > michael_segel (AT) hotmail.com
> >
> >
> >
> >
> >
> >
>
> The opinions expressed here are mine, while they may reflect a cognitive
> thought, that is purely accidental.
> Use at your own risk.
> Michael Segel
> michael_segel (AT) hotmail.com
>
>
>
>
>
>
Re: Nondeterministic outcome based on cell TTL and major compaction event order
Posted by Michael Segel <mi...@hotmail.com>.
Actually I just thought of a better example…
Credit Card Fraud detection.
Imagine you’re being sent to work on a project out of the country.
So suppose I head over across the pond and invaded Europe. ;-P
I would want the credit card companies to not weigh a foreign transaction heavily when determining fraud, so that if they know my location is in London, then spending $$ on a dinner in London is not fraud.
So I call ahead and tell my bank I’m going to be in Europe for XXX months..
>
> As to why you would want to TTL on a column that doesn’t always use a TTL?
>
> I used this example in a different post…
>
> Imagine you have a road link which has an attribute of speed.
>
> You could have construction, or variable speed limits.
> So you would want to change the speed limit with a TTL.
>
> Or you’re a retailer and you’re offering a 20% discount on a product for a limited time only?
>
> Sure, these are bad examples because in reality the database is a sync and the application would manage these type of issues.
>
>
>> On Apr 18, 2015, at 12:23 AM, lars hofhansl <la...@apache.org> wrote:
>>
>> The formatting did not come out right. Lemme try again...
>>
>>
>> Just came here to say that. From our (maybe not clearly enough) defined semantics this how it should behave.
>>
>> It _is_ confusing, though, since compactions are - in a sense - just optimizations that run in the background to prevent the number of HFiles to be unbounded.
>> In this case the schedule of the compactions influences the outcome.
>>
>> Note that even tombstone markers can be confusing. Here's another confusing example:
>> 1. delete (r1, f1, q1, T2)
>> 2. put (r1, f1, q1, v1, T1)
>>
>> If a compaction happens after #1 but before #2 the put will remain:
>> delete
>> compaction
>> put (remains visible)
>>
>> If the compaction happens after #2 the put will be affected by the delete and hence removed:
>> delete
>> put
>> compaction (will remove the put)
>>
>> Notice though that both of these examples _are_ a bit weird.
>> Why would only a newer version of the cell have a TTL?
>> Why would you date a delete into the future?
>>
>> -- Lars
>>
>>
>>
>>
>> ________________________________
>> From: lars hofhansl <la...@apache.org>
>> To: "dev@hbase.apache.org" <de...@hbase.apache.org>
>> Sent: Friday, April 17, 2015 10:18 PM
>> Subject: Re: Nondeterministic outcome based on cell TTL and major compaction event order
>>
>>
>> Just came here to say that. From our (maybe not clearly enough) defined semantics this how it should behave.
>>
>> It _is_ confusing, though, since compactions are - in a sense - just optimizations that run in the background to prevent the number of HFiles to be unbounded.In this case the schedule of the compactions influences the outcome.
>> Note that even tombstone markers can be confusing. Here's another confusing example:1. delete (r1, f1, q1, T2)2. put (r1, f1, q1, v1, T1)
>> If a compaction happens after #1 but before #2 the put will remain:deletecompactionput (remains visible)
>>
>> If the compaction happens after #2 the put will be affected by the delete and hence removed.deleteputcompaction (will remove the put)
>>
>> Notice though that both of these examples _are_ a bit weird.Why would only a newer version of the cell have a TTL?Why would you date a delete into the future?
>> -- Lars
>>
>> From: Sean Busbey <bu...@cloudera.com>
>>
>>
>>
>> To: dev <de...@hbase.apache.org>
>> Sent: Friday, April 17, 2015 4:52 PM
>> Subject: Re: Nondeterministic outcome based on cell TTL and major compaction event order
>>
>> If you have max versions set to 1 (the default), then c1 should be removed
>> at compaction time if c2 still exists then.
>>
>> --
>> Sean
>>
>>
>> On Apr 17, 2015 6:41 PM, "Michael Segel" <mi...@hotmail.com> wrote:
>>
>>> Ok,
>>> So then if you have a previous cell (c1) and you insert a new cell c2 that
>>> has a TTL of lets say 5 mins, then c1 should always exist?
>>> That is my understanding but from Cosmin’s post, he’s saying its
>>> different. And that’s why I don’t understand. You couldn’t lose the cell
>>> c1 at all.
>>> Compaction or no compaction.
>>>
>>> That’s why I’m confused. Current behavior doesn’t match the expected
>>> contract.
>>>
>>> -Mike
>>>
>>>> On Apr 17, 2015, at 4:37 PM, Andrew Purtell <ap...@apache.org> wrote:
>>>>
>>>> The way TTLs work today is they define the interval of time a cell
>>>> exists - exactly as that. There is no tombstone laid like a normal
>>>> delete. Once the TTL elapses the cell just ceases to exist to normal
>>>> scanners. The interaction of expired cells, multiple versions, minimum
>>>> versions, raw scanners, etc. can be confusing. We can absolutely
>>>> revisit this.
>>>>
>>>> A cell with an expired TTL could be treated as the combination of
>>>> tombstone and the most recent value it lays over. This is not how the
>>>> implementation works today, but could be changed for an upcoming major
>>>> version like 2.0 if there's consensus to do it.
>>>>
>>>>
>>>>> On Apr 10, 2015, at 7:26 AM, Cosmin Lehene <cl...@adobe.com> wrote:
>>>>>
>>>>> I've been initially puzzled by this, although I realize how it's likely
>>> as designed.
>>>>>
>>>>>
>>>>> The cell TTL expiration and compactions events can lead to either some
>>> (the older) data left or no data at all for a particular (row, family,
>>> qualifier, ts) coordinate.
>>>>>
>>>>>
>>>>>
>>>>> Write (r1, f1, q1, v1, 1)
>>>>>
>>>>> Write (r1, f1, q1, v1, 2) - TTL=1 minute
>>>>>
>>>>>
>>>>> Scenario 1:
>>>>>
>>>>>
>>>>> If a major compaction happens within a minute
>>>>>
>>>>>
>>>>> it will remove (r1, f1, q1, v1, 1)
>>>>>
>>>>> then after a minute (r1, f1, q1, v1, 2) will expire
>>>>>
>>>>> no data left
>>>>>
>>>>>
>>>>> Scenario 2:
>>>>>
>>>>>
>>>>> A minute passes
>>>>>
>>>>> (r1, f1, q1, v1, 2) expires
>>>>>
>>>>> Compaction runs..
>>>>>
>>>>> (r1, f1, q1, v1, 1) remains
>>>>>
>>>>>
>>>>>
>>>>> This seems, by and large expected behavior, but it still seems
>>> "uncomfortable" that the (overall) outcome is not decided by me, but by a
>>> chance of event ordering.
>>>>>
>>>>>
>>>>> I wonder we'd want this to behave differently (perhaps it has been
>>> discussed already), but if not, it's worth a more detailed documentation in
>>> the book.
>>>>>
>>>>>
>>>>> What do you think?
>>>>>
>>>>>
>>>>> Cosmin
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> Best regards,
>>>>
>>>> - Andy
>>>>
>>>> Problems worthy of attack prove their worth by hitting back. - Piet
>>>> Hein (via Tom White)
>>>>
>>>
>>> The opinions expressed here are mine, while they may reflect a cognitive
>>> thought, that is purely accidental.
>>> Use at your own risk.
>>> Michael Segel
>>> michael_segel (AT) hotmail.com
>>>
>>>
>>>
>>>
>>>
>>>
>>
>
> The opinions expressed here are mine, while they may reflect a cognitive thought, that is purely accidental.
> Use at your own risk.
> Michael Segel
> michael_segel (AT) hotmail.com
>
>
>
>
>
>
The opinions expressed here are mine, while they may reflect a cognitive thought, that is purely accidental.
Use at your own risk.
Michael Segel
michael_segel (AT) hotmail.com
Re: Nondeterministic outcome based on cell TTL and major compaction event order
Posted by Michael Segel <mi...@hotmail.com>.
Ok…
In your example the put has an earlier time T1 than T2?
I’m assuming that the orders are reversed due to something in the memcache?
Also what happens to the data within the memcache that hasn’t been written to file and you insert a delete after a put() ?
As to why you would want to TTL on a column that doesn’t always use a TTL?
I used this example in a different post…
Imagine you have a road link which has an attribute of speed.
You could have construction, or variable speed limits.
So you would want to change the speed limit with a TTL.
Or you’re a retailer and you’re offering a 20% discount on a product for a limited time only?
Sure, these are bad examples because in reality the database is a sync and the application would manage these type of issues.
> On Apr 18, 2015, at 12:23 AM, lars hofhansl <la...@apache.org> wrote:
>
> The formatting did not come out right. Lemme try again...
>
>
> Just came here to say that. From our (maybe not clearly enough) defined semantics this how it should behave.
>
> It _is_ confusing, though, since compactions are - in a sense - just optimizations that run in the background to prevent the number of HFiles to be unbounded.
> In this case the schedule of the compactions influences the outcome.
>
> Note that even tombstone markers can be confusing. Here's another confusing example:
> 1. delete (r1, f1, q1, T2)
> 2. put (r1, f1, q1, v1, T1)
>
> If a compaction happens after #1 but before #2 the put will remain:
> delete
> compaction
> put (remains visible)
>
> If the compaction happens after #2 the put will be affected by the delete and hence removed:
> delete
> put
> compaction (will remove the put)
>
> Notice though that both of these examples _are_ a bit weird.
> Why would only a newer version of the cell have a TTL?
> Why would you date a delete into the future?
>
> -- Lars
>
>
>
>
> ________________________________
> From: lars hofhansl <la...@apache.org>
> To: "dev@hbase.apache.org" <de...@hbase.apache.org>
> Sent: Friday, April 17, 2015 10:18 PM
> Subject: Re: Nondeterministic outcome based on cell TTL and major compaction event order
>
>
> Just came here to say that. From our (maybe not clearly enough) defined semantics this how it should behave.
>
> It _is_ confusing, though, since compactions are - in a sense - just optimizations that run in the background to prevent the number of HFiles to be unbounded.In this case the schedule of the compactions influences the outcome.
> Note that even tombstone markers can be confusing. Here's another confusing example:1. delete (r1, f1, q1, T2)2. put (r1, f1, q1, v1, T1)
> If a compaction happens after #1 but before #2 the put will remain:deletecompactionput (remains visible)
>
> If the compaction happens after #2 the put will be affected by the delete and hence removed.deleteputcompaction (will remove the put)
>
> Notice though that both of these examples _are_ a bit weird.Why would only a newer version of the cell have a TTL?Why would you date a delete into the future?
> -- Lars
>
> From: Sean Busbey <bu...@cloudera.com>
>
>
>
> To: dev <de...@hbase.apache.org>
> Sent: Friday, April 17, 2015 4:52 PM
> Subject: Re: Nondeterministic outcome based on cell TTL and major compaction event order
>
> If you have max versions set to 1 (the default), then c1 should be removed
> at compaction time if c2 still exists then.
>
> --
> Sean
>
>
> On Apr 17, 2015 6:41 PM, "Michael Segel" <mi...@hotmail.com> wrote:
>
>> Ok,
>> So then if you have a previous cell (c1) and you insert a new cell c2 that
>> has a TTL of lets say 5 mins, then c1 should always exist?
>> That is my understanding but from Cosmin’s post, he’s saying its
>> different. And that’s why I don’t understand. You couldn’t lose the cell
>> c1 at all.
>> Compaction or no compaction.
>>
>> That’s why I’m confused. Current behavior doesn’t match the expected
>> contract.
>>
>> -Mike
>>
>>> On Apr 17, 2015, at 4:37 PM, Andrew Purtell <ap...@apache.org> wrote:
>>>
>>> The way TTLs work today is they define the interval of time a cell
>>> exists - exactly as that. There is no tombstone laid like a normal
>>> delete. Once the TTL elapses the cell just ceases to exist to normal
>>> scanners. The interaction of expired cells, multiple versions, minimum
>>> versions, raw scanners, etc. can be confusing. We can absolutely
>>> revisit this.
>>>
>>> A cell with an expired TTL could be treated as the combination of
>>> tombstone and the most recent value it lays over. This is not how the
>>> implementation works today, but could be changed for an upcoming major
>>> version like 2.0 if there's consensus to do it.
>>>
>>>
>>>> On Apr 10, 2015, at 7:26 AM, Cosmin Lehene <cl...@adobe.com> wrote:
>>>>
>>>> I've been initially puzzled by this, although I realize how it's likely
>> as designed.
>>>>
>>>>
>>>> The cell TTL expiration and compactions events can lead to either some
>> (the older) data left or no data at all for a particular (row, family,
>> qualifier, ts) coordinate.
>>>>
>>>>
>>>>
>>>> Write (r1, f1, q1, v1, 1)
>>>>
>>>> Write (r1, f1, q1, v1, 2) - TTL=1 minute
>>>>
>>>>
>>>> Scenario 1:
>>>>
>>>>
>>>> If a major compaction happens within a minute
>>>>
>>>>
>>>> it will remove (r1, f1, q1, v1, 1)
>>>>
>>>> then after a minute (r1, f1, q1, v1, 2) will expire
>>>>
>>>> no data left
>>>>
>>>>
>>>> Scenario 2:
>>>>
>>>>
>>>> A minute passes
>>>>
>>>> (r1, f1, q1, v1, 2) expires
>>>>
>>>> Compaction runs..
>>>>
>>>> (r1, f1, q1, v1, 1) remains
>>>>
>>>>
>>>>
>>>> This seems, by and large expected behavior, but it still seems
>> "uncomfortable" that the (overall) outcome is not decided by me, but by a
>> chance of event ordering.
>>>>
>>>>
>>>> I wonder we'd want this to behave differently (perhaps it has been
>> discussed already), but if not, it's worth a more detailed documentation in
>> the book.
>>>>
>>>>
>>>> What do you think?
>>>>
>>>>
>>>> Cosmin
>>>>
>>>>
>>>>
>>>>
>>>
>>> --
>>> Best regards,
>>>
>>> - Andy
>>>
>>> Problems worthy of attack prove their worth by hitting back. - Piet
>>> Hein (via Tom White)
>>>
>>
>> The opinions expressed here are mine, while they may reflect a cognitive
>> thought, that is purely accidental.
>> Use at your own risk.
>> Michael Segel
>> michael_segel (AT) hotmail.com
>>
>>
>>
>>
>>
>>
>
The opinions expressed here are mine, while they may reflect a cognitive thought, that is purely accidental.
Use at your own risk.
Michael Segel
michael_segel (AT) hotmail.com
Re: Nondeterministic outcome based on cell TTL and major compaction
event order
Posted by lars hofhansl <la...@apache.org>.
The formatting did not come out right. Lemme try again...
Just came here to say that. From our (maybe not clearly enough) defined semantics this how it should behave.
It _is_ confusing, though, since compactions are - in a sense - just optimizations that run in the background to prevent the number of HFiles to be unbounded.
In this case the schedule of the compactions influences the outcome.
Note that even tombstone markers can be confusing. Here's another confusing example:
1. delete (r1, f1, q1, T2)
2. put (r1, f1, q1, v1, T1)
If a compaction happens after #1 but before #2 the put will remain:
delete
compaction
put (remains visible)
If the compaction happens after #2 the put will be affected by the delete and hence removed:
delete
put
compaction (will remove the put)
Notice though that both of these examples _are_ a bit weird.
Why would only a newer version of the cell have a TTL?
Why would you date a delete into the future?
-- Lars
________________________________
From: lars hofhansl <la...@apache.org>
To: "dev@hbase.apache.org" <de...@hbase.apache.org>
Sent: Friday, April 17, 2015 10:18 PM
Subject: Re: Nondeterministic outcome based on cell TTL and major compaction event order
Just came here to say that. From our (maybe not clearly enough) defined semantics this how it should behave.
It _is_ confusing, though, since compactions are - in a sense - just optimizations that run in the background to prevent the number of HFiles to be unbounded.In this case the schedule of the compactions influences the outcome.
Note that even tombstone markers can be confusing. Here's another confusing example:1. delete (r1, f1, q1, T2)2. put (r1, f1, q1, v1, T1)
If a compaction happens after #1 but before #2 the put will remain:deletecompactionput (remains visible)
If the compaction happens after #2 the put will be affected by the delete and hence removed.deleteputcompaction (will remove the put)
Notice though that both of these examples _are_ a bit weird.Why would only a newer version of the cell have a TTL?Why would you date a delete into the future?
-- Lars
From: Sean Busbey <bu...@cloudera.com>
To: dev <de...@hbase.apache.org>
Sent: Friday, April 17, 2015 4:52 PM
Subject: Re: Nondeterministic outcome based on cell TTL and major compaction event order
If you have max versions set to 1 (the default), then c1 should be removed
at compaction time if c2 still exists then.
--
Sean
On Apr 17, 2015 6:41 PM, "Michael Segel" <mi...@hotmail.com> wrote:
> Ok,
> So then if you have a previous cell (c1) and you insert a new cell c2 that
> has a TTL of lets say 5 mins, then c1 should always exist?
> That is my understanding but from Cosmin’s post, he’s saying its
> different. And that’s why I don’t understand. You couldn’t lose the cell
> c1 at all.
> Compaction or no compaction.
>
> That’s why I’m confused. Current behavior doesn’t match the expected
> contract.
>
> -Mike
>
> > On Apr 17, 2015, at 4:37 PM, Andrew Purtell <ap...@apache.org> wrote:
> >
> > The way TTLs work today is they define the interval of time a cell
> > exists - exactly as that. There is no tombstone laid like a normal
> > delete. Once the TTL elapses the cell just ceases to exist to normal
> > scanners. The interaction of expired cells, multiple versions, minimum
> > versions, raw scanners, etc. can be confusing. We can absolutely
> > revisit this.
> >
> > A cell with an expired TTL could be treated as the combination of
> > tombstone and the most recent value it lays over. This is not how the
> > implementation works today, but could be changed for an upcoming major
> > version like 2.0 if there's consensus to do it.
> >
> >
> >> On Apr 10, 2015, at 7:26 AM, Cosmin Lehene <cl...@adobe.com> wrote:
> >>
> >> I've been initially puzzled by this, although I realize how it's likely
> as designed.
> >>
> >>
> >> The cell TTL expiration and compactions events can lead to either some
> (the older) data left or no data at all for a particular (row, family,
> qualifier, ts) coordinate.
> >>
> >>
> >>
> >> Write (r1, f1, q1, v1, 1)
> >>
> >> Write (r1, f1, q1, v1, 2) - TTL=1 minute
> >>
> >>
> >> Scenario 1:
> >>
> >>
> >> If a major compaction happens within a minute
> >>
> >>
> >> it will remove (r1, f1, q1, v1, 1)
> >>
> >> then after a minute (r1, f1, q1, v1, 2) will expire
> >>
> >> no data left
> >>
> >>
> >> Scenario 2:
> >>
> >>
> >> A minute passes
> >>
> >> (r1, f1, q1, v1, 2) expires
> >>
> >> Compaction runs..
> >>
> >> (r1, f1, q1, v1, 1) remains
> >>
> >>
> >>
> >> This seems, by and large expected behavior, but it still seems
> "uncomfortable" that the (overall) outcome is not decided by me, but by a
> chance of event ordering.
> >>
> >>
> >> I wonder we'd want this to behave differently (perhaps it has been
> discussed already), but if not, it's worth a more detailed documentation in
> the book.
> >>
> >>
> >> What do you think?
> >>
> >>
> >> Cosmin
> >>
> >>
> >>
> >>
> >
> > --
> > Best regards,
> >
> > - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet
> > Hein (via Tom White)
> >
>
> The opinions expressed here are mine, while they may reflect a cognitive
> thought, that is purely accidental.
> Use at your own risk.
> Michael Segel
> michael_segel (AT) hotmail.com
>
>
>
>
>
>
Re: Nondeterministic outcome based on cell TTL and major compaction
event order
Posted by lars hofhansl <la...@apache.org>.
Just came here to say that. From our (maybe not clearly enough) defined semantics this how it should behave.
It _is_ confusing, though, since compactions are - in a sense - just optimizations that run in the background to prevent the number of HFiles to be unbounded.In this case the schedule of the compactions influences the outcome.
Note that even tombstone markers can be confusing. Here's another confusing example:1. delete (r1, f1, q1, T2)2. put (r1, f1, q1, v1, T1)
If a compaction happens after #1 but before #2 the put will remain:deletecompactionput (remains visible)
If the compaction happens after #2 the put will be affected by the delete and hence removed.deleteputcompaction (will remove the put)
Notice though that both of these examples _are_ a bit weird.Why would only a newer version of the cell have a TTL?Why would you date a delete into the future?
-- Lars
From: Sean Busbey <bu...@cloudera.com>
To: dev <de...@hbase.apache.org>
Sent: Friday, April 17, 2015 4:52 PM
Subject: Re: Nondeterministic outcome based on cell TTL and major compaction event order
If you have max versions set to 1 (the default), then c1 should be removed
at compaction time if c2 still exists then.
--
Sean
On Apr 17, 2015 6:41 PM, "Michael Segel" <mi...@hotmail.com> wrote:
> Ok,
> So then if you have a previous cell (c1) and you insert a new cell c2 that
> has a TTL of lets say 5 mins, then c1 should always exist?
> That is my understanding but from Cosmin’s post, he’s saying its
> different. And that’s why I don’t understand. You couldn’t lose the cell
> c1 at all.
> Compaction or no compaction.
>
> That’s why I’m confused. Current behavior doesn’t match the expected
> contract.
>
> -Mike
>
> > On Apr 17, 2015, at 4:37 PM, Andrew Purtell <ap...@apache.org> wrote:
> >
> > The way TTLs work today is they define the interval of time a cell
> > exists - exactly as that. There is no tombstone laid like a normal
> > delete. Once the TTL elapses the cell just ceases to exist to normal
> > scanners. The interaction of expired cells, multiple versions, minimum
> > versions, raw scanners, etc. can be confusing. We can absolutely
> > revisit this.
> >
> > A cell with an expired TTL could be treated as the combination of
> > tombstone and the most recent value it lays over. This is not how the
> > implementation works today, but could be changed for an upcoming major
> > version like 2.0 if there's consensus to do it.
> >
> >
> >> On Apr 10, 2015, at 7:26 AM, Cosmin Lehene <cl...@adobe.com> wrote:
> >>
> >> I've been initially puzzled by this, although I realize how it's likely
> as designed.
> >>
> >>
> >> The cell TTL expiration and compactions events can lead to either some
> (the older) data left or no data at all for a particular (row, family,
> qualifier, ts) coordinate.
> >>
> >>
> >>
> >> Write (r1, f1, q1, v1, 1)
> >>
> >> Write (r1, f1, q1, v1, 2) - TTL=1 minute
> >>
> >>
> >> Scenario 1:
> >>
> >>
> >> If a major compaction happens within a minute
> >>
> >>
> >> it will remove (r1, f1, q1, v1, 1)
> >>
> >> then after a minute (r1, f1, q1, v1, 2) will expire
> >>
> >> no data left
> >>
> >>
> >> Scenario 2:
> >>
> >>
> >> A minute passes
> >>
> >> (r1, f1, q1, v1, 2) expires
> >>
> >> Compaction runs..
> >>
> >> (r1, f1, q1, v1, 1) remains
> >>
> >>
> >>
> >> This seems, by and large expected behavior, but it still seems
> "uncomfortable" that the (overall) outcome is not decided by me, but by a
> chance of event ordering.
> >>
> >>
> >> I wonder we'd want this to behave differently (perhaps it has been
> discussed already), but if not, it's worth a more detailed documentation in
> the book.
> >>
> >>
> >> What do you think?
> >>
> >>
> >> Cosmin
> >>
> >>
> >>
> >>
> >
> > --
> > Best regards,
> >
> > - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet
> > Hein (via Tom White)
> >
>
> The opinions expressed here are mine, while they may reflect a cognitive
> thought, that is purely accidental.
> Use at your own risk.
> Michael Segel
> michael_segel (AT) hotmail.com
>
>
>
>
>
>
Re: Nondeterministic outcome based on cell TTL and major compaction event order
Posted by Michael Segel <mi...@hotmail.com>.
I said barring max versions… (in an earlier post on the thread.)
> On Apr 17, 2015, at 6:52 PM, Sean Busbey <bu...@cloudera.com> wrote:
>
> If you have max versions set to 1 (the default), then c1 should be removed
> at compaction time if c2 still exists then.
>
> --
> Sean
> On Apr 17, 2015 6:41 PM, "Michael Segel" <mi...@hotmail.com> wrote:
>
>> Ok,
>> So then if you have a previous cell (c1) and you insert a new cell c2 that
>> has a TTL of lets say 5 mins, then c1 should always exist?
>> That is my understanding but from Cosmin’s post, he’s saying its
>> different. And that’s why I don’t understand. You couldn’t lose the cell
>> c1 at all.
>> Compaction or no compaction.
>>
>> That’s why I’m confused. Current behavior doesn’t match the expected
>> contract.
>>
>> -Mike
>>
>>> On Apr 17, 2015, at 4:37 PM, Andrew Purtell <ap...@apache.org> wrote:
>>>
>>> The way TTLs work today is they define the interval of time a cell
>>> exists - exactly as that. There is no tombstone laid like a normal
>>> delete. Once the TTL elapses the cell just ceases to exist to normal
>>> scanners. The interaction of expired cells, multiple versions, minimum
>>> versions, raw scanners, etc. can be confusing. We can absolutely
>>> revisit this.
>>>
>>> A cell with an expired TTL could be treated as the combination of
>>> tombstone and the most recent value it lays over. This is not how the
>>> implementation works today, but could be changed for an upcoming major
>>> version like 2.0 if there's consensus to do it.
>>>
>>>
>>>> On Apr 10, 2015, at 7:26 AM, Cosmin Lehene <cl...@adobe.com> wrote:
>>>>
>>>> I've been initially puzzled by this, although I realize how it's likely
>> as designed.
>>>>
>>>>
>>>> The cell TTL expiration and compactions events can lead to either some
>> (the older) data left or no data at all for a particular (row, family,
>> qualifier, ts) coordinate.
>>>>
>>>>
>>>>
>>>> Write (r1, f1, q1, v1, 1)
>>>>
>>>> Write (r1, f1, q1, v1, 2) - TTL=1 minute
>>>>
>>>>
>>>> Scenario 1:
>>>>
>>>>
>>>> If a major compaction happens within a minute
>>>>
>>>>
>>>> it will remove (r1, f1, q1, v1, 1)
>>>>
>>>> then after a minute (r1, f1, q1, v1, 2) will expire
>>>>
>>>> no data left
>>>>
>>>>
>>>> Scenario 2:
>>>>
>>>>
>>>> A minute passes
>>>>
>>>> (r1, f1, q1, v1, 2) expires
>>>>
>>>> Compaction runs..
>>>>
>>>> (r1, f1, q1, v1, 1) remains
>>>>
>>>>
>>>>
>>>> This seems, by and large expected behavior, but it still seems
>> "uncomfortable" that the (overall) outcome is not decided by me, but by a
>> chance of event ordering.
>>>>
>>>>
>>>> I wonder we'd want this to behave differently (perhaps it has been
>> discussed already), but if not, it's worth a more detailed documentation in
>> the book.
>>>>
>>>>
>>>> What do you think?
>>>>
>>>>
>>>> Cosmin
>>>>
>>>>
>>>>
>>>>
>>>
>>> --
>>> Best regards,
>>>
>>> - Andy
>>>
>>> Problems worthy of attack prove their worth by hitting back. - Piet
>>> Hein (via Tom White)
>>>
>>
>> The opinions expressed here are mine, while they may reflect a cognitive
>> thought, that is purely accidental.
>> Use at your own risk.
>> Michael Segel
>> michael_segel (AT) hotmail.com
>>
>>
>>
>>
>>
>>
The opinions expressed here are mine, while they may reflect a cognitive thought, that is purely accidental.
Use at your own risk.
Michael Segel
michael_segel (AT) hotmail.com
Re: Nondeterministic outcome based on cell TTL and major compaction
event order
Posted by Sean Busbey <bu...@cloudera.com>.
If you have max versions set to 1 (the default), then c1 should be removed
at compaction time if c2 still exists then.
--
Sean
On Apr 17, 2015 6:41 PM, "Michael Segel" <mi...@hotmail.com> wrote:
> Ok,
> So then if you have a previous cell (c1) and you insert a new cell c2 that
> has a TTL of lets say 5 mins, then c1 should always exist?
> That is my understanding but from Cosmin’s post, he’s saying its
> different. And that’s why I don’t understand. You couldn’t lose the cell
> c1 at all.
> Compaction or no compaction.
>
> That’s why I’m confused. Current behavior doesn’t match the expected
> contract.
>
> -Mike
>
> > On Apr 17, 2015, at 4:37 PM, Andrew Purtell <ap...@apache.org> wrote:
> >
> > The way TTLs work today is they define the interval of time a cell
> > exists - exactly as that. There is no tombstone laid like a normal
> > delete. Once the TTL elapses the cell just ceases to exist to normal
> > scanners. The interaction of expired cells, multiple versions, minimum
> > versions, raw scanners, etc. can be confusing. We can absolutely
> > revisit this.
> >
> > A cell with an expired TTL could be treated as the combination of
> > tombstone and the most recent value it lays over. This is not how the
> > implementation works today, but could be changed for an upcoming major
> > version like 2.0 if there's consensus to do it.
> >
> >
> >> On Apr 10, 2015, at 7:26 AM, Cosmin Lehene <cl...@adobe.com> wrote:
> >>
> >> I've been initially puzzled by this, although I realize how it's likely
> as designed.
> >>
> >>
> >> The cell TTL expiration and compactions events can lead to either some
> (the older) data left or no data at all for a particular (row, family,
> qualifier, ts) coordinate.
> >>
> >>
> >>
> >> Write (r1, f1, q1, v1, 1)
> >>
> >> Write (r1, f1, q1, v1, 2) - TTL=1 minute
> >>
> >>
> >> Scenario 1:
> >>
> >>
> >> If a major compaction happens within a minute
> >>
> >>
> >> it will remove (r1, f1, q1, v1, 1)
> >>
> >> then after a minute (r1, f1, q1, v1, 2) will expire
> >>
> >> no data left
> >>
> >>
> >> Scenario 2:
> >>
> >>
> >> A minute passes
> >>
> >> (r1, f1, q1, v1, 2) expires
> >>
> >> Compaction runs..
> >>
> >> (r1, f1, q1, v1, 1) remains
> >>
> >>
> >>
> >> This seems, by and large expected behavior, but it still seems
> "uncomfortable" that the (overall) outcome is not decided by me, but by a
> chance of event ordering.
> >>
> >>
> >> I wonder we'd want this to behave differently (perhaps it has been
> discussed already), but if not, it's worth a more detailed documentation in
> the book.
> >>
> >>
> >> What do you think?
> >>
> >>
> >> Cosmin
> >>
> >>
> >>
> >>
> >
> > --
> > Best regards,
> >
> > - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet
> > Hein (via Tom White)
> >
>
> The opinions expressed here are mine, while they may reflect a cognitive
> thought, that is purely accidental.
> Use at your own risk.
> Michael Segel
> michael_segel (AT) hotmail.com
>
>
>
>
>
>
Re: Nondeterministic outcome based on cell TTL and major compaction event order
Posted by Michael Segel <mi...@hotmail.com>.
Ok,
So then if you have a previous cell (c1) and you insert a new cell c2 that has a TTL of lets say 5 mins, then c1 should always exist?
That is my understanding but from Cosmin’s post, he’s saying its different. And that’s why I don’t understand. You couldn’t lose the cell c1 at all.
Compaction or no compaction.
That’s why I’m confused. Current behavior doesn’t match the expected contract.
-Mike
> On Apr 17, 2015, at 4:37 PM, Andrew Purtell <ap...@apache.org> wrote:
>
> The way TTLs work today is they define the interval of time a cell
> exists - exactly as that. There is no tombstone laid like a normal
> delete. Once the TTL elapses the cell just ceases to exist to normal
> scanners. The interaction of expired cells, multiple versions, minimum
> versions, raw scanners, etc. can be confusing. We can absolutely
> revisit this.
>
> A cell with an expired TTL could be treated as the combination of
> tombstone and the most recent value it lays over. This is not how the
> implementation works today, but could be changed for an upcoming major
> version like 2.0 if there's consensus to do it.
>
>
>> On Apr 10, 2015, at 7:26 AM, Cosmin Lehene <cl...@adobe.com> wrote:
>>
>> I've been initially puzzled by this, although I realize how it's likely as designed.
>>
>>
>> The cell TTL expiration and compactions events can lead to either some (the older) data left or no data at all for a particular (row, family, qualifier, ts) coordinate.
>>
>>
>>
>> Write (r1, f1, q1, v1, 1)
>>
>> Write (r1, f1, q1, v1, 2) - TTL=1 minute
>>
>>
>> Scenario 1:
>>
>>
>> If a major compaction happens within a minute
>>
>>
>> it will remove (r1, f1, q1, v1, 1)
>>
>> then after a minute (r1, f1, q1, v1, 2) will expire
>>
>> no data left
>>
>>
>> Scenario 2:
>>
>>
>> A minute passes
>>
>> (r1, f1, q1, v1, 2) expires
>>
>> Compaction runs..
>>
>> (r1, f1, q1, v1, 1) remains
>>
>>
>>
>> This seems, by and large expected behavior, but it still seems "uncomfortable" that the (overall) outcome is not decided by me, but by a chance of event ordering.
>>
>>
>> I wonder we'd want this to behave differently (perhaps it has been discussed already), but if not, it's worth a more detailed documentation in the book.
>>
>>
>> What do you think?
>>
>>
>> Cosmin
>>
>>
>>
>>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet
> Hein (via Tom White)
>
The opinions expressed here are mine, while they may reflect a cognitive thought, that is purely accidental.
Use at your own risk.
Michael Segel
michael_segel (AT) hotmail.com
Re: Nondeterministic outcome based on cell TTL and major compaction
event order
Posted by Andrew Purtell <ap...@apache.org>.
The way TTLs work today is they define the interval of time a cell
exists - exactly as that. There is no tombstone laid like a normal
delete. Once the TTL elapses the cell just ceases to exist to normal
scanners. The interaction of expired cells, multiple versions, minimum
versions, raw scanners, etc. can be confusing. We can absolutely
revisit this.
A cell with an expired TTL could be treated as the combination of
tombstone and the most recent value it lays over. This is not how the
implementation works today, but could be changed for an upcoming major
version like 2.0 if there's consensus to do it.
> On Apr 10, 2015, at 7:26 AM, Cosmin Lehene <cl...@adobe.com> wrote:
>
> I've been initially puzzled by this, although I realize how it's likely as designed.
>
>
> The cell TTL expiration and compactions events can lead to either some (the older) data left or no data at all for a particular (row, family, qualifier, ts) coordinate.
>
>
>
> Write (r1, f1, q1, v1, 1)
>
> Write (r1, f1, q1, v1, 2) - TTL=1 minute
>
>
> Scenario 1:
>
>
> If a major compaction happens within a minute
>
>
> it will remove (r1, f1, q1, v1, 1)
>
> then after a minute (r1, f1, q1, v1, 2) will expire
>
> no data left
>
>
> Scenario 2:
>
>
> A minute passes
>
> (r1, f1, q1, v1, 2) expires
>
> Compaction runs..
>
> (r1, f1, q1, v1, 1) remains
>
>
>
> This seems, by and large expected behavior, but it still seems "uncomfortable" that the (overall) outcome is not decided by me, but by a chance of event ordering.
>
>
> I wonder we'd want this to behave differently (perhaps it has been discussed already), but if not, it's worth a more detailed documentation in the book.
>
>
> What do you think?
>
>
> Cosmin
>
>
>
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet
Hein (via Tom White)
Re: Nondeterministic outcome based on cell TTL and major compaction event order
Posted by Michael Segel <mi...@hotmail.com>.
No, you need to count them. The exist therefore they count. ;-)
But that doesn’t answer the question about the expected behavior of TTL.
> On Apr 17, 2015, at 3:09 PM, Jean-Marc Spaggiari <je...@spaggiari.org> wrote:
>
> Maybe we should just not counts the cells with TTL when counting for the
> versions? That way, what ever we do we always keep the initial value?
>
> Better to have an extra version and have a consistent behaviour I think...
>
> JMS
> Le 2015-04-17 15:28, "Michael Segel" <mi...@hotmail.com> a écrit :
>
>> I think you misunderstood the question(s).
>>
>> How do you define the meaning of the TTL attribute?
>>
>> Is the TTL a special form of the delete tombstone?
>>
>> The reason I ask this is that if we look at your example you’ve stated
>> that if a compaction occurred prior to the end of life of cell 2, cell 1
>> would be destroyed.
>> That doesn’t make sense as an expected result.
>>
>> This is why I am asking what is the defined contract of the TTL feature.
>>
>> Cell 1 should never be deleted. (unless you explicitly state the number of
>> versions to be 1 then it would be marked for deletion when you insert cell
>> 2.)
>>
>> Thx
>>
>> -Mike
>>
>>
>>> On Apr 13, 2015, at 6:12 PM, Cosmin Lehene <cl...@adobe.com> wrote:
>>>
>>> The ambiguity seems to lie at the intersection of TTL and version
>> "garbage collection" during compactions.
>>>
>>> Major compactions can lead to nondeterministic results when multiple
>> versions are involved (slightly captured in the book
>> http://hbase.apache.org/book.html#versions and
>> http://www.ngdata.com/bending-time-in-hbase/ )
>>> TTL expirations don't result in deletes (at least not in the classical
>> sense with a tombstone).
>>>
>>> Cosmin
>>>
>>> _______________________________________
>>> From: Michael Segel <mi...@hotmail.com>
>>> Sent: Friday, April 10, 2015 8:35 AM
>>> To: dev@hbase.apache.org
>>> Subject: Re: Nondeterministic outcome based on cell TTL and major
>> compaction event order
>>>
>>> Interesting.
>>> There seems to be some ambiguity in what happens between a TTL and a
>> deletion.
>>>
>>> Is the TTL a delete or is it a separate type of function?
>>>
>>> That is to say when you inserted version 2 of the cell, did you intend
>> to just have version 2 exist for a little while and then default to version
>> 1 or did you mean that when you inserted version 2, you wanted to delete
>> everything prior to version 2 and then when version 2 expires, it then goes
>> away?
>>>
>>> The documentation isn’t clear on this point.
>>>
>>> To give you an example where you wouldn’t want to have the TTL on a cell
>> also delete prior versions…
>>>
>>> Suppose you’re storing map data in HBase. You have an attribute (speed)
>> associated to a road link.
>>>
>>> If the road is a 65 MPH highway, then the base speed (default speed) is
>> 65MPH. However if there’s construction planned for the road then you need
>> to reset the speed to 45 mph while there is construction. You know that
>> the construction is supposed to last X months, so you reset the speed limit
>> to 45 with a TTL on that cell version only.
>>>
>>> Another example is if you’re storing price for a given sku in a given
>> region of your retail chain. So you want to reduce the price by 20% for a
>> 2 week period.
>>> Again, you set that discount to live for 2 weeks with a TTL, then revert
>> back to original price.
>>>
>>> So I guess there should be a clarification as to what is intended for
>> the TTL to do?
>>>
>>> Does that make sense?
>>>
>>>
>>>
>>>
>>>
>>>> On Apr 10, 2015, at 9:26 AM, Cosmin Lehene <cl...@adobe.com> wrote:
>>>>
>>>> I've been initially puzzled by this, although I realize how it's likely
>> as designed.
>>>>
>>>>
>>>> The cell TTL expiration and compactions events can lead to either some
>> (the older) data left or no data at all for a particular (row, family,
>> qualifier, ts) coordinate.
>>>>
>>>>
>>>>
>>>> Write (r1, f1, q1, v1, 1)
>>>>
>>>> Write (r1, f1, q1, v1, 2) - TTL=1 minute
>>>>
>>>>
>>>> Scenario 1:
>>>>
>>>>
>>>> If a major compaction happens within a minute
>>>>
>>>>
>>>> it will remove (r1, f1, q1, v1, 1)
>>>>
>>>> then after a minute (r1, f1, q1, v1, 2) will expire
>>>>
>>>> no data left
>>>>
>>>>
>>>> Scenario 2:
>>>>
>>>>
>>>> A minute passes
>>>>
>>>> (r1, f1, q1, v1, 2) expires
>>>>
>>>> Compaction runs..
>>>>
>>>> (r1, f1, q1, v1, 1) remains
>>>>
>>>>
>>>>
>>>> This seems, by and large expected behavior, but it still seems
>> "uncomfortable" that the (overall) outcome is not decided by me, but by a
>> chance of event ordering.
>>>>
>>>>
>>>> I wonder we'd want this to behave differently (perhaps it has been
>> discussed already), but if not, it's worth a more detailed documentation in
>> the book.
>>>>
>>>>
>>>> What do you think?
>>>>
>>>>
>>>> Cosmin
>>>>
>>>>
>>>>
>>>>
>>>
>>> The opinions expressed here are mine, while they may reflect a cognitive
>> thought, that is purely accidental.
>>> Use at your own risk.
>>> Michael Segel
>>> michael_segel (AT) hotmail.com
>>>
>>>
>>>
>>>
>>>
>>>
>>
>> The opinions expressed here are mine, while they may reflect a cognitive
>> thought, that is purely accidental.
>> Use at your own risk.
>> Michael Segel
>> michael_segel (AT) hotmail.com
>>
>>
>>
>>
>>
>>
The opinions expressed here are mine, while they may reflect a cognitive thought, that is purely accidental.
Use at your own risk.
Michael Segel
michael_segel (AT) hotmail.com
Re: Nondeterministic outcome based on cell TTL and major compaction
event order
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Maybe we should just not counts the cells with TTL when counting for the
versions? That way, what ever we do we always keep the initial value?
Better to have an extra version and have a consistent behaviour I think...
JMS
Le 2015-04-17 15:28, "Michael Segel" <mi...@hotmail.com> a écrit :
> I think you misunderstood the question(s).
>
> How do you define the meaning of the TTL attribute?
>
> Is the TTL a special form of the delete tombstone?
>
> The reason I ask this is that if we look at your example you’ve stated
> that if a compaction occurred prior to the end of life of cell 2, cell 1
> would be destroyed.
> That doesn’t make sense as an expected result.
>
> This is why I am asking what is the defined contract of the TTL feature.
>
> Cell 1 should never be deleted. (unless you explicitly state the number of
> versions to be 1 then it would be marked for deletion when you insert cell
> 2.)
>
> Thx
>
> -Mike
>
>
> > On Apr 13, 2015, at 6:12 PM, Cosmin Lehene <cl...@adobe.com> wrote:
> >
> > The ambiguity seems to lie at the intersection of TTL and version
> "garbage collection" during compactions.
> >
> > Major compactions can lead to nondeterministic results when multiple
> versions are involved (slightly captured in the book
> http://hbase.apache.org/book.html#versions and
> http://www.ngdata.com/bending-time-in-hbase/ )
> > TTL expirations don't result in deletes (at least not in the classical
> sense with a tombstone).
> >
> > Cosmin
> >
> > _______________________________________
> > From: Michael Segel <mi...@hotmail.com>
> > Sent: Friday, April 10, 2015 8:35 AM
> > To: dev@hbase.apache.org
> > Subject: Re: Nondeterministic outcome based on cell TTL and major
> compaction event order
> >
> > Interesting.
> > There seems to be some ambiguity in what happens between a TTL and a
> deletion.
> >
> > Is the TTL a delete or is it a separate type of function?
> >
> > That is to say when you inserted version 2 of the cell, did you intend
> to just have version 2 exist for a little while and then default to version
> 1 or did you mean that when you inserted version 2, you wanted to delete
> everything prior to version 2 and then when version 2 expires, it then goes
> away?
> >
> > The documentation isn’t clear on this point.
> >
> > To give you an example where you wouldn’t want to have the TTL on a cell
> also delete prior versions…
> >
> > Suppose you’re storing map data in HBase. You have an attribute (speed)
> associated to a road link.
> >
> > If the road is a 65 MPH highway, then the base speed (default speed) is
> 65MPH. However if there’s construction planned for the road then you need
> to reset the speed to 45 mph while there is construction. You know that
> the construction is supposed to last X months, so you reset the speed limit
> to 45 with a TTL on that cell version only.
> >
> > Another example is if you’re storing price for a given sku in a given
> region of your retail chain. So you want to reduce the price by 20% for a
> 2 week period.
> > Again, you set that discount to live for 2 weeks with a TTL, then revert
> back to original price.
> >
> > So I guess there should be a clarification as to what is intended for
> the TTL to do?
> >
> > Does that make sense?
> >
> >
> >
> >
> >
> >> On Apr 10, 2015, at 9:26 AM, Cosmin Lehene <cl...@adobe.com> wrote:
> >>
> >> I've been initially puzzled by this, although I realize how it's likely
> as designed.
> >>
> >>
> >> The cell TTL expiration and compactions events can lead to either some
> (the older) data left or no data at all for a particular (row, family,
> qualifier, ts) coordinate.
> >>
> >>
> >>
> >> Write (r1, f1, q1, v1, 1)
> >>
> >> Write (r1, f1, q1, v1, 2) - TTL=1 minute
> >>
> >>
> >> Scenario 1:
> >>
> >>
> >> If a major compaction happens within a minute
> >>
> >>
> >> it will remove (r1, f1, q1, v1, 1)
> >>
> >> then after a minute (r1, f1, q1, v1, 2) will expire
> >>
> >> no data left
> >>
> >>
> >> Scenario 2:
> >>
> >>
> >> A minute passes
> >>
> >> (r1, f1, q1, v1, 2) expires
> >>
> >> Compaction runs..
> >>
> >> (r1, f1, q1, v1, 1) remains
> >>
> >>
> >>
> >> This seems, by and large expected behavior, but it still seems
> "uncomfortable" that the (overall) outcome is not decided by me, but by a
> chance of event ordering.
> >>
> >>
> >> I wonder we'd want this to behave differently (perhaps it has been
> discussed already), but if not, it's worth a more detailed documentation in
> the book.
> >>
> >>
> >> What do you think?
> >>
> >>
> >> Cosmin
> >>
> >>
> >>
> >>
> >
> > The opinions expressed here are mine, while they may reflect a cognitive
> thought, that is purely accidental.
> > Use at your own risk.
> > Michael Segel
> > michael_segel (AT) hotmail.com
> >
> >
> >
> >
> >
> >
>
> The opinions expressed here are mine, while they may reflect a cognitive
> thought, that is purely accidental.
> Use at your own risk.
> Michael Segel
> michael_segel (AT) hotmail.com
>
>
>
>
>
>
Re: Nondeterministic outcome based on cell TTL and major compaction event order
Posted by Michael Segel <mi...@hotmail.com>.
I think you misunderstood the question(s).
How do you define the meaning of the TTL attribute?
Is the TTL a special form of the delete tombstone?
The reason I ask this is that if we look at your example you’ve stated that if a compaction occurred prior to the end of life of cell 2, cell 1 would be destroyed.
That doesn’t make sense as an expected result.
This is why I am asking what is the defined contract of the TTL feature.
Cell 1 should never be deleted. (unless you explicitly state the number of versions to be 1 then it would be marked for deletion when you insert cell 2.)
Thx
-Mike
> On Apr 13, 2015, at 6:12 PM, Cosmin Lehene <cl...@adobe.com> wrote:
>
> The ambiguity seems to lie at the intersection of TTL and version "garbage collection" during compactions.
>
> Major compactions can lead to nondeterministic results when multiple versions are involved (slightly captured in the book http://hbase.apache.org/book.html#versions and http://www.ngdata.com/bending-time-in-hbase/ )
> TTL expirations don't result in deletes (at least not in the classical sense with a tombstone).
>
> Cosmin
>
> _______________________________________
> From: Michael Segel <mi...@hotmail.com>
> Sent: Friday, April 10, 2015 8:35 AM
> To: dev@hbase.apache.org
> Subject: Re: Nondeterministic outcome based on cell TTL and major compaction event order
>
> Interesting.
> There seems to be some ambiguity in what happens between a TTL and a deletion.
>
> Is the TTL a delete or is it a separate type of function?
>
> That is to say when you inserted version 2 of the cell, did you intend to just have version 2 exist for a little while and then default to version 1 or did you mean that when you inserted version 2, you wanted to delete everything prior to version 2 and then when version 2 expires, it then goes away?
>
> The documentation isn’t clear on this point.
>
> To give you an example where you wouldn’t want to have the TTL on a cell also delete prior versions…
>
> Suppose you’re storing map data in HBase. You have an attribute (speed) associated to a road link.
>
> If the road is a 65 MPH highway, then the base speed (default speed) is 65MPH. However if there’s construction planned for the road then you need to reset the speed to 45 mph while there is construction. You know that the construction is supposed to last X months, so you reset the speed limit to 45 with a TTL on that cell version only.
>
> Another example is if you’re storing price for a given sku in a given region of your retail chain. So you want to reduce the price by 20% for a 2 week period.
> Again, you set that discount to live for 2 weeks with a TTL, then revert back to original price.
>
> So I guess there should be a clarification as to what is intended for the TTL to do?
>
> Does that make sense?
>
>
>
>
>
>> On Apr 10, 2015, at 9:26 AM, Cosmin Lehene <cl...@adobe.com> wrote:
>>
>> I've been initially puzzled by this, although I realize how it's likely as designed.
>>
>>
>> The cell TTL expiration and compactions events can lead to either some (the older) data left or no data at all for a particular (row, family, qualifier, ts) coordinate.
>>
>>
>>
>> Write (r1, f1, q1, v1, 1)
>>
>> Write (r1, f1, q1, v1, 2) - TTL=1 minute
>>
>>
>> Scenario 1:
>>
>>
>> If a major compaction happens within a minute
>>
>>
>> it will remove (r1, f1, q1, v1, 1)
>>
>> then after a minute (r1, f1, q1, v1, 2) will expire
>>
>> no data left
>>
>>
>> Scenario 2:
>>
>>
>> A minute passes
>>
>> (r1, f1, q1, v1, 2) expires
>>
>> Compaction runs..
>>
>> (r1, f1, q1, v1, 1) remains
>>
>>
>>
>> This seems, by and large expected behavior, but it still seems "uncomfortable" that the (overall) outcome is not decided by me, but by a chance of event ordering.
>>
>>
>> I wonder we'd want this to behave differently (perhaps it has been discussed already), but if not, it's worth a more detailed documentation in the book.
>>
>>
>> What do you think?
>>
>>
>> Cosmin
>>
>>
>>
>>
>
> The opinions expressed here are mine, while they may reflect a cognitive thought, that is purely accidental.
> Use at your own risk.
> Michael Segel
> michael_segel (AT) hotmail.com
>
>
>
>
>
>
The opinions expressed here are mine, while they may reflect a cognitive thought, that is purely accidental.
Use at your own risk.
Michael Segel
michael_segel (AT) hotmail.com
Re: Nondeterministic outcome based on cell TTL and major compaction
event order
Posted by Cosmin Lehene <cl...@adobe.com>.
The ambiguity seems to lie at the intersection of TTL and version "garbage collection" during compactions.
Major compactions can lead to nondeterministic results when multiple versions are involved (slightly captured in the book http://hbase.apache.org/book.html#versions and http://www.ngdata.com/bending-time-in-hbase/ )
TTL expirations don't result in deletes (at least not in the classical sense with a tombstone).
Cosmin
_______________________________________
From: Michael Segel <mi...@hotmail.com>
Sent: Friday, April 10, 2015 8:35 AM
To: dev@hbase.apache.org
Subject: Re: Nondeterministic outcome based on cell TTL and major compaction event order
Interesting.
There seems to be some ambiguity in what happens between a TTL and a deletion.
Is the TTL a delete or is it a separate type of function?
That is to say when you inserted version 2 of the cell, did you intend to just have version 2 exist for a little while and then default to version 1 or did you mean that when you inserted version 2, you wanted to delete everything prior to version 2 and then when version 2 expires, it then goes away?
The documentation isn’t clear on this point.
To give you an example where you wouldn’t want to have the TTL on a cell also delete prior versions…
Suppose you’re storing map data in HBase. You have an attribute (speed) associated to a road link.
If the road is a 65 MPH highway, then the base speed (default speed) is 65MPH. However if there’s construction planned for the road then you need to reset the speed to 45 mph while there is construction. You know that the construction is supposed to last X months, so you reset the speed limit to 45 with a TTL on that cell version only.
Another example is if you’re storing price for a given sku in a given region of your retail chain. So you want to reduce the price by 20% for a 2 week period.
Again, you set that discount to live for 2 weeks with a TTL, then revert back to original price.
So I guess there should be a clarification as to what is intended for the TTL to do?
Does that make sense?
> On Apr 10, 2015, at 9:26 AM, Cosmin Lehene <cl...@adobe.com> wrote:
>
> I've been initially puzzled by this, although I realize how it's likely as designed.
>
>
> The cell TTL expiration and compactions events can lead to either some (the older) data left or no data at all for a particular (row, family, qualifier, ts) coordinate.
>
>
>
> Write (r1, f1, q1, v1, 1)
>
> Write (r1, f1, q1, v1, 2) - TTL=1 minute
>
>
> Scenario 1:
>
>
> If a major compaction happens within a minute
>
>
> it will remove (r1, f1, q1, v1, 1)
>
> then after a minute (r1, f1, q1, v1, 2) will expire
>
> no data left
>
>
> Scenario 2:
>
>
> A minute passes
>
> (r1, f1, q1, v1, 2) expires
>
> Compaction runs..
>
> (r1, f1, q1, v1, 1) remains
>
>
>
> This seems, by and large expected behavior, but it still seems "uncomfortable" that the (overall) outcome is not decided by me, but by a chance of event ordering.
>
>
> I wonder we'd want this to behave differently (perhaps it has been discussed already), but if not, it's worth a more detailed documentation in the book.
>
>
> What do you think?
>
>
> Cosmin
>
>
>
>
The opinions expressed here are mine, while they may reflect a cognitive thought, that is purely accidental.
Use at your own risk.
Michael Segel
michael_segel (AT) hotmail.com
Re: Nondeterministic outcome based on cell TTL and major compaction event order
Posted by Michael Segel <mi...@hotmail.com>.
Interesting.
There seems to be some ambiguity in what happens between a TTL and a deletion.
Is the TTL a delete or is it a separate type of function?
That is to say when you inserted version 2 of the cell, did you intend to just have version 2 exist for a little while and then default to version 1 or did you mean that when you inserted version 2, you wanted to delete everything prior to version 2 and then when version 2 expires, it then goes away?
The documentation isn’t clear on this point.
To give you an example where you wouldn’t want to have the TTL on a cell also delete prior versions…
Suppose you’re storing map data in HBase. You have an attribute (speed) associated to a road link.
If the road is a 65 MPH highway, then the base speed (default speed) is 65MPH. However if there’s construction planned for the road then you need to reset the speed to 45 mph while there is construction. You know that the construction is supposed to last X months, so you reset the speed limit to 45 with a TTL on that cell version only.
Another example is if you’re storing price for a given sku in a given region of your retail chain. So you want to reduce the price by 20% for a 2 week period.
Again, you set that discount to live for 2 weeks with a TTL, then revert back to original price.
So I guess there should be a clarification as to what is intended for the TTL to do?
Does that make sense?
> On Apr 10, 2015, at 9:26 AM, Cosmin Lehene <cl...@adobe.com> wrote:
>
> I've been initially puzzled by this, although I realize how it's likely as designed.
>
>
> The cell TTL expiration and compactions events can lead to either some (the older) data left or no data at all for a particular (row, family, qualifier, ts) coordinate.
>
>
>
> Write (r1, f1, q1, v1, 1)
>
> Write (r1, f1, q1, v1, 2) - TTL=1 minute
>
>
> Scenario 1:
>
>
> If a major compaction happens within a minute
>
>
> it will remove (r1, f1, q1, v1, 1)
>
> then after a minute (r1, f1, q1, v1, 2) will expire
>
> no data left
>
>
> Scenario 2:
>
>
> A minute passes
>
> (r1, f1, q1, v1, 2) expires
>
> Compaction runs..
>
> (r1, f1, q1, v1, 1) remains
>
>
>
> This seems, by and large expected behavior, but it still seems "uncomfortable" that the (overall) outcome is not decided by me, but by a chance of event ordering.
>
>
> I wonder we'd want this to behave differently (perhaps it has been discussed already), but if not, it's worth a more detailed documentation in the book.
>
>
> What do you think?
>
>
> Cosmin
>
>
>
>
The opinions expressed here are mine, while they may reflect a cognitive thought, that is purely accidental.
Use at your own risk.
Michael Segel
michael_segel (AT) hotmail.com
Re: Nondeterministic outcome based on cell TTL and major compaction
event order
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Hum. Good catch. Maybe when we expire the cell when need to check
configured number of versions and place a delete tombstone to replace the
expired cell? That works when version = 1. But what when version is
greater? I mean, what should be a correct behavior?
JM
Le 2015-04-10 07:27, "Cosmin Lehene" <cl...@adobe.com> a écrit :
> I've been initially puzzled by this, although I realize how it's likely as
> designed.
>
>
> The cell TTL expiration and compactions events can lead to either some
> (the older) data left or no data at all for a particular (row, family,
> qualifier, ts) coordinate.
>
>
>
> Write (r1, f1, q1, v1, 1)
>
> Write (r1, f1, q1, v1, 2) - TTL=1 minute
>
>
> Scenario 1:
>
>
> If a major compaction happens within a minute
>
>
> it will remove (r1, f1, q1, v1, 1)
>
> then after a minute (r1, f1, q1, v1, 2) will expire
>
> no data left
>
>
> Scenario 2:
>
>
> A minute passes
>
> (r1, f1, q1, v1, 2) expires
>
> Compaction runs..
>
> (r1, f1, q1, v1, 1) remains
>
>
>
> This seems, by and large expected behavior, but it still seems
> "uncomfortable" that the (overall) outcome is not decided by me, but by a
> chance of event ordering.
>
>
> I wonder we'd want this to behave differently (perhaps it has been
> discussed already), but if not, it's worth a more detailed documentation in
> the book.
>
>
> What do you think?
>
>
> Cosmin
>
>
>
>
>