You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Niels Basjes <Ni...@basjes.nl> on 2013/02/27 14:47:09 UTC
Generic increments?
Hi,
Last year at a meetup I spoke with Lars George about the counters in hbase.
What I understood is that the counters are stored as increments (i.e.
increment without locking) and during compaction and querying a the
increments are aggregated into the actual value.
So far I've examined the API and this seems to work as long as the value is
a long.
Now incrementing longs is nice but I would like to do things like
- Calculating min, max
- Bloomfilters
- Average ( recording both the "count" and "sum" )
- Variance and Standard Deviation ( using
http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Parallel_algorithm)
All of those need more bytes of internal storage and need custom code for
storing, aggregating and querying.
Especially querying because perhaps I can ask several different questions
to a single byte[].
If I store both the count and the sum in a single byte[] then I can ask
getN(), getSum(), getAvg()
Now my question to you guys is how I can implement such a more generic form
of "lock free increments" with user defined setters, getters and a custom
aggregator (used for both compacting and querying).
Perhaps there is an example on how to do this?
--
Best regards / Met vriendelijke groeten,
Niels Basjes
Re: Generic increments?
Posted by Niels Basjes <Ni...@basjes.nl>.
Thanks.
On Mon, Mar 4, 2013 at 8:37 PM, Nick Dimiduk <nd...@gmail.com> wrote:
> Hi Niels,
>
> As Lars said, I would start by reading the code for Increment and Append.
> HRegion#append(Append, Boolean) should be interesting for you.
>
> -n
>
> On Thu, Feb 28, 2013 at 1:22 AM, Niels Basjes <Ni...@basjes.nl> wrote:
>
> > Do you have some suggestions on where i should start? Perhaps someone has
> > already created some rough design on how this can be done.
> >
> >
>
> > > We do not have such a facility as far as I know.
> > > We have Increment/Append, and these work by locking the row, retrieving
> > > the old value, storing the updated value, unlocking the row.
> > >
> > > -- Lars
> > >
> > >
> > >
> > > ________________________________
> > > From: Niels Basjes <Ni...@basjes.nl>
> > > To: dev@hbase.apache.org
> > > Sent: Wednesday, February 27, 2013 5:47 AM
> > > Subject: Generic increments?
> > >
> > > Hi,
> > >
> > > Last year at a meetup I spoke with Lars George about the counters in
> > hbase.
> > > What I understood is that the counters are stored as increments (i.e.
> > > increment without locking) and during compaction and querying a the
> > > increments are aggregated into the actual value.
> > >
> > > So far I've examined the API and this seems to work as long as the
> value
> > is
> > > a long.
> > >
> > > Now incrementing longs is nice but I would like to do things like
> > > - Calculating min, max
> > > - Bloomfilters
> > > - Average ( recording both the "count" and "sum" )
> > > - Variance and Standard Deviation ( using
> > >
> > >
> >
> http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Parallel_algorithm
> > > )
> > >
> > > All of those need more bytes of internal storage and need custom code
> for
> > > storing, aggregating and querying.
> > > Especially querying because perhaps I can ask several different
> questions
> > > to a single byte[].
> > > If I store both the count and the sum in a single byte[] then I can ask
> > > getN(), getSum(), getAvg()
> > >
> > > Now my question to you guys is how I can implement such a more generic
> > form
> > > of "lock free increments" with user defined setters, getters and a
> custom
> > > aggregator (used for both compacting and querying).
> > > Perhaps there is an example on how to do this?
> > >
> > > --
> > > Best regards / Met vriendelijke groeten,
> > >
> > > Niels Basjes
> >
>
--
Best regards / Met vriendelijke groeten,
Niels Basjes
Re: Generic increments?
Posted by Nick Dimiduk <nd...@gmail.com>.
Hi Niels,
As Lars said, I would start by reading the code for Increment and Append.
HRegion#append(Append, Boolean) should be interesting for you.
-n
On Thu, Feb 28, 2013 at 1:22 AM, Niels Basjes <Ni...@basjes.nl> wrote:
> Do you have some suggestions on where i should start? Perhaps someone has
> already created some rough design on how this can be done.
>
>
> > We do not have such a facility as far as I know.
> > We have Increment/Append, and these work by locking the row, retrieving
> > the old value, storing the updated value, unlocking the row.
> >
> > -- Lars
> >
> >
> >
> > ________________________________
> > From: Niels Basjes <Ni...@basjes.nl>
> > To: dev@hbase.apache.org
> > Sent: Wednesday, February 27, 2013 5:47 AM
> > Subject: Generic increments?
> >
> > Hi,
> >
> > Last year at a meetup I spoke with Lars George about the counters in
> hbase.
> > What I understood is that the counters are stored as increments (i.e.
> > increment without locking) and during compaction and querying a the
> > increments are aggregated into the actual value.
> >
> > So far I've examined the API and this seems to work as long as the value
> is
> > a long.
> >
> > Now incrementing longs is nice but I would like to do things like
> > - Calculating min, max
> > - Bloomfilters
> > - Average ( recording both the "count" and "sum" )
> > - Variance and Standard Deviation ( using
> >
> >
> http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Parallel_algorithm
> > )
> >
> > All of those need more bytes of internal storage and need custom code for
> > storing, aggregating and querying.
> > Especially querying because perhaps I can ask several different questions
> > to a single byte[].
> > If I store both the count and the sum in a single byte[] then I can ask
> > getN(), getSum(), getAvg()
> >
> > Now my question to you guys is how I can implement such a more generic
> form
> > of "lock free increments" with user defined setters, getters and a custom
> > aggregator (used for both compacting and querying).
> > Perhaps there is an example on how to do this?
> >
> > --
> > Best regards / Met vriendelijke groeten,
> >
> > Niels Basjes
>
Re: Generic increments?
Posted by Niels Basjes <Ni...@basjes.nl>.
Hi,
Just to be sure i understand you correctly: did i describe the workings of
the counters correctly? If not; was Lars perhaps referring to something he
was working on? Or just a good idea he was thinking of.
I'm willing to give this (generic) form a stab to see how far i can get.
Probably by extending the core hbase and/or by writing external code
(coprocessors?).
Do you have some suggestions on where i should start? Perhaps someone has
already created some rough design on how this can be done.
Thanks.
--
Met vriendelijke groet,
Niels Basjes
(Verstuurd vanaf mobiel )
Op 28 feb. 2013 05:30 schreef "lars hofhansl" <la...@apache.org> het
volgende:
> We do not have such a facility as far as I know.
> We have Increment/Append, and these work by locking the row, retrieving
> the old value, storing the updated value, unlocking the row.
>
> -- Lars
>
>
>
> ________________________________
> From: Niels Basjes <Ni...@basjes.nl>
> To: dev@hbase.apache.org
> Sent: Wednesday, February 27, 2013 5:47 AM
> Subject: Generic increments?
>
> Hi,
>
> Last year at a meetup I spoke with Lars George about the counters in hbase.
> What I understood is that the counters are stored as increments (i.e.
> increment without locking) and during compaction and querying a the
> increments are aggregated into the actual value.
>
> So far I've examined the API and this seems to work as long as the value is
> a long.
>
> Now incrementing longs is nice but I would like to do things like
> - Calculating min, max
> - Bloomfilters
> - Average ( recording both the "count" and "sum" )
> - Variance and Standard Deviation ( using
>
> http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Parallel_algorithm
> )
>
> All of those need more bytes of internal storage and need custom code for
> storing, aggregating and querying.
> Especially querying because perhaps I can ask several different questions
> to a single byte[].
> If I store both the count and the sum in a single byte[] then I can ask
> getN(), getSum(), getAvg()
>
> Now my question to you guys is how I can implement such a more generic form
> of "lock free increments" with user defined setters, getters and a custom
> aggregator (used for both compacting and querying).
> Perhaps there is an example on how to do this?
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
Re: Generic increments?
Posted by lars hofhansl <la...@apache.org>.
We do not have such a facility as far as I know.
We have Increment/Append, and these work by locking the row, retrieving the old value, storing the updated value, unlocking the row.
-- Lars
________________________________
From: Niels Basjes <Ni...@basjes.nl>
To: dev@hbase.apache.org
Sent: Wednesday, February 27, 2013 5:47 AM
Subject: Generic increments?
Hi,
Last year at a meetup I spoke with Lars George about the counters in hbase.
What I understood is that the counters are stored as increments (i.e.
increment without locking) and during compaction and querying a the
increments are aggregated into the actual value.
So far I've examined the API and this seems to work as long as the value is
a long.
Now incrementing longs is nice but I would like to do things like
- Calculating min, max
- Bloomfilters
- Average ( recording both the "count" and "sum" )
- Variance and Standard Deviation ( using
http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Parallel_algorithm)
All of those need more bytes of internal storage and need custom code for
storing, aggregating and querying.
Especially querying because perhaps I can ask several different questions
to a single byte[].
If I store both the count and the sum in a single byte[] then I can ask
getN(), getSum(), getAvg()
Now my question to you guys is how I can implement such a more generic form
of "lock free increments" with user defined setters, getters and a custom
aggregator (used for both compacting and querying).
Perhaps there is an example on how to do this?
--
Best regards / Met vriendelijke groeten,
Niels Basjes