You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Sreeram <sr...@gmail.com> on 2016/03/18 10:33:23 UTC
How to implement increment in an idempotent manner
Hi,
I am looking for suggestions from community on implementing HBase
increment in a idempotent manner.
My use case is a storm Hbase bolt that atomically increments a HBase
counter. Replay of the storm bolt results in a double increment.
Any suggestion on the approach to be taken is welcome.
Thank you.
Regards,
Sreeram
Re: How to implement increment in an idempotent manner
Posted by Sreeram <sr...@gmail.com>.
All my Hbase processing is going to be inside any given bolt. So I cannot
use the second option.
On Fri, Mar 18, 2016 at 3:46 PM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:
> What's about the other option where each bolt increment it's own column and
> at theend ou aggregate those few columns together?
>
> 2016-03-18 6:14 GMT-04:00 Sreeram <sr...@gmail.com>:
>
> > The incremented field is more like an amount field that will be storing
> the
> > aggregate amount. Since the field will be incremented concurrently by
> > multiple bolts running in parallel, storing the value before increment
> and
> > then doing a put in case of replay will not help.
> >
> > Reason to have this field is to pre-compute a certain aggregate amount
> and
> > materialize it in the Hbase table.
> >
> > On Fri, Mar 18, 2016 at 3:30 PM, Jean-Marc Spaggiari <
> > jean-marc@spaggiari.org> wrote:
> >
> > > At the beginning of your Storm bolt process can you not do a put of
> "0"?
> > So
> > > it start back from scratch? Or else you will need to query the value,
> and
> > > keep the value to put it back if you need to replay your bolt
> > >
> > > Other option is, you increment a specific difference column and at the
> > end
> > > if you are successful with your bolt you increment te initial column
> with
> > > the new total counter?
> > >
> > > JMS
> > >
> > > 2016-03-18 5:33 GMT-04:00 Sreeram <sr...@gmail.com>:
> > >
> > > > Hi,
> > > >
> > > > I am looking for suggestions from community on implementing HBase
> > > > increment in a idempotent manner.
> > > >
> > > > My use case is a storm Hbase bolt that atomically increments a HBase
> > > > counter. Replay of the storm bolt results in a double increment.
> > > >
> > > > Any suggestion on the approach to be taken is welcome.
> > > >
> > > > Thank you.
> > > >
> > > > Regards,
> > > > Sreeram
> > > >
> > >
> >
>
Re: How to implement increment in an idempotent manner
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
What's about the other option where each bolt increment it's own column and
at theend ou aggregate those few columns together?
2016-03-18 6:14 GMT-04:00 Sreeram <sr...@gmail.com>:
> The incremented field is more like an amount field that will be storing the
> aggregate amount. Since the field will be incremented concurrently by
> multiple bolts running in parallel, storing the value before increment and
> then doing a put in case of replay will not help.
>
> Reason to have this field is to pre-compute a certain aggregate amount and
> materialize it in the Hbase table.
>
> On Fri, Mar 18, 2016 at 3:30 PM, Jean-Marc Spaggiari <
> jean-marc@spaggiari.org> wrote:
>
> > At the beginning of your Storm bolt process can you not do a put of "0"?
> So
> > it start back from scratch? Or else you will need to query the value, and
> > keep the value to put it back if you need to replay your bolt
> >
> > Other option is, you increment a specific difference column and at the
> end
> > if you are successful with your bolt you increment te initial column with
> > the new total counter?
> >
> > JMS
> >
> > 2016-03-18 5:33 GMT-04:00 Sreeram <sr...@gmail.com>:
> >
> > > Hi,
> > >
> > > I am looking for suggestions from community on implementing HBase
> > > increment in a idempotent manner.
> > >
> > > My use case is a storm Hbase bolt that atomically increments a HBase
> > > counter. Replay of the storm bolt results in a double increment.
> > >
> > > Any suggestion on the approach to be taken is welcome.
> > >
> > > Thank you.
> > >
> > > Regards,
> > > Sreeram
> > >
> >
>
Re: How to implement increment in an idempotent manner
Posted by Sreeram <sr...@gmail.com>.
The incremented field is more like an amount field that will be storing the
aggregate amount. Since the field will be incremented concurrently by
multiple bolts running in parallel, storing the value before increment and
then doing a put in case of replay will not help.
Reason to have this field is to pre-compute a certain aggregate amount and
materialize it in the Hbase table.
On Fri, Mar 18, 2016 at 3:30 PM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:
> At the beginning of your Storm bolt process can you not do a put of "0"? So
> it start back from scratch? Or else you will need to query the value, and
> keep the value to put it back if you need to replay your bolt
>
> Other option is, you increment a specific difference column and at the end
> if you are successful with your bolt you increment te initial column with
> the new total counter?
>
> JMS
>
> 2016-03-18 5:33 GMT-04:00 Sreeram <sr...@gmail.com>:
>
> > Hi,
> >
> > I am looking for suggestions from community on implementing HBase
> > increment in a idempotent manner.
> >
> > My use case is a storm Hbase bolt that atomically increments a HBase
> > counter. Replay of the storm bolt results in a double increment.
> >
> > Any suggestion on the approach to be taken is welcome.
> >
> > Thank you.
> >
> > Regards,
> > Sreeram
> >
>
Re: How to implement increment in an idempotent manner
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
At the beginning of your Storm bolt process can you not do a put of "0"? So
it start back from scratch? Or else you will need to query the value, and
keep the value to put it back if you need to replay your bolt
Other option is, you increment a specific difference column and at the end
if you are successful with your bolt you increment te initial column with
the new total counter?
JMS
2016-03-18 5:33 GMT-04:00 Sreeram <sr...@gmail.com>:
> Hi,
>
> I am looking for suggestions from community on implementing HBase
> increment in a idempotent manner.
>
> My use case is a storm Hbase bolt that atomically increments a HBase
> counter. Replay of the storm bolt results in a double increment.
>
> Any suggestion on the approach to be taken is welcome.
>
> Thank you.
>
> Regards,
> Sreeram
>