You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Manjeet Singh <ma...@gmail.com> on 2016/11/02 17:57:04 UTC
How can I achieve HBase row level atomicity?
Hi All
I have ETL process for inserting data into hbase for this I have spark jobs
which are responsible for reading data from kafka topics. so my question is
if I have some rowkey already exist in hbase and I have 3 spark job running
and they all try to update on same rowkey how hbase deal for atomicity?
for more understanding i have 3 rowkey coming from 3 seprate spark job and
all trying to update same rowkey which is already exist in hbase table.
[image: Inline image 1]
--
luv all
Re: How can I achieve HBase row level atomicity?
Posted by Anoop John <an...@gmail.com>.
You can use increment API in Table then? Within RS it will do a get
and then do + and put the new Cell. All under rowlock so that
atomicity is guaranteed.
And when u call the API for the 1st time (No existing row) the
increment value as such (y) will get added. I believe this will make
things easy for u. Pls try. Let us know any further doubts/issues.
-Anoop-
On Tue, Nov 8, 2016 at 5:55 AM, Manjeet Singh
<ma...@gmail.com> wrote:
> Increment by y x+y
>
> On 7 Nov 2016 23:52, "Anoop John" <an...@gmail.com> wrote:
>
>> So what is the update op doing? Say the cur available value in table
>> is 'x' and incoming data from ETL is 'y' what is the updated value u
>> try put? Is it an increment by 'y"? Or some complex logic behind?
>>
>> -Anoop-
>>
>> On Mon, Nov 7, 2016 at 11:09 PM, Manjeet Singh
>> <ma...@gmail.com> wrote:
>> > Yes Anoop you are right
>> > my input source is kafka pipeline we have 7 spark ETL jobs which are
>> > responsible for aggregation, and to get last value from Hbase and put the
>> > updated one.
>> > we set zero version so we always have only one copy in Hbase.
>> > Problem is if my 2 different ETL job having the same rowkey and one ETL
>> > process get the value and at same time second also get that value and
>> > update it now first ETL job will replace that updated value.
>> >
>> > it can happen in same ETL job too.
>> >
>> > Thanks
>> > Manjeet
>> >
>> > On Mon, Nov 7, 2016 at 7:07 PM, Anoop John <an...@gmail.com>
>> wrote:
>> >
>> >> Seems u want to get an already present row and do some op and put the
>> >> updated value. What is that op? If u can explain those we can try
>> >> help u with ways (if available) As such, the above kind of API do not
>> >> guarantee u any atomicity.
>> >>
>> >> -Anoop-
>> >>
>> >> On Fri, Nov 4, 2016 at 4:12 AM, Ted Yu <yu...@gmail.com> wrote:
>> >> > bq. from some api its going for update (means get is performed)
>> >> >
>> >> > Update on hbase would correlate with Put or Delete (not sure what
>> 'get'
>> >> > above means).
>> >> >
>> >> > Looks like your concern is that two concurrent updates may overwrite
>> the
>> >> > data for same rowkey.
>> >> >
>> >> > Have you considered using:
>> >> >
>> >> > public boolean checkAndPut(final byte [] row,
>> >> >
>> >> > final byte [] family, final byte [] qualifier, final byte []
>> value,
>> >> >
>> >> > final Put put)
>> >> >
>> >> > Cheers
>> >> >
>> >> > On Thu, Nov 3, 2016 at 10:57 AM, Manjeet Singh <
>> >> manjeet.chandhok@gmail.com>
>> >> > wrote:
>> >> >
>> >> >> Hi Ted,
>> >> >>
>> >> >> code not required for this case and how MVCC can help?
>> >> >>
>> >> >> question is if I have record which come in hbase and from some api
>> its
>> >> >> going for update (means get is performed)
>> >> >> now other operation also performed get operation both update on same
>> >> rwokey
>> >> >> at end we will not have correct value.
>> >> >>
>> >> >> in seprate mail thread i asked same type of question regarding rowkey
>> >> lock,
>> >> >> but still i did't get correct anser
>> >> >>
>> >> >> Thanks
>> >> >> Manjeet
>> >> >>
>> >> >> On Wed, Nov 2, 2016 at 11:36 PM, Ted Yu <yu...@gmail.com> wrote:
>> >> >>
>> >> >> > Were you including code in the image (which didn't come through) ?
>> >> >> >
>> >> >> > MultiVersionConcurrencyControl is involved in answering your
>> question.
>> >> >> >
>> >> >> > See http://hbase.apache.org/book.html#hregion.scans
>> >> >> >
>> >> >> > Cheers
>> >> >> >
>> >> >> > On Wed, Nov 2, 2016 at 10:57 AM, Manjeet Singh <
>> >> >> manjeet.chandhok@gmail.com
>> >> >> > >
>> >> >> > wrote:
>> >> >> >
>> >> >> > > Hi All
>> >> >> > >
>> >> >> > > I have ETL process for inserting data into hbase for this I have
>> >> spark
>> >> >> > > jobs which are responsible for reading data from kafka topics.
>> so my
>> >> >> > > question is if I have some rowkey already exist in hbase and I
>> have
>> >> 3
>> >> >> > spark
>> >> >> > > job running and they all try to update on same rowkey how hbase
>> deal
>> >> >> for
>> >> >> > > atomicity?
>> >> >> > >
>> >> >> > > for more understanding i have 3 rowkey coming from 3 seprate
>> spark
>> >> job
>> >> >> > and
>> >> >> > > all trying to update same rowkey which is already exist in hbase
>> >> table.
>> >> >> > >
>> >> >> > > [image: Inline image 1]
>> >> >> > >
>> >> >> > > --
>> >> >> > > luv all
>> >> >> > >
>> >> >> >
>> >> >>
>> >> >>
>> >> >>
>> >> >> --
>> >> >> luv all
>> >> >>
>> >>
>> >
>> >
>> >
>> > --
>> > luv all
>>
Re: How can I achieve HBase row level atomicity?
Posted by Manjeet Singh <ma...@gmail.com>.
Increment by y x+y
On 7 Nov 2016 23:52, "Anoop John" <an...@gmail.com> wrote:
> So what is the update op doing? Say the cur available value in table
> is 'x' and incoming data from ETL is 'y' what is the updated value u
> try put? Is it an increment by 'y"? Or some complex logic behind?
>
> -Anoop-
>
> On Mon, Nov 7, 2016 at 11:09 PM, Manjeet Singh
> <ma...@gmail.com> wrote:
> > Yes Anoop you are right
> > my input source is kafka pipeline we have 7 spark ETL jobs which are
> > responsible for aggregation, and to get last value from Hbase and put the
> > updated one.
> > we set zero version so we always have only one copy in Hbase.
> > Problem is if my 2 different ETL job having the same rowkey and one ETL
> > process get the value and at same time second also get that value and
> > update it now first ETL job will replace that updated value.
> >
> > it can happen in same ETL job too.
> >
> > Thanks
> > Manjeet
> >
> > On Mon, Nov 7, 2016 at 7:07 PM, Anoop John <an...@gmail.com>
> wrote:
> >
> >> Seems u want to get an already present row and do some op and put the
> >> updated value. What is that op? If u can explain those we can try
> >> help u with ways (if available) As such, the above kind of API do not
> >> guarantee u any atomicity.
> >>
> >> -Anoop-
> >>
> >> On Fri, Nov 4, 2016 at 4:12 AM, Ted Yu <yu...@gmail.com> wrote:
> >> > bq. from some api its going for update (means get is performed)
> >> >
> >> > Update on hbase would correlate with Put or Delete (not sure what
> 'get'
> >> > above means).
> >> >
> >> > Looks like your concern is that two concurrent updates may overwrite
> the
> >> > data for same rowkey.
> >> >
> >> > Have you considered using:
> >> >
> >> > public boolean checkAndPut(final byte [] row,
> >> >
> >> > final byte [] family, final byte [] qualifier, final byte []
> value,
> >> >
> >> > final Put put)
> >> >
> >> > Cheers
> >> >
> >> > On Thu, Nov 3, 2016 at 10:57 AM, Manjeet Singh <
> >> manjeet.chandhok@gmail.com>
> >> > wrote:
> >> >
> >> >> Hi Ted,
> >> >>
> >> >> code not required for this case and how MVCC can help?
> >> >>
> >> >> question is if I have record which come in hbase and from some api
> its
> >> >> going for update (means get is performed)
> >> >> now other operation also performed get operation both update on same
> >> rwokey
> >> >> at end we will not have correct value.
> >> >>
> >> >> in seprate mail thread i asked same type of question regarding rowkey
> >> lock,
> >> >> but still i did't get correct anser
> >> >>
> >> >> Thanks
> >> >> Manjeet
> >> >>
> >> >> On Wed, Nov 2, 2016 at 11:36 PM, Ted Yu <yu...@gmail.com> wrote:
> >> >>
> >> >> > Were you including code in the image (which didn't come through) ?
> >> >> >
> >> >> > MultiVersionConcurrencyControl is involved in answering your
> question.
> >> >> >
> >> >> > See http://hbase.apache.org/book.html#hregion.scans
> >> >> >
> >> >> > Cheers
> >> >> >
> >> >> > On Wed, Nov 2, 2016 at 10:57 AM, Manjeet Singh <
> >> >> manjeet.chandhok@gmail.com
> >> >> > >
> >> >> > wrote:
> >> >> >
> >> >> > > Hi All
> >> >> > >
> >> >> > > I have ETL process for inserting data into hbase for this I have
> >> spark
> >> >> > > jobs which are responsible for reading data from kafka topics.
> so my
> >> >> > > question is if I have some rowkey already exist in hbase and I
> have
> >> 3
> >> >> > spark
> >> >> > > job running and they all try to update on same rowkey how hbase
> deal
> >> >> for
> >> >> > > atomicity?
> >> >> > >
> >> >> > > for more understanding i have 3 rowkey coming from 3 seprate
> spark
> >> job
> >> >> > and
> >> >> > > all trying to update same rowkey which is already exist in hbase
> >> table.
> >> >> > >
> >> >> > > [image: Inline image 1]
> >> >> > >
> >> >> > > --
> >> >> > > luv all
> >> >> > >
> >> >> >
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> luv all
> >> >>
> >>
> >
> >
> >
> > --
> > luv all
>
Re: How can I achieve HBase row level atomicity?
Posted by Anoop John <an...@gmail.com>.
So what is the update op doing? Say the cur available value in table
is 'x' and incoming data from ETL is 'y' what is the updated value u
try put? Is it an increment by 'y"? Or some complex logic behind?
-Anoop-
On Mon, Nov 7, 2016 at 11:09 PM, Manjeet Singh
<ma...@gmail.com> wrote:
> Yes Anoop you are right
> my input source is kafka pipeline we have 7 spark ETL jobs which are
> responsible for aggregation, and to get last value from Hbase and put the
> updated one.
> we set zero version so we always have only one copy in Hbase.
> Problem is if my 2 different ETL job having the same rowkey and one ETL
> process get the value and at same time second also get that value and
> update it now first ETL job will replace that updated value.
>
> it can happen in same ETL job too.
>
> Thanks
> Manjeet
>
> On Mon, Nov 7, 2016 at 7:07 PM, Anoop John <an...@gmail.com> wrote:
>
>> Seems u want to get an already present row and do some op and put the
>> updated value. What is that op? If u can explain those we can try
>> help u with ways (if available) As such, the above kind of API do not
>> guarantee u any atomicity.
>>
>> -Anoop-
>>
>> On Fri, Nov 4, 2016 at 4:12 AM, Ted Yu <yu...@gmail.com> wrote:
>> > bq. from some api its going for update (means get is performed)
>> >
>> > Update on hbase would correlate with Put or Delete (not sure what 'get'
>> > above means).
>> >
>> > Looks like your concern is that two concurrent updates may overwrite the
>> > data for same rowkey.
>> >
>> > Have you considered using:
>> >
>> > public boolean checkAndPut(final byte [] row,
>> >
>> > final byte [] family, final byte [] qualifier, final byte [] value,
>> >
>> > final Put put)
>> >
>> > Cheers
>> >
>> > On Thu, Nov 3, 2016 at 10:57 AM, Manjeet Singh <
>> manjeet.chandhok@gmail.com>
>> > wrote:
>> >
>> >> Hi Ted,
>> >>
>> >> code not required for this case and how MVCC can help?
>> >>
>> >> question is if I have record which come in hbase and from some api its
>> >> going for update (means get is performed)
>> >> now other operation also performed get operation both update on same
>> rwokey
>> >> at end we will not have correct value.
>> >>
>> >> in seprate mail thread i asked same type of question regarding rowkey
>> lock,
>> >> but still i did't get correct anser
>> >>
>> >> Thanks
>> >> Manjeet
>> >>
>> >> On Wed, Nov 2, 2016 at 11:36 PM, Ted Yu <yu...@gmail.com> wrote:
>> >>
>> >> > Were you including code in the image (which didn't come through) ?
>> >> >
>> >> > MultiVersionConcurrencyControl is involved in answering your question.
>> >> >
>> >> > See http://hbase.apache.org/book.html#hregion.scans
>> >> >
>> >> > Cheers
>> >> >
>> >> > On Wed, Nov 2, 2016 at 10:57 AM, Manjeet Singh <
>> >> manjeet.chandhok@gmail.com
>> >> > >
>> >> > wrote:
>> >> >
>> >> > > Hi All
>> >> > >
>> >> > > I have ETL process for inserting data into hbase for this I have
>> spark
>> >> > > jobs which are responsible for reading data from kafka topics. so my
>> >> > > question is if I have some rowkey already exist in hbase and I have
>> 3
>> >> > spark
>> >> > > job running and they all try to update on same rowkey how hbase deal
>> >> for
>> >> > > atomicity?
>> >> > >
>> >> > > for more understanding i have 3 rowkey coming from 3 seprate spark
>> job
>> >> > and
>> >> > > all trying to update same rowkey which is already exist in hbase
>> table.
>> >> > >
>> >> > > [image: Inline image 1]
>> >> > >
>> >> > > --
>> >> > > luv all
>> >> > >
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> luv all
>> >>
>>
>
>
>
> --
> luv all
Re: How can I achieve HBase row level atomicity?
Posted by Manjeet Singh <ma...@gmail.com>.
Yes Anoop you are right
my input source is kafka pipeline we have 7 spark ETL jobs which are
responsible for aggregation, and to get last value from Hbase and put the
updated one.
we set zero version so we always have only one copy in Hbase.
Problem is if my 2 different ETL job having the same rowkey and one ETL
process get the value and at same time second also get that value and
update it now first ETL job will replace that updated value.
it can happen in same ETL job too.
Thanks
Manjeet
On Mon, Nov 7, 2016 at 7:07 PM, Anoop John <an...@gmail.com> wrote:
> Seems u want to get an already present row and do some op and put the
> updated value. What is that op? If u can explain those we can try
> help u with ways (if available) As such, the above kind of API do not
> guarantee u any atomicity.
>
> -Anoop-
>
> On Fri, Nov 4, 2016 at 4:12 AM, Ted Yu <yu...@gmail.com> wrote:
> > bq. from some api its going for update (means get is performed)
> >
> > Update on hbase would correlate with Put or Delete (not sure what 'get'
> > above means).
> >
> > Looks like your concern is that two concurrent updates may overwrite the
> > data for same rowkey.
> >
> > Have you considered using:
> >
> > public boolean checkAndPut(final byte [] row,
> >
> > final byte [] family, final byte [] qualifier, final byte [] value,
> >
> > final Put put)
> >
> > Cheers
> >
> > On Thu, Nov 3, 2016 at 10:57 AM, Manjeet Singh <
> manjeet.chandhok@gmail.com>
> > wrote:
> >
> >> Hi Ted,
> >>
> >> code not required for this case and how MVCC can help?
> >>
> >> question is if I have record which come in hbase and from some api its
> >> going for update (means get is performed)
> >> now other operation also performed get operation both update on same
> rwokey
> >> at end we will not have correct value.
> >>
> >> in seprate mail thread i asked same type of question regarding rowkey
> lock,
> >> but still i did't get correct anser
> >>
> >> Thanks
> >> Manjeet
> >>
> >> On Wed, Nov 2, 2016 at 11:36 PM, Ted Yu <yu...@gmail.com> wrote:
> >>
> >> > Were you including code in the image (which didn't come through) ?
> >> >
> >> > MultiVersionConcurrencyControl is involved in answering your question.
> >> >
> >> > See http://hbase.apache.org/book.html#hregion.scans
> >> >
> >> > Cheers
> >> >
> >> > On Wed, Nov 2, 2016 at 10:57 AM, Manjeet Singh <
> >> manjeet.chandhok@gmail.com
> >> > >
> >> > wrote:
> >> >
> >> > > Hi All
> >> > >
> >> > > I have ETL process for inserting data into hbase for this I have
> spark
> >> > > jobs which are responsible for reading data from kafka topics. so my
> >> > > question is if I have some rowkey already exist in hbase and I have
> 3
> >> > spark
> >> > > job running and they all try to update on same rowkey how hbase deal
> >> for
> >> > > atomicity?
> >> > >
> >> > > for more understanding i have 3 rowkey coming from 3 seprate spark
> job
> >> > and
> >> > > all trying to update same rowkey which is already exist in hbase
> table.
> >> > >
> >> > > [image: Inline image 1]
> >> > >
> >> > > --
> >> > > luv all
> >> > >
> >> >
> >>
> >>
> >>
> >> --
> >> luv all
> >>
>
--
luv all
Re: How can I achieve HBase row level atomicity?
Posted by Anoop John <an...@gmail.com>.
Seems u want to get an already present row and do some op and put the
updated value. What is that op? If u can explain those we can try
help u with ways (if available) As such, the above kind of API do not
guarantee u any atomicity.
-Anoop-
On Fri, Nov 4, 2016 at 4:12 AM, Ted Yu <yu...@gmail.com> wrote:
> bq. from some api its going for update (means get is performed)
>
> Update on hbase would correlate with Put or Delete (not sure what 'get'
> above means).
>
> Looks like your concern is that two concurrent updates may overwrite the
> data for same rowkey.
>
> Have you considered using:
>
> public boolean checkAndPut(final byte [] row,
>
> final byte [] family, final byte [] qualifier, final byte [] value,
>
> final Put put)
>
> Cheers
>
> On Thu, Nov 3, 2016 at 10:57 AM, Manjeet Singh <ma...@gmail.com>
> wrote:
>
>> Hi Ted,
>>
>> code not required for this case and how MVCC can help?
>>
>> question is if I have record which come in hbase and from some api its
>> going for update (means get is performed)
>> now other operation also performed get operation both update on same rwokey
>> at end we will not have correct value.
>>
>> in seprate mail thread i asked same type of question regarding rowkey lock,
>> but still i did't get correct anser
>>
>> Thanks
>> Manjeet
>>
>> On Wed, Nov 2, 2016 at 11:36 PM, Ted Yu <yu...@gmail.com> wrote:
>>
>> > Were you including code in the image (which didn't come through) ?
>> >
>> > MultiVersionConcurrencyControl is involved in answering your question.
>> >
>> > See http://hbase.apache.org/book.html#hregion.scans
>> >
>> > Cheers
>> >
>> > On Wed, Nov 2, 2016 at 10:57 AM, Manjeet Singh <
>> manjeet.chandhok@gmail.com
>> > >
>> > wrote:
>> >
>> > > Hi All
>> > >
>> > > I have ETL process for inserting data into hbase for this I have spark
>> > > jobs which are responsible for reading data from kafka topics. so my
>> > > question is if I have some rowkey already exist in hbase and I have 3
>> > spark
>> > > job running and they all try to update on same rowkey how hbase deal
>> for
>> > > atomicity?
>> > >
>> > > for more understanding i have 3 rowkey coming from 3 seprate spark job
>> > and
>> > > all trying to update same rowkey which is already exist in hbase table.
>> > >
>> > > [image: Inline image 1]
>> > >
>> > > --
>> > > luv all
>> > >
>> >
>>
>>
>>
>> --
>> luv all
>>
Re: How can I achieve HBase row level atomicity?
Posted by Ted Yu <yu...@gmail.com>.
bq. from some api its going for update (means get is performed)
Update on hbase would correlate with Put or Delete (not sure what 'get'
above means).
Looks like your concern is that two concurrent updates may overwrite the
data for same rowkey.
Have you considered using:
public boolean checkAndPut(final byte [] row,
final byte [] family, final byte [] qualifier, final byte [] value,
final Put put)
Cheers
On Thu, Nov 3, 2016 at 10:57 AM, Manjeet Singh <ma...@gmail.com>
wrote:
> Hi Ted,
>
> code not required for this case and how MVCC can help?
>
> question is if I have record which come in hbase and from some api its
> going for update (means get is performed)
> now other operation also performed get operation both update on same rwokey
> at end we will not have correct value.
>
> in seprate mail thread i asked same type of question regarding rowkey lock,
> but still i did't get correct anser
>
> Thanks
> Manjeet
>
> On Wed, Nov 2, 2016 at 11:36 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > Were you including code in the image (which didn't come through) ?
> >
> > MultiVersionConcurrencyControl is involved in answering your question.
> >
> > See http://hbase.apache.org/book.html#hregion.scans
> >
> > Cheers
> >
> > On Wed, Nov 2, 2016 at 10:57 AM, Manjeet Singh <
> manjeet.chandhok@gmail.com
> > >
> > wrote:
> >
> > > Hi All
> > >
> > > I have ETL process for inserting data into hbase for this I have spark
> > > jobs which are responsible for reading data from kafka topics. so my
> > > question is if I have some rowkey already exist in hbase and I have 3
> > spark
> > > job running and they all try to update on same rowkey how hbase deal
> for
> > > atomicity?
> > >
> > > for more understanding i have 3 rowkey coming from 3 seprate spark job
> > and
> > > all trying to update same rowkey which is already exist in hbase table.
> > >
> > > [image: Inline image 1]
> > >
> > > --
> > > luv all
> > >
> >
>
>
>
> --
> luv all
>
Re: How can I achieve HBase row level atomicity?
Posted by Manjeet Singh <ma...@gmail.com>.
Hi Ted,
code not required for this case and how MVCC can help?
question is if I have record which come in hbase and from some api its
going for update (means get is performed)
now other operation also performed get operation both update on same rwokey
at end we will not have correct value.
in seprate mail thread i asked same type of question regarding rowkey lock,
but still i did't get correct anser
Thanks
Manjeet
On Wed, Nov 2, 2016 at 11:36 PM, Ted Yu <yu...@gmail.com> wrote:
> Were you including code in the image (which didn't come through) ?
>
> MultiVersionConcurrencyControl is involved in answering your question.
>
> See http://hbase.apache.org/book.html#hregion.scans
>
> Cheers
>
> On Wed, Nov 2, 2016 at 10:57 AM, Manjeet Singh <manjeet.chandhok@gmail.com
> >
> wrote:
>
> > Hi All
> >
> > I have ETL process for inserting data into hbase for this I have spark
> > jobs which are responsible for reading data from kafka topics. so my
> > question is if I have some rowkey already exist in hbase and I have 3
> spark
> > job running and they all try to update on same rowkey how hbase deal for
> > atomicity?
> >
> > for more understanding i have 3 rowkey coming from 3 seprate spark job
> and
> > all trying to update same rowkey which is already exist in hbase table.
> >
> > [image: Inline image 1]
> >
> > --
> > luv all
> >
>
--
luv all
Re: How can I achieve HBase row level atomicity?
Posted by Ted Yu <yu...@gmail.com>.
Were you including code in the image (which didn't come through) ?
MultiVersionConcurrencyControl is involved in answering your question.
See http://hbase.apache.org/book.html#hregion.scans
Cheers
On Wed, Nov 2, 2016 at 10:57 AM, Manjeet Singh <ma...@gmail.com>
wrote:
> Hi All
>
> I have ETL process for inserting data into hbase for this I have spark
> jobs which are responsible for reading data from kafka topics. so my
> question is if I have some rowkey already exist in hbase and I have 3 spark
> job running and they all try to update on same rowkey how hbase deal for
> atomicity?
>
> for more understanding i have 3 rowkey coming from 3 seprate spark job and
> all trying to update same rowkey which is already exist in hbase table.
>
> [image: Inline image 1]
>
> --
> luv all
>