You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Pablo Musa <pa...@psafe.com> on 2012/07/31 00:13:05 UTC
Retrieve Put timestamp
Hey guys,
in my application the HBase timestamp is used as version in my logic.
I would like to know what is the best way to insert a new record and get its timestamp.
I have come up with two possibilities:
/* I could force timestamp, but it is not a good idea since different servers
* write into HBase which could lead to crazy behavior */
new Put(row, timestamp);
/* Or I could write into HBase and read it back. But I don't know how much overhead
* this option causes.*/
@Override
public void put(Put put) throws IOException {
byte[] row = put.getRow();
hTableInterface.put(put);
KeyValue kv = hTableInterface.get(new Get(row)).getColumnLatest(family, qualifier);
long version = kv.getTimestamp();
}
Is there any better way to do it?
Thanks,
Pablo
RE: Retrieve Put timestamp
Posted by Pablo Musa <pa...@psafe.com>.
> What do you mean by using TS as version?
My application uses versioning between client and server. So I am using the timestamp as a version.
> Are you determining the ts long value before and then setting it in the Put object? If so, I think you can
> use a specific cell as a counter. In that case of course you need the value of the TS so that's not a problem.
The problem is that I would insert an increment in one column for different inserts along the table.
Every column has a timestamp, but just the "latest" timestamp is important to the client. By latest
I mean the latest when the client contacted the server, of course it will change.
> The latter is problematic because another client may have inserted a version between your put and the get.
Actually this is not an issue in my logic. This will not happen at all. Thus I am concerned with the overhead.
> The former could work but it depends on your applications version semantic.
It usually works, but server clocks can be unsynchronized and lead us into a loss of data scenario.
> Is it ok if another client does an update with the same 'version/timestamp'?
Not at all. The important thing is that the client has all data until a specific version, which is the
timestamp of the last insert in the last communication.
Thank you very much for the ideas!
-----Original Message-----
From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of Stack
Sent: terça-feira, 31 de julho de 2012 18:12
To: user@hbase.apache.org
Subject: Re: Retrieve Put timestamp
On Mon, Jul 30, 2012 at 11:13 PM, Pablo Musa <pa...@psafe.com> wrote:
> Hey guys,
> in my application the HBase timestamp is used as version in my logic.
> I would like to know what is the best way to insert a new record and get its timestamp.
>
> I have come up with two possibilities:
>
> /* I could force timestamp, but it is not a good idea since different
> servers
> * write into HBase which could lead to crazy behavior */ new Put(row,
> timestamp);
>
> /* Or I could write into HBase and read it back. But I don't know how
> much overhead
> * this option causes.*/
> @Override
> public void put(Put put) throws IOException {
> byte[] row = put.getRow();
> hTableInterface.put(put);
> KeyValue kv = hTableInterface.get(new Get(row)).getColumnLatest(family, qualifier);
> long version = kv.getTimestamp();
> }
>
> Is there any better way to do it?
>
Not that I know of.
The latter is problematic because another client may have inserted a version between your put and the get. The former could work but it depends on your applications version semantic. Is it ok if another client does an update with the same 'version/timestamp'?
St.Ack
Re: Retrieve Put timestamp
Posted by Stack <st...@duboce.net>.
On Mon, Jul 30, 2012 at 11:13 PM, Pablo Musa <pa...@psafe.com> wrote:
> Hey guys,
> in my application the HBase timestamp is used as version in my logic.
> I would like to know what is the best way to insert a new record and get its timestamp.
>
> I have come up with two possibilities:
>
> /* I could force timestamp, but it is not a good idea since different servers
> * write into HBase which could lead to crazy behavior */
> new Put(row, timestamp);
>
> /* Or I could write into HBase and read it back. But I don't know how much overhead
> * this option causes.*/
> @Override
> public void put(Put put) throws IOException {
> byte[] row = put.getRow();
> hTableInterface.put(put);
> KeyValue kv = hTableInterface.get(new Get(row)).getColumnLatest(family, qualifier);
> long version = kv.getTimestamp();
> }
>
> Is there any better way to do it?
>
Not that I know of.
The latter is problematic because another client may have inserted a
version between your put and the get. The former could work but it
depends on your applications version semantic. Is it ok if another
client does an update with the same 'version/timestamp'?
St.Ack
RE: Retrieve Put timestamp
Posted by Wei Tan <wt...@us.ibm.com>.
+1.
So far I think timestamp is very useful. I would imagine if we can
configure the return, say in pre/post put, it would be even nicer.
Thanks,
Wei
Wei Tan
Research Staff Member
IBM T. J. Watson Research Center
19 Skyline Dr, Hawthorne, NY 10532
wtan@us.ibm.com; 914-784-6752
From: "Ramkrishna.S.Vasudevan" <ra...@huawei.com>
To: <us...@hbase.apache.org>,
Date: 08/02/2012 12:54 AM
Subject: RE: Retrieve Put timestamp
+1. Anyway all mutations extends OperationsWithAttributes also.
Regards
Ram
> -----Original Message-----
> From: Anoop Sam John [mailto:anoopsj@huawei.com]
> Sent: Thursday, August 02, 2012 10:13 AM
> To: user@hbase.apache.org
> Subject: RE: Retrieve Put timestamp
>
> Currently in Append there is a setter to specify whether to return the
> result or not. Similar way we can use for Put? Only with specific use
> cases the return TS might be needed.
> May be in a generic way we can return the attributes of the Mutation?
> So any thing which the client needs back can be added into the
> attributes [Any byte[] value]
> and we can return the same to client [If the flag is turned on] User
> can add these attributes using pre/post CP hooks.
>
> -Anoop-
> ________________________________________
> From: saint.ack@gmail.com [saint.ack@gmail.com] on behalf of Stack
> [stack@duboce.net]
> Sent: Thursday, August 02, 2012 3:41 AM
> To: user@hbase.apache.org
> Subject: Re: Retrieve Put timestamp
>
> On Wed, Aug 1, 2012 at 7:12 PM, Wei Tan <wt...@us.ibm.com> wrote:
> > We have a similar requirement and here is the solution in our mind:
> > add a coprocessor, in prePut() get the current ms and set it to put -
> --
> > the current implementation get the current ms and set it in put()
> > return the ms generated to prePut() to client. For now put() does not
> > return any value. we need to change the behavior of it
> >
> > Any flaw in this design?
>
> In 0.96 we have moved to protobufs. The put/mutate call currently
> doesn't return anything:
>
> message MutateResponse {
> optional Result result = 1;
>
> // used for mutate to indicate processed only
> optional bool processed = 2;
> }
>
> Should be easy enough changing it to run timestamps? Should it do it
> always or should we return the request so you have to ask for it?
>
> St.Ack=
RE: Retrieve Put timestamp
Posted by Wei Tan <wt...@us.ibm.com>.
I wonder if there is any follow up on this issue, i.e., a put can return a
timestamp of the record? Thanks!
Best Regards,
Wei
From: Wei Tan/Watson/IBM
To: user@hbase.apache.org,
Date: 08/02/2012 12:37 PM
Subject: RE: Retrieve Put timestamp
+1.
So far I think timestamp is very useful. I would imagine if we can
configure the return, say in pre/post put, it would be even nicer.
Thanks,
Wei
Wei Tan
Research Staff Member
IBM T. J. Watson Research Center
19 Skyline Dr, Hawthorne, NY 10532
wtan@us.ibm.com; 914-784-6752
From: "Ramkrishna.S.Vasudevan" <ra...@huawei.com>
To: <us...@hbase.apache.org>,
Date: 08/02/2012 12:54 AM
Subject: RE: Retrieve Put timestamp
+1. Anyway all mutations extends OperationsWithAttributes also.
Regards
Ram
> -----Original Message-----
> From: Anoop Sam John [mailto:anoopsj@huawei.com]
> Sent: Thursday, August 02, 2012 10:13 AM
> To: user@hbase.apache.org
> Subject: RE: Retrieve Put timestamp
>
> Currently in Append there is a setter to specify whether to return the
> result or not. Similar way we can use for Put? Only with specific use
> cases the return TS might be needed.
> May be in a generic way we can return the attributes of the Mutation?
> So any thing which the client needs back can be added into the
> attributes [Any byte[] value]
> and we can return the same to client [If the flag is turned on] User
> can add these attributes using pre/post CP hooks.
>
> -Anoop-
> ________________________________________
> From: saint.ack@gmail.com [saint.ack@gmail.com] on behalf of Stack
> [stack@duboce.net]
> Sent: Thursday, August 02, 2012 3:41 AM
> To: user@hbase.apache.org
> Subject: Re: Retrieve Put timestamp
>
> On Wed, Aug 1, 2012 at 7:12 PM, Wei Tan <wt...@us.ibm.com> wrote:
> > We have a similar requirement and here is the solution in our mind:
> > add a coprocessor, in prePut() get the current ms and set it to put -
> --
> > the current implementation get the current ms and set it in put()
> > return the ms generated to prePut() to client. For now put() does not
> > return any value. we need to change the behavior of it
> >
> > Any flaw in this design?
>
> In 0.96 we have moved to protobufs. The put/mutate call currently
> doesn't return anything:
>
> message MutateResponse {
> optional Result result = 1;
>
> // used for mutate to indicate processed only
> optional bool processed = 2;
> }
>
> Should be easy enough changing it to run timestamps? Should it do it
> always or should we return the request so you have to ask for it?
>
> St.Ack=
RE: Retrieve Put timestamp
Posted by "Ramkrishna.S.Vasudevan" <ra...@huawei.com>.
+1. Anyway all mutations extends OperationsWithAttributes also.
Regards
Ram
> -----Original Message-----
> From: Anoop Sam John [mailto:anoopsj@huawei.com]
> Sent: Thursday, August 02, 2012 10:13 AM
> To: user@hbase.apache.org
> Subject: RE: Retrieve Put timestamp
>
> Currently in Append there is a setter to specify whether to return the
> result or not. Similar way we can use for Put? Only with specific use
> cases the return TS might be needed.
> May be in a generic way we can return the attributes of the Mutation?
> So any thing which the client needs back can be added into the
> attributes [Any byte[] value]
> and we can return the same to client [If the flag is turned on] User
> can add these attributes using pre/post CP hooks.
>
> -Anoop-
> ________________________________________
> From: saint.ack@gmail.com [saint.ack@gmail.com] on behalf of Stack
> [stack@duboce.net]
> Sent: Thursday, August 02, 2012 3:41 AM
> To: user@hbase.apache.org
> Subject: Re: Retrieve Put timestamp
>
> On Wed, Aug 1, 2012 at 7:12 PM, Wei Tan <wt...@us.ibm.com> wrote:
> > We have a similar requirement and here is the solution in our mind:
> > add a coprocessor, in prePut() get the current ms and set it to put -
> --
> > the current implementation get the current ms and set it in put()
> > return the ms generated to prePut() to client. For now put() does not
> > return any value. we need to change the behavior of it
> >
> > Any flaw in this design?
>
> In 0.96 we have moved to protobufs. The put/mutate call currently
> doesn't return anything:
>
> message MutateResponse {
> optional Result result = 1;
>
> // used for mutate to indicate processed only
> optional bool processed = 2;
> }
>
> Should be easy enough changing it to run timestamps? Should it do it
> always or should we return the request so you have to ask for it?
>
> St.Ack=
RE: Retrieve Put timestamp
Posted by Anoop Sam John <an...@huawei.com>.
Currently in Append there is a setter to specify whether to return the result or not. Similar way we can use for Put? Only with specific use cases the return TS might be needed.
May be in a generic way we can return the attributes of the Mutation? So any thing which the client needs back can be added into the attributes [Any byte[] value]
and we can return the same to client [If the flag is turned on] User can add these attributes using pre/post CP hooks.
-Anoop-
________________________________________
From: saint.ack@gmail.com [saint.ack@gmail.com] on behalf of Stack [stack@duboce.net]
Sent: Thursday, August 02, 2012 3:41 AM
To: user@hbase.apache.org
Subject: Re: Retrieve Put timestamp
On Wed, Aug 1, 2012 at 7:12 PM, Wei Tan <wt...@us.ibm.com> wrote:
> We have a similar requirement and here is the solution in our mind:
> add a coprocessor, in prePut() get the current ms and set it to put ---
> the current implementation get the current ms and set it in put()
> return the ms generated to prePut() to client. For now put() does not
> return any value. we need to change the behavior of it
>
> Any flaw in this design?
In 0.96 we have moved to protobufs. The put/mutate call currently
doesn't return anything:
message MutateResponse {
optional Result result = 1;
// used for mutate to indicate processed only
optional bool processed = 2;
}
Should be easy enough changing it to run timestamps? Should it do it
always or should we return the request so you have to ask for it?
St.Ack
Re: Retrieve Put timestamp
Posted by Stack <st...@duboce.net>.
On Wed, Aug 1, 2012 at 7:12 PM, Wei Tan <wt...@us.ibm.com> wrote:
> We have a similar requirement and here is the solution in our mind:
> add a coprocessor, in prePut() get the current ms and set it to put ---
> the current implementation get the current ms and set it in put()
> return the ms generated to prePut() to client. For now put() does not
> return any value. we need to change the behavior of it
>
> Any flaw in this design?
In 0.96 we have moved to protobufs. The put/mutate call currently
doesn't return anything:
message MutateResponse {
optional Result result = 1;
// used for mutate to indicate processed only
optional bool processed = 2;
}
Should be easy enough changing it to run timestamps? Should it do it
always or should we return the request so you have to ask for it?
St.Ack
Re: Retrieve Put timestamp
Posted by Wei Tan <wt...@us.ibm.com>.
We have a similar requirement and here is the solution in our mind:
add a coprocessor, in prePut() get the current ms and set it to put ---
the current implementation get the current ms and set it in put()
return the ms generated to prePut() to client. For now put() does not
return any value. we need to change the behavior of it
Any flaw in this design?
Thanks,
Wei
From: lars hofhansl <lh...@yahoo.com>
To: "user@hbase.apache.org" <us...@hbase.apache.org>,
Date: 08/01/2012 12:37 PM
Subject: Re: Retrieve Put timestamp
There is no HBase API for this.
However, this could useful in some scenario, so maybe we could add an API
for this.
It's not entirely trivial, though.
________________________________
From: Pablo Musa <pa...@psafe.com>
To: "user@hbase.apache.org" <us...@hbase.apache.org>
Sent: Monday, July 30, 2012 3:13 PM
Subject: Retrieve Put timestamp
Hey guys,
in my application the HBase timestamp is used as version in my logic.
I would like to know what is the best way to insert a new record and get
its timestamp.
I have come up with two possibilities:
/* I could force timestamp, but it is not a good idea since different
servers
* write into HBase which could lead to crazy behavior */
new Put(row, timestamp);
/* Or I could write into HBase and read it back. But I don't know how much
overhead
* this option causes.*/
@Override
public void put(Put put) throws IOException {
byte[] row = put.getRow();
hTableInterface.put(put);
KeyValue kv = hTableInterface.get(new
Get(row)).getColumnLatest(family, qualifier);
long version = kv.getTimestamp();
}
Is there any better way to do it?
Thanks,
Pablo
Re: Retrieve Put timestamp
Posted by lars hofhansl <lh...@yahoo.com>.
There is no HBase API for this.
However, this could useful in some scenario, so maybe we could add an API for this.
It's not entirely trivial, though.
________________________________
From: Pablo Musa <pa...@psafe.com>
To: "user@hbase.apache.org" <us...@hbase.apache.org>
Sent: Monday, July 30, 2012 3:13 PM
Subject: Retrieve Put timestamp
Hey guys,
in my application the HBase timestamp is used as version in my logic.
I would like to know what is the best way to insert a new record and get its timestamp.
I have come up with two possibilities:
/* I could force timestamp, but it is not a good idea since different servers
* write into HBase which could lead to crazy behavior */
new Put(row, timestamp);
/* Or I could write into HBase and read it back. But I don't know how much overhead
* this option causes.*/
@Override
public void put(Put put) throws IOException {
byte[] row = put.getRow();
hTableInterface.put(put);
KeyValue kv = hTableInterface.get(new Get(row)).getColumnLatest(family, qualifier);
long version = kv.getTimestamp();
}
Is there any better way to do it?
Thanks,
Pablo
Re: Retrieve Put timestamp
Posted by Asaf Mesika <as...@gmail.com>.
What do you mean by using TS as version? Are you determining the ts long value before and then setting it in the Put object? If so, I think you can use a specific cell as a counter (Sequence in Oracle language, or Auto Increment column in MySQL). In that case of course you need the value of the TS so that's not a problem.
On 31 ביול 2012, at 01:13, Pablo Musa <pa...@psafe.com> wrote:
> Hey guys,
> in my application the HBase timestamp is used as version in my logic.
> I would like to know what is the best way to insert a new record and get its timestamp.
>
> I have come up with two possibilities:
>
> /* I could force timestamp, but it is not a good idea since different servers
> * write into HBase which could lead to crazy behavior */
> new Put(row, timestamp);
>
> /* Or I could write into HBase and read it back. But I don't know how much overhead
> * this option causes.*/
> @Override
> public void put(Put put) throws IOException {
> byte[] row = put.getRow();
> hTableInterface.put(put);
> KeyValue kv = hTableInterface.get(new Get(row)).getColumnLatest(family, qualifier);
> long version = kv.getTimestamp();
> }
>
> Is there any better way to do it?
>
> Thanks,
> Pablo