You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by GSK Chaitanya <gs...@gmail.com> on 2014/04/14 19:05:55 UTC
HBase atomic append functionality (not just client)
Mighty Hbase users and developers,
I have few questions and I'd really appreciate it if someone can clarify
them.
1) I want to know if Hbase inherently supports *atomic
append*functionality like
*get* and *put*. For my work, I would be using OpenTSDB which is a layer on
top of AsynchHBase and AsynchHBase doesnt work with HBase client (which
supports *atomic append*).
2) If I understand correctly, atomic append of HBase client internally does
a get and put instead of actually appending to the end of the cell. If
that's the case, I wonder how does this functionality is of much use in
terms of performance. In our case, we would like a very light weight append
functionality. I'd like to know if there are any plans of adding this
feature to HBase main in the near future.
Thanks,
Chaitanya
Re: HBase atomic append functionality (not just client)
Posted by lars hofhansl <la...@apache.org>.
It would.
________________________________
From: Sergey Shelukhin <se...@hortonworks.com>
To: dev@hbase.apache.org
Cc: "user@hbase.apache.org" <us...@hbase.apache.org>
Sent: Tuesday, April 15, 2014 11:13 AM
Subject: Re: HBase atomic append functionality (not just client)
Hmm... Wouldn't mvcc prevent seeing partial append?
Append is just put in the end, the way it is currently implemented.
On Mon, Apr 14, 2014 at 10:41 AM, Vladimir Rodionov <vrodionov@carrieriq.com
> wrote:
> From HRegion.java:
>
> "Appends performed are done under row lock but reads do not take locks out
> so this can be seen partially complete by gets and scans."
>
> Appends are partially atomic (you can get partial reads but you will never
> get corrupted writes) and they are implemented on the server side.
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: vrodionov@carrieriq.com
>
> ________________________________________
> From: GSK Chaitanya [gskchaitanya12@gmail.com]
> Sent: Monday, April 14, 2014 10:05 AM
> To: user@hbase.apache.org; dev@hbase.apache.org
> Subject: HBase atomic append functionality (not just client)
>
> Mighty Hbase users and developers,
>
> I have few questions and I'd really appreciate it if someone can clarify
> them.
>
> 1) I want to know if Hbase inherently supports *atomic
> append*functionality like
> *get* and *put*. For my work, I would be using OpenTSDB which is a layer on
> top of AsynchHBase and AsynchHBase doesnt work with HBase client (which
> supports *atomic append*).
>
> 2) If I understand correctly, atomic append of HBase client internally does
> a get and put instead of actually appending to the end of the cell. If
> that's the case, I wonder how does this functionality is of much use in
> terms of performance. In our case, we would like a very light weight append
> functionality. I'd like to know if there are any plans of adding this
> feature to HBase main in the near future.
>
> Thanks,
> Chaitanya
>
> Confidentiality Notice: The information contained in this message,
> including any attachments hereto, may be confidential and is intended to be
> read only by the individual or entity to whom this message is addressed. If
> the reader of this message is not the intended recipient or an agent or
> designee of the intended recipient, please note that any review, use,
> disclosure or distribution of this message or its attachments, in any form,
> is strictly prohibited. If you have received this message in error, please
> immediately notify the sender and/or Notifications@carrieriq.com and
> delete or destroy any copy of this message and its attachments.
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: HBase atomic append functionality (not just client)
Posted by lars hofhansl <la...@apache.org>.
It would.
________________________________
From: Sergey Shelukhin <se...@hortonworks.com>
To: dev@hbase.apache.org
Cc: "user@hbase.apache.org" <us...@hbase.apache.org>
Sent: Tuesday, April 15, 2014 11:13 AM
Subject: Re: HBase atomic append functionality (not just client)
Hmm... Wouldn't mvcc prevent seeing partial append?
Append is just put in the end, the way it is currently implemented.
On Mon, Apr 14, 2014 at 10:41 AM, Vladimir Rodionov <vrodionov@carrieriq.com
> wrote:
> From HRegion.java:
>
> "Appends performed are done under row lock but reads do not take locks out
> so this can be seen partially complete by gets and scans."
>
> Appends are partially atomic (you can get partial reads but you will never
> get corrupted writes) and they are implemented on the server side.
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: vrodionov@carrieriq.com
>
> ________________________________________
> From: GSK Chaitanya [gskchaitanya12@gmail.com]
> Sent: Monday, April 14, 2014 10:05 AM
> To: user@hbase.apache.org; dev@hbase.apache.org
> Subject: HBase atomic append functionality (not just client)
>
> Mighty Hbase users and developers,
>
> I have few questions and I'd really appreciate it if someone can clarify
> them.
>
> 1) I want to know if Hbase inherently supports *atomic
> append*functionality like
> *get* and *put*. For my work, I would be using OpenTSDB which is a layer on
> top of AsynchHBase and AsynchHBase doesnt work with HBase client (which
> supports *atomic append*).
>
> 2) If I understand correctly, atomic append of HBase client internally does
> a get and put instead of actually appending to the end of the cell. If
> that's the case, I wonder how does this functionality is of much use in
> terms of performance. In our case, we would like a very light weight append
> functionality. I'd like to know if there are any plans of adding this
> feature to HBase main in the near future.
>
> Thanks,
> Chaitanya
>
> Confidentiality Notice: The information contained in this message,
> including any attachments hereto, may be confidential and is intended to be
> read only by the individual or entity to whom this message is addressed. If
> the reader of this message is not the intended recipient or an agent or
> designee of the intended recipient, please note that any review, use,
> disclosure or distribution of this message or its attachments, in any form,
> is strictly prohibited. If you have received this message in error, please
> immediately notify the sender and/or Notifications@carrieriq.com and
> delete or destroy any copy of this message and its attachments.
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: HBase atomic append functionality (not just client)
Posted by Sergey Shelukhin <se...@hortonworks.com>.
Hmm... Wouldn't mvcc prevent seeing partial append?
Append is just put in the end, the way it is currently implemented.
On Mon, Apr 14, 2014 at 10:41 AM, Vladimir Rodionov <vrodionov@carrieriq.com
> wrote:
> From HRegion.java:
>
> "Appends performed are done under row lock but reads do not take locks out
> so this can be seen partially complete by gets and scans."
>
> Appends are partially atomic (you can get partial reads but you will never
> get corrupted writes) and they are implemented on the server side.
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: vrodionov@carrieriq.com
>
> ________________________________________
> From: GSK Chaitanya [gskchaitanya12@gmail.com]
> Sent: Monday, April 14, 2014 10:05 AM
> To: user@hbase.apache.org; dev@hbase.apache.org
> Subject: HBase atomic append functionality (not just client)
>
> Mighty Hbase users and developers,
>
> I have few questions and I'd really appreciate it if someone can clarify
> them.
>
> 1) I want to know if Hbase inherently supports *atomic
> append*functionality like
> *get* and *put*. For my work, I would be using OpenTSDB which is a layer on
> top of AsynchHBase and AsynchHBase doesnt work with HBase client (which
> supports *atomic append*).
>
> 2) If I understand correctly, atomic append of HBase client internally does
> a get and put instead of actually appending to the end of the cell. If
> that's the case, I wonder how does this functionality is of much use in
> terms of performance. In our case, we would like a very light weight append
> functionality. I'd like to know if there are any plans of adding this
> feature to HBase main in the near future.
>
> Thanks,
> Chaitanya
>
> Confidentiality Notice: The information contained in this message,
> including any attachments hereto, may be confidential and is intended to be
> read only by the individual or entity to whom this message is addressed. If
> the reader of this message is not the intended recipient or an agent or
> designee of the intended recipient, please note that any review, use,
> disclosure or distribution of this message or its attachments, in any form,
> is strictly prohibited. If you have received this message in error, please
> immediately notify the sender and/or Notifications@carrieriq.com and
> delete or destroy any copy of this message and its attachments.
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: HBase atomic append functionality (not just client)
Posted by Sergey Shelukhin <se...@hortonworks.com>.
Hmm... Wouldn't mvcc prevent seeing partial append?
Append is just put in the end, the way it is currently implemented.
On Mon, Apr 14, 2014 at 10:41 AM, Vladimir Rodionov <vrodionov@carrieriq.com
> wrote:
> From HRegion.java:
>
> "Appends performed are done under row lock but reads do not take locks out
> so this can be seen partially complete by gets and scans."
>
> Appends are partially atomic (you can get partial reads but you will never
> get corrupted writes) and they are implemented on the server side.
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: vrodionov@carrieriq.com
>
> ________________________________________
> From: GSK Chaitanya [gskchaitanya12@gmail.com]
> Sent: Monday, April 14, 2014 10:05 AM
> To: user@hbase.apache.org; dev@hbase.apache.org
> Subject: HBase atomic append functionality (not just client)
>
> Mighty Hbase users and developers,
>
> I have few questions and I'd really appreciate it if someone can clarify
> them.
>
> 1) I want to know if Hbase inherently supports *atomic
> append*functionality like
> *get* and *put*. For my work, I would be using OpenTSDB which is a layer on
> top of AsynchHBase and AsynchHBase doesnt work with HBase client (which
> supports *atomic append*).
>
> 2) If I understand correctly, atomic append of HBase client internally does
> a get and put instead of actually appending to the end of the cell. If
> that's the case, I wonder how does this functionality is of much use in
> terms of performance. In our case, we would like a very light weight append
> functionality. I'd like to know if there are any plans of adding this
> feature to HBase main in the near future.
>
> Thanks,
> Chaitanya
>
> Confidentiality Notice: The information contained in this message,
> including any attachments hereto, may be confidential and is intended to be
> read only by the individual or entity to whom this message is addressed. If
> the reader of this message is not the intended recipient or an agent or
> designee of the intended recipient, please note that any review, use,
> disclosure or distribution of this message or its attachments, in any form,
> is strictly prohibited. If you have received this message in error, please
> immediately notify the sender and/or Notifications@carrieriq.com and
> delete or destroy any copy of this message and its attachments.
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
RE: HBase atomic append functionality (not just client)
Posted by Vladimir Rodionov <vr...@carrieriq.com>.
Correction:
If row being appended is hot it is going to be in a block cache anyway, therefore there will be no reads from HDFS in almost all cases,
but definitely, it is much faster to read from MemStore only than from MemStore and block cache and checking other store files if data
is not major compacted. One more thing to consider is BloomFilter false positives: with default 2% and say 5-10 hfiles per region
we will get 10-20% chances of going to HDFS directly.
Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodionov@carrieriq.com
________________________________________
From: Vladimir Rodionov
Sent: Monday, April 14, 2014 1:22 PM
To: dev@hbase.apache.org
Cc: user@hbase.apache.org
Subject: RE: HBase atomic append functionality (not just client)
Ted,
The general idea of improving read-modify-write performance is to always read data from closest and fastest store only: either from MemStore or from block cache.
The read pipeline in HBase will always read data from HFile as well to compare versions. This should be optimized with a read hint, for example.
I remember Lars has mentioned some related JIRA? Not sure, though.
Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodionov@carrieriq.com
________________________________________
From: Ted Yu [yuzhihong@gmail.com]
Sent: Monday, April 14, 2014 12:46 PM
To: dev@hbase.apache.org
Cc: user@hbase.apache.org
Subject: Re: HBase atomic append functionality (not just client)
There was discussion on speeding up Append operation by only doing Put at
write time.
When reading, the Puts would be consolidated to produce the correct result.
Let me try to find the JIRA related to the discussion.
Cheers
On Mon, Apr 14, 2014 at 10:41 AM, Vladimir Rodionov <vrodionov@carrieriq.com
> wrote:
> From HRegion.java:
>
> "Appends performed are done under row lock but reads do not take locks out
> so this can be seen partially complete by gets and scans."
>
> Appends are partially atomic (you can get partial reads but you will never
> get corrupted writes) and they are implemented on the server side.
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: vrodionov@carrieriq.com
>
> ________________________________________
> From: GSK Chaitanya [gskchaitanya12@gmail.com]
> Sent: Monday, April 14, 2014 10:05 AM
> To: user@hbase.apache.org; dev@hbase.apache.org
> Subject: HBase atomic append functionality (not just client)
>
> Mighty Hbase users and developers,
>
> I have few questions and I'd really appreciate it if someone can clarify
> them.
>
> 1) I want to know if Hbase inherently supports *atomic
> append*functionality like
> *get* and *put*. For my work, I would be using OpenTSDB which is a layer on
> top of AsynchHBase and AsynchHBase doesnt work with HBase client (which
> supports *atomic append*).
>
> 2) If I understand correctly, atomic append of HBase client internally does
> a get and put instead of actually appending to the end of the cell. If
> that's the case, I wonder how does this functionality is of much use in
> terms of performance. In our case, we would like a very light weight append
> functionality. I'd like to know if there are any plans of adding this
> feature to HBase main in the near future.
>
> Thanks,
> Chaitanya
>
> Confidentiality Notice: The information contained in this message,
> including any attachments hereto, may be confidential and is intended to be
> read only by the individual or entity to whom this message is addressed. If
> the reader of this message is not the intended recipient or an agent or
> designee of the intended recipient, please note that any review, use,
> disclosure or distribution of this message or its attachments, in any form,
> is strictly prohibited. If you have received this message in error, please
> immediately notify the sender and/or Notifications@carrieriq.com and
> delete or destroy any copy of this message and its attachments.
>
Confidentiality Notice: The information contained in this message, including any attachments hereto, may be confidential and is intended to be read only by the individual or entity to whom this message is addressed. If the reader of this message is not the intended recipient or an agent or designee of the intended recipient, please note that any review, use, disclosure or distribution of this message or its attachments, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or Notifications@carrieriq.com and delete or destroy any copy of this message and its attachments.
RE: HBase atomic append functionality (not just client)
Posted by Vladimir Rodionov <vr...@carrieriq.com>.
Correction:
If row being appended is hot it is going to be in a block cache anyway, therefore there will be no reads from HDFS in almost all cases,
but definitely, it is much faster to read from MemStore only than from MemStore and block cache and checking other store files if data
is not major compacted. One more thing to consider is BloomFilter false positives: with default 2% and say 5-10 hfiles per region
we will get 10-20% chances of going to HDFS directly.
Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodionov@carrieriq.com
________________________________________
From: Vladimir Rodionov
Sent: Monday, April 14, 2014 1:22 PM
To: dev@hbase.apache.org
Cc: user@hbase.apache.org
Subject: RE: HBase atomic append functionality (not just client)
Ted,
The general idea of improving read-modify-write performance is to always read data from closest and fastest store only: either from MemStore or from block cache.
The read pipeline in HBase will always read data from HFile as well to compare versions. This should be optimized with a read hint, for example.
I remember Lars has mentioned some related JIRA? Not sure, though.
Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodionov@carrieriq.com
________________________________________
From: Ted Yu [yuzhihong@gmail.com]
Sent: Monday, April 14, 2014 12:46 PM
To: dev@hbase.apache.org
Cc: user@hbase.apache.org
Subject: Re: HBase atomic append functionality (not just client)
There was discussion on speeding up Append operation by only doing Put at
write time.
When reading, the Puts would be consolidated to produce the correct result.
Let me try to find the JIRA related to the discussion.
Cheers
On Mon, Apr 14, 2014 at 10:41 AM, Vladimir Rodionov <vrodionov@carrieriq.com
> wrote:
> From HRegion.java:
>
> "Appends performed are done under row lock but reads do not take locks out
> so this can be seen partially complete by gets and scans."
>
> Appends are partially atomic (you can get partial reads but you will never
> get corrupted writes) and they are implemented on the server side.
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: vrodionov@carrieriq.com
>
> ________________________________________
> From: GSK Chaitanya [gskchaitanya12@gmail.com]
> Sent: Monday, April 14, 2014 10:05 AM
> To: user@hbase.apache.org; dev@hbase.apache.org
> Subject: HBase atomic append functionality (not just client)
>
> Mighty Hbase users and developers,
>
> I have few questions and I'd really appreciate it if someone can clarify
> them.
>
> 1) I want to know if Hbase inherently supports *atomic
> append*functionality like
> *get* and *put*. For my work, I would be using OpenTSDB which is a layer on
> top of AsynchHBase and AsynchHBase doesnt work with HBase client (which
> supports *atomic append*).
>
> 2) If I understand correctly, atomic append of HBase client internally does
> a get and put instead of actually appending to the end of the cell. If
> that's the case, I wonder how does this functionality is of much use in
> terms of performance. In our case, we would like a very light weight append
> functionality. I'd like to know if there are any plans of adding this
> feature to HBase main in the near future.
>
> Thanks,
> Chaitanya
>
> Confidentiality Notice: The information contained in this message,
> including any attachments hereto, may be confidential and is intended to be
> read only by the individual or entity to whom this message is addressed. If
> the reader of this message is not the intended recipient or an agent or
> designee of the intended recipient, please note that any review, use,
> disclosure or distribution of this message or its attachments, in any form,
> is strictly prohibited. If you have received this message in error, please
> immediately notify the sender and/or Notifications@carrieriq.com and
> delete or destroy any copy of this message and its attachments.
>
Confidentiality Notice: The information contained in this message, including any attachments hereto, may be confidential and is intended to be read only by the individual or entity to whom this message is addressed. If the reader of this message is not the intended recipient or an agent or designee of the intended recipient, please note that any review, use, disclosure or distribution of this message or its attachments, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or Notifications@carrieriq.com and delete or destroy any copy of this message and its attachments.
Re: HBase atomic append functionality (not just client)
Posted by lars hofhansl <la...@apache.org>.
Yep. And even further, imagine how we could improve scanning.
If we only look for the latest version, we can scan memstore/HFiles in order of file age, if we find a version we're done, no need to even seek the other HFiles.
-- Lars
----- Original Message -----
From: Vladimir Rodionov <vl...@gmail.com>
To: "dev@hbase.apache.org" <de...@hbase.apache.org>; lars hofhansl <la...@apache.org>
Cc: "user@hbase.apache.org" <us...@hbase.apache.org>
Sent: Monday, April 14, 2014 1:51 PM
Subject: Re: HBase atomic append functionality (not just client)
Thanks, Lars
This JIRA is worth looking at. This hint will improve all reads: simple
gets, increments and appends.
-Vladimir
On Mon, Apr 14, 2014 at 1:46 PM, lars hofhansl <la...@apache.org> wrote:
> The jira would be this one: HBASE-10247.
>
> If the client promises not muck with timestamps we first look into the
> memstore. If we find a version there, we know it is the right one, only
> otherwise do we need to check the HFiles.
> (This is not possible if the client can set timestamps, because the latest
> value added to the memstore might be a backdated Put, or there might be a
> futuredated Delete in some of the HFiles.)
>
> -- Lars
>
>
> ----- Original Message -----
> From: Vladimir Rodionov <vr...@carrieriq.com>
> To: "dev@hbase.apache.org" <de...@hbase.apache.org>
> Cc: "user@hbase.apache.org" <us...@hbase.apache.org>
> Sent: Monday, April 14, 2014 1:22 PM
> Subject: RE: HBase atomic append functionality (not just client)
>
> Ted,
>
> The general idea of improving read-modify-write performance is to always
> read data from closest and fastest store only: either from MemStore or from
> block cache.
> The read pipeline in HBase will always read data from HFile as well to
> compare versions. This should be optimized with a read hint, for example.
> I remember Lars has mentioned some related JIRA? Not sure, though.
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: vrodionov@carrieriq.com
>
> ________________________________________
> From: Ted Yu [yuzhihong@gmail.com]
> Sent: Monday, April 14, 2014 12:46 PM
> To: dev@hbase.apache.org
> Cc: user@hbase.apache.org
> Subject: Re: HBase atomic append functionality (not just client)
>
> There was discussion on speeding up Append operation by only doing Put at
> write time.
> When reading, the Puts would be consolidated to produce the correct result.
>
> Let me try to find the JIRA related to the discussion.
>
> Cheers
>
>
> On Mon, Apr 14, 2014 at 10:41 AM, Vladimir Rodionov <
> vrodionov@carrieriq.com
> > wrote:
>
> > From HRegion.java:
> >
> > "Appends performed are done under row lock but reads do not take locks
> out
> > so this can be seen partially complete by gets and scans."
> >
> > Appends are partially atomic (you can get partial reads but you will
> never
> > get corrupted writes) and they are implemented on the server side.
> >
> > Best regards,
> > Vladimir Rodionov
> > Principal Platform Engineer
> > Carrier IQ, www.carrieriq.com
> > e-mail: vrodionov@carrieriq.com
> >
> > ________________________________________
> > From: GSK Chaitanya [gskchaitanya12@gmail.com]
> > Sent: Monday, April 14, 2014 10:05 AM
> > To: user@hbase.apache.org; dev@hbase.apache.org
> > Subject: HBase atomic append functionality (not just client)
> >
> > Mighty Hbase users and developers,
> >
> > I have few questions and I'd really appreciate it if someone can clarify
> > them.
> >
> > 1) I want to know if Hbase inherently supports *atomic
> > append*functionality like
> > *get* and *put*. For my work, I would be using OpenTSDB which is a layer
> on
> > top of AsynchHBase and AsynchHBase doesnt work with HBase client (which
> > supports *atomic append*).
> >
> > 2) If I understand correctly, atomic append of HBase client internally
> does
> > a get and put instead of actually appending to the end of the cell. If
> > that's the case, I wonder how does this functionality is of much use in
> > terms of performance. In our case, we would like a very light weight
> append
> > functionality. I'd like to know if there are any plans of adding this
> > feature to HBase main in the near future.
> >
> > Thanks,
> > Chaitanya
> >
> > Confidentiality Notice: The information contained in this message,
> > including any attachments hereto, may be confidential and is intended to
> be
> > read only by the individual or entity to whom this message is addressed.
> If
> > the reader of this message is not the intended recipient or an agent or
> > designee of the intended recipient, please note that any review, use,
> > disclosure or distribution of this message or its attachments, in any
> form,
> > is strictly prohibited. If you have received this message in error,
> please
> > immediately notify the sender and/or Notifications@carrieriq.com and
> > delete or destroy any copy of this message and its attachments.
>
> >
>
> Confidentiality Notice: The information contained in this message,
> including any attachments hereto, may be confidential and is intended to be
> read only by the individual or entity to whom this message is addressed. If
> the reader of this message is not the intended recipient or an agent or
> designee of the intended recipient, please note that any review, use,
> disclosure or distribution of this message or its attachments, in any form,
> is strictly prohibited. If you have received this message in error, please
> immediately notify the sender and/or Notifications@carrieriq.com and
> delete or destroy any copy of this message and its attachments.
>
Re: HBase atomic append functionality (not just client)
Posted by lars hofhansl <la...@apache.org>.
Yep. And even further, imagine how we could improve scanning.
If we only look for the latest version, we can scan memstore/HFiles in order of file age, if we find a version we're done, no need to even seek the other HFiles.
-- Lars
----- Original Message -----
From: Vladimir Rodionov <vl...@gmail.com>
To: "dev@hbase.apache.org" <de...@hbase.apache.org>; lars hofhansl <la...@apache.org>
Cc: "user@hbase.apache.org" <us...@hbase.apache.org>
Sent: Monday, April 14, 2014 1:51 PM
Subject: Re: HBase atomic append functionality (not just client)
Thanks, Lars
This JIRA is worth looking at. This hint will improve all reads: simple
gets, increments and appends.
-Vladimir
On Mon, Apr 14, 2014 at 1:46 PM, lars hofhansl <la...@apache.org> wrote:
> The jira would be this one: HBASE-10247.
>
> If the client promises not muck with timestamps we first look into the
> memstore. If we find a version there, we know it is the right one, only
> otherwise do we need to check the HFiles.
> (This is not possible if the client can set timestamps, because the latest
> value added to the memstore might be a backdated Put, or there might be a
> futuredated Delete in some of the HFiles.)
>
> -- Lars
>
>
> ----- Original Message -----
> From: Vladimir Rodionov <vr...@carrieriq.com>
> To: "dev@hbase.apache.org" <de...@hbase.apache.org>
> Cc: "user@hbase.apache.org" <us...@hbase.apache.org>
> Sent: Monday, April 14, 2014 1:22 PM
> Subject: RE: HBase atomic append functionality (not just client)
>
> Ted,
>
> The general idea of improving read-modify-write performance is to always
> read data from closest and fastest store only: either from MemStore or from
> block cache.
> The read pipeline in HBase will always read data from HFile as well to
> compare versions. This should be optimized with a read hint, for example.
> I remember Lars has mentioned some related JIRA? Not sure, though.
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: vrodionov@carrieriq.com
>
> ________________________________________
> From: Ted Yu [yuzhihong@gmail.com]
> Sent: Monday, April 14, 2014 12:46 PM
> To: dev@hbase.apache.org
> Cc: user@hbase.apache.org
> Subject: Re: HBase atomic append functionality (not just client)
>
> There was discussion on speeding up Append operation by only doing Put at
> write time.
> When reading, the Puts would be consolidated to produce the correct result.
>
> Let me try to find the JIRA related to the discussion.
>
> Cheers
>
>
> On Mon, Apr 14, 2014 at 10:41 AM, Vladimir Rodionov <
> vrodionov@carrieriq.com
> > wrote:
>
> > From HRegion.java:
> >
> > "Appends performed are done under row lock but reads do not take locks
> out
> > so this can be seen partially complete by gets and scans."
> >
> > Appends are partially atomic (you can get partial reads but you will
> never
> > get corrupted writes) and they are implemented on the server side.
> >
> > Best regards,
> > Vladimir Rodionov
> > Principal Platform Engineer
> > Carrier IQ, www.carrieriq.com
> > e-mail: vrodionov@carrieriq.com
> >
> > ________________________________________
> > From: GSK Chaitanya [gskchaitanya12@gmail.com]
> > Sent: Monday, April 14, 2014 10:05 AM
> > To: user@hbase.apache.org; dev@hbase.apache.org
> > Subject: HBase atomic append functionality (not just client)
> >
> > Mighty Hbase users and developers,
> >
> > I have few questions and I'd really appreciate it if someone can clarify
> > them.
> >
> > 1) I want to know if Hbase inherently supports *atomic
> > append*functionality like
> > *get* and *put*. For my work, I would be using OpenTSDB which is a layer
> on
> > top of AsynchHBase and AsynchHBase doesnt work with HBase client (which
> > supports *atomic append*).
> >
> > 2) If I understand correctly, atomic append of HBase client internally
> does
> > a get and put instead of actually appending to the end of the cell. If
> > that's the case, I wonder how does this functionality is of much use in
> > terms of performance. In our case, we would like a very light weight
> append
> > functionality. I'd like to know if there are any plans of adding this
> > feature to HBase main in the near future.
> >
> > Thanks,
> > Chaitanya
> >
> > Confidentiality Notice: The information contained in this message,
> > including any attachments hereto, may be confidential and is intended to
> be
> > read only by the individual or entity to whom this message is addressed.
> If
> > the reader of this message is not the intended recipient or an agent or
> > designee of the intended recipient, please note that any review, use,
> > disclosure or distribution of this message or its attachments, in any
> form,
> > is strictly prohibited. If you have received this message in error,
> please
> > immediately notify the sender and/or Notifications@carrieriq.com and
> > delete or destroy any copy of this message and its attachments.
>
> >
>
> Confidentiality Notice: The information contained in this message,
> including any attachments hereto, may be confidential and is intended to be
> read only by the individual or entity to whom this message is addressed. If
> the reader of this message is not the intended recipient or an agent or
> designee of the intended recipient, please note that any review, use,
> disclosure or distribution of this message or its attachments, in any form,
> is strictly prohibited. If you have received this message in error, please
> immediately notify the sender and/or Notifications@carrieriq.com and
> delete or destroy any copy of this message and its attachments.
>
Re: HBase atomic append functionality (not just client)
Posted by Vladimir Rodionov <vl...@gmail.com>.
Thanks, Lars
This JIRA is worth looking at. This hint will improve all reads: simple
gets, increments and appends.
-Vladimir
On Mon, Apr 14, 2014 at 1:46 PM, lars hofhansl <la...@apache.org> wrote:
> The jira would be this one: HBASE-10247.
>
> If the client promises not muck with timestamps we first look into the
> memstore. If we find a version there, we know it is the right one, only
> otherwise do we need to check the HFiles.
> (This is not possible if the client can set timestamps, because the latest
> value added to the memstore might be a backdated Put, or there might be a
> futuredated Delete in some of the HFiles.)
>
> -- Lars
>
>
> ----- Original Message -----
> From: Vladimir Rodionov <vr...@carrieriq.com>
> To: "dev@hbase.apache.org" <de...@hbase.apache.org>
> Cc: "user@hbase.apache.org" <us...@hbase.apache.org>
> Sent: Monday, April 14, 2014 1:22 PM
> Subject: RE: HBase atomic append functionality (not just client)
>
> Ted,
>
> The general idea of improving read-modify-write performance is to always
> read data from closest and fastest store only: either from MemStore or from
> block cache.
> The read pipeline in HBase will always read data from HFile as well to
> compare versions. This should be optimized with a read hint, for example.
> I remember Lars has mentioned some related JIRA? Not sure, though.
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: vrodionov@carrieriq.com
>
> ________________________________________
> From: Ted Yu [yuzhihong@gmail.com]
> Sent: Monday, April 14, 2014 12:46 PM
> To: dev@hbase.apache.org
> Cc: user@hbase.apache.org
> Subject: Re: HBase atomic append functionality (not just client)
>
> There was discussion on speeding up Append operation by only doing Put at
> write time.
> When reading, the Puts would be consolidated to produce the correct result.
>
> Let me try to find the JIRA related to the discussion.
>
> Cheers
>
>
> On Mon, Apr 14, 2014 at 10:41 AM, Vladimir Rodionov <
> vrodionov@carrieriq.com
> > wrote:
>
> > From HRegion.java:
> >
> > "Appends performed are done under row lock but reads do not take locks
> out
> > so this can be seen partially complete by gets and scans."
> >
> > Appends are partially atomic (you can get partial reads but you will
> never
> > get corrupted writes) and they are implemented on the server side.
> >
> > Best regards,
> > Vladimir Rodionov
> > Principal Platform Engineer
> > Carrier IQ, www.carrieriq.com
> > e-mail: vrodionov@carrieriq.com
> >
> > ________________________________________
> > From: GSK Chaitanya [gskchaitanya12@gmail.com]
> > Sent: Monday, April 14, 2014 10:05 AM
> > To: user@hbase.apache.org; dev@hbase.apache.org
> > Subject: HBase atomic append functionality (not just client)
> >
> > Mighty Hbase users and developers,
> >
> > I have few questions and I'd really appreciate it if someone can clarify
> > them.
> >
> > 1) I want to know if Hbase inherently supports *atomic
> > append*functionality like
> > *get* and *put*. For my work, I would be using OpenTSDB which is a layer
> on
> > top of AsynchHBase and AsynchHBase doesnt work with HBase client (which
> > supports *atomic append*).
> >
> > 2) If I understand correctly, atomic append of HBase client internally
> does
> > a get and put instead of actually appending to the end of the cell. If
> > that's the case, I wonder how does this functionality is of much use in
> > terms of performance. In our case, we would like a very light weight
> append
> > functionality. I'd like to know if there are any plans of adding this
> > feature to HBase main in the near future.
> >
> > Thanks,
> > Chaitanya
> >
> > Confidentiality Notice: The information contained in this message,
> > including any attachments hereto, may be confidential and is intended to
> be
> > read only by the individual or entity to whom this message is addressed.
> If
> > the reader of this message is not the intended recipient or an agent or
> > designee of the intended recipient, please note that any review, use,
> > disclosure or distribution of this message or its attachments, in any
> form,
> > is strictly prohibited. If you have received this message in error,
> please
> > immediately notify the sender and/or Notifications@carrieriq.com and
> > delete or destroy any copy of this message and its attachments.
>
> >
>
> Confidentiality Notice: The information contained in this message,
> including any attachments hereto, may be confidential and is intended to be
> read only by the individual or entity to whom this message is addressed. If
> the reader of this message is not the intended recipient or an agent or
> designee of the intended recipient, please note that any review, use,
> disclosure or distribution of this message or its attachments, in any form,
> is strictly prohibited. If you have received this message in error, please
> immediately notify the sender and/or Notifications@carrieriq.com and
> delete or destroy any copy of this message and its attachments.
>
Re: HBase atomic append functionality (not just client)
Posted by Vladimir Rodionov <vl...@gmail.com>.
Thanks, Lars
This JIRA is worth looking at. This hint will improve all reads: simple
gets, increments and appends.
-Vladimir
On Mon, Apr 14, 2014 at 1:46 PM, lars hofhansl <la...@apache.org> wrote:
> The jira would be this one: HBASE-10247.
>
> If the client promises not muck with timestamps we first look into the
> memstore. If we find a version there, we know it is the right one, only
> otherwise do we need to check the HFiles.
> (This is not possible if the client can set timestamps, because the latest
> value added to the memstore might be a backdated Put, or there might be a
> futuredated Delete in some of the HFiles.)
>
> -- Lars
>
>
> ----- Original Message -----
> From: Vladimir Rodionov <vr...@carrieriq.com>
> To: "dev@hbase.apache.org" <de...@hbase.apache.org>
> Cc: "user@hbase.apache.org" <us...@hbase.apache.org>
> Sent: Monday, April 14, 2014 1:22 PM
> Subject: RE: HBase atomic append functionality (not just client)
>
> Ted,
>
> The general idea of improving read-modify-write performance is to always
> read data from closest and fastest store only: either from MemStore or from
> block cache.
> The read pipeline in HBase will always read data from HFile as well to
> compare versions. This should be optimized with a read hint, for example.
> I remember Lars has mentioned some related JIRA? Not sure, though.
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: vrodionov@carrieriq.com
>
> ________________________________________
> From: Ted Yu [yuzhihong@gmail.com]
> Sent: Monday, April 14, 2014 12:46 PM
> To: dev@hbase.apache.org
> Cc: user@hbase.apache.org
> Subject: Re: HBase atomic append functionality (not just client)
>
> There was discussion on speeding up Append operation by only doing Put at
> write time.
> When reading, the Puts would be consolidated to produce the correct result.
>
> Let me try to find the JIRA related to the discussion.
>
> Cheers
>
>
> On Mon, Apr 14, 2014 at 10:41 AM, Vladimir Rodionov <
> vrodionov@carrieriq.com
> > wrote:
>
> > From HRegion.java:
> >
> > "Appends performed are done under row lock but reads do not take locks
> out
> > so this can be seen partially complete by gets and scans."
> >
> > Appends are partially atomic (you can get partial reads but you will
> never
> > get corrupted writes) and they are implemented on the server side.
> >
> > Best regards,
> > Vladimir Rodionov
> > Principal Platform Engineer
> > Carrier IQ, www.carrieriq.com
> > e-mail: vrodionov@carrieriq.com
> >
> > ________________________________________
> > From: GSK Chaitanya [gskchaitanya12@gmail.com]
> > Sent: Monday, April 14, 2014 10:05 AM
> > To: user@hbase.apache.org; dev@hbase.apache.org
> > Subject: HBase atomic append functionality (not just client)
> >
> > Mighty Hbase users and developers,
> >
> > I have few questions and I'd really appreciate it if someone can clarify
> > them.
> >
> > 1) I want to know if Hbase inherently supports *atomic
> > append*functionality like
> > *get* and *put*. For my work, I would be using OpenTSDB which is a layer
> on
> > top of AsynchHBase and AsynchHBase doesnt work with HBase client (which
> > supports *atomic append*).
> >
> > 2) If I understand correctly, atomic append of HBase client internally
> does
> > a get and put instead of actually appending to the end of the cell. If
> > that's the case, I wonder how does this functionality is of much use in
> > terms of performance. In our case, we would like a very light weight
> append
> > functionality. I'd like to know if there are any plans of adding this
> > feature to HBase main in the near future.
> >
> > Thanks,
> > Chaitanya
> >
> > Confidentiality Notice: The information contained in this message,
> > including any attachments hereto, may be confidential and is intended to
> be
> > read only by the individual or entity to whom this message is addressed.
> If
> > the reader of this message is not the intended recipient or an agent or
> > designee of the intended recipient, please note that any review, use,
> > disclosure or distribution of this message or its attachments, in any
> form,
> > is strictly prohibited. If you have received this message in error,
> please
> > immediately notify the sender and/or Notifications@carrieriq.com and
> > delete or destroy any copy of this message and its attachments.
>
> >
>
> Confidentiality Notice: The information contained in this message,
> including any attachments hereto, may be confidential and is intended to be
> read only by the individual or entity to whom this message is addressed. If
> the reader of this message is not the intended recipient or an agent or
> designee of the intended recipient, please note that any review, use,
> disclosure or distribution of this message or its attachments, in any form,
> is strictly prohibited. If you have received this message in error, please
> immediately notify the sender and/or Notifications@carrieriq.com and
> delete or destroy any copy of this message and its attachments.
>
Re: HBase atomic append functionality (not just client)
Posted by lars hofhansl <la...@apache.org>.
The jira would be this one: HBASE-10247.
If the client promises not muck with timestamps we first look into the memstore. If we find a version there, we know it is the right one, only otherwise do we need to check the HFiles.
(This is not possible if the client can set timestamps, because the latest value added to the memstore might be a backdated Put, or there might be a futuredated Delete in some of the HFiles.)
-- Lars
----- Original Message -----
From: Vladimir Rodionov <vr...@carrieriq.com>
To: "dev@hbase.apache.org" <de...@hbase.apache.org>
Cc: "user@hbase.apache.org" <us...@hbase.apache.org>
Sent: Monday, April 14, 2014 1:22 PM
Subject: RE: HBase atomic append functionality (not just client)
Ted,
The general idea of improving read-modify-write performance is to always read data from closest and fastest store only: either from MemStore or from block cache.
The read pipeline in HBase will always read data from HFile as well to compare versions. This should be optimized with a read hint, for example.
I remember Lars has mentioned some related JIRA? Not sure, though.
Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodionov@carrieriq.com
________________________________________
From: Ted Yu [yuzhihong@gmail.com]
Sent: Monday, April 14, 2014 12:46 PM
To: dev@hbase.apache.org
Cc: user@hbase.apache.org
Subject: Re: HBase atomic append functionality (not just client)
There was discussion on speeding up Append operation by only doing Put at
write time.
When reading, the Puts would be consolidated to produce the correct result.
Let me try to find the JIRA related to the discussion.
Cheers
On Mon, Apr 14, 2014 at 10:41 AM, Vladimir Rodionov <vrodionov@carrieriq.com
> wrote:
> From HRegion.java:
>
> "Appends performed are done under row lock but reads do not take locks out
> so this can be seen partially complete by gets and scans."
>
> Appends are partially atomic (you can get partial reads but you will never
> get corrupted writes) and they are implemented on the server side.
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: vrodionov@carrieriq.com
>
> ________________________________________
> From: GSK Chaitanya [gskchaitanya12@gmail.com]
> Sent: Monday, April 14, 2014 10:05 AM
> To: user@hbase.apache.org; dev@hbase.apache.org
> Subject: HBase atomic append functionality (not just client)
>
> Mighty Hbase users and developers,
>
> I have few questions and I'd really appreciate it if someone can clarify
> them.
>
> 1) I want to know if Hbase inherently supports *atomic
> append*functionality like
> *get* and *put*. For my work, I would be using OpenTSDB which is a layer on
> top of AsynchHBase and AsynchHBase doesnt work with HBase client (which
> supports *atomic append*).
>
> 2) If I understand correctly, atomic append of HBase client internally does
> a get and put instead of actually appending to the end of the cell. If
> that's the case, I wonder how does this functionality is of much use in
> terms of performance. In our case, we would like a very light weight append
> functionality. I'd like to know if there are any plans of adding this
> feature to HBase main in the near future.
>
> Thanks,
> Chaitanya
>
> Confidentiality Notice: The information contained in this message,
> including any attachments hereto, may be confidential and is intended to be
> read only by the individual or entity to whom this message is addressed. If
> the reader of this message is not the intended recipient or an agent or
> designee of the intended recipient, please note that any review, use,
> disclosure or distribution of this message or its attachments, in any form,
> is strictly prohibited. If you have received this message in error, please
> immediately notify the sender and/or Notifications@carrieriq.com and
> delete or destroy any copy of this message and its attachments.
>
Confidentiality Notice: The information contained in this message, including any attachments hereto, may be confidential and is intended to be read only by the individual or entity to whom this message is addressed. If the reader of this message is not the intended recipient or an agent or designee of the intended recipient, please note that any review, use, disclosure or distribution of this message or its attachments, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or Notifications@carrieriq.com and delete or destroy any copy of this message and its attachments.
Re: HBase atomic append functionality (not just client)
Posted by lars hofhansl <la...@apache.org>.
The jira would be this one: HBASE-10247.
If the client promises not muck with timestamps we first look into the memstore. If we find a version there, we know it is the right one, only otherwise do we need to check the HFiles.
(This is not possible if the client can set timestamps, because the latest value added to the memstore might be a backdated Put, or there might be a futuredated Delete in some of the HFiles.)
-- Lars
----- Original Message -----
From: Vladimir Rodionov <vr...@carrieriq.com>
To: "dev@hbase.apache.org" <de...@hbase.apache.org>
Cc: "user@hbase.apache.org" <us...@hbase.apache.org>
Sent: Monday, April 14, 2014 1:22 PM
Subject: RE: HBase atomic append functionality (not just client)
Ted,
The general idea of improving read-modify-write performance is to always read data from closest and fastest store only: either from MemStore or from block cache.
The read pipeline in HBase will always read data from HFile as well to compare versions. This should be optimized with a read hint, for example.
I remember Lars has mentioned some related JIRA? Not sure, though.
Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodionov@carrieriq.com
________________________________________
From: Ted Yu [yuzhihong@gmail.com]
Sent: Monday, April 14, 2014 12:46 PM
To: dev@hbase.apache.org
Cc: user@hbase.apache.org
Subject: Re: HBase atomic append functionality (not just client)
There was discussion on speeding up Append operation by only doing Put at
write time.
When reading, the Puts would be consolidated to produce the correct result.
Let me try to find the JIRA related to the discussion.
Cheers
On Mon, Apr 14, 2014 at 10:41 AM, Vladimir Rodionov <vrodionov@carrieriq.com
> wrote:
> From HRegion.java:
>
> "Appends performed are done under row lock but reads do not take locks out
> so this can be seen partially complete by gets and scans."
>
> Appends are partially atomic (you can get partial reads but you will never
> get corrupted writes) and they are implemented on the server side.
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: vrodionov@carrieriq.com
>
> ________________________________________
> From: GSK Chaitanya [gskchaitanya12@gmail.com]
> Sent: Monday, April 14, 2014 10:05 AM
> To: user@hbase.apache.org; dev@hbase.apache.org
> Subject: HBase atomic append functionality (not just client)
>
> Mighty Hbase users and developers,
>
> I have few questions and I'd really appreciate it if someone can clarify
> them.
>
> 1) I want to know if Hbase inherently supports *atomic
> append*functionality like
> *get* and *put*. For my work, I would be using OpenTSDB which is a layer on
> top of AsynchHBase and AsynchHBase doesnt work with HBase client (which
> supports *atomic append*).
>
> 2) If I understand correctly, atomic append of HBase client internally does
> a get and put instead of actually appending to the end of the cell. If
> that's the case, I wonder how does this functionality is of much use in
> terms of performance. In our case, we would like a very light weight append
> functionality. I'd like to know if there are any plans of adding this
> feature to HBase main in the near future.
>
> Thanks,
> Chaitanya
>
> Confidentiality Notice: The information contained in this message,
> including any attachments hereto, may be confidential and is intended to be
> read only by the individual or entity to whom this message is addressed. If
> the reader of this message is not the intended recipient or an agent or
> designee of the intended recipient, please note that any review, use,
> disclosure or distribution of this message or its attachments, in any form,
> is strictly prohibited. If you have received this message in error, please
> immediately notify the sender and/or Notifications@carrieriq.com and
> delete or destroy any copy of this message and its attachments.
>
Confidentiality Notice: The information contained in this message, including any attachments hereto, may be confidential and is intended to be read only by the individual or entity to whom this message is addressed. If the reader of this message is not the intended recipient or an agent or designee of the intended recipient, please note that any review, use, disclosure or distribution of this message or its attachments, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or Notifications@carrieriq.com and delete or destroy any copy of this message and its attachments.
RE: HBase atomic append functionality (not just client)
Posted by Vladimir Rodionov <vr...@carrieriq.com>.
Ted,
The general idea of improving read-modify-write performance is to always read data from closest and fastest store only: either from MemStore or from block cache.
The read pipeline in HBase will always read data from HFile as well to compare versions. This should be optimized with a read hint, for example.
I remember Lars has mentioned some related JIRA? Not sure, though.
Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodionov@carrieriq.com
________________________________________
From: Ted Yu [yuzhihong@gmail.com]
Sent: Monday, April 14, 2014 12:46 PM
To: dev@hbase.apache.org
Cc: user@hbase.apache.org
Subject: Re: HBase atomic append functionality (not just client)
There was discussion on speeding up Append operation by only doing Put at
write time.
When reading, the Puts would be consolidated to produce the correct result.
Let me try to find the JIRA related to the discussion.
Cheers
On Mon, Apr 14, 2014 at 10:41 AM, Vladimir Rodionov <vrodionov@carrieriq.com
> wrote:
> From HRegion.java:
>
> "Appends performed are done under row lock but reads do not take locks out
> so this can be seen partially complete by gets and scans."
>
> Appends are partially atomic (you can get partial reads but you will never
> get corrupted writes) and they are implemented on the server side.
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: vrodionov@carrieriq.com
>
> ________________________________________
> From: GSK Chaitanya [gskchaitanya12@gmail.com]
> Sent: Monday, April 14, 2014 10:05 AM
> To: user@hbase.apache.org; dev@hbase.apache.org
> Subject: HBase atomic append functionality (not just client)
>
> Mighty Hbase users and developers,
>
> I have few questions and I'd really appreciate it if someone can clarify
> them.
>
> 1) I want to know if Hbase inherently supports *atomic
> append*functionality like
> *get* and *put*. For my work, I would be using OpenTSDB which is a layer on
> top of AsynchHBase and AsynchHBase doesnt work with HBase client (which
> supports *atomic append*).
>
> 2) If I understand correctly, atomic append of HBase client internally does
> a get and put instead of actually appending to the end of the cell. If
> that's the case, I wonder how does this functionality is of much use in
> terms of performance. In our case, we would like a very light weight append
> functionality. I'd like to know if there are any plans of adding this
> feature to HBase main in the near future.
>
> Thanks,
> Chaitanya
>
> Confidentiality Notice: The information contained in this message,
> including any attachments hereto, may be confidential and is intended to be
> read only by the individual or entity to whom this message is addressed. If
> the reader of this message is not the intended recipient or an agent or
> designee of the intended recipient, please note that any review, use,
> disclosure or distribution of this message or its attachments, in any form,
> is strictly prohibited. If you have received this message in error, please
> immediately notify the sender and/or Notifications@carrieriq.com and
> delete or destroy any copy of this message and its attachments.
>
Confidentiality Notice: The information contained in this message, including any attachments hereto, may be confidential and is intended to be read only by the individual or entity to whom this message is addressed. If the reader of this message is not the intended recipient or an agent or designee of the intended recipient, please note that any review, use, disclosure or distribution of this message or its attachments, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or Notifications@carrieriq.com and delete or destroy any copy of this message and its attachments.
RE: HBase atomic append functionality (not just client)
Posted by Vladimir Rodionov <vr...@carrieriq.com>.
Ted,
The general idea of improving read-modify-write performance is to always read data from closest and fastest store only: either from MemStore or from block cache.
The read pipeline in HBase will always read data from HFile as well to compare versions. This should be optimized with a read hint, for example.
I remember Lars has mentioned some related JIRA? Not sure, though.
Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodionov@carrieriq.com
________________________________________
From: Ted Yu [yuzhihong@gmail.com]
Sent: Monday, April 14, 2014 12:46 PM
To: dev@hbase.apache.org
Cc: user@hbase.apache.org
Subject: Re: HBase atomic append functionality (not just client)
There was discussion on speeding up Append operation by only doing Put at
write time.
When reading, the Puts would be consolidated to produce the correct result.
Let me try to find the JIRA related to the discussion.
Cheers
On Mon, Apr 14, 2014 at 10:41 AM, Vladimir Rodionov <vrodionov@carrieriq.com
> wrote:
> From HRegion.java:
>
> "Appends performed are done under row lock but reads do not take locks out
> so this can be seen partially complete by gets and scans."
>
> Appends are partially atomic (you can get partial reads but you will never
> get corrupted writes) and they are implemented on the server side.
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: vrodionov@carrieriq.com
>
> ________________________________________
> From: GSK Chaitanya [gskchaitanya12@gmail.com]
> Sent: Monday, April 14, 2014 10:05 AM
> To: user@hbase.apache.org; dev@hbase.apache.org
> Subject: HBase atomic append functionality (not just client)
>
> Mighty Hbase users and developers,
>
> I have few questions and I'd really appreciate it if someone can clarify
> them.
>
> 1) I want to know if Hbase inherently supports *atomic
> append*functionality like
> *get* and *put*. For my work, I would be using OpenTSDB which is a layer on
> top of AsynchHBase and AsynchHBase doesnt work with HBase client (which
> supports *atomic append*).
>
> 2) If I understand correctly, atomic append of HBase client internally does
> a get and put instead of actually appending to the end of the cell. If
> that's the case, I wonder how does this functionality is of much use in
> terms of performance. In our case, we would like a very light weight append
> functionality. I'd like to know if there are any plans of adding this
> feature to HBase main in the near future.
>
> Thanks,
> Chaitanya
>
> Confidentiality Notice: The information contained in this message,
> including any attachments hereto, may be confidential and is intended to be
> read only by the individual or entity to whom this message is addressed. If
> the reader of this message is not the intended recipient or an agent or
> designee of the intended recipient, please note that any review, use,
> disclosure or distribution of this message or its attachments, in any form,
> is strictly prohibited. If you have received this message in error, please
> immediately notify the sender and/or Notifications@carrieriq.com and
> delete or destroy any copy of this message and its attachments.
>
Confidentiality Notice: The information contained in this message, including any attachments hereto, may be confidential and is intended to be read only by the individual or entity to whom this message is addressed. If the reader of this message is not the intended recipient or an agent or designee of the intended recipient, please note that any review, use, disclosure or distribution of this message or its attachments, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or Notifications@carrieriq.com and delete or destroy any copy of this message and its attachments.
Re: HBase atomic append functionality (not just client)
Posted by Ted Yu <yu...@gmail.com>.
There was discussion on speeding up Append operation by only doing Put at
write time.
When reading, the Puts would be consolidated to produce the correct result.
Let me try to find the JIRA related to the discussion.
Cheers
On Mon, Apr 14, 2014 at 10:41 AM, Vladimir Rodionov <vrodionov@carrieriq.com
> wrote:
> From HRegion.java:
>
> "Appends performed are done under row lock but reads do not take locks out
> so this can be seen partially complete by gets and scans."
>
> Appends are partially atomic (you can get partial reads but you will never
> get corrupted writes) and they are implemented on the server side.
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: vrodionov@carrieriq.com
>
> ________________________________________
> From: GSK Chaitanya [gskchaitanya12@gmail.com]
> Sent: Monday, April 14, 2014 10:05 AM
> To: user@hbase.apache.org; dev@hbase.apache.org
> Subject: HBase atomic append functionality (not just client)
>
> Mighty Hbase users and developers,
>
> I have few questions and I'd really appreciate it if someone can clarify
> them.
>
> 1) I want to know if Hbase inherently supports *atomic
> append*functionality like
> *get* and *put*. For my work, I would be using OpenTSDB which is a layer on
> top of AsynchHBase and AsynchHBase doesnt work with HBase client (which
> supports *atomic append*).
>
> 2) If I understand correctly, atomic append of HBase client internally does
> a get and put instead of actually appending to the end of the cell. If
> that's the case, I wonder how does this functionality is of much use in
> terms of performance. In our case, we would like a very light weight append
> functionality. I'd like to know if there are any plans of adding this
> feature to HBase main in the near future.
>
> Thanks,
> Chaitanya
>
> Confidentiality Notice: The information contained in this message,
> including any attachments hereto, may be confidential and is intended to be
> read only by the individual or entity to whom this message is addressed. If
> the reader of this message is not the intended recipient or an agent or
> designee of the intended recipient, please note that any review, use,
> disclosure or distribution of this message or its attachments, in any form,
> is strictly prohibited. If you have received this message in error, please
> immediately notify the sender and/or Notifications@carrieriq.com and
> delete or destroy any copy of this message and its attachments.
>
Re: HBase atomic append functionality (not just client)
Posted by Ted Yu <yu...@gmail.com>.
There was discussion on speeding up Append operation by only doing Put at
write time.
When reading, the Puts would be consolidated to produce the correct result.
Let me try to find the JIRA related to the discussion.
Cheers
On Mon, Apr 14, 2014 at 10:41 AM, Vladimir Rodionov <vrodionov@carrieriq.com
> wrote:
> From HRegion.java:
>
> "Appends performed are done under row lock but reads do not take locks out
> so this can be seen partially complete by gets and scans."
>
> Appends are partially atomic (you can get partial reads but you will never
> get corrupted writes) and they are implemented on the server side.
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: vrodionov@carrieriq.com
>
> ________________________________________
> From: GSK Chaitanya [gskchaitanya12@gmail.com]
> Sent: Monday, April 14, 2014 10:05 AM
> To: user@hbase.apache.org; dev@hbase.apache.org
> Subject: HBase atomic append functionality (not just client)
>
> Mighty Hbase users and developers,
>
> I have few questions and I'd really appreciate it if someone can clarify
> them.
>
> 1) I want to know if Hbase inherently supports *atomic
> append*functionality like
> *get* and *put*. For my work, I would be using OpenTSDB which is a layer on
> top of AsynchHBase and AsynchHBase doesnt work with HBase client (which
> supports *atomic append*).
>
> 2) If I understand correctly, atomic append of HBase client internally does
> a get and put instead of actually appending to the end of the cell. If
> that's the case, I wonder how does this functionality is of much use in
> terms of performance. In our case, we would like a very light weight append
> functionality. I'd like to know if there are any plans of adding this
> feature to HBase main in the near future.
>
> Thanks,
> Chaitanya
>
> Confidentiality Notice: The information contained in this message,
> including any attachments hereto, may be confidential and is intended to be
> read only by the individual or entity to whom this message is addressed. If
> the reader of this message is not the intended recipient or an agent or
> designee of the intended recipient, please note that any review, use,
> disclosure or distribution of this message or its attachments, in any form,
> is strictly prohibited. If you have received this message in error, please
> immediately notify the sender and/or Notifications@carrieriq.com and
> delete or destroy any copy of this message and its attachments.
>
RE: HBase atomic append functionality (not just client)
Posted by Vladimir Rodionov <vr...@carrieriq.com>.
>From HRegion.java:
"Appends performed are done under row lock but reads do not take locks out so this can be seen partially complete by gets and scans."
Appends are partially atomic (you can get partial reads but you will never get corrupted writes) and they are implemented on the server side.
Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodionov@carrieriq.com
________________________________________
From: GSK Chaitanya [gskchaitanya12@gmail.com]
Sent: Monday, April 14, 2014 10:05 AM
To: user@hbase.apache.org; dev@hbase.apache.org
Subject: HBase atomic append functionality (not just client)
Mighty Hbase users and developers,
I have few questions and I'd really appreciate it if someone can clarify
them.
1) I want to know if Hbase inherently supports *atomic
append*functionality like
*get* and *put*. For my work, I would be using OpenTSDB which is a layer on
top of AsynchHBase and AsynchHBase doesnt work with HBase client (which
supports *atomic append*).
2) If I understand correctly, atomic append of HBase client internally does
a get and put instead of actually appending to the end of the cell. If
that's the case, I wonder how does this functionality is of much use in
terms of performance. In our case, we would like a very light weight append
functionality. I'd like to know if there are any plans of adding this
feature to HBase main in the near future.
Thanks,
Chaitanya
Confidentiality Notice: The information contained in this message, including any attachments hereto, may be confidential and is intended to be read only by the individual or entity to whom this message is addressed. If the reader of this message is not the intended recipient or an agent or designee of the intended recipient, please note that any review, use, disclosure or distribution of this message or its attachments, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or Notifications@carrieriq.com and delete or destroy any copy of this message and its attachments.
RE: HBase atomic append functionality (not just client)
Posted by Vladimir Rodionov <vr...@carrieriq.com>.
>From HRegion.java:
"Appends performed are done under row lock but reads do not take locks out so this can be seen partially complete by gets and scans."
Appends are partially atomic (you can get partial reads but you will never get corrupted writes) and they are implemented on the server side.
Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodionov@carrieriq.com
________________________________________
From: GSK Chaitanya [gskchaitanya12@gmail.com]
Sent: Monday, April 14, 2014 10:05 AM
To: user@hbase.apache.org; dev@hbase.apache.org
Subject: HBase atomic append functionality (not just client)
Mighty Hbase users and developers,
I have few questions and I'd really appreciate it if someone can clarify
them.
1) I want to know if Hbase inherently supports *atomic
append*functionality like
*get* and *put*. For my work, I would be using OpenTSDB which is a layer on
top of AsynchHBase and AsynchHBase doesnt work with HBase client (which
supports *atomic append*).
2) If I understand correctly, atomic append of HBase client internally does
a get and put instead of actually appending to the end of the cell. If
that's the case, I wonder how does this functionality is of much use in
terms of performance. In our case, we would like a very light weight append
functionality. I'd like to know if there are any plans of adding this
feature to HBase main in the near future.
Thanks,
Chaitanya
Confidentiality Notice: The information contained in this message, including any attachments hereto, may be confidential and is intended to be read only by the individual or entity to whom this message is addressed. If the reader of this message is not the intended recipient or an agent or designee of the intended recipient, please note that any review, use, disclosure or distribution of this message or its attachments, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or Notifications@carrieriq.com and delete or destroy any copy of this message and its attachments.