You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Anthony Nguyen <an...@gmail.com> on 2015/09/10 01:23:51 UTC
Ramifications of minimizing use of .tmp directories / renames in
HBase when using S3 as backing store
Hi all,
I'm investigating the use of S3 as a backing store for HBase. Would there
be any major issues with modifying HBase in such a way where when an S3
location is set for the rootdir, writes to .tmp are removed and minimized,
instead writing directly to the final destination? The reason I'd like to
do this is because renames in S3 are expensive and performance for
operations such as compactions and snapshot restores that have many renames
suffer.
Thanks!
Re: Ramifications of minimizing use of .tmp directories / renames in
HBase when using S3 as backing store
Posted by Anthony Nguyen <an...@gmail.com>.
Thanks Matteo. If I understand correctly, one example of how the .tmp
directories help prevent issues is as follows: If HBase were to crash
during a compaction, since these .tmp directories are cleared out at start,
cleanup is much easier, right?
On Wed, Sep 9, 2015 at 7:31 PM, Matteo Bertozzi <th...@gmail.com>
wrote:
> hbase relies on .tmp directories to do some sort of "atomic" file creation.
> and avoid problems like half data written when it crashes.
>
> there is a jira open, to solve that problem in one of the next major
> releases:
> https://issues.apache.org/jira/browse/HBASE-14090
> There is a document in it, if you are interested in reading about the
> internals.
>
> Matteo
>
>
> On Wed, Sep 9, 2015 at 4:23 PM, Anthony Nguyen <
> anthony.an.nguyen@gmail.com>
> wrote:
>
> > Hi all,
> >
> > I'm investigating the use of S3 as a backing store for HBase. Would there
> > be any major issues with modifying HBase in such a way where when an S3
> > location is set for the rootdir, writes to .tmp are removed and
> minimized,
> > instead writing directly to the final destination? The reason I'd like to
> > do this is because renames in S3 are expensive and performance for
> > operations such as compactions and snapshot restores that have many
> renames
> > suffer.
> >
> > Thanks!
> >
>
Re: Ramifications of minimizing use of .tmp directories / renames in
HBase when using S3 as backing store
Posted by Matteo Bertozzi <th...@gmail.com>.
hbase relies on .tmp directories to do some sort of "atomic" file creation.
and avoid problems like half data written when it crashes.
there is a jira open, to solve that problem in one of the next major
releases:
https://issues.apache.org/jira/browse/HBASE-14090
There is a document in it, if you are interested in reading about the
internals.
Matteo
On Wed, Sep 9, 2015 at 4:23 PM, Anthony Nguyen <an...@gmail.com>
wrote:
> Hi all,
>
> I'm investigating the use of S3 as a backing store for HBase. Would there
> be any major issues with modifying HBase in such a way where when an S3
> location is set for the rootdir, writes to .tmp are removed and minimized,
> instead writing directly to the final destination? The reason I'd like to
> do this is because renames in S3 are expensive and performance for
> operations such as compactions and snapshot restores that have many renames
> suffer.
>
> Thanks!
>
Re: Ramifications of minimizing use of .tmp directories / renames in
HBase when using S3 as backing store
Posted by iain wright <ia...@gmail.com>.
I see -
If it was only M/R, Hive, or a similar reporting/analytics workload without
the low latency get's/read requirement, I was going to suggest writing to
S3 directly and using spark+hivecontext.
Cheers,
--
Iain Wright
This email message is confidential, intended only for the recipient(s)
named above and may contain information that is privileged, exempt from
disclosure under applicable law. If you are not the intended recipient, do
not disclose or disseminate the message to anyone except the intended
recipient. If you have received this message in error, or are not the named
recipient(s), please immediately notify the sender by return email, and
delete all copies of this message.
On Wed, Sep 9, 2015 at 6:38 PM, Anthony Nguyen <an...@gmail.com>
wrote:
> Hi Iain,
>
> Random reads for now, with further MR work w/ Hive possibly down the line.
>
> Thanks,
> -t
>
> On Wed, Sep 9, 2015 at 9:31 PM, iain wright <ia...@gmail.com> wrote:
>
> > Hi Anthony,
> >
> > Just curious, you mention your access pattern is mostly reads. Is it
> random
> > reads, M/R jobs over a portion of the dataset, or other?
> >
> > Best,
> >
> > --
> > Iain Wright
> >
> > This email message is confidential, intended only for the recipient(s)
> > named above and may contain information that is privileged, exempt from
> > disclosure under applicable law. If you are not the intended recipient,
> do
> > not disclose or disseminate the message to anyone except the intended
> > recipient. If you have received this message in error, or are not the
> named
> > recipient(s), please immediately notify the sender by return email, and
> > delete all copies of this message.
> >
> > On Wed, Sep 9, 2015 at 5:55 PM, Anthony Nguyen <
> > anthony.an.nguyen@gmail.com>
> > wrote:
> >
> > > Hi Andrew,
> > >
> > > Thanks for your help! I'm attempting to give it Hbase on S3 another try
> > :).
> > > Let me see if I can address the two things you pointed out and you can
> > call
> > > me crazy for even thinking that they'd be okay:
> > >
> > > First, some background - my initial use case will be using HBase in a
> > > mostly read-only fashion (bulk loads are the primary method of data
> > load).
> > >
> > > Consistency issues: Since S3 has read after write consistency for new
> > > objects, we could potentially structure the naming scheme within HBase
> > file
> > > writes such that the same location is never overwritten. In this way,
> any
> > > new files (which would include updates / appends to an ultimately new
> > file)
> > > in S3 will be immediately consistent. Files can be cleaned up after a
> set
> > > period of time.
> > >
> > > Appends: An append in S3 can be modeled as a read / copy / delete
> > > operation. Performance will definitely suffer, but we can configure WAL
> > > settings to perhaps roll the WAL very often to minimize appends, right?
> > >
> > > Atomic folder renames can also probably be worked around too, similar
> to
> > > how hadoop-azure does it.
> > >
> > > Thanks again. Looking forward to your insight.
> > >
> > > On Wed, Sep 9, 2015 at 7:32 PM, Andrew Purtell <ap...@apache.org>
> > > wrote:
> > >
> > > > It cannot work to use S3 as a backing store for HBase. This has been
> > > > attempted in the past (although not by me, so this isn't firsthand
> > > > knowledge). One basic problem is HBase expects to be able to read
> what
> > it
> > > > has written immediately after the write completes. For example,
> > opening a
> > > > store file after a flush to service reads, or reopening store files
> > > after a
> > > > compaction. S3's notion of availability is too eventual. You can get
> > 404
> > > or
> > > > 500 responses for some time after completing a file write. HBase will
> > > > panic. I'm not sure we'd accept kludges around this problem. Another
> > > > fundamental problem is S3 doesn't provide any means for appending to
> > > files,
> > > > so the HBase write ahead log cannot work. I suppose this could be
> > worked
> > > > around with changes that allow specification of one type of
> filesystem
> > > for
> > > > HFiles and another for write ahead logs. There are numerous finer
> > points
> > > > about HDFS behavior used as synchronization primitive, such as atomic
> > > > renames.
> > > >
> > > >
> > > > On Wed, Sep 9, 2015 at 4:23 PM, Anthony Nguyen <
> > > > anthony.an.nguyen@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I'm investigating the use of S3 as a backing store for HBase. Would
> > > there
> > > > > be any major issues with modifying HBase in such a way where when
> an
> > S3
> > > > > location is set for the rootdir, writes to .tmp are removed and
> > > > minimized,
> > > > > instead writing directly to the final destination? The reason I'd
> > like
> > > to
> > > > > do this is because renames in S3 are expensive and performance for
> > > > > operations such as compactions and snapshot restores that have many
> > > > renames
> > > > > suffer.
> > > > >
> > > > > Thanks!
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > >
> > > > - Andy
> > > >
> > > > Problems worthy of attack prove their worth by hitting back. - Piet
> > Hein
> > > > (via Tom White)
> > > >
> > >
> >
>
Re: Ramifications of minimizing use of .tmp directories / renames in
HBase when using S3 as backing store
Posted by Anthony Nguyen <an...@gmail.com>.
Hi Iain,
Random reads for now, with further MR work w/ Hive possibly down the line.
Thanks,
-t
On Wed, Sep 9, 2015 at 9:31 PM, iain wright <ia...@gmail.com> wrote:
> Hi Anthony,
>
> Just curious, you mention your access pattern is mostly reads. Is it random
> reads, M/R jobs over a portion of the dataset, or other?
>
> Best,
>
> --
> Iain Wright
>
> This email message is confidential, intended only for the recipient(s)
> named above and may contain information that is privileged, exempt from
> disclosure under applicable law. If you are not the intended recipient, do
> not disclose or disseminate the message to anyone except the intended
> recipient. If you have received this message in error, or are not the named
> recipient(s), please immediately notify the sender by return email, and
> delete all copies of this message.
>
> On Wed, Sep 9, 2015 at 5:55 PM, Anthony Nguyen <
> anthony.an.nguyen@gmail.com>
> wrote:
>
> > Hi Andrew,
> >
> > Thanks for your help! I'm attempting to give it Hbase on S3 another try
> :).
> > Let me see if I can address the two things you pointed out and you can
> call
> > me crazy for even thinking that they'd be okay:
> >
> > First, some background - my initial use case will be using HBase in a
> > mostly read-only fashion (bulk loads are the primary method of data
> load).
> >
> > Consistency issues: Since S3 has read after write consistency for new
> > objects, we could potentially structure the naming scheme within HBase
> file
> > writes such that the same location is never overwritten. In this way, any
> > new files (which would include updates / appends to an ultimately new
> file)
> > in S3 will be immediately consistent. Files can be cleaned up after a set
> > period of time.
> >
> > Appends: An append in S3 can be modeled as a read / copy / delete
> > operation. Performance will definitely suffer, but we can configure WAL
> > settings to perhaps roll the WAL very often to minimize appends, right?
> >
> > Atomic folder renames can also probably be worked around too, similar to
> > how hadoop-azure does it.
> >
> > Thanks again. Looking forward to your insight.
> >
> > On Wed, Sep 9, 2015 at 7:32 PM, Andrew Purtell <ap...@apache.org>
> > wrote:
> >
> > > It cannot work to use S3 as a backing store for HBase. This has been
> > > attempted in the past (although not by me, so this isn't firsthand
> > > knowledge). One basic problem is HBase expects to be able to read what
> it
> > > has written immediately after the write completes. For example,
> opening a
> > > store file after a flush to service reads, or reopening store files
> > after a
> > > compaction. S3's notion of availability is too eventual. You can get
> 404
> > or
> > > 500 responses for some time after completing a file write. HBase will
> > > panic. I'm not sure we'd accept kludges around this problem. Another
> > > fundamental problem is S3 doesn't provide any means for appending to
> > files,
> > > so the HBase write ahead log cannot work. I suppose this could be
> worked
> > > around with changes that allow specification of one type of filesystem
> > for
> > > HFiles and another for write ahead logs. There are numerous finer
> points
> > > about HDFS behavior used as synchronization primitive, such as atomic
> > > renames.
> > >
> > >
> > > On Wed, Sep 9, 2015 at 4:23 PM, Anthony Nguyen <
> > > anthony.an.nguyen@gmail.com>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I'm investigating the use of S3 as a backing store for HBase. Would
> > there
> > > > be any major issues with modifying HBase in such a way where when an
> S3
> > > > location is set for the rootdir, writes to .tmp are removed and
> > > minimized,
> > > > instead writing directly to the final destination? The reason I'd
> like
> > to
> > > > do this is because renames in S3 are expensive and performance for
> > > > operations such as compactions and snapshot restores that have many
> > > renames
> > > > suffer.
> > > >
> > > > Thanks!
> > > >
> > >
> > >
> > >
> > > --
> > > Best regards,
> > >
> > > - Andy
> > >
> > > Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> > > (via Tom White)
> > >
> >
>
Re: Ramifications of minimizing use of .tmp directories / renames in
HBase when using S3 as backing store
Posted by iain wright <ia...@gmail.com>.
Hi Anthony,
Just curious, you mention your access pattern is mostly reads. Is it random
reads, M/R jobs over a portion of the dataset, or other?
Best,
--
Iain Wright
This email message is confidential, intended only for the recipient(s)
named above and may contain information that is privileged, exempt from
disclosure under applicable law. If you are not the intended recipient, do
not disclose or disseminate the message to anyone except the intended
recipient. If you have received this message in error, or are not the named
recipient(s), please immediately notify the sender by return email, and
delete all copies of this message.
On Wed, Sep 9, 2015 at 5:55 PM, Anthony Nguyen <an...@gmail.com>
wrote:
> Hi Andrew,
>
> Thanks for your help! I'm attempting to give it Hbase on S3 another try :).
> Let me see if I can address the two things you pointed out and you can call
> me crazy for even thinking that they'd be okay:
>
> First, some background - my initial use case will be using HBase in a
> mostly read-only fashion (bulk loads are the primary method of data load).
>
> Consistency issues: Since S3 has read after write consistency for new
> objects, we could potentially structure the naming scheme within HBase file
> writes such that the same location is never overwritten. In this way, any
> new files (which would include updates / appends to an ultimately new file)
> in S3 will be immediately consistent. Files can be cleaned up after a set
> period of time.
>
> Appends: An append in S3 can be modeled as a read / copy / delete
> operation. Performance will definitely suffer, but we can configure WAL
> settings to perhaps roll the WAL very often to minimize appends, right?
>
> Atomic folder renames can also probably be worked around too, similar to
> how hadoop-azure does it.
>
> Thanks again. Looking forward to your insight.
>
> On Wed, Sep 9, 2015 at 7:32 PM, Andrew Purtell <ap...@apache.org>
> wrote:
>
> > It cannot work to use S3 as a backing store for HBase. This has been
> > attempted in the past (although not by me, so this isn't firsthand
> > knowledge). One basic problem is HBase expects to be able to read what it
> > has written immediately after the write completes. For example, opening a
> > store file after a flush to service reads, or reopening store files
> after a
> > compaction. S3's notion of availability is too eventual. You can get 404
> or
> > 500 responses for some time after completing a file write. HBase will
> > panic. I'm not sure we'd accept kludges around this problem. Another
> > fundamental problem is S3 doesn't provide any means for appending to
> files,
> > so the HBase write ahead log cannot work. I suppose this could be worked
> > around with changes that allow specification of one type of filesystem
> for
> > HFiles and another for write ahead logs. There are numerous finer points
> > about HDFS behavior used as synchronization primitive, such as atomic
> > renames.
> >
> >
> > On Wed, Sep 9, 2015 at 4:23 PM, Anthony Nguyen <
> > anthony.an.nguyen@gmail.com>
> > wrote:
> >
> > > Hi all,
> > >
> > > I'm investigating the use of S3 as a backing store for HBase. Would
> there
> > > be any major issues with modifying HBase in such a way where when an S3
> > > location is set for the rootdir, writes to .tmp are removed and
> > minimized,
> > > instead writing directly to the final destination? The reason I'd like
> to
> > > do this is because renames in S3 are expensive and performance for
> > > operations such as compactions and snapshot restores that have many
> > renames
> > > suffer.
> > >
> > > Thanks!
> > >
> >
> >
> >
> > --
> > Best regards,
> >
> > - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>
Re: Ramifications of minimizing use of .tmp directories / renames in
HBase when using S3 as backing store
Posted by Andrew Purtell <ap...@apache.org>.
> Consistency issues: Since S3 has read after write consistency for new
>
objects
Eventually. The problem is reads for the new objects may fail for some
arbitrary time first, with 404 or 500 responses as I mentioned before.
>
Appends: An append in S3 can be modeled as a read / copy / delete
operation. Performance will definitely suffer, but we can configure WAL
settings to perhaps roll the WAL very often to minimize appends, right?
No, every single mutation sent to HBase is committed to the WAL first
(with an append then a flush) and only then do we return to the client. If
you batch edits before writing this abandons HBase's durability guarantees.
If you write exactly one edit per WAL this will produce terrible
performance. A client may be able to write a few values to HBase per second.
S3 storage isn't workable.
On Wed, Sep 9, 2015 at 5:55 PM, Anthony Nguyen <an...@gmail.com>
wrote:
> Hi Andrew,
>
> Thanks for your help! I'm attempting to give it Hbase on S3 another try :).
> Let me see if I can address the two things you pointed out and you can call
> me crazy for even thinking that they'd be okay:
>
> First, some background - my initial use case will be using HBase in a
> mostly read-only fashion (bulk loads are the primary method of data load).
>
> Consistency issues: Since S3 has read after write consistency for new
> objects, we could potentially structure the naming scheme within HBase file
> writes such that the same location is never overwritten. In this way, any
> new files (which would include updates / appends to an ultimately new file)
> in S3 will be immediately consistent. Files can be cleaned up after a set
> period of time.
>
> Appends: An append in S3 can be modeled as a read / copy / delete
> operation. Performance will definitely suffer, but we can configure WAL
> settings to perhaps roll the WAL very often to minimize appends, right?
>
> Atomic folder renames can also probably be worked around too, similar to
> how hadoop-azure does it.
>
> Thanks again. Looking forward to your insight.
>
> On Wed, Sep 9, 2015 at 7:32 PM, Andrew Purtell <ap...@apache.org>
> wrote:
>
> > It cannot work to use S3 as a backing store for HBase. This has been
> > attempted in the past (although not by me, so this isn't firsthand
> > knowledge). One basic problem is HBase expects to be able to read what it
> > has written immediately after the write completes. For example, opening a
> > store file after a flush to service reads, or reopening store files
> after a
> > compaction. S3's notion of availability is too eventual. You can get 404
> or
> > 500 responses for some time after completing a file write. HBase will
> > panic. I'm not sure we'd accept kludges around this problem. Another
> > fundamental problem is S3 doesn't provide any means for appending to
> files,
> > so the HBase write ahead log cannot work. I suppose this could be worked
> > around with changes that allow specification of one type of filesystem
> for
> > HFiles and another for write ahead logs. There are numerous finer points
> > about HDFS behavior used as synchronization primitive, such as atomic
> > renames.
> >
> >
> > On Wed, Sep 9, 2015 at 4:23 PM, Anthony Nguyen <
> > anthony.an.nguyen@gmail.com>
> > wrote:
> >
> > > Hi all,
> > >
> > > I'm investigating the use of S3 as a backing store for HBase. Would
> there
> > > be any major issues with modifying HBase in such a way where when an S3
> > > location is set for the rootdir, writes to .tmp are removed and
> > minimized,
> > > instead writing directly to the final destination? The reason I'd like
> to
> > > do this is because renames in S3 are expensive and performance for
> > > operations such as compactions and snapshot restores that have many
> > renames
> > > suffer.
> > >
> > > Thanks!
> > >
> >
> >
> >
> > --
> > Best regards,
> >
> > - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: Ramifications of minimizing use of .tmp directories / renames in
HBase when using S3 as backing store
Posted by Anthony Nguyen <an...@gmail.com>.
Hi Andrew,
Thanks for your help! I'm attempting to give it Hbase on S3 another try :).
Let me see if I can address the two things you pointed out and you can call
me crazy for even thinking that they'd be okay:
First, some background - my initial use case will be using HBase in a
mostly read-only fashion (bulk loads are the primary method of data load).
Consistency issues: Since S3 has read after write consistency for new
objects, we could potentially structure the naming scheme within HBase file
writes such that the same location is never overwritten. In this way, any
new files (which would include updates / appends to an ultimately new file)
in S3 will be immediately consistent. Files can be cleaned up after a set
period of time.
Appends: An append in S3 can be modeled as a read / copy / delete
operation. Performance will definitely suffer, but we can configure WAL
settings to perhaps roll the WAL very often to minimize appends, right?
Atomic folder renames can also probably be worked around too, similar to
how hadoop-azure does it.
Thanks again. Looking forward to your insight.
On Wed, Sep 9, 2015 at 7:32 PM, Andrew Purtell <ap...@apache.org> wrote:
> It cannot work to use S3 as a backing store for HBase. This has been
> attempted in the past (although not by me, so this isn't firsthand
> knowledge). One basic problem is HBase expects to be able to read what it
> has written immediately after the write completes. For example, opening a
> store file after a flush to service reads, or reopening store files after a
> compaction. S3's notion of availability is too eventual. You can get 404 or
> 500 responses for some time after completing a file write. HBase will
> panic. I'm not sure we'd accept kludges around this problem. Another
> fundamental problem is S3 doesn't provide any means for appending to files,
> so the HBase write ahead log cannot work. I suppose this could be worked
> around with changes that allow specification of one type of filesystem for
> HFiles and another for write ahead logs. There are numerous finer points
> about HDFS behavior used as synchronization primitive, such as atomic
> renames.
>
>
> On Wed, Sep 9, 2015 at 4:23 PM, Anthony Nguyen <
> anthony.an.nguyen@gmail.com>
> wrote:
>
> > Hi all,
> >
> > I'm investigating the use of S3 as a backing store for HBase. Would there
> > be any major issues with modifying HBase in such a way where when an S3
> > location is set for the rootdir, writes to .tmp are removed and
> minimized,
> > instead writing directly to the final destination? The reason I'd like to
> > do this is because renames in S3 are expensive and performance for
> > operations such as compactions and snapshot restores that have many
> renames
> > suffer.
> >
> > Thanks!
> >
>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
Re: Ramifications of minimizing use of .tmp directories / renames in
HBase when using S3 as backing store
Posted by Andrew Purtell <ap...@apache.org>.
It cannot work to use S3 as a backing store for HBase. This has been
attempted in the past (although not by me, so this isn't firsthand
knowledge). One basic problem is HBase expects to be able to read what it
has written immediately after the write completes. For example, opening a
store file after a flush to service reads, or reopening store files after a
compaction. S3's notion of availability is too eventual. You can get 404 or
500 responses for some time after completing a file write. HBase will
panic. I'm not sure we'd accept kludges around this problem. Another
fundamental problem is S3 doesn't provide any means for appending to files,
so the HBase write ahead log cannot work. I suppose this could be worked
around with changes that allow specification of one type of filesystem for
HFiles and another for write ahead logs. There are numerous finer points
about HDFS behavior used as synchronization primitive, such as atomic
renames.
On Wed, Sep 9, 2015 at 4:23 PM, Anthony Nguyen <an...@gmail.com>
wrote:
> Hi all,
>
> I'm investigating the use of S3 as a backing store for HBase. Would there
> be any major issues with modifying HBase in such a way where when an S3
> location is set for the rootdir, writes to .tmp are removed and minimized,
> instead writing directly to the final destination? The reason I'd like to
> do this is because renames in S3 are expensive and performance for
> operations such as compactions and snapshot restores that have many renames
> suffer.
>
> Thanks!
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)