Re: Disk space leak when using HBase and HDFS ShortCircuit

[Colin McCabe <cm...@alumni.cmu.edu>]

HDFS now notifies the client when the block file has been deleted. See
HDFS-6750 for details.

Best,
Colin
 On Jun 26, 2014 11:39 AM, "Giuseppe Reina" <g....@gmail.com> wrote:

> On Thu, Jun 26, 2014 at 1:02 AM, Enis Söztutar <en...@apache.org> wrote:
>
> > Agreed, this seems like an hdfs issue unless hbase itself does not close
> > the hfiles properly. But judging from the fact that you were able to
> > circumvent the problem by getting reducing the cache size, it does seem
> > unlikely.
> >
>
>
> Well, we had the same issue with two different HDP 2.1 installations (with
> default parameters) and two different client applications on top of HBase.
> I agree this is an HDFS issue, I posted on the HBase mailing list because I
> was wondering if anyone else had the same problem. Thanks for forwarding
> it.
>
> In both applications we are injesting lots of data on one HBase table with
> one column family and about 10 regions for each RS.
> So the RegionServer does a lot of flushes, merges and split which involves
> lots of HDFS block creations and deletions.
>
> What was your dfs.client.read.shortcircuit.streams.cache.size ?
>
>
> The default value is 4096 on HDP 2.1 (not 256 as in the vanilla
> distribution).
> When the shortcircuit cache is too big (like in this case), the deleted
> files remain open until the cache is full. This is bad when big WAL/HFiles
> are merged or split as the space on disk is still occupied and the file is
> never accessed (so it's a leak). Of course any application that perform
> file creation, read, and deletion of big hdfs files will be affected by the
> same problem.
>
>
> >
> > I don't think the local block reader will be notified when a file/block
> is
> > deleted.
> >
>
> That's what I thought too, but I would be easy for the HDFS client library
> to periodically scan through all the open file descriptors verifying that
> the file has not been deleted (by "stat"-ing the filename for example). It
> is useless to keep it in memory as the file will never be accessed, and if
> it will this could potentially lead to inconsistent states.
>
>
>
> Giuseppe
>
>
>
>
> >
> > Enis
> >
> >
> > On Wed, Jun 25, 2014 at 3:29 PM, Andrew Purtell <ap...@apache.org>
> > wrote:
> >
> > > Forwarded
> > >
> > > ---------- Forwarded message ----------
> > > From: Vladimir Rodionov <vr...@carrieriq.com>
> > > Date: Wed, Jun 25, 2014 at 12:03 PM
> > > Subject: RE: Disk space leak when using HBase and HDFS ShortCircuit
> > > To: "user@hbase.apache.org" <us...@hbase.apache.org>
> > >
> > >
> > > >> Apparently those file descriptors were stored by the HDFS
> > > >> ShortCircuit cache.
> > >
> > > As far as I understand this is issue of HDFS shorty-circuit-reads
> > > implementation not HBase. HBase uses HDFS API to access
> > > files. Did you ask this question on hdfs dev list? This looks like a
> very
> > > serious bug.
> > >
> > > Best regards,
> > > Vladimir Rodionov
> > > Principal Platform Engineer
> > > Carrier IQ, www.carrieriq.com
> > > e-mail: vrodionov@carrieriq.com
> > >
> > > ________________________________________
> > > From: Giuseppe Reina [g.reina@gmail.com]
> > > Sent: Wednesday, June 25, 2014 2:54 AM
> > > To: user@hbase.apache.org
> > > Subject: Disk space leak when using HBase and HDFS ShortCircuit
> > >
> > > Hi all,
> > >    we have been experiencing the same problem with 2 of our clusters.
> We
> > > are currently using HDP 2.1 that comes with HBase 0.98.
> > >
> > > The problem manifested by showing a huge differences (hundreds of GB)
> > > between the output of "df" and "du" of the hdfs data directories.
> > > Eventually, other systems complained for the lack of space before
> > shutting
> > > down. We identified the problem and discovered that all the
> RegionServers
> > > were holding lots of open file descriptors to deleted files, which
> > > prevented the OS to free the disk space occupied (hence the difference
> > > between "df" and "du"). The deleted files were pointing to the local
> HDFS
> > > blocks of old HFiles deleted from HDFS during the compaction and/or
> split
> > > operations. Apparently those file descriptors were stored by the HDFS
> > > ShortCircuit cache.
> > >
> > > My question is, isn't the shortcircuit feautre supposed to get
> "notified"
> > > somehow of file deletion on a file on HDFS so it can remove the open
> fds
> > > from the cache? This creates huge leaks whenever HBase is heavily
> loaded
> > > and we had to restart the RegionServer periodically until before
> > > identifying the problem. We solved the problem first by disabling
> > > shortcircuit from HDFS and then enabling it and reducing the cache size
> > so
> > > to trigger often the caching policies (this leads to some performance
> > > loss).
> > >
> > >
> > > p.s. I am aware of the "
> > > dfs.client.read.shortcircuit.streams.cache.expiry.ms
> > > " directoparameter, but for some reason the default value (5 mins) does
> > not
> > > work out-of-the-box on HDP 2.1, moreover the problem persists for high
> > > timeouts and big cache sizes.
> > >
> > > Kind Regards
> > >
> > > Confidentiality Notice:  The information contained in this message,
> > > including any attachments hereto, may be confidential and is intended
> to
> > be
> > > read only by the individual or entity to whom this message is
> addressed.
> > If
> > > the reader of this message is not the intended recipient or an agent or
> > > designee of the intended recipient, please note that any review, use,
> > > disclosure or distribution of this message or its attachments, in any
> > form,
> > > is strictly prohibited.  If you have received this message in error,
> > please
> > > immediately notify the sender and/or Notifications@carrieriq.com and
> > > delete
> > > or destroy any copy of this message and its attachments.
> > >
> > >
> > >
> > > --
> > > Best regards,
> > >
> > >    - Andy
> > >
> > > Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> > > (via Tom White)
> > >
> >
>