You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by sathyafmt <sa...@gmail.com> on 2015/01/13 07:02:58 UTC
HBase with opentsdb creates huge .tmp file & runs out of hdfs space
CDH5.1.2 (hbase 0.98.1) (running on a vm - vmware esx)
We use opentsdb(2.1.0RC) with hbase & after ingesting 200-300MB of data,
hbase tries to compact the table, and ends up creating a .tmp file which
grows to fill up the entire hdfs space.. and dies eventually. I tried to
remove this .tmp file & restart hbase, but it goes back to creating this
gigantic .tmp file & ends up dying again...
Any ideas, what could be causing this? I thought this was a one-off thing &
tried again by clearing out hbase & starting over again. But it ends up back
in this same state after ingesting 200-300MB of data.
Thanks,
sathya
==
2015-01-07 16:22:27,855 INFO
[regionserver60020-smallCompactions-1420676547848] regionserver.HStore:
Starting compaction of 3 file(s) in t of
tsdb,,1419996612897.b881b798f5a7a766932a1e59bc2bd738. into
tmpdir=hdfs://localhost/hbase/data/default/tsdb/b881b798f5a7a766932a1e59bc2bd738/.tmp,
totalSize=300.4 M
2015-01-07 16:22:27,856 DEBUG [RS_OPEN_REGION-XXXX:60020-1]
zookeeper.ZKAssign: regionserver:60020-0x14ac6e25f5c0003,
quorum=localhost:2181, baseZNode=/hbase Transitioned node
15abb3e48aeb8b0b43e8cabb4db459bf from M_ZK_REGION_OFFLINE to
RS_ZK_REGION_OPENING
2015-01-07 16:22:27,857 DEBUG
[regionserver60020-smallCompactions-1420676547848]
compactions.Compactor:Compacting
hdfs://localhost/hbase/data/default/tsdb/b881b798f5a7a766932a1e59bc2bd738/t/19fb0a1e6fb54b868c0845ce6589ca05,
keycount=75, bloomtype=ROW, size=24.8 M, encoding=NONE, seqNum=98187,
earliestPutTs=1420224086959
2015-01-07 16:22:27,855 INFO
[regionserver60020-smallCompactions-1420676547848] regionserver.HStore:
Starting compaction of 3 file(s) in t of
tsdb,,1419996612897.b881b798f5a7a766932a1e59bc2bd738. into
tmpdir=hdfs://localhost/hbase/data/default/tsdb/b881b798f5a7a766932a1e59bc2bd738/.tmp,
totalSize=300.4 M
2015-01-07 16:22:27,856 DEBUG [RS_OPEN_REGION-XXXX:60020-1]
zookeeper.ZKAssign: regionserver:60020-0x14ac6e25f5c0003,
quorum=localhost:2181, baseZNode=/hbase Transitioned node
15abb3e48aeb8b0b43e8cabb4db459bf from M_ZK_REGION_OFFLINE to
RS_ZK_REGION_OPENING
2015-01-07 16:22:27,857 DEBUG
[regionserver60020-smallCompactions-1420676547848]
compactions.Compactor:Compacting
hdfs://localhost/hbase/data/default/tsdb/b881b798f5a7a766932a1e59bc2bd738/t/19fb0a1e6fb54b868c0845ce6589ca05,
keycount=75, bloomtype=ROW, size=24.8 M, encoding=NONE, seqNum=98187,
earliestPutTs=1420224086959
2015-01-07 16:22:27,855 INFO
[regionserver60020-smallCompactions-1420676547848] regionserver.HStore:
Starting compaction of 3 file(s) in t of
tsdb,,1419996612897.b881b798f5a7a766932a1e59bc2bd738. into
tmpdir=hdfs://localhost/hbase/data/default/tsdb/b881b798f5a7a766932a1e59bc2bd738/.tmp,
totalSize=300.4 M
2015-01-07 16:22:27,856 DEBUG [RS_OPEN_REGION-XXXX:60020-1]
zookeeper.ZKAssign: regionserver:60020-0x14ac6e25f5c0003,
quorum=localhost:2181, baseZNode=/hbase Transitioned node
15abb3e48aeb8b0b43e8cabb4db459bf from M_ZK_REGION_OFFLINE to
RS_ZK_REGION_OPENING
2015-01-07 16:22:27,857 DEBUG
[regionserver60020-smallCompactions-1420676547848]
compactions.Compactor:Compacting
hdfs://localhost/hbase/data/default/tsdb/b881b798f5a7a766932a1e59bc2bd738/t/19fb0a1e6fb54b868c0845ce6589ca05,
keycount=75, bloomtype=ROW, size=24.8 M, encoding=NONE, seqNum=98187,
earliestPutTs=1420224086959
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/HBase-with-opentsdb-creates-huge-tmp-file-runs-out-of-hdfs-space-tp4067577.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: HBase with opentsdb creates huge .tmp file & runs out of hdfs
space
Posted by sathyafmt <sa...@gmail.com>.
Hi John,
What's the fix (committed on Jan11) you mentioned ? Is it in opentsdb/hbase
? Do you have a JIRA #.. ?
Thanks
On Wed, Feb 25, 2015 at 5:49 AM, brady2 [via Apache HBase] <
ml-node+s679495n4068627h88@n3.nabble.com> wrote:
> Hi Sathya and Nick,
>
> Here are the stack traces of the region server dumps when the huge .tmp
> files are created:
>
> https://drive.google.com/open?id=0B1tQg4D17jKQNDdFZkFQTlg4ZjQ&authuser=0
>
> As background we are not using compression. Compaction is occurs every
> hour. Everything else is default.
>
> OpenTSDB v2.0 is running on top of Cloudera 5.3.1 in AWS. We have a 7 node
> Cloudera cluster(each node with 32GB ram and 3TB disk space), with 5
> OpenTSDB instances dedicated for writing and 2 for reading. We are using
> AWS ELB’s in front of OpenTSDB to balance the read/writes.
>
> We are load testing OpenTSDB using SOCKETS, but running into several
> issues. Let me explain first how we do this load testing:
>
> 1.From another AWS system, we have written a testing framework to generate
> load.
>
> 2. The framework takes several parameters, we can specify the number of
> threads, the loop size (i.e. the number of sockets that each thread will
> open) and the batch size (i.e. the number of PUT’s, or inserts, that each
> socket connection will handle).
>
> 3. To simplify troubleshooting, we removed variables from the tests, we
> have just 1 OpenTSDB instance behind the AWS ELB so the load is being sent
> to 1 instance only.
>
> 4. We are initially creating the openTSDB tables without any pre-splitting
> of regions.
>
> 5. We are doing the loading with 1 metric only for ease of querying in the
> UI.
>
> 6. We are sending under 5000 inserts per second:
>
> 7. At the top of the hour, the row compaction kicks in and the region
> server is too busy so we lose data. it recovers the first time. But the 2nd
> hour, there is so much data presumably, that it doesn’t recover. To fix it,
> we have to restart cloudera, reboot the nodes, drop the tsdb tables and
> re-create them. Otherwise the .tmp file keeps growing until it fills the
> 3TB disks and the system is unresponsive.
>
> 8. We see problems with region splits happening under heavy load. We noted
> a code fix committed on Jan 11 for this but I presume that is not in RC2.1.
>
> Thanks
>
>
>
>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-hbase.679495.n3.nabble.com/HBase-with-opentsdb-creates-huge-tmp-file-runs-out-of-hdfs-space-tp4067577p4068627.html
> To unsubscribe from HBase with opentsdb creates huge .tmp file & runs out
> of hdfs space, click here
> <http://apache-hbase.679495.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4067577&code=c2F0aHlhZm10QGdtYWlsLmNvbXw0MDY3NTc3fDUxNzU0MjkyMA==>
> .
> NAML
> <http://apache-hbase.679495.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/HBase-with-opentsdb-creates-huge-tmp-file-runs-out-of-hdfs-space-tp4067577p4068650.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: HBase with opentsdb creates huge .tmp file & runs out of hdfs
space
Posted by brady2 <jo...@intel.com>.
Hi Sathya and Nick,
Here are the stack traces of the region server dumps when the huge .tmp
files are created:
https://drive.google.com/open?id=0B1tQg4D17jKQNDdFZkFQTlg4ZjQ&authuser=0
<https://drive.google.com/open?id=0B1tQg4D17jKQNDdFZkFQTlg4ZjQ&authuser=0>
As background we are not using compression. Compaction is occurs every hour.
Everything else is default.
OpenTSDB v2.0 is running on top of Cloudera 5.3.1 in AWS. We have a 7 node
Cloudera cluster(each node with 32GB ram and 3TB disk space), with 5
OpenTSDB instances dedicated for writing and 2 for reading. We are using AWS
ELB’s in front of OpenTSDB to balance the read/writes.
We are load testing OpenTSDB using SOCKETS, but running into several issues.
Let me explain first how we do this load testing:
1.From another AWS system, we have written a testing framework to generate
load.
2. The framework takes several parameters, we can specify the number of
threads, the loop size (i.e. the number of sockets that each thread will
open) and the batch size (i.e. the number of PUT’s, or inserts, that each
socket connection will handle).
3. To simplify troubleshooting, we removed variables from the tests, we have
just 1 OpenTSDB instance behind the AWS ELB so the load is being sent to 1
instance only.
4. We are initially creating the openTSDB tables without any pre-splitting
of regions.
5. We are doing the loading with 1 metric only for ease of querying in the
UI.
6. We are sending under 5000 inserts per second:
7. At the top of the hour, the row compaction kicks in and the region server
is too busy so we lose data. it recovers the first time. But the 2nd hour,
there is so much data presumably, that it doesn’t recover. To fix it, we
have to restart cloudera, reboot the nodes, drop the tsdb tables and
re-create them. Otherwise the .tmp file keeps growing until it fills the 3TB
disks and the system is unresponsive.
8. We see problems with region splits happening under heavy load. We noted a
code fix committed on Jan 11 for this but I presume that is not in RC2.1.
Thanks
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/HBase-with-opentsdb-creates-huge-tmp-file-runs-out-of-hdfs-space-tp4067577p4068627.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: HBase with opentsdb creates huge .tmp file & runs out of hdfs space
Posted by anil gupta <an...@gmail.com>.
Did you guys tried posting your question to opentsdb group(
https://groups.google.com/forum/#!forum/opentsdb)? They might be able to
provide insight with respect to OpenTSDB working.
~Anil
On Tue, Feb 24, 2015 at 10:32 AM, sathyafmt <sa...@gmail.com> wrote:
> Find out the regionserver causing this & then take 10 thread dumps (with a
> delay of 10s) with curl http://rregionserver:60030/dump
>
> for i in {1..10} ; do echo "Dump: $i"; echo "+++++++++++++++++++++++++";
> curl http://rs:60030/dump; sleep 10; done
>
>
>
> On Tue, Feb 24, 2015 at 10:05 AM, brady2 [via Apache HBase] <
> ml-node+s679495n4068589h10@n3.nabble.com> wrote:
>
> > Hi Sathya,
> >
> > Could you post the command that you use to captured the stack traces of
> > the region server dumps that you attached please?
> >
> > I don't have have enough knowledge of hbase/opentsdb.
> >
> > I'll post the exact conditions under which the .tmp is being created
> > tomorrow.
> >
> > Thanks
> > John
> >
> >
> >
> >
> > ------------------------------
> > If you reply to this email, your message will be added to the discussion
> > below:
> >
> >
> http://apache-hbase.679495.n3.nabble.com/HBase-with-opentsdb-creates-huge-tmp-file-runs-out-of-hdfs-space-tp4067577p4068589.html
> > To unsubscribe from HBase with opentsdb creates huge .tmp file & runs
> out
> > of hdfs space, click here
> > <
> http://apache-hbase.679495.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4067577&code=c2F0aHlhZm10QGdtYWlsLmNvbXw0MDY3NTc3fDUxNzU0MjkyMA==
> >
> > .
> > NAML
> > <
> http://apache-hbase.679495.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml
> >
> >
>
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/HBase-with-opentsdb-creates-huge-tmp-file-runs-out-of-hdfs-space-tp4067577p4068592.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
--
Thanks & Regards,
Anil Gupta
Re: HBase with opentsdb creates huge .tmp file & runs out of hdfs space
Posted by Esteban Gutierrez <es...@cloudera.com>.
On Wed, Jun 3, 2015 at 11:28 AM, Vladimir Rodionov <vl...@gmail.com>
wrote:
> >> Sounds like a possible
> >> workaround would have been to bump the value
> >> of hfile.index.block.max.size, did you try that?
>
> ? How is that?
>
I'm still looking into that, but I assume that Sathya would have tried to
bump that value in order to find that was the problem.
>
> -Vlad
>
> On Wed, Jun 3, 2015 at 11:03 AM, Esteban Gutierrez <es...@cloudera.com>
> wrote:
>
> > Thats a very interesting finding Sathya. Could you please open an HBase
> > JIRA and additional information about this behavior? Sounds like a
> possible
> > workaround would have been to bump the value
> > of hfile.index.block.max.size, did you try that?
> >
> > thanks,
> > esteban.
> >
> > --
> > Cloudera, Inc.
> >
> >
> > On Tue, Jun 2, 2015 at 9:42 PM, sathyafmt <sa...@gmail.com> wrote:
> >
> > > FYI: haven't seen this issue since we turned off tsdb compaction. See
> > > github
> > > opentsdb issue #490 <https://github.com/OpenTSDB/opentsdb/issues/490>
> > > for
> > > more info..
> > >
> > >
> > >
> > > --
> > > View this message in context:
> > >
> >
> http://apache-hbase.679495.n3.nabble.com/HBase-with-opentsdb-creates-huge-tmp-file-runs-out-of-hdfs-space-tp4067577p4072091.html
> > > Sent from the HBase User mailing list archive at Nabble.com.
> > >
> >
>
Re: HBase with opentsdb creates huge .tmp file & runs out of hdfs space
Posted by Vladimir Rodionov <vl...@gmail.com>.
>> Sounds like a possible
>> workaround would have been to bump the value
>> of hfile.index.block.max.size, did you try that?
? How is that?
-Vlad
On Wed, Jun 3, 2015 at 11:03 AM, Esteban Gutierrez <es...@cloudera.com>
wrote:
> Thats a very interesting finding Sathya. Could you please open an HBase
> JIRA and additional information about this behavior? Sounds like a possible
> workaround would have been to bump the value
> of hfile.index.block.max.size, did you try that?
>
> thanks,
> esteban.
>
> --
> Cloudera, Inc.
>
>
> On Tue, Jun 2, 2015 at 9:42 PM, sathyafmt <sa...@gmail.com> wrote:
>
> > FYI: haven't seen this issue since we turned off tsdb compaction. See
> > github
> > opentsdb issue #490 <https://github.com/OpenTSDB/opentsdb/issues/490>
> > for
> > more info..
> >
> >
> >
> > --
> > View this message in context:
> >
> http://apache-hbase.679495.n3.nabble.com/HBase-with-opentsdb-creates-huge-tmp-file-runs-out-of-hdfs-space-tp4067577p4072091.html
> > Sent from the HBase User mailing list archive at Nabble.com.
> >
>
Re: HBase with opentsdb creates huge .tmp file & runs out of hdfs space
Posted by Esteban Gutierrez <es...@cloudera.com>.
Thats a very interesting finding Sathya. Could you please open an HBase
JIRA and additional information about this behavior? Sounds like a possible
workaround would have been to bump the value
of hfile.index.block.max.size, did you try that?
thanks,
esteban.
--
Cloudera, Inc.
On Tue, Jun 2, 2015 at 9:42 PM, sathyafmt <sa...@gmail.com> wrote:
> FYI: haven't seen this issue since we turned off tsdb compaction. See
> github
> opentsdb issue #490 <https://github.com/OpenTSDB/opentsdb/issues/490>
> for
> more info..
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/HBase-with-opentsdb-creates-huge-tmp-file-runs-out-of-hdfs-space-tp4067577p4072091.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
Re: HBase with opentsdb creates huge .tmp file & runs out of hdfs
space
Posted by sathyafmt <sa...@gmail.com>.
FYI: haven't seen this issue since we turned off tsdb compaction. See github
opentsdb issue #490 <https://github.com/OpenTSDB/opentsdb/issues/490> for
more info..
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/HBase-with-opentsdb-creates-huge-tmp-file-runs-out-of-hdfs-space-tp4067577p4072091.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: HBase with opentsdb creates huge .tmp file & runs out of hdfs
space
Posted by sathyafmt <sa...@gmail.com>.
Find out the regionserver causing this & then take 10 thread dumps (with a
delay of 10s) with curl http://rregionserver:60030/dump
for i in {1..10} ; do echo "Dump: $i"; echo "+++++++++++++++++++++++++";
curl http://rs:60030/dump; sleep 10; done
On Tue, Feb 24, 2015 at 10:05 AM, brady2 [via Apache HBase] <
ml-node+s679495n4068589h10@n3.nabble.com> wrote:
> Hi Sathya,
>
> Could you post the command that you use to captured the stack traces of
> the region server dumps that you attached please?
>
> I don't have have enough knowledge of hbase/opentsdb.
>
> I'll post the exact conditions under which the .tmp is being created
> tomorrow.
>
> Thanks
> John
>
>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-hbase.679495.n3.nabble.com/HBase-with-opentsdb-creates-huge-tmp-file-runs-out-of-hdfs-space-tp4067577p4068589.html
> To unsubscribe from HBase with opentsdb creates huge .tmp file & runs out
> of hdfs space, click here
> <http://apache-hbase.679495.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4067577&code=c2F0aHlhZm10QGdtYWlsLmNvbXw0MDY3NTc3fDUxNzU0MjkyMA==>
> .
> NAML
> <http://apache-hbase.679495.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/HBase-with-opentsdb-creates-huge-tmp-file-runs-out-of-hdfs-space-tp4067577p4068592.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: HBase with opentsdb creates huge .tmp file & runs out of hdfs
space
Posted by brady2 <jo...@intel.com>.
Hi Sathya,
Could you post the command that you use to captured the stack traces of the
region server dumps that you attached please?
I don't have have enough knowledge of hbase/opentsdb.
I'll post the exact conditions under which the .tmp is being created
tomorrow.
Thanks
John
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/HBase-with-opentsdb-creates-huge-tmp-file-runs-out-of-hdfs-space-tp4067577p4068589.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: HBase with opentsdb creates huge .tmp file & runs out of hdfs
space
Posted by brady2 <jo...@intel.com>.
Thanks Sathya and Nick, I dropped the Hbase tables but I will reproduce the
issue today and post the same logs as Sathya. Thanks John
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/HBase-with-opentsdb-creates-huge-tmp-file-runs-out-of-hdfs-space-tp4067577p4068567.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: HBase with opentsdb creates huge .tmp file & runs out of hdfs
space
Posted by sathyafmt <sa...@gmail.com>.
Nick,
Look at my reply from 02/06/2015, I have the stack traces on my google
drive...
===
We ran into this issue again at the customer site & I collected the region
server dumps (25 of them at 10s intervals). I uploaded it to my google
drive
<https://drive.google.com/file/d/0B53HyylRdzUuZi1aMXRBd2V2V2s/view?usp=sharing>
(apaste.info has a 1M cap, this file is around 6M)
===
thanks
-sathya
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/HBase-with-opentsdb-creates-huge-tmp-file-runs-out-of-hdfs-space-tp4067577p4068553.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: HBase with opentsdb creates huge .tmp file & runs out of hdfs space
Posted by Nick Dimiduk <nd...@gmail.com>.
Oh I see snappy and no block encoder. How about stack traces while the
endless file is being created (like, a couple/sec)? Poor man's sampler.
On Monday, February 23, 2015, Nick Dimiduk <nd...@gmail.com> wrote:
> Can anyone reproducing this provide additional details requested earlier:
>
> you using any BlockEncoding or Compression with this column family? Any
> other store/table configuration? This happens repeatably? Can you provide
> jstack of the RS process along with log lines while this file is growing
> excessively?
>
> On Monday, February 23, 2015, sathyafmt <sathyafmt@gmail.com
> <javascript:_e(%7B%7D,'cvml','sathyafmt@gmail.com');>> wrote:
>
>> John - No solution yet, I didn't hear anything back from the group.. I am
>> still running into this issue. Are you running on a VM or bare-metal ?
>>
>> Thanks
>> -sathya
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-hbase.679495.n3.nabble.com/HBase-with-opentsdb-creates-huge-tmp-file-runs-out-of-hdfs-space-tp4067577p4068547.html
>> Sent from the HBase User mailing list archive at Nabble.com.
>>
>
Re: HBase with opentsdb creates huge .tmp file & runs out of hdfs space
Posted by Nick Dimiduk <nd...@gmail.com>.
Can anyone reproducing this provide additional details requested earlier:
you using any BlockEncoding or Compression with this column family? Any
other store/table configuration? This happens repeatably? Can you provide
jstack of the RS process along with log lines while this file is growing
excessively?
On Monday, February 23, 2015, sathyafmt <sa...@gmail.com> wrote:
> John - No solution yet, I didn't hear anything back from the group.. I am
> still running into this issue. Are you running on a VM or bare-metal ?
>
> Thanks
> -sathya
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/HBase-with-opentsdb-creates-huge-tmp-file-runs-out-of-hdfs-space-tp4067577p4068547.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
Re: HBase with opentsdb creates huge .tmp file & runs out of hdfs
space
Posted by sathyafmt <sa...@gmail.com>.
John - No solution yet, I didn't hear anything back from the group.. I am
still running into this issue. Are you running on a VM or bare-metal ?
Thanks
-sathya
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/HBase-with-opentsdb-creates-huge-tmp-file-runs-out-of-hdfs-space-tp4067577p4068547.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: HBase with opentsdb creates huge .tmp file & runs out of hdfs
space
Posted by brady2 <jo...@intel.com>.
Hello,
I am having the exact same issue. Opentsdb is creating a huge .tmp file and
is runs out of space under after ingesting a similar amount of data.
Could you post the solution please?
Many thanks
John
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/HBase-with-opentsdb-creates-huge-tmp-file-runs-out-of-hdfs-space-tp4067577p4068529.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: HBase with opentsdb creates huge .tmp file & runs out of hdfs
space
Posted by brady2 <jo...@intel.com>.
Hello,
I am having the exact same issue. Opentsdb is creating a huge .tmp file and
is runs out of space under after ingesting a similar amount of data.
Could you post the solution please?
Many thanks
John
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/HBase-with-opentsdb-creates-huge-tmp-file-runs-out-of-hdfs-space-tp4067577p4068530.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: HBase with opentsdb creates huge .tmp file & runs out of hdfs
space
Posted by 3h_william <Wi...@newegg.com>.
I have meet the same question that hbase generate huge file in .tmp folder
then do compaction.
And the same condition is use OpenTSDB 2.1.0 .
Who have resolved this problem or can give me some suggestion ?
Thanks very much
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/HBase-with-opentsdb-creates-huge-tmp-file-runs-out-of-hdfs-space-tp4067577p4069867.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: HBase with opentsdb creates huge .tmp file & runs out of hdfs
space
Posted by sathyafmt <sa...@gmail.com>.
I have dfs.datanode.data.dir=/var/vcap/store/hadoop/hdfs/data,
dfs.datanode.failed.volumes.tolerated=0. [I don't have
dfs.datanode.du.reserved: thanks for mentioning it, I'll set it to 10G going
forward]
The CF has compression=SNAPPY. I have only hbase.cluster.distributed=true,
hbase.rootdir=hdfs://n.n.n.n/hbase & hbase.zookeeper.quorum=n.n.n.n in
hbase-site.xml (nothing else)
Here's the tune2fs o/p, pdfs-site.xml & abase-site.xml: (The /var/vcap disk
is shared with other processes, so I can't have any fs params like noatime,
etc..)
-sathya
tune2fs 1.42.9 (4-Feb-2014)
Filesystem volume name: <none>
Last mounted on: /var/vcap
Filesystem UUID: 7eca2b17-c0df-4e55-811f-ed549a2c1663
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr resize_inode dir_index
filetype needs_recovery extent flex_bg sparse_super large_file huge_file
uninit_bg dir_nlink extra_isize
Filesystem flags: signed_directory_hash
Default mount options: user_xattr acl
Filesystem state: clean
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 16384000
Block count: 65535744
Reserved block count: 3276787
Free blocks: 63203358
Free inodes: 16381182
First block: 0
Block size: 4096
Fragment size: 4096
Reserved GDT blocks: 1008
Blocks per group: 32768
Fragments per group: 32768
Inodes per group: 8192
Inode blocks per group: 512
Flex block group size: 16
Filesystem created: Tue Jan 6 12:34:14 2015
Last mount time: Thu Jan 8 22:04:49 2015
Last write time: Thu Jan 8 22:04:49 2015
Mount count: 2
Maximum mount count: -1
Last checked: Tue Jan 6 12:34:14 2015
Check interval: 0 (<none>)
Lifetime writes: 31 GB
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 256
Required extra isize: 28
Desired extra isize: 28
Journal inode: 8
Default directory hash: half_md4
Directory Hash Seed: 37e6cced-a9fb-465d-bcc2-5c54d5c6533e
Journal backup: inode blocks
===
hbase(main):002:0> describe 'tsdb'
'tsdb', {NAME => 't', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW',
REPLICATION_SCOPE => '0', COMPRESSION => 'SNAPPY', VERSIONS => '1', TTL =>
'FOREVER', MIN_VERSIONS => '0', KEEP_DELETED_CELLS => 'false', BLOCKSIZE =>
'65536', IN_MEMORY => 'false',
BLOCKCACHE => 'true'}
1 row(s) in 3.5200 seconds
==
hdfs-site.xml
===
<configuration>
<property>
<name>dfs.name.dir</name>
<value>/var/vcap/store/hadoop/hdfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/var/vcap/store/hadoop/hdfs/data</value>
</property>
<property>
<name>dfs.namenode.http-address</name>
<value>0.0.0.0:50070</value>
</property>
<property>
<name>dfs.namenode.checkpoint.dir</name>
<value>/var/vcap/store/hadoop/hdfs/secondarynn</value>
</property>
<property>
<name>dfs.datanode.failed.volumes.tolerated</name>
<value>0</value>
</property>
</configuration>
===
hbase-site.xml
===
<configuration>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.rootdir</name>
<value>hdfs://n.n.n.n/hbase</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>n.n.n.n</value>
</property>
</configuration>
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/HBase-with-opentsdb-creates-huge-tmp-file-runs-out-of-hdfs-space-tp4067577p4067600.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: HBase with opentsdb creates huge .tmp file & runs out of hdfs space
Posted by Sean Busbey <bu...@cloudera.com>.
On Tue, Jan 13, 2015 at 5:44 PM, Sean Busbey <bu...@cloudera.com> wrote:
> Hiya!
>
> You'll have to post the config file somewhere (e.g. http://apaste.info/)
> and then provide a link to it. ASF mailing lists strip out attachments.
>
>
>
Never mind me; I see the configs now. :/
--
Sean
Re: HBase with opentsdb creates huge .tmp file & runs out of hdfs space
Posted by Sean Busbey <bu...@cloudera.com>.
Hiya!
You'll have to post the config file somewhere (e.g. http://apaste.info/)
and then provide a link to it. ASF mailing lists strip out attachments.
On Tue, Jan 13, 2015 at 4:34 PM, sathyafmt <sa...@gmail.com> wrote:
> yes, painful :-)
>
> This happens at a customer deployment & the next time I hit this I'll get
> you the stack trace. I have the hbase-site.xml attached to my response to
> Sean, take a look.
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/HBase-with-opentsdb-creates-huge-tmp-file-runs-out-of-hdfs-space-tp4067577p4067601.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
--
Sean
Re: HBase with opentsdb creates huge .tmp file & runs out of hdfs
space
Posted by sathyafmt <sa...@gmail.com>.
We ran into this issue again at the customer site & I collected the region
server dumps (25 of them at 10s intervals). I uploaded it to my google drive
<https://drive.google.com/file/d/0B53HyylRdzUuZi1aMXRBd2V2V2s/view?usp=sharing>
(apaste.info has a 1M cap, this file is around 6M)
thanks
-sathya
On Tue, Jan 13, 2015 at 4:34 PM, sathyafmt [via Apache HBase] <
ml-node+s679495n4067601h82@n3.nabble.com> wrote:
> yes, painful :-)
>
> This happens at a customer deployment & the next time I hit this I'll get
> you the stack trace. I have the hbase-site.xml attached to my response to
> Sean, take a look.
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-hbase.679495.n3.nabble.com/HBase-with-opentsdb-creates-huge-tmp-file-runs-out-of-hdfs-space-tp4067577p4067601.html
> To unsubscribe from HBase with opentsdb creates huge .tmp file & runs out
> of hdfs space, click here
> <http://apache-hbase.679495.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4067577&code=c2F0aHlhZm10QGdtYWlsLmNvbXw0MDY3NTc3fDUxNzU0MjkyMA==>
> .
> NAML
> <http://apache-hbase.679495.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/HBase-with-opentsdb-creates-huge-tmp-file-runs-out-of-hdfs-space-tp4067577p4068148.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: HBase with opentsdb creates huge .tmp file & runs out of hdfs
space
Posted by sathyafmt <sa...@gmail.com>.
yes, painful :-)
This happens at a customer deployment & the next time I hit this I'll get
you the stack trace. I have the hbase-site.xml attached to my response to
Sean, take a look.
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/HBase-with-opentsdb-creates-huge-tmp-file-runs-out-of-hdfs-space-tp4067577p4067601.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: HBase with opentsdb creates huge .tmp file & runs out of hdfs space
Posted by Esteban Gutierrez <es...@cloudera.com>.
Hi sathya,
I agree with Nik: Ouch.
Do you see this file? It doesn't look right as you mention:
/hbase/data/default/tsdb/22681186bbd538cc6f284d923feeed71/.tmp/eca51fd20f464525a3a611fdcd61496e
It is 455GB and that times 3 can easily cause the DNs to run out of space.
It would be really useful if you could provide all the configurations that
the RS is using along with some jstacks. Perhaps just loop curl to grab
the output from the /dump servlet from the RS:
$ (for i in {1..30}; do echo curl http://rs:port/dump ; sleep 10; done) >
rs.dumps.txt
regards,
esteban.
--
Cloudera, Inc.
On Tue, Jan 13, 2015 at 2:53 PM, Sean Busbey <bu...@cloudera.com> wrote:
> what are the configuration settings for
>
> * dfs.data.dir / dfs.datanode.data.dir
> * dfs.datanode.du.reserved
>
> For each of the directories in the first list, can you get the output of
> "tune2fs -l" ?
>
> Can you get us any non-default settings for the hbase cluster and table?
> Are you using compression on the CF?
>
>
> On Tue, Jan 13, 2015 at 1:37 PM, sathyafmt <sa...@gmail.com> wrote:
>
> > Yes, we normally run a 4 node hbase instance with 500G on each of the
> nodes
> > for HDFS. Here's the hadoop fs -ls from a single node instance.
> >
> > sathya
> > =====
> >
> > user@node1:/var/lib$ hadoop fs -ls -R /hbase
> > drwxr-xr-x - hbase hbase 0 2015-01-05 10:59 /hbase/.tmp
> > drwxr-xr-x - hbase hbase 0 2015-01-05 10:59 /hbase/WALs
> > drwxr-xr-x - hbase hbase 0 2015-01-06 13:00
> > /hbase/WALs/node1,60020,1420484330328
> > -rw-r--r-- 3 hbase hbase 29728249 2015-01-06 13:00
> >
> >
> /hbase/WALs/node1,60020,1420484330328/node1%2C60020%2C1420484330328.1420574422072
> > -rw-r--r-- 3 hbase hbase 9 2015-01-06 12:59
> >
> >
> /hbase/WALs/node1,60020,1420484330328/node1%2C60020%2C1420484330328.1420577990013.meta
> > -rw-r--r-- 3 hbase hbase 9 2015-01-06 13:00
> >
> >
> /hbase/WALs/node1,60020,1420484330328/node1%2C60020%2C1420484330328.1420578022196
> > drwxr-xr-x - hbase hbase 0 2015-01-06 11:36 /hbase/archive
> > drwxr-xr-x - hbase hbase 0 2015-01-05 10:53 /hbase/corrupt
> > drwxr-xr-x - hbase hbase 0 2014-12-17 06:38 /hbase/data
> > drwxr-xr-x - hbase hbase 0 2014-12-17 06:45
> /hbase/data/default
> > drwxr-xr-x - hbase hbase 0 2015-01-05 10:59
> > /hbase/data/default/tsdb
> > drwxr-xr-x - hbase hbase 0 2014-12-17 06:45
> > /hbase/data/default/tsdb/.tabledesc
> > -rw-r--r-- 3 hbase hbase 283 2014-12-17 06:45
> > /hbase/data/default/tsdb/.tabledesc/.tableinfo.0000000001
> > drwxr-xr-x - hbase hbase 0 2014-12-17 06:45
> > /hbase/data/default/tsdb/.tmp
> > drwxr-xr-x - hbase hbase 0 2015-01-06 11:34
> > /hbase/data/default/tsdb/22681186bbd538cc6f284d923feeed71
> > -rwxr-xr-x 3 hbase hbase 50 2014-12-30 16:14
> > /hbase/data/default/tsdb/22681186bbd538cc6f284d923feeed71/.regioninfo
> > drwxr-xr-x - hbase hbase 0 2015-01-06 12:34
> > /hbase/data/default/tsdb/22681186bbd538cc6f284d923feeed71/.tmp
> >
> > -rwxr-xr-x 3 hbase hbase 488820965376 2015-01-06 12:34
> >
> >
> /hbase/data/default/tsdb/22681186bbd538cc6f284d923feeed71/.tmp/eca51fd20f464525a3a611fdcd61496e
> >
> > -drwxr-xr-x - hbase hbase 0 2015-01-05 10:53
> > /hbase/data/default/tsdb/22681186bbd538cc6f284d923feeed71/recovered.edits
> > drwxr-xr-x - hbase hbase 0 2015-01-06 12:34
> > /hbase/data/default/tsdb/22681186bbd538cc6f284d923feeed71/t
> > -rwxr-xr-x 3 hbase hbase 977014 2015-01-06 12:34
> >
> >
> /hbase/data/default/tsdb/22681186bbd538cc6f284d923feeed71/t/0eea561cb7b841a7b7dbb6ead58f4556
> > -rwxr-xr-x 3 hbase hbase 775564 2015-01-05 10:53
> >
> >
> /hbase/data/default/tsdb/22681186bbd538cc6f284d923feeed71/t/130b7f0ca2e646e29771367740713911
> > -rwxr-xr-x 3 hbase hbase 216732 2015-01-06 11:34
> >
> >
> /hbase/data/default/tsdb/22681186bbd538cc6f284d923feeed71/t/3bbb590e4f1e4b57a9cca41a80271738
> > drwxr-xr-x - hbase hbase 0 2015-01-05 13:00
> > /hbase/data/default/tsdb/8bb27e884a944b7e87422cf6b4258453
> > -rwxr-xr-x 3 hbase hbase 50 2014-12-30 16:14
> > /hbase/data/default/tsdb/8bb27e884a944b7e87422cf6b4258453/.regioninfo
> > drwxr-xr-x - hbase hbase 0 2015-01-06 13:16
> > /hbase/data/default/tsdb/8bb27e884a944b7e87422cf6b4258453/.tmp
> > -rwxr-xr-x 3 hbase hbase 0 2015-01-06 13:16
> >
> >
> /hbase/data/default/tsdb/8bb27e884a944b7e87422cf6b4258453/.tmp/20679e20318c479ca30d980d9d26293e
> >
> >
> >
> > --
> > View this message in context:
> >
> http://apache-hbase.679495.n3.nabble.com/HBase-with-opentsdb-creates-huge-tmp-file-runs-out-of-hdfs-space-tp4067577p4067589.html
> > Sent from the HBase User mailing list archive at Nabble.com.
> >
>
>
>
> --
> Sean
>
Re: HBase with opentsdb creates huge .tmp file & runs out of hdfs space
Posted by Sean Busbey <bu...@cloudera.com>.
what are the configuration settings for
* dfs.data.dir / dfs.datanode.data.dir
* dfs.datanode.du.reserved
For each of the directories in the first list, can you get the output of
"tune2fs -l" ?
Can you get us any non-default settings for the hbase cluster and table?
Are you using compression on the CF?
On Tue, Jan 13, 2015 at 1:37 PM, sathyafmt <sa...@gmail.com> wrote:
> Yes, we normally run a 4 node hbase instance with 500G on each of the nodes
> for HDFS. Here's the hadoop fs -ls from a single node instance.
>
> sathya
> =====
>
> user@node1:/var/lib$ hadoop fs -ls -R /hbase
> drwxr-xr-x - hbase hbase 0 2015-01-05 10:59 /hbase/.tmp
> drwxr-xr-x - hbase hbase 0 2015-01-05 10:59 /hbase/WALs
> drwxr-xr-x - hbase hbase 0 2015-01-06 13:00
> /hbase/WALs/node1,60020,1420484330328
> -rw-r--r-- 3 hbase hbase 29728249 2015-01-06 13:00
>
> /hbase/WALs/node1,60020,1420484330328/node1%2C60020%2C1420484330328.1420574422072
> -rw-r--r-- 3 hbase hbase 9 2015-01-06 12:59
>
> /hbase/WALs/node1,60020,1420484330328/node1%2C60020%2C1420484330328.1420577990013.meta
> -rw-r--r-- 3 hbase hbase 9 2015-01-06 13:00
>
> /hbase/WALs/node1,60020,1420484330328/node1%2C60020%2C1420484330328.1420578022196
> drwxr-xr-x - hbase hbase 0 2015-01-06 11:36 /hbase/archive
> drwxr-xr-x - hbase hbase 0 2015-01-05 10:53 /hbase/corrupt
> drwxr-xr-x - hbase hbase 0 2014-12-17 06:38 /hbase/data
> drwxr-xr-x - hbase hbase 0 2014-12-17 06:45 /hbase/data/default
> drwxr-xr-x - hbase hbase 0 2015-01-05 10:59
> /hbase/data/default/tsdb
> drwxr-xr-x - hbase hbase 0 2014-12-17 06:45
> /hbase/data/default/tsdb/.tabledesc
> -rw-r--r-- 3 hbase hbase 283 2014-12-17 06:45
> /hbase/data/default/tsdb/.tabledesc/.tableinfo.0000000001
> drwxr-xr-x - hbase hbase 0 2014-12-17 06:45
> /hbase/data/default/tsdb/.tmp
> drwxr-xr-x - hbase hbase 0 2015-01-06 11:34
> /hbase/data/default/tsdb/22681186bbd538cc6f284d923feeed71
> -rwxr-xr-x 3 hbase hbase 50 2014-12-30 16:14
> /hbase/data/default/tsdb/22681186bbd538cc6f284d923feeed71/.regioninfo
> drwxr-xr-x - hbase hbase 0 2015-01-06 12:34
> /hbase/data/default/tsdb/22681186bbd538cc6f284d923feeed71/.tmp
>
> -rwxr-xr-x 3 hbase hbase 488820965376 2015-01-06 12:34
>
> /hbase/data/default/tsdb/22681186bbd538cc6f284d923feeed71/.tmp/eca51fd20f464525a3a611fdcd61496e
>
> -drwxr-xr-x - hbase hbase 0 2015-01-05 10:53
> /hbase/data/default/tsdb/22681186bbd538cc6f284d923feeed71/recovered.edits
> drwxr-xr-x - hbase hbase 0 2015-01-06 12:34
> /hbase/data/default/tsdb/22681186bbd538cc6f284d923feeed71/t
> -rwxr-xr-x 3 hbase hbase 977014 2015-01-06 12:34
>
> /hbase/data/default/tsdb/22681186bbd538cc6f284d923feeed71/t/0eea561cb7b841a7b7dbb6ead58f4556
> -rwxr-xr-x 3 hbase hbase 775564 2015-01-05 10:53
>
> /hbase/data/default/tsdb/22681186bbd538cc6f284d923feeed71/t/130b7f0ca2e646e29771367740713911
> -rwxr-xr-x 3 hbase hbase 216732 2015-01-06 11:34
>
> /hbase/data/default/tsdb/22681186bbd538cc6f284d923feeed71/t/3bbb590e4f1e4b57a9cca41a80271738
> drwxr-xr-x - hbase hbase 0 2015-01-05 13:00
> /hbase/data/default/tsdb/8bb27e884a944b7e87422cf6b4258453
> -rwxr-xr-x 3 hbase hbase 50 2014-12-30 16:14
> /hbase/data/default/tsdb/8bb27e884a944b7e87422cf6b4258453/.regioninfo
> drwxr-xr-x - hbase hbase 0 2015-01-06 13:16
> /hbase/data/default/tsdb/8bb27e884a944b7e87422cf6b4258453/.tmp
> -rwxr-xr-x 3 hbase hbase 0 2015-01-06 13:16
>
> /hbase/data/default/tsdb/8bb27e884a944b7e87422cf6b4258453/.tmp/20679e20318c479ca30d980d9d26293e
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/HBase-with-opentsdb-creates-huge-tmp-file-runs-out-of-hdfs-space-tp4067577p4067589.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
--
Sean
Re: HBase with opentsdb creates huge .tmp file & runs out of hdfs
space
Posted by sathyafmt <sa...@gmail.com>.
Yes, we normally run a 4 node hbase instance with 500G on each of the nodes
for HDFS. Here's the hadoop fs -ls from a single node instance.
sathya
=====
user@node1:/var/lib$ hadoop fs -ls -R /hbase
drwxr-xr-x - hbase hbase 0 2015-01-05 10:59 /hbase/.tmp
drwxr-xr-x - hbase hbase 0 2015-01-05 10:59 /hbase/WALs
drwxr-xr-x - hbase hbase 0 2015-01-06 13:00
/hbase/WALs/node1,60020,1420484330328
-rw-r--r-- 3 hbase hbase 29728249 2015-01-06 13:00
/hbase/WALs/node1,60020,1420484330328/node1%2C60020%2C1420484330328.1420574422072
-rw-r--r-- 3 hbase hbase 9 2015-01-06 12:59
/hbase/WALs/node1,60020,1420484330328/node1%2C60020%2C1420484330328.1420577990013.meta
-rw-r--r-- 3 hbase hbase 9 2015-01-06 13:00
/hbase/WALs/node1,60020,1420484330328/node1%2C60020%2C1420484330328.1420578022196
drwxr-xr-x - hbase hbase 0 2015-01-06 11:36 /hbase/archive
drwxr-xr-x - hbase hbase 0 2015-01-05 10:53 /hbase/corrupt
drwxr-xr-x - hbase hbase 0 2014-12-17 06:38 /hbase/data
drwxr-xr-x - hbase hbase 0 2014-12-17 06:45 /hbase/data/default
drwxr-xr-x - hbase hbase 0 2015-01-05 10:59
/hbase/data/default/tsdb
drwxr-xr-x - hbase hbase 0 2014-12-17 06:45
/hbase/data/default/tsdb/.tabledesc
-rw-r--r-- 3 hbase hbase 283 2014-12-17 06:45
/hbase/data/default/tsdb/.tabledesc/.tableinfo.0000000001
drwxr-xr-x - hbase hbase 0 2014-12-17 06:45
/hbase/data/default/tsdb/.tmp
drwxr-xr-x - hbase hbase 0 2015-01-06 11:34
/hbase/data/default/tsdb/22681186bbd538cc6f284d923feeed71
-rwxr-xr-x 3 hbase hbase 50 2014-12-30 16:14
/hbase/data/default/tsdb/22681186bbd538cc6f284d923feeed71/.regioninfo
drwxr-xr-x - hbase hbase 0 2015-01-06 12:34
/hbase/data/default/tsdb/22681186bbd538cc6f284d923feeed71/.tmp
-rwxr-xr-x 3 hbase hbase 488820965376 2015-01-06 12:34
/hbase/data/default/tsdb/22681186bbd538cc6f284d923feeed71/.tmp/eca51fd20f464525a3a611fdcd61496e
-drwxr-xr-x - hbase hbase 0 2015-01-05 10:53
/hbase/data/default/tsdb/22681186bbd538cc6f284d923feeed71/recovered.edits
drwxr-xr-x - hbase hbase 0 2015-01-06 12:34
/hbase/data/default/tsdb/22681186bbd538cc6f284d923feeed71/t
-rwxr-xr-x 3 hbase hbase 977014 2015-01-06 12:34
/hbase/data/default/tsdb/22681186bbd538cc6f284d923feeed71/t/0eea561cb7b841a7b7dbb6ead58f4556
-rwxr-xr-x 3 hbase hbase 775564 2015-01-05 10:53
/hbase/data/default/tsdb/22681186bbd538cc6f284d923feeed71/t/130b7f0ca2e646e29771367740713911
-rwxr-xr-x 3 hbase hbase 216732 2015-01-06 11:34
/hbase/data/default/tsdb/22681186bbd538cc6f284d923feeed71/t/3bbb590e4f1e4b57a9cca41a80271738
drwxr-xr-x - hbase hbase 0 2015-01-05 13:00
/hbase/data/default/tsdb/8bb27e884a944b7e87422cf6b4258453
-rwxr-xr-x 3 hbase hbase 50 2014-12-30 16:14
/hbase/data/default/tsdb/8bb27e884a944b7e87422cf6b4258453/.regioninfo
drwxr-xr-x - hbase hbase 0 2015-01-06 13:16
/hbase/data/default/tsdb/8bb27e884a944b7e87422cf6b4258453/.tmp
-rwxr-xr-x 3 hbase hbase 0 2015-01-06 13:16
/hbase/data/default/tsdb/8bb27e884a944b7e87422cf6b4258453/.tmp/20679e20318c479ca30d980d9d26293e
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/HBase-with-opentsdb-creates-huge-tmp-file-runs-out-of-hdfs-space-tp4067577p4067589.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: HBase with opentsdb creates huge .tmp file & runs out of hdfs space
Posted by Nick Dimiduk <nd...@gmail.com>.
Ouch :/
Not to be a pedant, but you're sure HDFS is configured against the 2TB of
space (and not, say, the root partition)? You're sure the the temporary
file is growing to 2TB (hadoop fs -ls output)? Are you using any
BlockEncoding or Compression with this column family? Any other store/table
configuration? This happens repeatably? Can you provide jstack of the RS
process along with log lines while this file is growing excessively?
Thanks,
Nick
On Tue, Jan 13, 2015 at 9:47 AM, sathyafmt <sa...@gmail.com> wrote:
> Thanks Esteban. We do have lots of space ~2TB. The compaction starts on
> around a 300MB column and dies after consuming all the 2TB of space.
>
> sathya
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/HBase-with-opentsdb-creates-huge-tmp-file-runs-out-of-hdfs-space-tp4067577p4067583.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
Re: HBase with opentsdb creates huge .tmp file & runs out of hdfs
space
Posted by sathyafmt <sa...@gmail.com>.
Thanks Esteban. We do have lots of space ~2TB. The compaction starts on
around a 300MB column and dies after consuming all the 2TB of space.
sathya
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/HBase-with-opentsdb-creates-huge-tmp-file-runs-out-of-hdfs-space-tp4067577p4067583.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: HBase with opentsdb creates huge .tmp file & runs out of hdfs space
Posted by Esteban Gutierrez <es...@cloudera.com>.
Hello sathya,
Those files under .tmp are created as part of the normal operations from
HBase since it needs to compact the existing store files into into a new
larger file. From your description seems that your VM doesn't have enough
space for HDFS. Have you tried to increase the space allocated space for
HDFS first?
regards,
esteban.
--
Cloudera, Inc.
On Mon, Jan 12, 2015 at 10:02 PM, sathyafmt <sa...@gmail.com> wrote:
> CDH5.1.2 (hbase 0.98.1) (running on a vm - vmware esx)
>
> We use opentsdb(2.1.0RC) with hbase & after ingesting 200-300MB of data,
> hbase tries to compact the table, and ends up creating a .tmp file which
> grows to fill up the entire hdfs space.. and dies eventually. I tried to
> remove this .tmp file & restart hbase, but it goes back to creating this
> gigantic .tmp file & ends up dying again...
>
> Any ideas, what could be causing this? I thought this was a one-off thing &
> tried again by clearing out hbase & starting over again. But it ends up
> back
> in this same state after ingesting 200-300MB of data.
>
> Thanks,
> sathya
>
> ==
> 2015-01-07 16:22:27,855 INFO
> [regionserver60020-smallCompactions-1420676547848] regionserver.HStore:
> Starting compaction of 3 file(s) in t of
> tsdb,,1419996612897.b881b798f5a7a766932a1e59bc2bd738. into
>
> tmpdir=hdfs://localhost/hbase/data/default/tsdb/b881b798f5a7a766932a1e59bc2bd738/.tmp,
> totalSize=300.4 M
> 2015-01-07 16:22:27,856 DEBUG [RS_OPEN_REGION-XXXX:60020-1]
> zookeeper.ZKAssign: regionserver:60020-0x14ac6e25f5c0003,
> quorum=localhost:2181, baseZNode=/hbase Transitioned node
> 15abb3e48aeb8b0b43e8cabb4db459bf from M_ZK_REGION_OFFLINE to
> RS_ZK_REGION_OPENING
> 2015-01-07 16:22:27,857 DEBUG
> [regionserver60020-smallCompactions-1420676547848]
> compactions.Compactor:Compacting
>
> hdfs://localhost/hbase/data/default/tsdb/b881b798f5a7a766932a1e59bc2bd738/t/19fb0a1e6fb54b868c0845ce6589ca05,
> keycount=75, bloomtype=ROW, size=24.8 M, encoding=NONE, seqNum=98187,
> earliestPutTs=1420224086959
> 2015-01-07 16:22:27,855 INFO
> [regionserver60020-smallCompactions-1420676547848] regionserver.HStore:
> Starting compaction of 3 file(s) in t of
> tsdb,,1419996612897.b881b798f5a7a766932a1e59bc2bd738. into
>
> tmpdir=hdfs://localhost/hbase/data/default/tsdb/b881b798f5a7a766932a1e59bc2bd738/.tmp,
> totalSize=300.4 M
> 2015-01-07 16:22:27,856 DEBUG [RS_OPEN_REGION-XXXX:60020-1]
> zookeeper.ZKAssign: regionserver:60020-0x14ac6e25f5c0003,
> quorum=localhost:2181, baseZNode=/hbase Transitioned node
> 15abb3e48aeb8b0b43e8cabb4db459bf from M_ZK_REGION_OFFLINE to
> RS_ZK_REGION_OPENING
> 2015-01-07 16:22:27,857 DEBUG
> [regionserver60020-smallCompactions-1420676547848]
> compactions.Compactor:Compacting
>
> hdfs://localhost/hbase/data/default/tsdb/b881b798f5a7a766932a1e59bc2bd738/t/19fb0a1e6fb54b868c0845ce6589ca05,
> keycount=75, bloomtype=ROW, size=24.8 M, encoding=NONE, seqNum=98187,
> earliestPutTs=1420224086959
> 2015-01-07 16:22:27,855 INFO
> [regionserver60020-smallCompactions-1420676547848] regionserver.HStore:
> Starting compaction of 3 file(s) in t of
> tsdb,,1419996612897.b881b798f5a7a766932a1e59bc2bd738. into
>
> tmpdir=hdfs://localhost/hbase/data/default/tsdb/b881b798f5a7a766932a1e59bc2bd738/.tmp,
> totalSize=300.4 M
> 2015-01-07 16:22:27,856 DEBUG [RS_OPEN_REGION-XXXX:60020-1]
> zookeeper.ZKAssign: regionserver:60020-0x14ac6e25f5c0003,
> quorum=localhost:2181, baseZNode=/hbase Transitioned node
> 15abb3e48aeb8b0b43e8cabb4db459bf from M_ZK_REGION_OFFLINE to
> RS_ZK_REGION_OPENING
> 2015-01-07 16:22:27,857 DEBUG
> [regionserver60020-smallCompactions-1420676547848]
> compactions.Compactor:Compacting
>
> hdfs://localhost/hbase/data/default/tsdb/b881b798f5a7a766932a1e59bc2bd738/t/19fb0a1e6fb54b868c0845ce6589ca05,
> keycount=75, bloomtype=ROW, size=24.8 M, encoding=NONE, seqNum=98187,
> earliestPutTs=1420224086959
>
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/HBase-with-opentsdb-creates-huge-tmp-file-runs-out-of-hdfs-space-tp4067577.html
> Sent from the HBase User mailing list archive at Nabble.com.
>