You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@trafodion.apache.org by Gunnar Tapper <gu...@esgyn.com> on 2015/07/14 18:27:07 UTC
Fwd: Disk space usage
Hi Folks,
Please see the below questions from Layla regarding disk-space usage.
Any thoughts on this subject?
Thanks,
Gunnar
---------- Forwarded message ----------
From: Martin, Layla (HP DualStudy) <la...@hp.com>
Date: Tue, Jul 14, 2015 at 2:08 AM
Subject: Disk space usage
To: "gunnar.tapper@esgyn.com" <gu...@esgyn.com>
Hi Gunnar,
Thanks for offering help!
I ran a few tests this morning regarding the disk space consumption of
Trafodion. My results are rather surprising:
Used disk space grows whenever I create a table (as expected)
Used disk space (usually) grows whenever I insert new data sets (stumbling:
in one occurrence, the used disk space actually got less)
Used disk space grows whenever I delete data sets (unlike I expected, but
this is ok, as HBase just outdates the datasets instead of truly deleting
them)
(It even grows when I now run a major compaction using the HBase Shell)
Used disk space grows even further, if I now drop this table (this is
surprising to me, as HBase should be able to delete all HFiles related to
this table)
This really becomes troubling, as my disk was 100% full twice (causing
trouble …). Obviously, not even a garbage collection process running later
is able to permanently delete this from disk.
A colleague of mine here in the EPC suggested that it could be disk space
allocated to Trafodion but not really used. I searched the Trafodion SQL
Reference Handbook, but didn’t find a query to return the disk space
comsumption of the Trafodion data. My colleague Rebecca suggested something
similar to
SELECT TABLESPACE_NAME, SUM(BYTES)/1024/1024/1024 "FREE SPACE(GB)"
FROM DBA_FREE_SPACE GROUP BY TABLESPACE_NAME;
(I’ve checked the disk space consumption using the df . command in the
Linux shell)
Thank you very much in advance
Layla
*Layla Martin*
HP DualStudy
Hewlett-Packard GmbH
Telefon +49 7031 4504682
*layla.martin@hp.com <la...@hp.com>*
Herrenberger Strasse 140| 71034 Böblingen| www.hp.com/de/dualstudy
[image: Description: Description:
http://intranet.hp.com/country/germany/live/PublishingImages/NeuesLogoSmall.png]
Follow us on: [image: facebook] <https://www.facebook.com/DualStudy.hp> [image:
twitter] <http://twitter.com/hpdualstudy> [image: youtube]
<http://www.youtube.com/user/hpDualStudy>
Geschäftsführer: Heiko Meyer (Vorsitzender), Thomas Bässler, Volkhard
Bregulla, Michael Eberhardt, Jochen Erlach, Angelika Gifford, Ernst Reichart
Vorsitzender des Aufsichtsrats: Jörg Menno Harms
Sitz der Gesellschaft: Böblingen, Amtsgericht Stuttgart HRB 244081
WEEE-Reg.-Nr. DE 30409072
--
Thank you,
Gunnar Tapper
Presales Manager
Esgyn Corp.
Re: Disk space usage
Posted by D&P Markt <dm...@gmail.com>.
Hi,
My first reaction was many of these findings make sense, but certainly not
the one where the disk space is not reclaimed after data is deleted and a
major compaction has been run. Hans has given one possible explanation.
Just to recap:
1) Trafodion uses HBase/Hadoop to store the base files so it should be
possible to find where the space is being used by using the HBase and Hadoop
utilities. Each Region Server can be queried through port 60030 to see the
store file sizes. Likewise the Hadoop Data Nodes provide information at
their level; the DN uses port 50075. If you have access to root or the
userid that owns the HDFS files I find the output of this command to be very
useful: "hadoop fsck /hbase/data/default -locations -files -blocks" (though
there will be a lot of output so direct it to a file). You might have to
change the "/hbase/data/default" portion depending on your directory
structure. Once you get the output you'll see output like:
/hbase/data/default/TRAFODION.JAVABENCH.OE_CUSTOMER_512/00a97580aeaec5a0490764ec21e07016/#1/c5fdfd982705413ab88227717d4e77e7
265076004 bytes, 2 block(s): OK
0. BP-2144980289-172.31.0.2-1411967709852:blk_1078015084_4274539
len=134217728 repl=3 [172.31.0.1:50010, 172.31.0.6:50010, 172.31.0.7:50010]
1. BP-2144980289-172.31.0.2-1411967709852:blk_1078015092_4274547
len=130858276 repl=3 [172.31.0.1:50010, 172.31.0.10:50010, 172.31.0.7:50010]
/hbase/data/default/TRAFODION.JAVABENCH.OE_CUSTOMER_512/00a97580aeaec5a0490764ec21e07016/#1/d72c6f2cbfa540889580ded8a846d406
927627125 bytes, 7 block(s): OK
0. BP-2144980289-172.31.0.2-1411967709852:blk_1078014856_4274311
len=134217728 repl=3 [172.31.0.1:50010, 172.31.0.8:50010, 172.31.0.4:50010]
1. BP-2144980289-172.31.0.2-1411967709852:blk_1078014861_4274316
len=134217728 repl=3 [172.31.0.1:50010, 172.31.0.7:50010, 172.31.0.2:50010]
2. BP-2144980289-172.31.0.2-1411967709852:blk_1078014870_4274325
len=134217728 repl=3 [172.31.0.1:50010, 172.31.0.4:50010, 172.31.0.7:50010]
3. BP-2144980289-172.31.0.2-1411967709852:blk_1078014877_4274332
len=134217728 repl=3 [172.31.0.1:50010, 172.31.0.6:50010, 172.31.0.9:50010]
4. BP-2144980289-172.31.0.2-1411967709852:blk_1078014884_4274339
len=134217728 repl=3 [172.31.0.1:50010, 172.31.0.6:50010, 172.31.0.2:50010]
5. BP-2144980289-172.31.0.2-1411967709852:blk_1078014896_4274351
len=134217728 repl=3 [172.31.0.1:50010, 172.31.0.7:50010, 172.31.0.3:50010]
6. BP-2144980289-172.31.0.2-1411967709852:blk_1078014904_4274359
len=122320757 repl=3 [172.31.0.1:50010, 172.31.0.3:50010, 172.31.0.6:50010]
/hbase/data/default/TRAFODION.JAVABENCH.OE_CUSTOMER_512/00a97580aeaec5a0490764ec21e07016/#1/dc5a445cd74348efaad3240149c39d2b
42007773 bytes, 1 block(s): OK
0. BP-2144980289-172.31.0.2-1411967709852:blk_1078021890_4281345
len=42007773 repl=3 [172.31.0.1:50010, 172.31.0.7:50010, 172.31.0.3:50010]
As Hans mentioned the Trafodion files per se are under the "#1" directory.
So you see in this output that region has 3 Hfiles stored in 10 blocks
(though with the replication factor of 3 it's using 3x that disk space
overall). Note that not all of the blocks are full because the contents
were flushed before the block became full. If you can't use the hadoop fsck
command using the hadoop dfs commands mentioned by Hans will get you close
to what this data shows.
2) Hfiles are immutable, so once written they aren't changed. So deleting
data doesn't allow any space to be reclaimed until a major compaction takes
place that removes the deleted rows in the new files that are being written.
I don't ever recall a major compaction resulting in a larger store file
after the compaction. I guess if your memstore contained only new data and
you did a major compaction the end result might look like the store file got
larger. If you made sure to flush the memstore, check the file sizes, and
then do the major compaction and check the sizes without adding any data, I
can't imagine the file sizes will have grown. If you see an overall
increase then I would think it has to be one of the examples Hans mentioned.
Regards,
Dennis
-----Original Message-----
From: Hans Zeller
Sent: Tuesday, July 14, 2015 1:27 PM
To: dev@trafodion.incubator.apache.org
Subject: Re: Disk space usage
Hi,
Thanks, Layla, for sharing the results of this experiment. Two things come
to my mind. One is the WAL that might be growing and the other, less likely
one is shapshots, created in the loader. A possible third one is the HBase
or HDFS trash. See
http://www.cloudera.com/content/cloudera/en/documentation/cloudera-manager/v4-8-1/Cloudera-Manager-Managing-Clusters/cmmc_hdfs_trash.html
.
To investigate this further, it might be good to list the space consumption
of individual HFiles, using the hadoop dfs commands. For example:
hadoop dfs -ls -R /hbase
hadoop dfs -du /hbase/data/default
hadoop dfs -du /hbase/WALs
We may then need to go one step deeper to look at the HFiles of a table. As
far as I can tell, the HFiles are in a directory
/hbase/data/default/TRAFODION.<schema>.<tablename>/<region id>/#1/. It may
be somewhat different in your setup.
Regards,
Hans
On Tue, Jul 14, 2015 at 9:27 AM, Gunnar Tapper <gu...@esgyn.com>
wrote:
> Hi Folks,
>
> Please see the below questions from Layla regarding disk-space usage.
>
> Any thoughts on this subject?
>
> Thanks,
>
> Gunnar
>
> ---------- Forwarded message ----------
> From: Martin, Layla (HP DualStudy) <la...@hp.com>
> Date: Tue, Jul 14, 2015 at 2:08 AM
> Subject: Disk space usage
> To: "gunnar.tapper@esgyn.com" <gu...@esgyn.com>
>
>
> Hi Gunnar,
>
>
>
> Thanks for offering help!
>
>
>
> I ran a few tests this morning regarding the disk space consumption of
> Trafodion. My results are rather surprising:
>
> Used disk space grows whenever I create a table (as expected)
>
> Used disk space (usually) grows whenever I insert new data sets
> (stumbling: in one occurrence, the used disk space actually got less)
>
> Used disk space grows whenever I delete data sets (unlike I expected, but
> this is ok, as HBase just outdates the datasets instead of truly deleting
> them)
>
> (It even grows when I now run a major compaction using the HBase Shell)
>
> Used disk space grows even further, if I now drop this table (this is
> surprising to me, as HBase should be able to delete all HFiles related to
> this table)
>
>
>
> This really becomes troubling, as my disk was 100% full twice (causing
> trouble …). Obviously, not even a garbage collection process running later
> is able to permanently delete this from disk.
>
>
>
> A colleague of mine here in the EPC suggested that it could be disk space
> allocated to Trafodion but not really used. I searched the Trafodion SQL
> Reference Handbook, but didn’t find a query to return the disk space
> comsumption of the Trafodion data. My colleague Rebecca suggested
> something
> similar to
>
>
>
> SELECT TABLESPACE_NAME, SUM(BYTES)/1024/1024/1024 "FREE SPACE(GB)"
>
> FROM DBA_FREE_SPACE GROUP BY TABLESPACE_NAME;
>
>
>
> (I’ve checked the disk space consumption using the df . command in the
> Linux shell)
>
>
>
> Thank you very much in advance
>
> Layla
>
>
>
> *Layla Martin*
>
> HP DualStudy
>
> Hewlett-Packard GmbH
>
> Telefon +49 7031 4504682
>
> *layla.martin@hp.com <la...@hp.com>*
>
> Herrenberger Strasse 140| 71034 Böblingen| www.hp.com/de/dualstudy
>
> [image: Description: Description:
> http://intranet.hp.com/country/germany/live/PublishingImages/NeuesLogoSmall.png]
>
> Follow us on: [image: facebook] <https://www.facebook.com/DualStudy.hp>
> [image:
> twitter] <http://twitter.com/hpdualstudy> [image: youtube]
> <http://www.youtube.com/user/hpDualStudy>
>
> Geschäftsführer: Heiko Meyer (Vorsitzender), Thomas Bässler, Volkhard
> Bregulla, Michael Eberhardt, Jochen Erlach, Angelika Gifford, Ernst
> Reichart
> Vorsitzender des Aufsichtsrats: Jörg Menno Harms
> Sitz der Gesellschaft: Böblingen, Amtsgericht Stuttgart HRB 244081
> WEEE-Reg.-Nr. DE 30409072
>
>
>
>
>
>
>
> --
> Thank you,
>
> Gunnar Tapper
> Presales Manager
> Esgyn Corp.
>
Re: Disk space usage
Posted by Hans Zeller <ha...@esgyn.com>.
Hi,
Thanks, Layla, for sharing the results of this experiment. Two things come
to my mind. One is the WAL that might be growing and the other, less likely
one is shapshots, created in the loader. A possible third one is the HBase
or HDFS trash. See
http://www.cloudera.com/content/cloudera/en/documentation/cloudera-manager/v4-8-1/Cloudera-Manager-Managing-Clusters/cmmc_hdfs_trash.html
.
To investigate this further, it might be good to list the space consumption
of individual HFiles, using the hadoop dfs commands. For example:
hadoop dfs -ls -R /hbase
hadoop dfs -du /hbase/data/default
hadoop dfs -du /hbase/WALs
We may then need to go one step deeper to look at the HFiles of a table. As
far as I can tell, the HFiles are in a directory
/hbase/data/default/TRAFODION.<schema>.<tablename>/<region id>/#1/. It may
be somewhat different in your setup.
Regards,
Hans
On Tue, Jul 14, 2015 at 9:27 AM, Gunnar Tapper <gu...@esgyn.com>
wrote:
> Hi Folks,
>
> Please see the below questions from Layla regarding disk-space usage.
>
> Any thoughts on this subject?
>
> Thanks,
>
> Gunnar
>
> ---------- Forwarded message ----------
> From: Martin, Layla (HP DualStudy) <la...@hp.com>
> Date: Tue, Jul 14, 2015 at 2:08 AM
> Subject: Disk space usage
> To: "gunnar.tapper@esgyn.com" <gu...@esgyn.com>
>
>
> Hi Gunnar,
>
>
>
> Thanks for offering help!
>
>
>
> I ran a few tests this morning regarding the disk space consumption of
> Trafodion. My results are rather surprising:
>
> Used disk space grows whenever I create a table (as expected)
>
> Used disk space (usually) grows whenever I insert new data sets
> (stumbling: in one occurrence, the used disk space actually got less)
>
> Used disk space grows whenever I delete data sets (unlike I expected, but
> this is ok, as HBase just outdates the datasets instead of truly deleting
> them)
>
> (It even grows when I now run a major compaction using the HBase Shell)
>
> Used disk space grows even further, if I now drop this table (this is
> surprising to me, as HBase should be able to delete all HFiles related to
> this table)
>
>
>
> This really becomes troubling, as my disk was 100% full twice (causing
> trouble …). Obviously, not even a garbage collection process running later
> is able to permanently delete this from disk.
>
>
>
> A colleague of mine here in the EPC suggested that it could be disk space
> allocated to Trafodion but not really used. I searched the Trafodion SQL
> Reference Handbook, but didn’t find a query to return the disk space
> comsumption of the Trafodion data. My colleague Rebecca suggested something
> similar to
>
>
>
> SELECT TABLESPACE_NAME, SUM(BYTES)/1024/1024/1024 "FREE SPACE(GB)"
>
> FROM DBA_FREE_SPACE GROUP BY TABLESPACE_NAME;
>
>
>
> (I’ve checked the disk space consumption using the df . command in the
> Linux shell)
>
>
>
> Thank you very much in advance
>
> Layla
>
>
>
> *Layla Martin*
>
> HP DualStudy
>
> Hewlett-Packard GmbH
>
> Telefon +49 7031 4504682
>
> *layla.martin@hp.com <la...@hp.com>*
>
> Herrenberger Strasse 140| 71034 Böblingen| www.hp.com/de/dualstudy
>
> [image: Description: Description:
> http://intranet.hp.com/country/germany/live/PublishingImages/NeuesLogoSmall.png]
>
> Follow us on: [image: facebook] <https://www.facebook.com/DualStudy.hp> [image:
> twitter] <http://twitter.com/hpdualstudy> [image: youtube]
> <http://www.youtube.com/user/hpDualStudy>
>
> Geschäftsführer: Heiko Meyer (Vorsitzender), Thomas Bässler, Volkhard
> Bregulla, Michael Eberhardt, Jochen Erlach, Angelika Gifford, Ernst
> Reichart
> Vorsitzender des Aufsichtsrats: Jörg Menno Harms
> Sitz der Gesellschaft: Böblingen, Amtsgericht Stuttgart HRB 244081
> WEEE-Reg.-Nr. DE 30409072
>
>
>
>
>
>
>
> --
> Thank you,
>
> Gunnar Tapper
> Presales Manager
> Esgyn Corp.
>