You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Elaine Gan <el...@gmo.jp> on 2012/11/06 09:11:13 UTC
Capacity of hdfs
Hi,
I'm a bit confused on the local disk and the hdfs disk.
Through my admin page, i see information like the following.
Configured Capacity : 254.05 GB
DFS Used : 207.09 GB
Non DFS Used : 16.87 GB
DFS Remaining : 30.09 GB
DFS Used% : 81.52 %
DFS Remaining% : 11.84 %
I only made configuration on the heap memory and not the disk capacity.
Where does "Configured Capacity : 254.05 GB " this comes from?
I have 4 nodes, each with 64GB of disk (local).
Will HDFS use it fully ?
I still need some place for local files.
Thank you.
Re: Capacity of hdfs
Posted by Elaine Gan <el...@gmo.jp>.
Dear Harsh,
Thank you for the very clear explanation.
Very well noted now :)
> Hi,
>
> Inline explanation:
>
> On Tue, Nov 6, 2012 at 1:41 PM, Elaine Gan <el...@gmo.jp> wrote:
> > Hi,
> >
> > I'm a bit confused on the local disk and the hdfs disk.
> > Through my admin page, i see information like the following.
> >
> > Configured Capacity : 254.05 GB
>
> Configured Capacity = Sum(Configured DN Capacity)
> = 64*4 = 256.
>
> > DFS Used : 207.09 GB
>
> HDFS files are cumulatively using 207 GB of data. This includes
> replicated blocks (default replication factor is 3).
>
> > Non DFS Used : 16.87 GB
>
> This is Sum(Configured DN Space) that lies outside the HDFS files.
> HDFS runs utilities such as "du"/"df" on disks and can pick up space
> that is being utilized by non HDFS, local files such as log files,
> etc. lying in the same disk as the one used for DNs.
>
> > DFS Remaining : 30.09 GB
>
> Your DFS has only 30 GB more to spare, which with a replication factor
> of 3, is less than 10 GB of true file length space.
>
> > DFS Used% : 81.52 %
> > DFS Remaining% : 11.84 %
>
> These should be self-explainatory.
>
> > I only made configuration on the heap memory and not the disk capacity.
> > Where does "Configured Capacity : 254.05 GB " this comes from?
> > I have 4 nodes, each with 64GB of disk (local).
> > Will HDFS use it fully ?
>
> Yes it would, unless you configure the "dfs.datanode.du.reserved"
> config param at each DN to a space value in bytes that must be left
> free on all configured volumes.
>
> > I still need some place for local files.
> >
> > Thank you.
> >
>
> Hope this helps!
>
> --
> Harsh J
Re: Capacity of hdfs
Posted by Elaine Gan <el...@gmo.jp>.
Dear Harsh,
Thank you for the very clear explanation.
Very well noted now :)
> Hi,
>
> Inline explanation:
>
> On Tue, Nov 6, 2012 at 1:41 PM, Elaine Gan <el...@gmo.jp> wrote:
> > Hi,
> >
> > I'm a bit confused on the local disk and the hdfs disk.
> > Through my admin page, i see information like the following.
> >
> > Configured Capacity : 254.05 GB
>
> Configured Capacity = Sum(Configured DN Capacity)
> = 64*4 = 256.
>
> > DFS Used : 207.09 GB
>
> HDFS files are cumulatively using 207 GB of data. This includes
> replicated blocks (default replication factor is 3).
>
> > Non DFS Used : 16.87 GB
>
> This is Sum(Configured DN Space) that lies outside the HDFS files.
> HDFS runs utilities such as "du"/"df" on disks and can pick up space
> that is being utilized by non HDFS, local files such as log files,
> etc. lying in the same disk as the one used for DNs.
>
> > DFS Remaining : 30.09 GB
>
> Your DFS has only 30 GB more to spare, which with a replication factor
> of 3, is less than 10 GB of true file length space.
>
> > DFS Used% : 81.52 %
> > DFS Remaining% : 11.84 %
>
> These should be self-explainatory.
>
> > I only made configuration on the heap memory and not the disk capacity.
> > Where does "Configured Capacity : 254.05 GB " this comes from?
> > I have 4 nodes, each with 64GB of disk (local).
> > Will HDFS use it fully ?
>
> Yes it would, unless you configure the "dfs.datanode.du.reserved"
> config param at each DN to a space value in bytes that must be left
> free on all configured volumes.
>
> > I still need some place for local files.
> >
> > Thank you.
> >
>
> Hope this helps!
>
> --
> Harsh J
Re: Capacity of hdfs
Posted by Elaine Gan <el...@gmo.jp>.
Dear Harsh,
Thank you for the very clear explanation.
Very well noted now :)
> Hi,
>
> Inline explanation:
>
> On Tue, Nov 6, 2012 at 1:41 PM, Elaine Gan <el...@gmo.jp> wrote:
> > Hi,
> >
> > I'm a bit confused on the local disk and the hdfs disk.
> > Through my admin page, i see information like the following.
> >
> > Configured Capacity : 254.05 GB
>
> Configured Capacity = Sum(Configured DN Capacity)
> = 64*4 = 256.
>
> > DFS Used : 207.09 GB
>
> HDFS files are cumulatively using 207 GB of data. This includes
> replicated blocks (default replication factor is 3).
>
> > Non DFS Used : 16.87 GB
>
> This is Sum(Configured DN Space) that lies outside the HDFS files.
> HDFS runs utilities such as "du"/"df" on disks and can pick up space
> that is being utilized by non HDFS, local files such as log files,
> etc. lying in the same disk as the one used for DNs.
>
> > DFS Remaining : 30.09 GB
>
> Your DFS has only 30 GB more to spare, which with a replication factor
> of 3, is less than 10 GB of true file length space.
>
> > DFS Used% : 81.52 %
> > DFS Remaining% : 11.84 %
>
> These should be self-explainatory.
>
> > I only made configuration on the heap memory and not the disk capacity.
> > Where does "Configured Capacity : 254.05 GB " this comes from?
> > I have 4 nodes, each with 64GB of disk (local).
> > Will HDFS use it fully ?
>
> Yes it would, unless you configure the "dfs.datanode.du.reserved"
> config param at each DN to a space value in bytes that must be left
> free on all configured volumes.
>
> > I still need some place for local files.
> >
> > Thank you.
> >
>
> Hope this helps!
>
> --
> Harsh J
Re: Capacity of hdfs
Posted by Elaine Gan <el...@gmo.jp>.
Dear Harsh,
Thank you for the very clear explanation.
Very well noted now :)
> Hi,
>
> Inline explanation:
>
> On Tue, Nov 6, 2012 at 1:41 PM, Elaine Gan <el...@gmo.jp> wrote:
> > Hi,
> >
> > I'm a bit confused on the local disk and the hdfs disk.
> > Through my admin page, i see information like the following.
> >
> > Configured Capacity : 254.05 GB
>
> Configured Capacity = Sum(Configured DN Capacity)
> = 64*4 = 256.
>
> > DFS Used : 207.09 GB
>
> HDFS files are cumulatively using 207 GB of data. This includes
> replicated blocks (default replication factor is 3).
>
> > Non DFS Used : 16.87 GB
>
> This is Sum(Configured DN Space) that lies outside the HDFS files.
> HDFS runs utilities such as "du"/"df" on disks and can pick up space
> that is being utilized by non HDFS, local files such as log files,
> etc. lying in the same disk as the one used for DNs.
>
> > DFS Remaining : 30.09 GB
>
> Your DFS has only 30 GB more to spare, which with a replication factor
> of 3, is less than 10 GB of true file length space.
>
> > DFS Used% : 81.52 %
> > DFS Remaining% : 11.84 %
>
> These should be self-explainatory.
>
> > I only made configuration on the heap memory and not the disk capacity.
> > Where does "Configured Capacity : 254.05 GB " this comes from?
> > I have 4 nodes, each with 64GB of disk (local).
> > Will HDFS use it fully ?
>
> Yes it would, unless you configure the "dfs.datanode.du.reserved"
> config param at each DN to a space value in bytes that must be left
> free on all configured volumes.
>
> > I still need some place for local files.
> >
> > Thank you.
> >
>
> Hope this helps!
>
> --
> Harsh J
Re: Capacity of hdfs
Posted by Harsh J <ha...@cloudera.com>.
Hi,
Inline explanation:
On Tue, Nov 6, 2012 at 1:41 PM, Elaine Gan <el...@gmo.jp> wrote:
> Hi,
>
> I'm a bit confused on the local disk and the hdfs disk.
> Through my admin page, i see information like the following.
>
> Configured Capacity : 254.05 GB
Configured Capacity = Sum(Configured DN Capacity)
= 64*4 = 256.
> DFS Used : 207.09 GB
HDFS files are cumulatively using 207 GB of data. This includes
replicated blocks (default replication factor is 3).
> Non DFS Used : 16.87 GB
This is Sum(Configured DN Space) that lies outside the HDFS files.
HDFS runs utilities such as "du"/"df" on disks and can pick up space
that is being utilized by non HDFS, local files such as log files,
etc. lying in the same disk as the one used for DNs.
> DFS Remaining : 30.09 GB
Your DFS has only 30 GB more to spare, which with a replication factor
of 3, is less than 10 GB of true file length space.
> DFS Used% : 81.52 %
> DFS Remaining% : 11.84 %
These should be self-explainatory.
> I only made configuration on the heap memory and not the disk capacity.
> Where does "Configured Capacity : 254.05 GB " this comes from?
> I have 4 nodes, each with 64GB of disk (local).
> Will HDFS use it fully ?
Yes it would, unless you configure the "dfs.datanode.du.reserved"
config param at each DN to a space value in bytes that must be left
free on all configured volumes.
> I still need some place for local files.
>
> Thank you.
>
Hope this helps!
--
Harsh J
Re: Capacity of hdfs
Posted by Harsh J <ha...@cloudera.com>.
Hi,
Inline explanation:
On Tue, Nov 6, 2012 at 1:41 PM, Elaine Gan <el...@gmo.jp> wrote:
> Hi,
>
> I'm a bit confused on the local disk and the hdfs disk.
> Through my admin page, i see information like the following.
>
> Configured Capacity : 254.05 GB
Configured Capacity = Sum(Configured DN Capacity)
= 64*4 = 256.
> DFS Used : 207.09 GB
HDFS files are cumulatively using 207 GB of data. This includes
replicated blocks (default replication factor is 3).
> Non DFS Used : 16.87 GB
This is Sum(Configured DN Space) that lies outside the HDFS files.
HDFS runs utilities such as "du"/"df" on disks and can pick up space
that is being utilized by non HDFS, local files such as log files,
etc. lying in the same disk as the one used for DNs.
> DFS Remaining : 30.09 GB
Your DFS has only 30 GB more to spare, which with a replication factor
of 3, is less than 10 GB of true file length space.
> DFS Used% : 81.52 %
> DFS Remaining% : 11.84 %
These should be self-explainatory.
> I only made configuration on the heap memory and not the disk capacity.
> Where does "Configured Capacity : 254.05 GB " this comes from?
> I have 4 nodes, each with 64GB of disk (local).
> Will HDFS use it fully ?
Yes it would, unless you configure the "dfs.datanode.du.reserved"
config param at each DN to a space value in bytes that must be left
free on all configured volumes.
> I still need some place for local files.
>
> Thank you.
>
Hope this helps!
--
Harsh J
Re: Capacity of hdfs
Posted by Harsh J <ha...@cloudera.com>.
Hi,
Inline explanation:
On Tue, Nov 6, 2012 at 1:41 PM, Elaine Gan <el...@gmo.jp> wrote:
> Hi,
>
> I'm a bit confused on the local disk and the hdfs disk.
> Through my admin page, i see information like the following.
>
> Configured Capacity : 254.05 GB
Configured Capacity = Sum(Configured DN Capacity)
= 64*4 = 256.
> DFS Used : 207.09 GB
HDFS files are cumulatively using 207 GB of data. This includes
replicated blocks (default replication factor is 3).
> Non DFS Used : 16.87 GB
This is Sum(Configured DN Space) that lies outside the HDFS files.
HDFS runs utilities such as "du"/"df" on disks and can pick up space
that is being utilized by non HDFS, local files such as log files,
etc. lying in the same disk as the one used for DNs.
> DFS Remaining : 30.09 GB
Your DFS has only 30 GB more to spare, which with a replication factor
of 3, is less than 10 GB of true file length space.
> DFS Used% : 81.52 %
> DFS Remaining% : 11.84 %
These should be self-explainatory.
> I only made configuration on the heap memory and not the disk capacity.
> Where does "Configured Capacity : 254.05 GB " this comes from?
> I have 4 nodes, each with 64GB of disk (local).
> Will HDFS use it fully ?
Yes it would, unless you configure the "dfs.datanode.du.reserved"
config param at each DN to a space value in bytes that must be left
free on all configured volumes.
> I still need some place for local files.
>
> Thank you.
>
Hope this helps!
--
Harsh J
Re: Capacity of hdfs
Posted by Harsh J <ha...@cloudera.com>.
Hi,
Inline explanation:
On Tue, Nov 6, 2012 at 1:41 PM, Elaine Gan <el...@gmo.jp> wrote:
> Hi,
>
> I'm a bit confused on the local disk and the hdfs disk.
> Through my admin page, i see information like the following.
>
> Configured Capacity : 254.05 GB
Configured Capacity = Sum(Configured DN Capacity)
= 64*4 = 256.
> DFS Used : 207.09 GB
HDFS files are cumulatively using 207 GB of data. This includes
replicated blocks (default replication factor is 3).
> Non DFS Used : 16.87 GB
This is Sum(Configured DN Space) that lies outside the HDFS files.
HDFS runs utilities such as "du"/"df" on disks and can pick up space
that is being utilized by non HDFS, local files such as log files,
etc. lying in the same disk as the one used for DNs.
> DFS Remaining : 30.09 GB
Your DFS has only 30 GB more to spare, which with a replication factor
of 3, is less than 10 GB of true file length space.
> DFS Used% : 81.52 %
> DFS Remaining% : 11.84 %
These should be self-explainatory.
> I only made configuration on the heap memory and not the disk capacity.
> Where does "Configured Capacity : 254.05 GB " this comes from?
> I have 4 nodes, each with 64GB of disk (local).
> Will HDFS use it fully ?
Yes it would, unless you configure the "dfs.datanode.du.reserved"
config param at each DN to a space value in bytes that must be left
free on all configured volumes.
> I still need some place for local files.
>
> Thank you.
>
Hope this helps!
--
Harsh J