You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Mohammad Tariq <do...@gmail.com> on 2012/12/20 13:01:50 UTC
Metadata size for 1 TB HDFS data?
Hello group,
What could be the approx. size of the metadata if I have 1 TB of
data in my HDFS?I am not doing anything additional but just a simple put.
Will it be ((1*1024*1024)/64)*200 Bytes?
*Keeping 64M as the block size.
Is my understanding right?Please correct me if i'm wrong.
Many thanks.
Best Regards,
Tariq
+91-9741563634
https://mtariq.jux.com/
Re: Metadata size for 1 TB HDFS data?
Posted by Mohammad Tariq <do...@gmail.com>.
Thank you so much for the valuable response Stephen. But I have a few
questions to ask here. Could you please elaborate a bit, if possible?
Each of the specified objects are totally different from each other. A file
will be smaller than a directory in size, and a directory might be smaller
than a block. They might have totally different attributes as well. But
still the space required by each object is same as the other. How is it
possible? Is there any formula or rule of thumb to calculate this?
Many thanks.
Best Regards,
Tariq
+91-9741563634
https://mtariq.jux.com/
On Thu, Dec 20, 2012 at 8:10 PM, Stephen Fritz <st...@cloudera.com>wrote:
> Each block, file, and directory is an object in the namenodes heap, so it
> depends on how you're storing your data. You may need to account for those
> in your calculations.
>
>
>
> On Thu, Dec 20, 2012 at 7:01 AM, Mohammad Tariq <do...@gmail.com>wrote:
>
>> Hello group,
>>
>> What could be the approx. size of the metadata if I have 1 TB of
>> data in my HDFS?I am not doing anything additional but just a simple put.
>> Will it be ((1*1024*1024)/64)*200 Bytes?
>> *Keeping 64M as the block size.
>>
>> Is my understanding right?Please correct me if i'm wrong.
>>
>> Many thanks.
>>
>> Best Regards,
>> Tariq
>> +91-9741563634
>> https://mtariq.jux.com/
>>
>
>
Re: Metadata size for 1 TB HDFS data?
Posted by Mohammad Tariq <do...@gmail.com>.
Thank you so much for the valuable response Stephen. But I have a few
questions to ask here. Could you please elaborate a bit, if possible?
Each of the specified objects are totally different from each other. A file
will be smaller than a directory in size, and a directory might be smaller
than a block. They might have totally different attributes as well. But
still the space required by each object is same as the other. How is it
possible? Is there any formula or rule of thumb to calculate this?
Many thanks.
Best Regards,
Tariq
+91-9741563634
https://mtariq.jux.com/
On Thu, Dec 20, 2012 at 8:10 PM, Stephen Fritz <st...@cloudera.com>wrote:
> Each block, file, and directory is an object in the namenodes heap, so it
> depends on how you're storing your data. You may need to account for those
> in your calculations.
>
>
>
> On Thu, Dec 20, 2012 at 7:01 AM, Mohammad Tariq <do...@gmail.com>wrote:
>
>> Hello group,
>>
>> What could be the approx. size of the metadata if I have 1 TB of
>> data in my HDFS?I am not doing anything additional but just a simple put.
>> Will it be ((1*1024*1024)/64)*200 Bytes?
>> *Keeping 64M as the block size.
>>
>> Is my understanding right?Please correct me if i'm wrong.
>>
>> Many thanks.
>>
>> Best Regards,
>> Tariq
>> +91-9741563634
>> https://mtariq.jux.com/
>>
>
>
Re: Metadata size for 1 TB HDFS data?
Posted by Mohammad Tariq <do...@gmail.com>.
Thank you so much for the valuable response Stephen. But I have a few
questions to ask here. Could you please elaborate a bit, if possible?
Each of the specified objects are totally different from each other. A file
will be smaller than a directory in size, and a directory might be smaller
than a block. They might have totally different attributes as well. But
still the space required by each object is same as the other. How is it
possible? Is there any formula or rule of thumb to calculate this?
Many thanks.
Best Regards,
Tariq
+91-9741563634
https://mtariq.jux.com/
On Thu, Dec 20, 2012 at 8:10 PM, Stephen Fritz <st...@cloudera.com>wrote:
> Each block, file, and directory is an object in the namenodes heap, so it
> depends on how you're storing your data. You may need to account for those
> in your calculations.
>
>
>
> On Thu, Dec 20, 2012 at 7:01 AM, Mohammad Tariq <do...@gmail.com>wrote:
>
>> Hello group,
>>
>> What could be the approx. size of the metadata if I have 1 TB of
>> data in my HDFS?I am not doing anything additional but just a simple put.
>> Will it be ((1*1024*1024)/64)*200 Bytes?
>> *Keeping 64M as the block size.
>>
>> Is my understanding right?Please correct me if i'm wrong.
>>
>> Many thanks.
>>
>> Best Regards,
>> Tariq
>> +91-9741563634
>> https://mtariq.jux.com/
>>
>
>
Re: Metadata size for 1 TB HDFS data?
Posted by Mohammad Tariq <do...@gmail.com>.
Thank you so much for the valuable response Stephen. But I have a few
questions to ask here. Could you please elaborate a bit, if possible?
Each of the specified objects are totally different from each other. A file
will be smaller than a directory in size, and a directory might be smaller
than a block. They might have totally different attributes as well. But
still the space required by each object is same as the other. How is it
possible? Is there any formula or rule of thumb to calculate this?
Many thanks.
Best Regards,
Tariq
+91-9741563634
https://mtariq.jux.com/
On Thu, Dec 20, 2012 at 8:10 PM, Stephen Fritz <st...@cloudera.com>wrote:
> Each block, file, and directory is an object in the namenodes heap, so it
> depends on how you're storing your data. You may need to account for those
> in your calculations.
>
>
>
> On Thu, Dec 20, 2012 at 7:01 AM, Mohammad Tariq <do...@gmail.com>wrote:
>
>> Hello group,
>>
>> What could be the approx. size of the metadata if I have 1 TB of
>> data in my HDFS?I am not doing anything additional but just a simple put.
>> Will it be ((1*1024*1024)/64)*200 Bytes?
>> *Keeping 64M as the block size.
>>
>> Is my understanding right?Please correct me if i'm wrong.
>>
>> Many thanks.
>>
>> Best Regards,
>> Tariq
>> +91-9741563634
>> https://mtariq.jux.com/
>>
>
>
Re: Metadata size for 1 TB HDFS data?
Posted by Stephen Fritz <st...@cloudera.com>.
Each block, file, and directory is an object in the namenodes heap, so it
depends on how you're storing your data. You may need to account for those
in your calculations.
On Thu, Dec 20, 2012 at 7:01 AM, Mohammad Tariq <do...@gmail.com> wrote:
> Hello group,
>
> What could be the approx. size of the metadata if I have 1 TB of
> data in my HDFS?I am not doing anything additional but just a simple put.
> Will it be ((1*1024*1024)/64)*200 Bytes?
> *Keeping 64M as the block size.
>
> Is my understanding right?Please correct me if i'm wrong.
>
> Many thanks.
>
> Best Regards,
> Tariq
> +91-9741563634
> https://mtariq.jux.com/
>
Re: Metadata size for 1 TB HDFS data?
Posted by Stephen Fritz <st...@cloudera.com>.
Each block, file, and directory is an object in the namenodes heap, so it
depends on how you're storing your data. You may need to account for those
in your calculations.
On Thu, Dec 20, 2012 at 7:01 AM, Mohammad Tariq <do...@gmail.com> wrote:
> Hello group,
>
> What could be the approx. size of the metadata if I have 1 TB of
> data in my HDFS?I am not doing anything additional but just a simple put.
> Will it be ((1*1024*1024)/64)*200 Bytes?
> *Keeping 64M as the block size.
>
> Is my understanding right?Please correct me if i'm wrong.
>
> Many thanks.
>
> Best Regards,
> Tariq
> +91-9741563634
> https://mtariq.jux.com/
>
Re: Metadata size for 1 TB HDFS data?
Posted by Stephen Fritz <st...@cloudera.com>.
Each block, file, and directory is an object in the namenodes heap, so it
depends on how you're storing your data. You may need to account for those
in your calculations.
On Thu, Dec 20, 2012 at 7:01 AM, Mohammad Tariq <do...@gmail.com> wrote:
> Hello group,
>
> What could be the approx. size of the metadata if I have 1 TB of
> data in my HDFS?I am not doing anything additional but just a simple put.
> Will it be ((1*1024*1024)/64)*200 Bytes?
> *Keeping 64M as the block size.
>
> Is my understanding right?Please correct me if i'm wrong.
>
> Many thanks.
>
> Best Regards,
> Tariq
> +91-9741563634
> https://mtariq.jux.com/
>
Re: Metadata size for 1 TB HDFS data?
Posted by Stephen Fritz <st...@cloudera.com>.
Each block, file, and directory is an object in the namenodes heap, so it
depends on how you're storing your data. You may need to account for those
in your calculations.
On Thu, Dec 20, 2012 at 7:01 AM, Mohammad Tariq <do...@gmail.com> wrote:
> Hello group,
>
> What could be the approx. size of the metadata if I have 1 TB of
> data in my HDFS?I am not doing anything additional but just a simple put.
> Will it be ((1*1024*1024)/64)*200 Bytes?
> *Keeping 64M as the block size.
>
> Is my understanding right?Please correct me if i'm wrong.
>
> Many thanks.
>
> Best Regards,
> Tariq
> +91-9741563634
> https://mtariq.jux.com/
>