You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Cagdas Gerede <ca...@gmail.com> on 2008/05/02 19:25:19 UTC
Master Heap Size and Master Startup Time vs. Number of Blocks
In the system I am working, we have 6 million blocks total and the namenode
heap size is about 600 MB and it takes about 5 minutes for namenode to leave
the safemode.
I try to estimate what would be the heap size if we have 100 - 150 million
blocks, and what would be the amount of time for namenode to leave the
safemode.
>From the extrapolation based on the numbers I have, I am calculating very
scary numbers for both (Terabytes for heap size) and half an hour or so
startup time. I am hoping that my extrapolation is not accurate.
>From your clusters, could you provide some numbers for number of files and
blocks in the system vs. the master heap size and master startup time.
I really appreciate your help.
Thanks.
Cagdas
--
------------
Best Regards, Cagdas Evren Gerede
Home Page: http://cagdasgerede.info
Re: Master Heap Size and Master Startup Time vs. Number of Blocks
Posted by Cagdas Gerede <ca...@gmail.com>.
I see your point.
I have one more question. If you are writing 10 blocks for a file and let's
say in 10th block namenode fails, all previous 9 blocks are lost because you
were not able to close the file and therefore namenode did not persist the
information about 9 blocks to the fsimage file.
How would you solve this problem? Why does the namenode not persist every
block and wait until closing of the file? Then, don't you need to keep a
copy of all the data you are writing to hadoop file until you close the file
successfully. Does it make sense to have this semantics with the assumption
of very large files?
Thanks for your response,
Cagdas
On Fri, May 2, 2008 at 2:29 PM, Ted Dunning <td...@veoh.com> wrote:
>
>
> First of all, your 10MB files will only be 10MB long and will take one
> block
> rather than 5. This is a win already.
>
> Secondly, you can have a consolidation process that merges your small
> files
> every hour or day into large files. By building the consolidated file in
> a
> side directory that is moved into the right place when it is complete, you
> can have a very short window when things don't look quite right. If you
> use
> an external transactional store such as zookeeper to keep the state
> straight, you can guarantee data integrity.
>
> When the archive file capability is available, this will provide you with
> a
> very nice way to do this.
>
> This does require a bit of machinery, but it provides order of magnitude
> or
> better improvement in hadoop performance. These two suggestions together
> can give you >30x boost.
>
>
> On 5/2/08 2:21 PM, "Cagdas Gerede" <ca...@gmail.com> wrote:
>
> >> But you could do all this with larger blocks as well. Having a large
> > block
> >> size only says that a block CAN be that long, not that it MUST be that
> > long.
> >
> > No you cannot.
> >
> > Imagine a streaming server where users send real time generated data to
> your
> > server and each file is not more than 100MB. Let's assume user do not
> have
> > more than 10 MB of local cache space. So user cannot keep more than 10
> MB of
> > data while he is generating the data. So user caches the data, and
> streams
> > it to your server. As one chunk of data accumulates, your server writes
> that
> > chunk to Hadoop, gets confirmation from Hadoop and sends an ack to the
> user
> > so that user can delete data from his cache (because data is persisted).
> > This way you are making the system tolerant to the failure of your
> servers.
> >
> > How would you do the same thing with a block size of 100MB?
> > What am I missing?
> >
> > Cagdas
> >
> > On Fri, May 2, 2008 at 1:20 PM, Ted Dunning <td...@veoh.com> wrote:
> >
> >>
> >>
> >>
> >> Also, you said that the average size was ~ 40MB (20 x 2MB blocks). If
> >> that
> >> is so, then you should be able to radically decrease the number of
> blocks
> >> with a larger block size.
> >>
>
>
--
------------
Best Regards, Cagdas Evren Gerede
Home Page: http://cagdasgerede.info
Re: Master Heap Size and Master Startup Time vs. Number of Blocks
Posted by Ted Dunning <td...@veoh.com>.
First of all, your 10MB files will only be 10MB long and will take one block
rather than 5. This is a win already.
Secondly, you can have a consolidation process that merges your small files
every hour or day into large files. By building the consolidated file in a
side directory that is moved into the right place when it is complete, you
can have a very short window when things don't look quite right. If you use
an external transactional store such as zookeeper to keep the state
straight, you can guarantee data integrity.
When the archive file capability is available, this will provide you with a
very nice way to do this.
This does require a bit of machinery, but it provides order of magnitude or
better improvement in hadoop performance. These two suggestions together
can give you >30x boost.
On 5/2/08 2:21 PM, "Cagdas Gerede" <ca...@gmail.com> wrote:
>> But you could do all this with larger blocks as well. Having a large
> block
>> size only says that a block CAN be that long, not that it MUST be that
> long.
>
> No you cannot.
>
> Imagine a streaming server where users send real time generated data to your
> server and each file is not more than 100MB. Let's assume user do not have
> more than 10 MB of local cache space. So user cannot keep more than 10 MB of
> data while he is generating the data. So user caches the data, and streams
> it to your server. As one chunk of data accumulates, your server writes that
> chunk to Hadoop, gets confirmation from Hadoop and sends an ack to the user
> so that user can delete data from his cache (because data is persisted).
> This way you are making the system tolerant to the failure of your servers.
>
> How would you do the same thing with a block size of 100MB?
> What am I missing?
>
> Cagdas
>
> On Fri, May 2, 2008 at 1:20 PM, Ted Dunning <td...@veoh.com> wrote:
>
>>
>>
>>
>> Also, you said that the average size was ~ 40MB (20 x 2MB blocks). If
>> that
>> is so, then you should be able to radically decrease the number of blocks
>> with a larger block size.
>>
Re: Master Heap Size and Master Startup Time vs. Number of Blocks
Posted by Cagdas Gerede <ca...@gmail.com>.
> But you could do all this with larger blocks as well. Having a large
block
> size only says that a block CAN be that long, not that it MUST be that
long.
No you cannot.
Imagine a streaming server where users send real time generated data to your
server and each file is not more than 100MB. Let's assume user do not have
more than 10 MB of local cache space. So user cannot keep more than 10 MB of
data while he is generating the data. So user caches the data, and streams
it to your server. As one chunk of data accumulates, your server writes that
chunk to Hadoop, gets confirmation from Hadoop and sends an ack to the user
so that user can delete data from his cache (because data is persisted).
This way you are making the system tolerant to the failure of your servers.
How would you do the same thing with a block size of 100MB?
What am I missing?
Cagdas
On Fri, May 2, 2008 at 1:20 PM, Ted Dunning <td...@veoh.com> wrote:
>
>
>
> Also, you said that the average size was ~ 40MB (20 x 2MB blocks). If
> that
> is so, then you should be able to radically decrease the number of blocks
> with a larger block size.
>
Re: Master Heap Size and Master Startup Time vs. Number of Blocks
Posted by Ted Dunning <td...@veoh.com>.
But you could do all this with larger blocks as well. Having a large block
size only says that a block CAN be that long, not that it MUST be that long.
Also, you said that the average size was ~ 40MB (20 x 2MB blocks). If that
is so, then you should be able to radically decrease the number of blocks
with a larger block size.
On 5/2/08 1:17 PM, "Cagdas Gerede" <ca...@gmail.com> wrote:
> fault tolerance. As files are uploaded into our server, we can continuously
> write the data in small chunks and if our server fails, we can tolerate this
> failure by switching our user to another server and the user can continue to
> write. Otherwise we have to wait on the server until we get the whole file
> to write it to Hadoop (if server fails then we lose all the data), or we
> need the user to cash all the data he is generating which is not feasible
> for our requirements.
>
> I appreciate your comment on this.
>
> Cagdas
>
> On Fri, May 2, 2008 at 1:09 PM, Ted Dunning <td...@veoh.com> wrote:
>
>>
>> Why did you pick such a small block size?
>>
>> Why not go with the default of 64MB?
>>
>> That would give you only 10 million blocks for your 600TB.
>>
>> I don't see any advantage to the tiny block size.
>>
>> On 5/2/08 1:06 PM, "Cagdas Gerede" <ca...@gmail.com> wrote:
>>
>>> Thanks Doug for your answers. Our interest is on more distributed file
>>> system part rather than map reduce.
>>> I must confess that our block size is not as large as how a lot of
>> people
>>> configure. I appreciate if I can get your and others' input.
>>>
>>> Do you think these numbers are suitable?
>>>
>>> We will have 5 million files each having 20 blocks of 2MB. With the
>> minimum
>>> replication of 3, we would have 300 million blocks.
>>> 300 million blocks would store 600TB. At ~10TB/node, this means a 60
>> node
>>> system.
>>>
>>> Do you think these numbers are suitable for Hadoop DFS.
>>>
>>> Cagdas
>>>
>>>
>>>
>>> At ~100M per block, 100M blocks would store 10PB. At ~1TB/node, this
>> means
>>>> a ~10,000 node system, larger than Hadoop currently supports well (for
>> this
>>>> and other reasons).
>>>>
>>>> Doug
>>>>
>>>>
>>>
>>
>>
>
Re: Master Heap Size and Master Startup Time vs. Number of Blocks
Posted by Cagdas Gerede <ca...@gmail.com>.
fault tolerance. As files are uploaded into our server, we can continuously
write the data in small chunks and if our server fails, we can tolerate this
failure by switching our user to another server and the user can continue to
write. Otherwise we have to wait on the server until we get the whole file
to write it to Hadoop (if server fails then we lose all the data), or we
need the user to cash all the data he is generating which is not feasible
for our requirements.
I appreciate your comment on this.
Cagdas
On Fri, May 2, 2008 at 1:09 PM, Ted Dunning <td...@veoh.com> wrote:
>
> Why did you pick such a small block size?
>
> Why not go with the default of 64MB?
>
> That would give you only 10 million blocks for your 600TB.
>
> I don't see any advantage to the tiny block size.
>
> On 5/2/08 1:06 PM, "Cagdas Gerede" <ca...@gmail.com> wrote:
>
> > Thanks Doug for your answers. Our interest is on more distributed file
> > system part rather than map reduce.
> > I must confess that our block size is not as large as how a lot of
> people
> > configure. I appreciate if I can get your and others' input.
> >
> > Do you think these numbers are suitable?
> >
> > We will have 5 million files each having 20 blocks of 2MB. With the
> minimum
> > replication of 3, we would have 300 million blocks.
> > 300 million blocks would store 600TB. At ~10TB/node, this means a 60
> node
> > system.
> >
> > Do you think these numbers are suitable for Hadoop DFS.
> >
> > Cagdas
> >
> >
> >
> > At ~100M per block, 100M blocks would store 10PB. At ~1TB/node, this
> means
> >> a ~10,000 node system, larger than Hadoop currently supports well (for
> this
> >> and other reasons).
> >>
> >> Doug
> >>
> >>
> >
>
>
--
------------
Best Regards, Cagdas Evren Gerede
Home Page: http://cagdasgerede.info
Re: Master Heap Size and Master Startup Time vs. Number of Blocks
Posted by Doug Cutting <cu...@apache.org>.
Cagdas Gerede wrote:
> We will have 5 million files each having 20 blocks of 2MB. With the minimum
> replication of 3, we would have 300 million blocks.
> 300 million blocks would store 600TB. At ~10TB/node, this means a 60 node
> system.
>
> Do you think these numbers are suitable for Hadoop DFS.
Why are you using such small blocks? A larger block size will decrease
the strain on Hadoop, but perhaps you have reasons?
Doug
Re: Master Heap Size and Master Startup Time vs. Number of Blocks
Posted by Ted Dunning <td...@veoh.com>.
Why did you pick such a small block size?
Why not go with the default of 64MB?
That would give you only 10 million blocks for your 600TB.
I don't see any advantage to the tiny block size.
On 5/2/08 1:06 PM, "Cagdas Gerede" <ca...@gmail.com> wrote:
> Thanks Doug for your answers. Our interest is on more distributed file
> system part rather than map reduce.
> I must confess that our block size is not as large as how a lot of people
> configure. I appreciate if I can get your and others' input.
>
> Do you think these numbers are suitable?
>
> We will have 5 million files each having 20 blocks of 2MB. With the minimum
> replication of 3, we would have 300 million blocks.
> 300 million blocks would store 600TB. At ~10TB/node, this means a 60 node
> system.
>
> Do you think these numbers are suitable for Hadoop DFS.
>
> Cagdas
>
>
>
> At ~100M per block, 100M blocks would store 10PB. At ~1TB/node, this means
>> a ~10,000 node system, larger than Hadoop currently supports well (for this
>> and other reasons).
>>
>> Doug
>>
>>
>
Re: Master Heap Size and Master Startup Time vs. Number of Blocks
Posted by Cagdas Gerede <ca...@gmail.com>.
Thanks Doug for your answers. Our interest is on more distributed file
system part rather than map reduce.
I must confess that our block size is not as large as how a lot of people
configure. I appreciate if I can get your and others' input.
Do you think these numbers are suitable?
We will have 5 million files each having 20 blocks of 2MB. With the minimum
replication of 3, we would have 300 million blocks.
300 million blocks would store 600TB. At ~10TB/node, this means a 60 node
system.
Do you think these numbers are suitable for Hadoop DFS.
Cagdas
At ~100M per block, 100M blocks would store 10PB. At ~1TB/node, this means
> a ~10,000 node system, larger than Hadoop currently supports well (for this
> and other reasons).
>
> Doug
>
>
--
------------
Best Regards, Cagdas Evren Gerede
Home Page: http://cagdasgerede.info
Re: Master Heap Size and Master Startup Time vs. Number of Blocks
Posted by Doug Cutting <cu...@apache.org>.
Cagdas Gerede wrote:
> In the system I am working, we have 6 million blocks total and the namenode
> heap size is about 600 MB and it takes about 5 minutes for namenode to leave
> the safemode.
How big is are your files? Are they several blocks on average? Hadoop
is not designed for small files, but rather for larger files. An
Archive system is currently being designed to help with this.
https://issues.apache.org/jira/browse/HADOOP-3307
> I try to estimate what would be the heap size if we have 100 - 150 million
> blocks, and what would be the amount of time for namenode to leave the
> safemode.
At ~100M per block, 100M blocks would store 10PB. At ~1TB/node, this
means a ~10,000 node system, larger than Hadoop currently supports well
(for this and other reasons).
If your files are generally large, you can increase your block size to
250MB to decrease the number of blocks in the system.
Doug