Disk Usage Overhead of Hadoop Upgrade

[Stu Hood <st...@rackspace.com>]

Hey gang,

We're preparing to upgrade our cluster from Hadoop 0.15.3 to 0.18.3.

How much disk usage overhead can we expect from the block conversion before we finalize the upgrade? In the worst case, will the upgrade cause our disk usage to double?

Thanks,

Stu Hood
Search Team Technical Lead
Email & Apps Division, Rackspace Hosting

Re: Disk Usage Overhead of Hadoop Upgrade

[Pankil Doshi <fo...@gmail.com>]

hi Stu,

which block conversion are you talking about? If you are talking abt block
size of data then it remains same in upgrade unless and until you change it.

Pankil

On Tue, Jun 16, 2009 at 5:16 PM, Stu Hood <st...@rackspace.com> wrote:

> Hey gang,
>
> We're preparing to upgrade our cluster from Hadoop 0.15.3 to 0.18.3.
>
> How much disk usage overhead can we expect from the block conversion before
> we finalize the upgrade? In the worst case, will the upgrade cause our disk
> usage to double?
>
> Thanks,
>
> Stu Hood
> Search Team Technical Lead
> Email & Apps Division, Rackspace Hosting
>
>

Re: Disk Usage Overhead of Hadoop Upgrade

[Raghu Angadi <ra...@yahoo-inc.com>]

The initial overhead is fairly small (extra hard link for each file).

After that, the overhead grows as you delete the files (thus its blocks) 
that existed before the upgrade.. since the physical files for blocks 
are deleted only after you finalize.

So the overhead == (the blocks that got deleted after the upgrade).

Raghu.

Stu Hood wrote:
> Hey gang,
> 
> We're preparing to upgrade our cluster from Hadoop 0.15.3 to 0.18.3.
> 
> How much disk usage overhead can we expect from the block conversion before we finalize the upgrade? In the worst case, will the upgrade cause our disk usage to double?
> 
> Thanks,
> 
> Stu Hood
> Search Team Technical Lead
> Email & Apps Division, Rackspace Hosting
>