what to do with the remaind space in the last block

[cheng xu <xc...@gmail.com>]

 Hi
all.

as we know. the last block of file may be less then 64MB(as default
configged), then what will hdfs do with the remained space in the block??
left for other files or just left unused??

any answer or related url is appreciated!
thanks
xu

Re: what to do with the remaind space in the last block

[cheng xu <xc...@gmail.com>]

Hi

thanks for your reply!  it really help alot!  thanks again .

2011/5/6 Matthew Foley <ma...@yahoo-inc.com>

> Xu, the datanode only stores the amount of data provided.  If the last
> block only has, e.g., 10 bytes in it, then the datanode will use one
> low-level block (the minimum space allocation from the underlying file
> system) to store those 10 bytes.  Typically, low-level blocks are 8KB, but
> that depends on the server's OS and FS configuration -- it's not visible to
> Hadoop.  It definitely won't use up 64MB of datanode storage.
>
> --Matt
>
>
> On May 5, 2011, at 12:49 AM, cheng xu wrote:
>
> Hi
> all.
>
> as we know. the last block of file may be less then 64MB(as default
> configged), then what will hdfs do with the remained space in the block??
> left for other files or just left unused??
>
> any answer or related url is appreciated!
> thanks
> xu
>
>

Re: what to do with the remaind space in the last block

[Matthew Foley <ma...@yahoo-inc.com>]

Xu, the datanode only stores the amount of data provided.  If the last block only has, e.g., 10 bytes in it, then the datanode will use one low-level block (the minimum space allocation from the underlying file system) to store those 10 bytes.  Typically, low-level blocks are 8KB, but that depends on the server's OS and FS configuration -- it's not visible to Hadoop.  It definitely won't use up 64MB of datanode storage.

--Matt


On May 5, 2011, at 12:49 AM, cheng xu wrote:

Hi
all.

as we know. the last block of file may be less then 64MB(as default
configged), then what will hdfs do with the remained space in the block??
left for other files or just left unused??

any answer or related url is appreciated!
thanks
xu

Re: what to do with the remaind space in the last block

[Harsh J <ha...@cloudera.com>]

Hello xu,

On Thu, May 5, 2011 at 1:19 PM, cheng xu <xc...@gmail.com> wrote:
> as we know. the last block of file may be less then 64MB(as default
> configged), then what will hdfs do with the remained space in the block??
> left for other files or just left unused??

Since no form of pre-allocation is performed while 'creating' blocks,
it only consumes space as is required of the data being streamed. A 64
MB block size is a logical term in HDFS to signify the 'split size' of
incoming data.

-- 
Harsh J