You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Mark question <ma...@gmail.com> on 2011/10/05 08:41:41 UTC
hadoop input buffer size
Hello,
Correct me if I'm wrong, but when a program opens n-files at the same time
to read from, and start reading from each file at a time 1 line at a time.
Isn't hadoop actually fetching dfs.block.size of lines into a buffer? and
not actually one line.
If this is correct, I set up my dfs.block.size = 3MB and each line takes
about 650 bytes only, then I would assume the performance for reading 1-4000
lines would be the same, but it isn't ! Do you know a way to find #n of
lines to be read at once?
Thank you,
Mark
Re: hadoop input buffer size
Posted by Mark question <ma...@gmail.com>.
Thanks for the clarifications guys :)
Mark
On Mon, Oct 10, 2011 at 8:27 AM, Uma Maheswara Rao G 72686 <
maheswara@huawei.com> wrote:
> I think below can give you more info about it.
>
> http://developer.yahoo.com/blogs/hadoop/posts/2009/08/the_anatomy_of_hadoop_io_pipel/
> Nice explanation by Owen here.
>
> Regards,
> Uma
>
> ----- Original Message -----
> From: Yang Xiaoliang <ya...@gmail.com>
> Date: Wednesday, October 5, 2011 4:27 pm
> Subject: Re: hadoop input buffer size
> To: common-user@hadoop.apache.org
>
> > Hi,
> >
> > Hadoop neither read one line each time, nor fetching
> > dfs.block.size of lines
> > into a buffer,
> > Actually, for the TextInputFormat, it read io.file.buffer.size
> > bytes of text
> > into a buffer each time,
> > this can be seen from the hadoop source file LineReader.java
> >
> >
> >
> > 2011/10/5 Mark question <ma...@gmail.com>
> >
> > > Hello,
> > >
> > > Correct me if I'm wrong, but when a program opens n-files at
> > the same time
> > > to read from, and start reading from each file at a time 1 line
> > at a time.
> > > Isn't hadoop actually fetching dfs.block.size of lines into a
> > buffer? and
> > > not actually one line.
> > >
> > > If this is correct, I set up my dfs.block.size = 3MB and each
> > line takes
> > > about 650 bytes only, then I would assume the performance for
> > reading> 1-4000
> > > lines would be the same, but it isn't ! Do you know a way to
> > find #n of
> > > lines to be read at once?
> > >
> > > Thank you,
> > > Mark
> > >
> >
>
Re: hadoop input buffer size
Posted by Uma Maheswara Rao G 72686 <ma...@huawei.com>.
I think below can give you more info about it.
http://developer.yahoo.com/blogs/hadoop/posts/2009/08/the_anatomy_of_hadoop_io_pipel/
Nice explanation by Owen here.
Regards,
Uma
----- Original Message -----
From: Yang Xiaoliang <ya...@gmail.com>
Date: Wednesday, October 5, 2011 4:27 pm
Subject: Re: hadoop input buffer size
To: common-user@hadoop.apache.org
> Hi,
>
> Hadoop neither read one line each time, nor fetching
> dfs.block.size of lines
> into a buffer,
> Actually, for the TextInputFormat, it read io.file.buffer.size
> bytes of text
> into a buffer each time,
> this can be seen from the hadoop source file LineReader.java
>
>
>
> 2011/10/5 Mark question <ma...@gmail.com>
>
> > Hello,
> >
> > Correct me if I'm wrong, but when a program opens n-files at
> the same time
> > to read from, and start reading from each file at a time 1 line
> at a time.
> > Isn't hadoop actually fetching dfs.block.size of lines into a
> buffer? and
> > not actually one line.
> >
> > If this is correct, I set up my dfs.block.size = 3MB and each
> line takes
> > about 650 bytes only, then I would assume the performance for
> reading> 1-4000
> > lines would be the same, but it isn't ! Do you know a way to
> find #n of
> > lines to be read at once?
> >
> > Thank you,
> > Mark
> >
>
Re: hadoop input buffer size
Posted by Yang Xiaoliang <ya...@gmail.com>.
Hi,
Hadoop neither read one line each time, nor fetching dfs.block.size of lines
into a buffer,
Actually, for the TextInputFormat, it read io.file.buffer.size bytes of text
into a buffer each time,
this can be seen from the hadoop source file LineReader.java
2011/10/5 Mark question <ma...@gmail.com>
> Hello,
>
> Correct me if I'm wrong, but when a program opens n-files at the same time
> to read from, and start reading from each file at a time 1 line at a time.
> Isn't hadoop actually fetching dfs.block.size of lines into a buffer? and
> not actually one line.
>
> If this is correct, I set up my dfs.block.size = 3MB and each line takes
> about 650 bytes only, then I would assume the performance for reading
> 1-4000
> lines would be the same, but it isn't ! Do you know a way to find #n of
> lines to be read at once?
>
> Thank you,
> Mark
>