You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by elton sky <el...@gmail.com> on 2010/10/18 12:33:34 UTC
BUG: Anyone use block size more than 2GB before?
Hello,
In
hdfs.org.apache.hadoop.hdfs.DFSClient<eclipse-javadoc:%E2%98%82=HadoopSrcCode/src%3Chdfs.org.apache.hadoop.hdfs%7BDFSClient.java%E2%98%83DFSClient>
.DFSOutputStream<eclipse-javadoc:%E2%98%82=HadoopSrcCode/src%3Chdfs.org.apache.hadoop.hdfs%7BDFSClient.java%E2%98%83DFSClient%E2%98%83DFSOutputStream>.writeChunk(byte[]
b, int offset, int len, byte[] checksum)
The second last line:
int psize = Math.min((int)(blockSize-bytesCurBlock), writePacketSize);
When I use blockSize bigger than 2GB, which is out of the boundary of
integer something weird would happen. For example, for a 3GB block it will
create more than 2Million packets.
Anyone noticed this before?
Elton
Re: BUG: Anyone use block size more than 2GB before?
Posted by Milind A Bhandarkar <mi...@yahoo-inc.com>.
That's correct. That is why teragen, the program that generates data to be sorted in terasort is a MR program :-)
- Milind
On Oct 21, 2010, at 9:47 PM, elton sky wrote:
> Milind,
>
> You are right. But that only happens when your client is one of the data
> nodes in HDFS. otherwise a random node will be picked up for the first
> replica.
>
> On Fri, Oct 22, 2010 at 3:37 PM, Milind A Bhandarkar
> <mi...@yahoo-inc.com>wrote:
>
>> If a file of say, 12.5 GB were produced by a single task with replication
>> 3, the default replication policy will ensure that the first replica of each
>> block will be created on local datanode. So, there will be one datanode in
>> the cluster that contains one replica of all blocks of that file. Map
>> placement hint specifies that node.
>>
>> It's evil, I know :-)
>>
>> - Milind
>>
>> On Oct 21, 2010, at 1:30 PM, Alex Kozlov wrote:
>>
>>> Hmm, this is interesting: how did it manage to keep the blocks local?
>> Why
>>> performance was better?
>>>
>>> On Thu, Oct 21, 2010 at 11:43 AM, Owen O'Malley <om...@apache.org>
>> wrote:
>>>
>>>> The block sizes were 2G. The input format made splits that were more
>> than a
>>>> block because that led to better performance.
>>>>
>>>> -- Owen
>>>>
>>
>> --
>> Milind Bhandarkar
>> (mailto:milindb@yahoo-inc.com)
>> (phone: 408-203-5213 W)
>>
>>
>>
--
Milind Bhandarkar
(mailto:milindb@yahoo-inc.com)
(phone: 408-203-5213 W)
Re: BUG: Anyone use block size more than 2GB before?
Posted by elton sky <el...@gmail.com>.
Milind,
You are right. But that only happens when your client is one of the data
nodes in HDFS. otherwise a random node will be picked up for the first
replica.
On Fri, Oct 22, 2010 at 3:37 PM, Milind A Bhandarkar
<mi...@yahoo-inc.com>wrote:
> If a file of say, 12.5 GB were produced by a single task with replication
> 3, the default replication policy will ensure that the first replica of each
> block will be created on local datanode. So, there will be one datanode in
> the cluster that contains one replica of all blocks of that file. Map
> placement hint specifies that node.
>
> It's evil, I know :-)
>
> - Milind
>
> On Oct 21, 2010, at 1:30 PM, Alex Kozlov wrote:
>
> > Hmm, this is interesting: how did it manage to keep the blocks local?
> Why
> > performance was better?
> >
> > On Thu, Oct 21, 2010 at 11:43 AM, Owen O'Malley <om...@apache.org>
> wrote:
> >
> >> The block sizes were 2G. The input format made splits that were more
> than a
> >> block because that led to better performance.
> >>
> >> -- Owen
> >>
>
> --
> Milind Bhandarkar
> (mailto:milindb@yahoo-inc.com)
> (phone: 408-203-5213 W)
>
>
>
Re: BUG: Anyone use block size more than 2GB before?
Posted by Milind A Bhandarkar <mi...@yahoo-inc.com>.
If a file of say, 12.5 GB were produced by a single task with replication 3, the default replication policy will ensure that the first replica of each block will be created on local datanode. So, there will be one datanode in the cluster that contains one replica of all blocks of that file. Map placement hint specifies that node.
It's evil, I know :-)
- Milind
On Oct 21, 2010, at 1:30 PM, Alex Kozlov wrote:
> Hmm, this is interesting: how did it manage to keep the blocks local? Why
> performance was better?
>
> On Thu, Oct 21, 2010 at 11:43 AM, Owen O'Malley <om...@apache.org> wrote:
>
>> The block sizes were 2G. The input format made splits that were more than a
>> block because that led to better performance.
>>
>> -- Owen
>>
--
Milind Bhandarkar
(mailto:milindb@yahoo-inc.com)
(phone: 408-203-5213 W)
Re: BUG: Anyone use block size more than 2GB before?
Posted by Alex Kozlov <al...@cloudera.com>.
Hmm, this is interesting: how did it manage to keep the blocks local? Why
performance was better?
On Thu, Oct 21, 2010 at 11:43 AM, Owen O'Malley <om...@apache.org> wrote:
> The block sizes were 2G. The input format made splits that were more than a
> block because that led to better performance.
>
> -- Owen
>
Re: BUG: Anyone use block size more than 2GB before?
Posted by Owen O'Malley <om...@apache.org>.
The block sizes were 2G. The input format made splits that were more than a
block because that led to better performance.
-- Owen
Re: BUG: Anyone use block size more than 2GB before?
Posted by "M. C. Srivas" <mc...@gmail.com>.
I thought the petasort benchmark you published used 12.5G block sizes. How
did you make that work?
On Mon, Oct 18, 2010 at 4:27 PM, Owen O'Malley <om...@apache.org> wrote:
> Block sizes larger than 2**31 are known to not work. I haven't ever tracked
> down the problem, just set my block size to be smaller than that.
>
> -- Owen
>
Re: BUG: Anyone use block size more than 2GB before?
Posted by elton sky <el...@gmail.com>.
I am curious, any specific reason to make it smaller than 2**31?
On Tue, Oct 19, 2010 at 10:27 AM, Owen O'Malley <om...@apache.org> wrote:
> Block sizes larger than 2**31 are known to not work. I haven't ever tracked
> down the problem, just set my block size to be smaller than that.
>
> -- Owen
>
Re: BUG: Anyone use block size more than 2GB before?
Posted by Owen O'Malley <om...@apache.org>.
Block sizes larger than 2**31 are known to not work. I haven't ever
tracked down the problem, just set my block size to be smaller than
that.
-- Owen
Re: BUG: Anyone use block size more than 2GB before?
Posted by Allen Wittenauer <aw...@linkedin.com>.
On Oct 18, 2010, at 3:33 AM, elton sky wrote:
>
>
> When I use blockSize bigger than 2GB, which is out of the boundary of
> integer something weird would happen. For example, for a 3GB block it will
> create more than 2Million packets.
>
> Anyone noticed this before?
https://issues.apache.org/jira/browse/HDFS-96
Re: BUG: Anyone use block size more than 2GB before?
Posted by Steve Loughran <st...@apache.org>.
On 18/10/10 23:07, Michael Segel wrote:
>
> Ok, I'll bite.
> Why would you want to use a block size of> 2GB?
>
1. Some of the events coming off large physics devices are single
self-contained files of 3+ GB size; having a block size which has an
event in a single block guarantees locality for the entirety of the
operation.
2. Some very fast clusters are finding problems with smaller block sizes
(256MB) causing TT's to finish too fast and so overload the JT. It's a
combination of TT-JVM-Reuse, cluster size and improved disk and CPU
performance.
Re: BUG: Anyone use block size more than 2GB before?
Posted by Allen Wittenauer <aw...@linkedin.com>.
On Oct 18, 2010, at 4:08 PM, elton sky wrote:
>> Why would you want to use a block size of > 2GB?
> For keeping a maps input split in a single block~
Just use mapred.min.split.size + multifileinputformat.
Re: BUG: Anyone use block size more than 2GB before?
Posted by James Seigel <ja...@tynt.com>.
If there is a hard requirement for input split being one block you could just make your input split fit a smaller block size.
Just saying, in case you can't overcome the 2G ceiling
J
Sent from my mobile. Please excuse the typos.
On 2010-10-18, at 5:08 PM, "elton sky" <el...@gmail.com> wrote:
>> Why would you want to use a block size of > 2GB?
> For keeping a maps input split in a single block~
>
> On Tue, Oct 19, 2010 at 9:07 AM, Michael Segel <mi...@hotmail.com>wrote:
>
>>
>> Ok, I'll bite.
>> Why would you want to use a block size of > 2GB?
>>
>>
>>
>>> Date: Mon, 18 Oct 2010 21:33:34 +1100
>>> Subject: BUG: Anyone use block size more than 2GB before?
>>> From: eltonsky9404@gmail.com
>>> To: common-user@hadoop.apache.org
>>>
>>> Hello,
>>>
>>> In
>>>
>> hdfs.org.apache.hadoop.hdfs.DFSClient<eclipse-javadoc:%E2%98%82=HadoopSrcCode/src%3Chdfs.org.apache.hadoop.hdfs%7BDFSClient.java%E2%98%83DFSClient>
>>>
>> .DFSOutputStream<eclipse-javadoc:%E2%98%82=HadoopSrcCode/src%3Chdfs.org.apache.hadoop.hdfs%7BDFSClient.java%E2%98%83DFSClient%E2%98%83DFSOutputStream>.writeChunk(byte[]
>>> b, int offset, int len, byte[] checksum)
>>> The second last line:
>>>
>>> int psize = Math.min((int)(blockSize-bytesCurBlock), writePacketSize);
>>>
>>> When I use blockSize bigger than 2GB, which is out of the boundary of
>>> integer something weird would happen. For example, for a 3GB block it
>> will
>>> create more than 2Million packets.
>>>
>>> Anyone noticed this before?
>>>
>>> Elton
>>
>>
Re: BUG: Anyone use block size more than 2GB before?
Posted by elton sky <el...@gmail.com>.
>Why would you want to use a block size of > 2GB?
For keeping a maps input split in a single block~
On Tue, Oct 19, 2010 at 9:07 AM, Michael Segel <mi...@hotmail.com>wrote:
>
> Ok, I'll bite.
> Why would you want to use a block size of > 2GB?
>
>
>
> > Date: Mon, 18 Oct 2010 21:33:34 +1100
> > Subject: BUG: Anyone use block size more than 2GB before?
> > From: eltonsky9404@gmail.com
> > To: common-user@hadoop.apache.org
> >
> > Hello,
> >
> > In
> >
> hdfs.org.apache.hadoop.hdfs.DFSClient<eclipse-javadoc:%E2%98%82=HadoopSrcCode/src%3Chdfs.org.apache.hadoop.hdfs%7BDFSClient.java%E2%98%83DFSClient>
> >
> .DFSOutputStream<eclipse-javadoc:%E2%98%82=HadoopSrcCode/src%3Chdfs.org.apache.hadoop.hdfs%7BDFSClient.java%E2%98%83DFSClient%E2%98%83DFSOutputStream>.writeChunk(byte[]
> > b, int offset, int len, byte[] checksum)
> > The second last line:
> >
> > int psize = Math.min((int)(blockSize-bytesCurBlock), writePacketSize);
> >
> > When I use blockSize bigger than 2GB, which is out of the boundary of
> > integer something weird would happen. For example, for a 3GB block it
> will
> > create more than 2Million packets.
> >
> > Anyone noticed this before?
> >
> > Elton
>
>
RE: BUG: Anyone use block size more than 2GB before?
Posted by Michael Segel <mi...@hotmail.com>.
Ok, I'll bite.
Why would you want to use a block size of > 2GB?
> Date: Mon, 18 Oct 2010 21:33:34 +1100
> Subject: BUG: Anyone use block size more than 2GB before?
> From: eltonsky9404@gmail.com
> To: common-user@hadoop.apache.org
>
> Hello,
>
> In
> hdfs.org.apache.hadoop.hdfs.DFSClient<eclipse-javadoc:%E2%98%82=HadoopSrcCode/src%3Chdfs.org.apache.hadoop.hdfs%7BDFSClient.java%E2%98%83DFSClient>
> .DFSOutputStream<eclipse-javadoc:%E2%98%82=HadoopSrcCode/src%3Chdfs.org.apache.hadoop.hdfs%7BDFSClient.java%E2%98%83DFSClient%E2%98%83DFSOutputStream>.writeChunk(byte[]
> b, int offset, int len, byte[] checksum)
> The second last line:
>
> int psize = Math.min((int)(blockSize-bytesCurBlock), writePacketSize);
>
> When I use blockSize bigger than 2GB, which is out of the boundary of
> integer something weird would happen. For example, for a 3GB block it will
> create more than 2Million packets.
>
> Anyone noticed this before?
>
> Elton