You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by kaveh minooie <ka...@plutoz.com> on 2013/08/26 21:26:47 UTC

1.2.1 ouput, what is bufvoid

I am getting these in job output in the 1.2.1

Spilling map output: record full = true
2013-08-26 11:40:33,413 INFO org.apache.hadoop.mapred.MapTask: bufstart 
= 111149676; bufend = 122264640; bufvoid = 318767104
2013-08-26 11:40:33,413 INFO org.apache.hadoop.mapred.MapTask: kvstart = 
1048561; kvend = 838844; length = 1048576
2013-08-26 11:40:35,879 INFO org.apache.hadoop.mapred.MapTask: Finished 
spill 10


my question is what is bufvoid? and how can I increase the amount of the 
length?

thanks,
-- 
Kaveh Minooie

Re: 1.2.1 ouput, what is bufvoid

Posted by kaveh minooie <ka...@plutoz.com>.
:) thanks for the clarification.

On 08/26/2013 03:51 PM, Harsh J wrote:
> The length printed there is the # of elements in an internal array,
> not a data size length. If you wish to raise that you can raise up
> io.sort.spill.percent (default 0.80, or 80%).
>
> On Tue, Aug 27, 2013 at 4:03 AM, kaveh minooie <ka...@plutoz.com> wrote:
>> Thaks Harsh, but my io.sort.mb is 320 while, as you can see, length is
>> only 1mb (1048576)
>>
>>
>> On 08/26/2013 03:14 PM, Harsh J wrote:
>>>
>>> You need to raise io.sort.mb to increase the sort buffer capacity.
>>>
>>> On Tue, Aug 27, 2013 at 12:56 AM, kaveh minooie <ka...@plutoz.com> wrote:
>>>>
>>>> I am getting these in job output in the 1.2.1
>>>>
>>>> Spilling map output: record full = true
>>>> 2013-08-26 11:40:33,413 INFO org.apache.hadoop.mapred.MapTask: bufstart =
>>>> 111149676; bufend = 122264640; bufvoid = 318767104
>>>> 2013-08-26 11:40:33,413 INFO org.apache.hadoop.mapred.MapTask: kvstart =
>>>> 1048561; kvend = 838844; length = 1048576
>>>> 2013-08-26 11:40:35,879 INFO org.apache.hadoop.mapred.MapTask: Finished
>>>> spill 10
>>>>
>>>>
>>>> my question is what is bufvoid? and how can I increase the amount of the
>>>> length?
>>>>
>>>> thanks,
>>>> --
>>>> Kaveh Minooie
>>>
>>>
>>>
>>>
>>
>> --
>> Kaveh Minooie
>
>
>

-- 
Kaveh Minooie

Re: 1.2.1 ouput, what is bufvoid

Posted by kaveh minooie <ka...@plutoz.com>.
:) thanks for the clarification.

On 08/26/2013 03:51 PM, Harsh J wrote:
> The length printed there is the # of elements in an internal array,
> not a data size length. If you wish to raise that you can raise up
> io.sort.spill.percent (default 0.80, or 80%).
>
> On Tue, Aug 27, 2013 at 4:03 AM, kaveh minooie <ka...@plutoz.com> wrote:
>> Thaks Harsh, but my io.sort.mb is 320 while, as you can see, length is
>> only 1mb (1048576)
>>
>>
>> On 08/26/2013 03:14 PM, Harsh J wrote:
>>>
>>> You need to raise io.sort.mb to increase the sort buffer capacity.
>>>
>>> On Tue, Aug 27, 2013 at 12:56 AM, kaveh minooie <ka...@plutoz.com> wrote:
>>>>
>>>> I am getting these in job output in the 1.2.1
>>>>
>>>> Spilling map output: record full = true
>>>> 2013-08-26 11:40:33,413 INFO org.apache.hadoop.mapred.MapTask: bufstart =
>>>> 111149676; bufend = 122264640; bufvoid = 318767104
>>>> 2013-08-26 11:40:33,413 INFO org.apache.hadoop.mapred.MapTask: kvstart =
>>>> 1048561; kvend = 838844; length = 1048576
>>>> 2013-08-26 11:40:35,879 INFO org.apache.hadoop.mapred.MapTask: Finished
>>>> spill 10
>>>>
>>>>
>>>> my question is what is bufvoid? and how can I increase the amount of the
>>>> length?
>>>>
>>>> thanks,
>>>> --
>>>> Kaveh Minooie
>>>
>>>
>>>
>>>
>>
>> --
>> Kaveh Minooie
>
>
>

-- 
Kaveh Minooie

Re: 1.2.1 ouput, what is bufvoid

Posted by kaveh minooie <ka...@plutoz.com>.
:) thanks for the clarification.

On 08/26/2013 03:51 PM, Harsh J wrote:
> The length printed there is the # of elements in an internal array,
> not a data size length. If you wish to raise that you can raise up
> io.sort.spill.percent (default 0.80, or 80%).
>
> On Tue, Aug 27, 2013 at 4:03 AM, kaveh minooie <ka...@plutoz.com> wrote:
>> Thaks Harsh, but my io.sort.mb is 320 while, as you can see, length is
>> only 1mb (1048576)
>>
>>
>> On 08/26/2013 03:14 PM, Harsh J wrote:
>>>
>>> You need to raise io.sort.mb to increase the sort buffer capacity.
>>>
>>> On Tue, Aug 27, 2013 at 12:56 AM, kaveh minooie <ka...@plutoz.com> wrote:
>>>>
>>>> I am getting these in job output in the 1.2.1
>>>>
>>>> Spilling map output: record full = true
>>>> 2013-08-26 11:40:33,413 INFO org.apache.hadoop.mapred.MapTask: bufstart =
>>>> 111149676; bufend = 122264640; bufvoid = 318767104
>>>> 2013-08-26 11:40:33,413 INFO org.apache.hadoop.mapred.MapTask: kvstart =
>>>> 1048561; kvend = 838844; length = 1048576
>>>> 2013-08-26 11:40:35,879 INFO org.apache.hadoop.mapred.MapTask: Finished
>>>> spill 10
>>>>
>>>>
>>>> my question is what is bufvoid? and how can I increase the amount of the
>>>> length?
>>>>
>>>> thanks,
>>>> --
>>>> Kaveh Minooie
>>>
>>>
>>>
>>>
>>
>> --
>> Kaveh Minooie
>
>
>

-- 
Kaveh Minooie

Re: 1.2.1 ouput, what is bufvoid

Posted by kaveh minooie <ka...@plutoz.com>.
:) thanks for the clarification.

On 08/26/2013 03:51 PM, Harsh J wrote:
> The length printed there is the # of elements in an internal array,
> not a data size length. If you wish to raise that you can raise up
> io.sort.spill.percent (default 0.80, or 80%).
>
> On Tue, Aug 27, 2013 at 4:03 AM, kaveh minooie <ka...@plutoz.com> wrote:
>> Thaks Harsh, but my io.sort.mb is 320 while, as you can see, length is
>> only 1mb (1048576)
>>
>>
>> On 08/26/2013 03:14 PM, Harsh J wrote:
>>>
>>> You need to raise io.sort.mb to increase the sort buffer capacity.
>>>
>>> On Tue, Aug 27, 2013 at 12:56 AM, kaveh minooie <ka...@plutoz.com> wrote:
>>>>
>>>> I am getting these in job output in the 1.2.1
>>>>
>>>> Spilling map output: record full = true
>>>> 2013-08-26 11:40:33,413 INFO org.apache.hadoop.mapred.MapTask: bufstart =
>>>> 111149676; bufend = 122264640; bufvoid = 318767104
>>>> 2013-08-26 11:40:33,413 INFO org.apache.hadoop.mapred.MapTask: kvstart =
>>>> 1048561; kvend = 838844; length = 1048576
>>>> 2013-08-26 11:40:35,879 INFO org.apache.hadoop.mapred.MapTask: Finished
>>>> spill 10
>>>>
>>>>
>>>> my question is what is bufvoid? and how can I increase the amount of the
>>>> length?
>>>>
>>>> thanks,
>>>> --
>>>> Kaveh Minooie
>>>
>>>
>>>
>>>
>>
>> --
>> Kaveh Minooie
>
>
>

-- 
Kaveh Minooie

Re: 1.2.1 ouput, what is bufvoid

Posted by Harsh J <ha...@cloudera.com>.
The length printed there is the # of elements in an internal array,
not a data size length. If you wish to raise that you can raise up
io.sort.spill.percent (default 0.80, or 80%).

On Tue, Aug 27, 2013 at 4:03 AM, kaveh minooie <ka...@plutoz.com> wrote:
> Thaks Harsh, but my io.sort.mb is 320 while, as you can see, length is
> only 1mb (1048576)
>
>
> On 08/26/2013 03:14 PM, Harsh J wrote:
>>
>> You need to raise io.sort.mb to increase the sort buffer capacity.
>>
>> On Tue, Aug 27, 2013 at 12:56 AM, kaveh minooie <ka...@plutoz.com> wrote:
>>>
>>> I am getting these in job output in the 1.2.1
>>>
>>> Spilling map output: record full = true
>>> 2013-08-26 11:40:33,413 INFO org.apache.hadoop.mapred.MapTask: bufstart =
>>> 111149676; bufend = 122264640; bufvoid = 318767104
>>> 2013-08-26 11:40:33,413 INFO org.apache.hadoop.mapred.MapTask: kvstart =
>>> 1048561; kvend = 838844; length = 1048576
>>> 2013-08-26 11:40:35,879 INFO org.apache.hadoop.mapred.MapTask: Finished
>>> spill 10
>>>
>>>
>>> my question is what is bufvoid? and how can I increase the amount of the
>>> length?
>>>
>>> thanks,
>>> --
>>> Kaveh Minooie
>>
>>
>>
>>
>
> --
> Kaveh Minooie



-- 
Harsh J

Re: 1.2.1 ouput, what is bufvoid

Posted by Harsh J <ha...@cloudera.com>.
The length printed there is the # of elements in an internal array,
not a data size length. If you wish to raise that you can raise up
io.sort.spill.percent (default 0.80, or 80%).

On Tue, Aug 27, 2013 at 4:03 AM, kaveh minooie <ka...@plutoz.com> wrote:
> Thaks Harsh, but my io.sort.mb is 320 while, as you can see, length is
> only 1mb (1048576)
>
>
> On 08/26/2013 03:14 PM, Harsh J wrote:
>>
>> You need to raise io.sort.mb to increase the sort buffer capacity.
>>
>> On Tue, Aug 27, 2013 at 12:56 AM, kaveh minooie <ka...@plutoz.com> wrote:
>>>
>>> I am getting these in job output in the 1.2.1
>>>
>>> Spilling map output: record full = true
>>> 2013-08-26 11:40:33,413 INFO org.apache.hadoop.mapred.MapTask: bufstart =
>>> 111149676; bufend = 122264640; bufvoid = 318767104
>>> 2013-08-26 11:40:33,413 INFO org.apache.hadoop.mapred.MapTask: kvstart =
>>> 1048561; kvend = 838844; length = 1048576
>>> 2013-08-26 11:40:35,879 INFO org.apache.hadoop.mapred.MapTask: Finished
>>> spill 10
>>>
>>>
>>> my question is what is bufvoid? and how can I increase the amount of the
>>> length?
>>>
>>> thanks,
>>> --
>>> Kaveh Minooie
>>
>>
>>
>>
>
> --
> Kaveh Minooie



-- 
Harsh J

Re: 1.2.1 ouput, what is bufvoid

Posted by Harsh J <ha...@cloudera.com>.
The length printed there is the # of elements in an internal array,
not a data size length. If you wish to raise that you can raise up
io.sort.spill.percent (default 0.80, or 80%).

On Tue, Aug 27, 2013 at 4:03 AM, kaveh minooie <ka...@plutoz.com> wrote:
> Thaks Harsh, but my io.sort.mb is 320 while, as you can see, length is
> only 1mb (1048576)
>
>
> On 08/26/2013 03:14 PM, Harsh J wrote:
>>
>> You need to raise io.sort.mb to increase the sort buffer capacity.
>>
>> On Tue, Aug 27, 2013 at 12:56 AM, kaveh minooie <ka...@plutoz.com> wrote:
>>>
>>> I am getting these in job output in the 1.2.1
>>>
>>> Spilling map output: record full = true
>>> 2013-08-26 11:40:33,413 INFO org.apache.hadoop.mapred.MapTask: bufstart =
>>> 111149676; bufend = 122264640; bufvoid = 318767104
>>> 2013-08-26 11:40:33,413 INFO org.apache.hadoop.mapred.MapTask: kvstart =
>>> 1048561; kvend = 838844; length = 1048576
>>> 2013-08-26 11:40:35,879 INFO org.apache.hadoop.mapred.MapTask: Finished
>>> spill 10
>>>
>>>
>>> my question is what is bufvoid? and how can I increase the amount of the
>>> length?
>>>
>>> thanks,
>>> --
>>> Kaveh Minooie
>>
>>
>>
>>
>
> --
> Kaveh Minooie



-- 
Harsh J

Re: 1.2.1 ouput, what is bufvoid

Posted by Harsh J <ha...@cloudera.com>.
The length printed there is the # of elements in an internal array,
not a data size length. If you wish to raise that you can raise up
io.sort.spill.percent (default 0.80, or 80%).

On Tue, Aug 27, 2013 at 4:03 AM, kaveh minooie <ka...@plutoz.com> wrote:
> Thaks Harsh, but my io.sort.mb is 320 while, as you can see, length is
> only 1mb (1048576)
>
>
> On 08/26/2013 03:14 PM, Harsh J wrote:
>>
>> You need to raise io.sort.mb to increase the sort buffer capacity.
>>
>> On Tue, Aug 27, 2013 at 12:56 AM, kaveh minooie <ka...@plutoz.com> wrote:
>>>
>>> I am getting these in job output in the 1.2.1
>>>
>>> Spilling map output: record full = true
>>> 2013-08-26 11:40:33,413 INFO org.apache.hadoop.mapred.MapTask: bufstart =
>>> 111149676; bufend = 122264640; bufvoid = 318767104
>>> 2013-08-26 11:40:33,413 INFO org.apache.hadoop.mapred.MapTask: kvstart =
>>> 1048561; kvend = 838844; length = 1048576
>>> 2013-08-26 11:40:35,879 INFO org.apache.hadoop.mapred.MapTask: Finished
>>> spill 10
>>>
>>>
>>> my question is what is bufvoid? and how can I increase the amount of the
>>> length?
>>>
>>> thanks,
>>> --
>>> Kaveh Minooie
>>
>>
>>
>>
>
> --
> Kaveh Minooie



-- 
Harsh J

Re: 1.2.1 ouput, what is bufvoid

Posted by kaveh minooie <ka...@plutoz.com>.
Thaks Harsh, but my io.sort.mb is 320 while, as you can see, length is
only 1mb (1048576)

On 08/26/2013 03:14 PM, Harsh J wrote:
> You need to raise io.sort.mb to increase the sort buffer capacity.
>
> On Tue, Aug 27, 2013 at 12:56 AM, kaveh minooie <ka...@plutoz.com> wrote:
>> I am getting these in job output in the 1.2.1
>>
>> Spilling map output: record full = true
>> 2013-08-26 11:40:33,413 INFO org.apache.hadoop.mapred.MapTask: bufstart =
>> 111149676; bufend = 122264640; bufvoid = 318767104
>> 2013-08-26 11:40:33,413 INFO org.apache.hadoop.mapred.MapTask: kvstart =
>> 1048561; kvend = 838844; length = 1048576
>> 2013-08-26 11:40:35,879 INFO org.apache.hadoop.mapred.MapTask: Finished
>> spill 10
>>
>>
>> my question is what is bufvoid? and how can I increase the amount of the
>> length?
>>
>> thanks,
>> --
>> Kaveh Minooie
>
>
>

-- 
Kaveh Minooie

Re: 1.2.1 ouput, what is bufvoid

Posted by kaveh minooie <ka...@plutoz.com>.
Thaks Harsh, but my io.sort.mb is 320 while, as you can see, length is
only 1mb (1048576)

On 08/26/2013 03:14 PM, Harsh J wrote:
> You need to raise io.sort.mb to increase the sort buffer capacity.
>
> On Tue, Aug 27, 2013 at 12:56 AM, kaveh minooie <ka...@plutoz.com> wrote:
>> I am getting these in job output in the 1.2.1
>>
>> Spilling map output: record full = true
>> 2013-08-26 11:40:33,413 INFO org.apache.hadoop.mapred.MapTask: bufstart =
>> 111149676; bufend = 122264640; bufvoid = 318767104
>> 2013-08-26 11:40:33,413 INFO org.apache.hadoop.mapred.MapTask: kvstart =
>> 1048561; kvend = 838844; length = 1048576
>> 2013-08-26 11:40:35,879 INFO org.apache.hadoop.mapred.MapTask: Finished
>> spill 10
>>
>>
>> my question is what is bufvoid? and how can I increase the amount of the
>> length?
>>
>> thanks,
>> --
>> Kaveh Minooie
>
>
>

-- 
Kaveh Minooie

Re: 1.2.1 ouput, what is bufvoid

Posted by kaveh minooie <ka...@plutoz.com>.
Thaks Harsh, but my io.sort.mb is 320 while, as you can see, length is
only 1mb (1048576)

On 08/26/2013 03:14 PM, Harsh J wrote:
> You need to raise io.sort.mb to increase the sort buffer capacity.
>
> On Tue, Aug 27, 2013 at 12:56 AM, kaveh minooie <ka...@plutoz.com> wrote:
>> I am getting these in job output in the 1.2.1
>>
>> Spilling map output: record full = true
>> 2013-08-26 11:40:33,413 INFO org.apache.hadoop.mapred.MapTask: bufstart =
>> 111149676; bufend = 122264640; bufvoid = 318767104
>> 2013-08-26 11:40:33,413 INFO org.apache.hadoop.mapred.MapTask: kvstart =
>> 1048561; kvend = 838844; length = 1048576
>> 2013-08-26 11:40:35,879 INFO org.apache.hadoop.mapred.MapTask: Finished
>> spill 10
>>
>>
>> my question is what is bufvoid? and how can I increase the amount of the
>> length?
>>
>> thanks,
>> --
>> Kaveh Minooie
>
>
>

-- 
Kaveh Minooie

Re: 1.2.1 ouput, what is bufvoid

Posted by kaveh minooie <ka...@plutoz.com>.
Thaks Harsh, but my io.sort.mb is 320 while, as you can see, length is
only 1mb (1048576)

On 08/26/2013 03:14 PM, Harsh J wrote:
> You need to raise io.sort.mb to increase the sort buffer capacity.
>
> On Tue, Aug 27, 2013 at 12:56 AM, kaveh minooie <ka...@plutoz.com> wrote:
>> I am getting these in job output in the 1.2.1
>>
>> Spilling map output: record full = true
>> 2013-08-26 11:40:33,413 INFO org.apache.hadoop.mapred.MapTask: bufstart =
>> 111149676; bufend = 122264640; bufvoid = 318767104
>> 2013-08-26 11:40:33,413 INFO org.apache.hadoop.mapred.MapTask: kvstart =
>> 1048561; kvend = 838844; length = 1048576
>> 2013-08-26 11:40:35,879 INFO org.apache.hadoop.mapred.MapTask: Finished
>> spill 10
>>
>>
>> my question is what is bufvoid? and how can I increase the amount of the
>> length?
>>
>> thanks,
>> --
>> Kaveh Minooie
>
>
>

-- 
Kaveh Minooie

Re: 1.2.1 ouput, what is bufvoid

Posted by Harsh J <ha...@cloudera.com>.
You need to raise io.sort.mb to increase the sort buffer capacity.

On Tue, Aug 27, 2013 at 12:56 AM, kaveh minooie <ka...@plutoz.com> wrote:
> I am getting these in job output in the 1.2.1
>
> Spilling map output: record full = true
> 2013-08-26 11:40:33,413 INFO org.apache.hadoop.mapred.MapTask: bufstart =
> 111149676; bufend = 122264640; bufvoid = 318767104
> 2013-08-26 11:40:33,413 INFO org.apache.hadoop.mapred.MapTask: kvstart =
> 1048561; kvend = 838844; length = 1048576
> 2013-08-26 11:40:35,879 INFO org.apache.hadoop.mapred.MapTask: Finished
> spill 10
>
>
> my question is what is bufvoid? and how can I increase the amount of the
> length?
>
> thanks,
> --
> Kaveh Minooie



-- 
Harsh J

Re: 1.2.1 ouput, what is bufvoid

Posted by Harsh J <ha...@cloudera.com>.
You need to raise io.sort.mb to increase the sort buffer capacity.

On Tue, Aug 27, 2013 at 12:56 AM, kaveh minooie <ka...@plutoz.com> wrote:
> I am getting these in job output in the 1.2.1
>
> Spilling map output: record full = true
> 2013-08-26 11:40:33,413 INFO org.apache.hadoop.mapred.MapTask: bufstart =
> 111149676; bufend = 122264640; bufvoid = 318767104
> 2013-08-26 11:40:33,413 INFO org.apache.hadoop.mapred.MapTask: kvstart =
> 1048561; kvend = 838844; length = 1048576
> 2013-08-26 11:40:35,879 INFO org.apache.hadoop.mapred.MapTask: Finished
> spill 10
>
>
> my question is what is bufvoid? and how can I increase the amount of the
> length?
>
> thanks,
> --
> Kaveh Minooie



-- 
Harsh J

Re: 1.2.1 ouput, what is bufvoid

Posted by Harsh J <ha...@cloudera.com>.
You need to raise io.sort.mb to increase the sort buffer capacity.

On Tue, Aug 27, 2013 at 12:56 AM, kaveh minooie <ka...@plutoz.com> wrote:
> I am getting these in job output in the 1.2.1
>
> Spilling map output: record full = true
> 2013-08-26 11:40:33,413 INFO org.apache.hadoop.mapred.MapTask: bufstart =
> 111149676; bufend = 122264640; bufvoid = 318767104
> 2013-08-26 11:40:33,413 INFO org.apache.hadoop.mapred.MapTask: kvstart =
> 1048561; kvend = 838844; length = 1048576
> 2013-08-26 11:40:35,879 INFO org.apache.hadoop.mapred.MapTask: Finished
> spill 10
>
>
> my question is what is bufvoid? and how can I increase the amount of the
> length?
>
> thanks,
> --
> Kaveh Minooie



-- 
Harsh J

Re: 1.2.1 ouput, what is bufvoid

Posted by Harsh J <ha...@cloudera.com>.
You need to raise io.sort.mb to increase the sort buffer capacity.

On Tue, Aug 27, 2013 at 12:56 AM, kaveh minooie <ka...@plutoz.com> wrote:
> I am getting these in job output in the 1.2.1
>
> Spilling map output: record full = true
> 2013-08-26 11:40:33,413 INFO org.apache.hadoop.mapred.MapTask: bufstart =
> 111149676; bufend = 122264640; bufvoid = 318767104
> 2013-08-26 11:40:33,413 INFO org.apache.hadoop.mapred.MapTask: kvstart =
> 1048561; kvend = 838844; length = 1048576
> 2013-08-26 11:40:35,879 INFO org.apache.hadoop.mapred.MapTask: Finished
> spill 10
>
>
> my question is what is bufvoid? and how can I increase the amount of the
> length?
>
> thanks,
> --
> Kaveh Minooie



-- 
Harsh J