You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Chris MacKenzie <st...@chrismackenziephotography.co.uk> on 2014/07/16 19:02:22 UTC
Can someone shed some light on this ? - java.io.IOException: Spill
failed
Hi,
Is this a coding or a setup issue ?
I¹m using Hadoop 2.41
My program is doing a concordance on 500,000 sequences of 400 chars.
My cluster set is 32 data nodes and two masters.
The exact error is:
Error: java.io.IOException: Spill failed
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.checkSpillException(MapTas
k.java:1535)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1062)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:692)
at
org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInput
OutputContextImpl.java:89)
at
org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapp
er.java:112)
at
par.gene.align.v3.concordance.ConcordanceMapper.map(ConcordanceMapper.java:
96)
at
par.gene.align.v3.concordance.ConcordanceMapper.map(ConcordanceMapper.java:
1)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.j
ava:1556)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could
not find any valid local directory for
attempt_1405523201400_0006_m_000000_0_spill_8.out
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForW
rite(LocalDirAllocator.java:402)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocat
or.java:150)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocat
or.java:131)
at
org.apache.hadoop.mapred.YarnOutputFiles.getSpillFileForWrite(YarnOutputFil
es.java:159)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:
1566)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$900(MapTask.java:85
3)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.ja
va:1505)
Regards,
Chris
Re: Can someone shed some light on this ? - java.io.IOException:
Spill failed
Posted by Chris Mawata <ch...@gmail.com>.
I would post the configuration files -- easier for someone to spot
something wrong than to imagine what configuration would get you to that
stacktrace. The part
Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could
not find any valid local directory for
attempt_1405523201400_0006_m_000000_0_spill_8.out
would suggest you might not have hadoop.tmp.dir set (?)
On Wed, Jul 16, 2014 at 1:02 PM, Chris MacKenzie <
studio@chrismackenziephotography.co.uk> wrote:
> Hi,
>
> Is this a coding or a setup issue ?
>
> I¹m using Hadoop 2.41
> My program is doing a concordance on 500,000 sequences of 400 chars.
> My cluster set is 32 data nodes and two masters.
>
> The exact error is:
> Error: java.io.IOException: Spill failed
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.checkSpillException(MapTas
> k.java:1535)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1062)
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:692)
> at
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInput
> OutputContextImpl.java:89)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapp
> er.java:112)
> at
> par.gene.align.v3.concordance.ConcordanceMapper.map(ConcordanceMapper.java:
> 96)
> at
> par.gene.align.v3.concordance.ConcordanceMapper.map(ConcordanceMapper.java:
> 1)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.j
> ava:1556)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could
> not find any valid local directory for
> attempt_1405523201400_0006_m_000000_0_spill_8.out
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForW
> rite(LocalDirAllocator.java:402)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocat
> or.java:150)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocat
> or.java:131)
> at
> org.apache.hadoop.mapred.YarnOutputFiles.getSpillFileForWrite(YarnOutputFil
> es.java:159)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:
> 1566)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$900(MapTask.java:85
> 3)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.ja
> va:1505)
>
> Regards,
>
> Chris
>
>
>
Re: Can someone shed some light on this ? - java.io.IOException:
Spill failed
Posted by Chris Mawata <ch...@gmail.com>.
I would post the configuration files -- easier for someone to spot
something wrong than to imagine what configuration would get you to that
stacktrace. The part
Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could
not find any valid local directory for
attempt_1405523201400_0006_m_000000_0_spill_8.out
would suggest you might not have hadoop.tmp.dir set (?)
On Wed, Jul 16, 2014 at 1:02 PM, Chris MacKenzie <
studio@chrismackenziephotography.co.uk> wrote:
> Hi,
>
> Is this a coding or a setup issue ?
>
> I¹m using Hadoop 2.41
> My program is doing a concordance on 500,000 sequences of 400 chars.
> My cluster set is 32 data nodes and two masters.
>
> The exact error is:
> Error: java.io.IOException: Spill failed
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.checkSpillException(MapTas
> k.java:1535)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1062)
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:692)
> at
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInput
> OutputContextImpl.java:89)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapp
> er.java:112)
> at
> par.gene.align.v3.concordance.ConcordanceMapper.map(ConcordanceMapper.java:
> 96)
> at
> par.gene.align.v3.concordance.ConcordanceMapper.map(ConcordanceMapper.java:
> 1)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.j
> ava:1556)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could
> not find any valid local directory for
> attempt_1405523201400_0006_m_000000_0_spill_8.out
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForW
> rite(LocalDirAllocator.java:402)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocat
> or.java:150)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocat
> or.java:131)
> at
> org.apache.hadoop.mapred.YarnOutputFiles.getSpillFileForWrite(YarnOutputFil
> es.java:159)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:
> 1566)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$900(MapTask.java:85
> 3)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.ja
> va:1505)
>
> Regards,
>
> Chris
>
>
>
Re: Can someone shed some light on this ? - java.io.IOException:
Spill failed
Posted by Chris Mawata <ch...@gmail.com>.
I would post the configuration files -- easier for someone to spot
something wrong than to imagine what configuration would get you to that
stacktrace. The part
Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could
not find any valid local directory for
attempt_1405523201400_0006_m_000000_0_spill_8.out
would suggest you might not have hadoop.tmp.dir set (?)
On Wed, Jul 16, 2014 at 1:02 PM, Chris MacKenzie <
studio@chrismackenziephotography.co.uk> wrote:
> Hi,
>
> Is this a coding or a setup issue ?
>
> I¹m using Hadoop 2.41
> My program is doing a concordance on 500,000 sequences of 400 chars.
> My cluster set is 32 data nodes and two masters.
>
> The exact error is:
> Error: java.io.IOException: Spill failed
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.checkSpillException(MapTas
> k.java:1535)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1062)
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:692)
> at
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInput
> OutputContextImpl.java:89)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapp
> er.java:112)
> at
> par.gene.align.v3.concordance.ConcordanceMapper.map(ConcordanceMapper.java:
> 96)
> at
> par.gene.align.v3.concordance.ConcordanceMapper.map(ConcordanceMapper.java:
> 1)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.j
> ava:1556)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could
> not find any valid local directory for
> attempt_1405523201400_0006_m_000000_0_spill_8.out
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForW
> rite(LocalDirAllocator.java:402)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocat
> or.java:150)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocat
> or.java:131)
> at
> org.apache.hadoop.mapred.YarnOutputFiles.getSpillFileForWrite(YarnOutputFil
> es.java:159)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:
> 1566)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$900(MapTask.java:85
> 3)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.ja
> va:1505)
>
> Regards,
>
> Chris
>
>
>
Re: Can someone shed some light on this ? - java.io.IOException:
Spill failed
Posted by Chris Mawata <ch...@gmail.com>.
I would post the configuration files -- easier for someone to spot
something wrong than to imagine what configuration would get you to that
stacktrace. The part
Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could
not find any valid local directory for
attempt_1405523201400_0006_m_000000_0_spill_8.out
would suggest you might not have hadoop.tmp.dir set (?)
On Wed, Jul 16, 2014 at 1:02 PM, Chris MacKenzie <
studio@chrismackenziephotography.co.uk> wrote:
> Hi,
>
> Is this a coding or a setup issue ?
>
> I¹m using Hadoop 2.41
> My program is doing a concordance on 500,000 sequences of 400 chars.
> My cluster set is 32 data nodes and two masters.
>
> The exact error is:
> Error: java.io.IOException: Spill failed
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.checkSpillException(MapTas
> k.java:1535)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1062)
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:692)
> at
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInput
> OutputContextImpl.java:89)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapp
> er.java:112)
> at
> par.gene.align.v3.concordance.ConcordanceMapper.map(ConcordanceMapper.java:
> 96)
> at
> par.gene.align.v3.concordance.ConcordanceMapper.map(ConcordanceMapper.java:
> 1)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.j
> ava:1556)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could
> not find any valid local directory for
> attempt_1405523201400_0006_m_000000_0_spill_8.out
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForW
> rite(LocalDirAllocator.java:402)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocat
> or.java:150)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocat
> or.java:131)
> at
> org.apache.hadoop.mapred.YarnOutputFiles.getSpillFileForWrite(YarnOutputFil
> es.java:159)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:
> 1566)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$900(MapTask.java:85
> 3)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.ja
> va:1505)
>
> Regards,
>
> Chris
>
>
>