You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Abhay Ratnaparkhi <ab...@gmail.com> on 2012/08/28 11:22:27 UTC
error in shuffle in InMemoryMerger
Hello,
I am getting following error when reduce task is running.
"mapreduce.reduce.shuffle.parallelcopies" property is set to 5.
org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in
shuffle in InMemoryMerger - Thread to merge in-memory shuffled map-outputs
at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:124) at
org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:362) at
org.apache.hadoop.mapred.Child$4.run(Child.java:217) at
java.security.AccessController.doPrivileged(AccessController.java:284) at
javax.security.auth.Subject.doAs(Subject.java:573) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:773)
at org.apache.hadoop.mapred.Child.main(Child.java:211) Caused by:
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any
valid local directory for output/map_128.out at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:351)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132)
at org.apache.hadoop.mapred.MapOutputFile.getInputFileForWrite(MapOutputF
org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in
shuffle in InMemoryMerger - Thread to merge in-memory shuffled map-outputs
at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:124) at
org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:362) at
org.apache.hadoop.mapred.Child$4.run(Child.java:217) at
java.security.AccessController.doPrivileged(AccessController.java:284) at
javax.security.auth.Subject.doAs(Subject.java:573) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:773)
at org.apache.hadoop.mapred.Child.main(Child.java:211) Caused by:
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any
valid local directory for output/map_119.out at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:351)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132)
at org.apache.hadoop.mapred.MapOutputFile.getInputFileForWrite(MapOutputF
Regards,
Abhay
Re: error in shuffle in InMemoryMerger
Posted by Abhay Ratnaparkhi <ab...@gmail.com>.
Checked the "mapred.tmp.local" directory on the node which is running the
reducer attempt and seems that there is available space around 1G(though
it's less).
On Tue, Aug 28, 2012 at 3:55 PM, Joshi, Rekha <Re...@intuit.com>wrote:
> Hi Abhay,
>
> Ideally the error line - "Caused by:
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any
> valid local directory for output/map_128.out" suggests you either do not
> have permissions for output folder or disk is full.
>
> Also 5 is not a big number on thread spawning, (infact, default on
> parallelcopies) to recommend reducing it, but a lower value might work.only
> long-term indications are for your system to under-go node maintenance.
>
> Thanks
> Rekha
>
> From: Abhay Ratnaparkhi <ab...@gmail.com>
> Reply-To: <us...@hadoop.apache.org>
> Date: Tue, 28 Aug 2012 14:52:27 +0530
> To: <us...@hadoop.apache.org>
> Subject: error in shuffle in InMemoryMerger
>
> Hello,
>
> I am getting following error when reduce task is running.
> "mapreduce.reduce.shuffle.parallelcopies" property is set to 5.
>
> org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in
> shuffle in InMemoryMerger - Thread to merge in-memory shuffled map-outputs
> at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:124) at
> org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:362) at
> org.apache.hadoop.mapred.Child$4.run(Child.java:217) at
> java.security.AccessController.doPrivileged(AccessController.java:284) at
> javax.security.auth.Subject.doAs(Subject.java:573) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:773)
> at org.apache.hadoop.mapred.Child.main(Child.java:211) Caused by:
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any
> valid local directory for output/map_128.out at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:351)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132)
> at org.apache.hadoop.mapred.MapOutputFile.getInputFileForWrite(MapOutputF
> org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in
> shuffle in InMemoryMerger - Thread to merge in-memory shuffled map-outputs
> at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:124) at
> org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:362) at
> org.apache.hadoop.mapred.Child$4.run(Child.java:217) at
> java.security.AccessController.doPrivileged(AccessController.java:284) at
> javax.security.auth.Subject.doAs(Subject.java:573) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:773)
> at org.apache.hadoop.mapred.Child.main(Child.java:211) Caused by:
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any
> valid local directory for output/map_119.out at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:351)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132)
> at org.apache.hadoop.mapred.MapOutputFile.getInputFileForWrite(MapOutputF
>
> Regards,
> Abhay
>
>
Re: error in shuffle in InMemoryMerger
Posted by Abhay Ratnaparkhi <ab...@gmail.com>.
Checked the "mapred.tmp.local" directory on the node which is running the
reducer attempt and seems that there is available space around 1G(though
it's less).
On Tue, Aug 28, 2012 at 3:55 PM, Joshi, Rekha <Re...@intuit.com>wrote:
> Hi Abhay,
>
> Ideally the error line - "Caused by:
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any
> valid local directory for output/map_128.out" suggests you either do not
> have permissions for output folder or disk is full.
>
> Also 5 is not a big number on thread spawning, (infact, default on
> parallelcopies) to recommend reducing it, but a lower value might work.only
> long-term indications are for your system to under-go node maintenance.
>
> Thanks
> Rekha
>
> From: Abhay Ratnaparkhi <ab...@gmail.com>
> Reply-To: <us...@hadoop.apache.org>
> Date: Tue, 28 Aug 2012 14:52:27 +0530
> To: <us...@hadoop.apache.org>
> Subject: error in shuffle in InMemoryMerger
>
> Hello,
>
> I am getting following error when reduce task is running.
> "mapreduce.reduce.shuffle.parallelcopies" property is set to 5.
>
> org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in
> shuffle in InMemoryMerger - Thread to merge in-memory shuffled map-outputs
> at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:124) at
> org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:362) at
> org.apache.hadoop.mapred.Child$4.run(Child.java:217) at
> java.security.AccessController.doPrivileged(AccessController.java:284) at
> javax.security.auth.Subject.doAs(Subject.java:573) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:773)
> at org.apache.hadoop.mapred.Child.main(Child.java:211) Caused by:
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any
> valid local directory for output/map_128.out at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:351)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132)
> at org.apache.hadoop.mapred.MapOutputFile.getInputFileForWrite(MapOutputF
> org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in
> shuffle in InMemoryMerger - Thread to merge in-memory shuffled map-outputs
> at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:124) at
> org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:362) at
> org.apache.hadoop.mapred.Child$4.run(Child.java:217) at
> java.security.AccessController.doPrivileged(AccessController.java:284) at
> javax.security.auth.Subject.doAs(Subject.java:573) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:773)
> at org.apache.hadoop.mapred.Child.main(Child.java:211) Caused by:
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any
> valid local directory for output/map_119.out at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:351)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132)
> at org.apache.hadoop.mapred.MapOutputFile.getInputFileForWrite(MapOutputF
>
> Regards,
> Abhay
>
>
Re: error in shuffle in InMemoryMerger
Posted by Abhay Ratnaparkhi <ab...@gmail.com>.
Checked the "mapred.tmp.local" directory on the node which is running the
reducer attempt and seems that there is available space around 1G(though
it's less).
On Tue, Aug 28, 2012 at 3:55 PM, Joshi, Rekha <Re...@intuit.com>wrote:
> Hi Abhay,
>
> Ideally the error line - "Caused by:
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any
> valid local directory for output/map_128.out" suggests you either do not
> have permissions for output folder or disk is full.
>
> Also 5 is not a big number on thread spawning, (infact, default on
> parallelcopies) to recommend reducing it, but a lower value might work.only
> long-term indications are for your system to under-go node maintenance.
>
> Thanks
> Rekha
>
> From: Abhay Ratnaparkhi <ab...@gmail.com>
> Reply-To: <us...@hadoop.apache.org>
> Date: Tue, 28 Aug 2012 14:52:27 +0530
> To: <us...@hadoop.apache.org>
> Subject: error in shuffle in InMemoryMerger
>
> Hello,
>
> I am getting following error when reduce task is running.
> "mapreduce.reduce.shuffle.parallelcopies" property is set to 5.
>
> org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in
> shuffle in InMemoryMerger - Thread to merge in-memory shuffled map-outputs
> at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:124) at
> org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:362) at
> org.apache.hadoop.mapred.Child$4.run(Child.java:217) at
> java.security.AccessController.doPrivileged(AccessController.java:284) at
> javax.security.auth.Subject.doAs(Subject.java:573) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:773)
> at org.apache.hadoop.mapred.Child.main(Child.java:211) Caused by:
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any
> valid local directory for output/map_128.out at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:351)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132)
> at org.apache.hadoop.mapred.MapOutputFile.getInputFileForWrite(MapOutputF
> org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in
> shuffle in InMemoryMerger - Thread to merge in-memory shuffled map-outputs
> at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:124) at
> org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:362) at
> org.apache.hadoop.mapred.Child$4.run(Child.java:217) at
> java.security.AccessController.doPrivileged(AccessController.java:284) at
> javax.security.auth.Subject.doAs(Subject.java:573) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:773)
> at org.apache.hadoop.mapred.Child.main(Child.java:211) Caused by:
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any
> valid local directory for output/map_119.out at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:351)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132)
> at org.apache.hadoop.mapred.MapOutputFile.getInputFileForWrite(MapOutputF
>
> Regards,
> Abhay
>
>
Re: error in shuffle in InMemoryMerger
Posted by Abhay Ratnaparkhi <ab...@gmail.com>.
Checked the "mapred.tmp.local" directory on the node which is running the
reducer attempt and seems that there is available space around 1G(though
it's less).
On Tue, Aug 28, 2012 at 3:55 PM, Joshi, Rekha <Re...@intuit.com>wrote:
> Hi Abhay,
>
> Ideally the error line - "Caused by:
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any
> valid local directory for output/map_128.out" suggests you either do not
> have permissions for output folder or disk is full.
>
> Also 5 is not a big number on thread spawning, (infact, default on
> parallelcopies) to recommend reducing it, but a lower value might work.only
> long-term indications are for your system to under-go node maintenance.
>
> Thanks
> Rekha
>
> From: Abhay Ratnaparkhi <ab...@gmail.com>
> Reply-To: <us...@hadoop.apache.org>
> Date: Tue, 28 Aug 2012 14:52:27 +0530
> To: <us...@hadoop.apache.org>
> Subject: error in shuffle in InMemoryMerger
>
> Hello,
>
> I am getting following error when reduce task is running.
> "mapreduce.reduce.shuffle.parallelcopies" property is set to 5.
>
> org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in
> shuffle in InMemoryMerger - Thread to merge in-memory shuffled map-outputs
> at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:124) at
> org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:362) at
> org.apache.hadoop.mapred.Child$4.run(Child.java:217) at
> java.security.AccessController.doPrivileged(AccessController.java:284) at
> javax.security.auth.Subject.doAs(Subject.java:573) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:773)
> at org.apache.hadoop.mapred.Child.main(Child.java:211) Caused by:
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any
> valid local directory for output/map_128.out at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:351)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132)
> at org.apache.hadoop.mapred.MapOutputFile.getInputFileForWrite(MapOutputF
> org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in
> shuffle in InMemoryMerger - Thread to merge in-memory shuffled map-outputs
> at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:124) at
> org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:362) at
> org.apache.hadoop.mapred.Child$4.run(Child.java:217) at
> java.security.AccessController.doPrivileged(AccessController.java:284) at
> javax.security.auth.Subject.doAs(Subject.java:573) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:773)
> at org.apache.hadoop.mapred.Child.main(Child.java:211) Caused by:
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any
> valid local directory for output/map_119.out at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:351)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132)
> at org.apache.hadoop.mapred.MapOutputFile.getInputFileForWrite(MapOutputF
>
> Regards,
> Abhay
>
>
Re: error in shuffle in InMemoryMerger
Posted by "Joshi, Rekha" <Re...@intuit.com>.
Hi Abhay,
Ideally the error line - "Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for output/map_128.out" suggests you either do not have permissions for output folder or disk is full.
Also 5 is not a big number on thread spawning, (infact, default on parallelcopies) to recommend reducing it, but a lower value might work.only long-term indications are for your system to under-go node maintenance.
Thanks
Rekha
From: Abhay Ratnaparkhi <ab...@gmail.com>>
Reply-To: <us...@hadoop.apache.org>>
Date: Tue, 28 Aug 2012 14:52:27 +0530
To: <us...@hadoop.apache.org>>
Subject: error in shuffle in InMemoryMerger
Hello,
I am getting following error when reduce task is running.
"mapreduce.reduce.shuffle.parallelcopies" property is set to 5.
org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in InMemoryMerger - Thread to merge in-memory shuffled map-outputs at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:124) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:362) at org.apache.hadoop.mapred.Child$4.run(Child.java:217) at java.security.AccessController.doPrivileged(AccessController.java:284) at javax.security.auth.Subject.doAs(Subject.java:573) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:773) at org.apache.hadoop.mapred.Child.main(Child.java:211) Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for output/map_128.out at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:351) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.mapred.MapOutputFile.getInputFileForWrite(MapOutputF org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in InMemoryMerger - Thread to merge in-memory shuffled map-outputs at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:124) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:362) at org.apache.hadoop.mapred.Child$4.run(Child.java:217) at java.security.AccessController.doPrivileged(AccessController.java:284) at javax.security.auth.Subject.doAs(Subject.java:573) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:773) at org.apache.hadoop.mapred.Child.main(Child.java:211) Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for output/map_119.out at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:351) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.mapred.MapOutputFile.getInputFileForWrite(MapOutputF
Regards,
Abhay
Re: error in shuffle in InMemoryMerger
Posted by "Joshi, Rekha" <Re...@intuit.com>.
Hi Abhay,
Ideally the error line - "Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for output/map_128.out" suggests you either do not have permissions for output folder or disk is full.
Also 5 is not a big number on thread spawning, (infact, default on parallelcopies) to recommend reducing it, but a lower value might work.only long-term indications are for your system to under-go node maintenance.
Thanks
Rekha
From: Abhay Ratnaparkhi <ab...@gmail.com>>
Reply-To: <us...@hadoop.apache.org>>
Date: Tue, 28 Aug 2012 14:52:27 +0530
To: <us...@hadoop.apache.org>>
Subject: error in shuffle in InMemoryMerger
Hello,
I am getting following error when reduce task is running.
"mapreduce.reduce.shuffle.parallelcopies" property is set to 5.
org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in InMemoryMerger - Thread to merge in-memory shuffled map-outputs at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:124) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:362) at org.apache.hadoop.mapred.Child$4.run(Child.java:217) at java.security.AccessController.doPrivileged(AccessController.java:284) at javax.security.auth.Subject.doAs(Subject.java:573) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:773) at org.apache.hadoop.mapred.Child.main(Child.java:211) Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for output/map_128.out at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:351) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.mapred.MapOutputFile.getInputFileForWrite(MapOutputF org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in InMemoryMerger - Thread to merge in-memory shuffled map-outputs at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:124) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:362) at org.apache.hadoop.mapred.Child$4.run(Child.java:217) at java.security.AccessController.doPrivileged(AccessController.java:284) at javax.security.auth.Subject.doAs(Subject.java:573) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:773) at org.apache.hadoop.mapred.Child.main(Child.java:211) Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for output/map_119.out at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:351) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.mapred.MapOutputFile.getInputFileForWrite(MapOutputF
Regards,
Abhay
Re: error in shuffle in InMemoryMerger
Posted by "Joshi, Rekha" <Re...@intuit.com>.
Hi Abhay,
Ideally the error line - "Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for output/map_128.out" suggests you either do not have permissions for output folder or disk is full.
Also 5 is not a big number on thread spawning, (infact, default on parallelcopies) to recommend reducing it, but a lower value might work.only long-term indications are for your system to under-go node maintenance.
Thanks
Rekha
From: Abhay Ratnaparkhi <ab...@gmail.com>>
Reply-To: <us...@hadoop.apache.org>>
Date: Tue, 28 Aug 2012 14:52:27 +0530
To: <us...@hadoop.apache.org>>
Subject: error in shuffle in InMemoryMerger
Hello,
I am getting following error when reduce task is running.
"mapreduce.reduce.shuffle.parallelcopies" property is set to 5.
org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in InMemoryMerger - Thread to merge in-memory shuffled map-outputs at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:124) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:362) at org.apache.hadoop.mapred.Child$4.run(Child.java:217) at java.security.AccessController.doPrivileged(AccessController.java:284) at javax.security.auth.Subject.doAs(Subject.java:573) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:773) at org.apache.hadoop.mapred.Child.main(Child.java:211) Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for output/map_128.out at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:351) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.mapred.MapOutputFile.getInputFileForWrite(MapOutputF org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in InMemoryMerger - Thread to merge in-memory shuffled map-outputs at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:124) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:362) at org.apache.hadoop.mapred.Child$4.run(Child.java:217) at java.security.AccessController.doPrivileged(AccessController.java:284) at javax.security.auth.Subject.doAs(Subject.java:573) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:773) at org.apache.hadoop.mapred.Child.main(Child.java:211) Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for output/map_119.out at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:351) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.mapred.MapOutputFile.getInputFileForWrite(MapOutputF
Regards,
Abhay
Re: error in shuffle in InMemoryMerger
Posted by "Joshi, Rekha" <Re...@intuit.com>.
Hi Abhay,
Ideally the error line - "Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for output/map_128.out" suggests you either do not have permissions for output folder or disk is full.
Also 5 is not a big number on thread spawning, (infact, default on parallelcopies) to recommend reducing it, but a lower value might work.only long-term indications are for your system to under-go node maintenance.
Thanks
Rekha
From: Abhay Ratnaparkhi <ab...@gmail.com>>
Reply-To: <us...@hadoop.apache.org>>
Date: Tue, 28 Aug 2012 14:52:27 +0530
To: <us...@hadoop.apache.org>>
Subject: error in shuffle in InMemoryMerger
Hello,
I am getting following error when reduce task is running.
"mapreduce.reduce.shuffle.parallelcopies" property is set to 5.
org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in InMemoryMerger - Thread to merge in-memory shuffled map-outputs at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:124) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:362) at org.apache.hadoop.mapred.Child$4.run(Child.java:217) at java.security.AccessController.doPrivileged(AccessController.java:284) at javax.security.auth.Subject.doAs(Subject.java:573) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:773) at org.apache.hadoop.mapred.Child.main(Child.java:211) Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for output/map_128.out at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:351) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.mapred.MapOutputFile.getInputFileForWrite(MapOutputF org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in InMemoryMerger - Thread to merge in-memory shuffled map-outputs at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:124) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:362) at org.apache.hadoop.mapred.Child$4.run(Child.java:217) at java.security.AccessController.doPrivileged(AccessController.java:284) at javax.security.auth.Subject.doAs(Subject.java:573) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:773) at org.apache.hadoop.mapred.Child.main(Child.java:211) Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for output/map_119.out at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:351) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132) at org.apache.hadoop.mapred.MapOutputFile.getInputFileForWrite(MapOutputF
Regards,
Abhay