You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Xuan Dzung Doan <do...@yahoo.com> on 2008/07/08 18:57:22 UTC
RemoteException - IOException on cached file

My mapred program uses a custom library AlignmentLib.jar that is distributed using DistributedCache. I'm testing my program in the pseudo distributed mode in my workstation, but somehow it doesn't seem able to open the library. This is the exception I got for the task (note the line in bold):

08/07/03 13:34:23 INFO mapred.JobClient: Task Id : task_200807031331_0001_m_000000_0, Status : FAILED
java.lang.NullPointerException
    at java.lang.String.<init>(String.java:176)
    at edu.bio.ec2alignment.AlignmentApp$MapClass.map(AlignmentApp.java:49)
    at edu.bio.ec2alignment.AlignmentApp$MapClass.map(AlignmentApp.java:1)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:208)
    at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2084)

task_200807031331_0001_m_000000_0: Caught exception while getting cached files: org.apache.hadoop.ipc.RemoteException: java.io.IOException: Cannot open filename /tmp/hadoop-root/mapred/local/taskTracker/archive/localhost/input/AlignmentLib.jar/AlignmentLib.jar
task_200807031331_0001_m_000000_0:     at org.apache.hadoop.dfs.NameNode.open(NameNode.java:243)
task_200807031331_0001_m_000000_0:     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
task_200807031331_0001_m_000000_0:     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
task_200807031331_0001_m_000000_0:     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
task_200807031331_0001_m_000000_0:     at java.lang.reflect.Method.invoke(Method.java:597)
task_200807031331_0001_m_000000_0:     at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:409)
task_200807031331_0001_m_000000_0:     at org.apache.hadoop.ipc.Server$Handler.run(Server.java:901)
task_200807031331_0001_m_000000_0:     at org.apache.hadoop.ipc.Client.call(Client.java:512)
task_200807031331_0001_m_000000_0:     at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:198)
task_200807031331_0001_m_000000_0:     at org.apache.hadoop.dfs.$Proxy1.open(Unknown Source)
task_200807031331_0001_m_000000_0:     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
task_200807031331_0001_m_000000_0:     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
task_200807031331_0001_m_000000_0:     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
task_200807031331_0001_m_000000_0:     at java.lang.reflect.Method.invoke(Method.java:597)
task_200807031331_0001_m_000000_0:     at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
task_200807031331_0001_m_000000_0:     at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
task_200807031331_0001_m_000000_0:     at org.apache.hadoop.dfs.$Proxy1.open(Unknown Source)
task_200807031331_0001_m_000000_0:     at org.apache.hadoop.dfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:925)
task_200807031331_0001_m_000000_0:     at org.apache.hadoop.dfs.DFSClient$DFSInputStream.<init>(DFSClient.java:917)
task_200807031331_0001_m_000000_0:     at org.apache.hadoop.dfs.DFSClient.open(DFSClient.java:274)
task_200807031331_0001_m_000000_0:     at org.apache.hadoop.dfs.DistributedFileSystem.open(DistributedFileSystem.java:120)
task_200807031331_0001_m_000000_0:     at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:282)
task_200807031331_0001_m_000000_0:     at edu.bio.ec2alignment.AlignmentApp$MapClass.configure(AlignmentApp.java:30)
task_200807031331_0001_m_000000_0:     at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
task_200807031331_0001_m_000000_0:     at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82)
task_200807031331_0001_m_000000_0:     at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:36)
task_200807031331_0001_m_000000_0:     at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
task_200807031331_0001_m_000000_0:     at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82)
task_200807031331_0001_m_000000_0:     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:204)
task_200807031331_0001_m_000000_0:     at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2084)

The local path/name of the file mentioned in the exception is correct. There doesn't seem to have any permission issue for the local file system (I always work as user root; and the permission for the local file for root is rwx). And from what I read in the mapred tutorial, execution permission for the jar file in cache is always automatically set.

To add this library to the cache, I used DistributedCache.addFileToClassPath()

Any idea about what the issue might be and how I should resolve it?

Thanks a bunch,
David.