You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Karl Wettin <ka...@gmail.com> on 2008/04/16 18:23:45 UTC
java.io.IOException: Cannot open filename
I have this problem where a job creates a sequence file output that
exists on the dfs, but it cant be opened by the sequence reader.
I'm on a single local node and on yesterdays trunk. This is the first
thing I do with Hadoop and suspect it has something to do with that.
> job.setOutputFormat(SequenceFileOutputFormat.class);
> job.setOutputPath(outputPath);
> JobClient.runJob(job);
> System.out.println(outputPath + " exists? " + fs.exists(outputPath));
> SequenceFile.Reader resultsReader = new SequenceFile.Reader(fs, outputPath, job);
This is the output:
> 08/04/16 18:07:22 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> 08/04/16 18:07:23 WARN fs.FileSystem: "192.168.1.202:7001" is a deprecated filesystem name. Use "hdfs://192.168.1.202:7001/" instead.
> 08/04/16 18:07:24 INFO mapred.FileInputFormat: Total input paths to process : 1
> 08/04/16 18:07:28 INFO mapred.JobClient: Running job: job_200804150848_0020
> 08/04/16 18:07:29 INFO mapred.JobClient: map 0% reduce 0%
> 08/04/16 18:08:04 INFO mapred.JobClient: map 100% reduce 0%
> 08/04/16 18:08:12 INFO mapred.JobClient: map 100% reduce 100%
> 08/04/16 18:08:13 INFO mapred.JobClient: Job complete: job_200804150848_0020
> 08/04/16 18:08:13 INFO mapred.JobClient: Counters: 16
> 08/04/16 18:08:13 INFO mapred.JobClient: Job Counters
> 08/04/16 18:08:13 INFO mapred.JobClient: Launched map tasks=1
> 08/04/16 18:08:13 INFO mapred.JobClient: Launched reduce tasks=1
> 08/04/16 18:08:13 INFO mapred.JobClient: Data-local map tasks=1
> 08/04/16 18:08:13 INFO mapred.JobClient: Map-Reduce Framework
> 08/04/16 18:08:13 INFO mapred.JobClient: Map input records=2
> 08/04/16 18:08:13 INFO mapred.JobClient: Map output records=2
> 08/04/16 18:08:13 INFO mapred.JobClient: Map input bytes=127
> 08/04/16 18:08:13 INFO mapred.JobClient: Map output bytes=32
> 08/04/16 18:08:13 INFO mapred.JobClient: Combine input records=1
> 08/04/16 18:08:13 INFO mapred.JobClient: Combine output records=2
> 08/04/16 18:08:13 INFO mapred.JobClient: Reduce input groups=1
> 08/04/16 18:08:13 INFO mapred.JobClient: Reduce input records=1
> 08/04/16 18:08:13 INFO mapred.JobClient: Reduce output records=1
> 08/04/16 18:08:13 INFO mapred.JobClient: File Systems
> 08/04/16 18:08:13 INFO mapred.JobClient: Local bytes read=152
> 08/04/16 18:08:13 INFO mapred.JobClient: Local bytes written=312
> 08/04/16 18:08:13 INFO mapred.JobClient: HDFS bytes read=367
> 08/04/16 18:08:13 INFO mapred.JobClient: HDFS bytes written=120
> /hier/results exists? true
> java.io.IOException: Cannot open filename /hier/results
> at org.apache.hadoop.dfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1139)
> at org.apache.hadoop.dfs.DFSClient$DFSInputStream.<init>(DFSClient.java:1130)
> at org.apache.hadoop.dfs.DFSClient.open(DFSClient.java:363)
> at org.apache.hadoop.dfs.DistributedFileSystem.open(DistributedFileSystem.java:129)
> at org.apache.hadoop.io.SequenceFile$Reader.openFile(SequenceFile.java:1451)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1438)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1431)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1426)
> at org.apache.mahout.clustering.hierarchial.HierarchialClusterDriver.run(HierarchialClusterDriver.java:172)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.mahout.clustering.hierarchial.HierarchialClusterDriver.main(HierarchialClusterDriver.java:43)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:155)
> at org.apache.hadoop.mapred.JobShell.run(JobShell.java:194)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at org.apache.hadoop.mapred.JobShell.main(JobShell.java:220)
>
> kodapan:bin kalle$ ./hadoop fs -ls /hier/results
> Found 1 items
> /hier/results/part-00000 <r 3> 120 2008-04-16 18:08 rw-r--r-- kalle supergroup
> kodapan:bin kalle$
DFSClient.java:
> /**
> * Grab the open-file info from namenode
> */
> synchronized void openInfo() throws IOException {
> LocatedBlocks newInfo = callGetBlockLocations(src, 0, prefetchSize);
> if (newInfo == null) {
> throw new IOException("Cannot open filename " + src);
> }
>
It tries to load the block from the fs, but the block is not there? But
I'm on a single node, should not the block already be there? Is there
something that I forgot to flush or wait for? In deperation I also tried
to sleep for a minute between executing job and reading the file.
The complete source code can be made available on request.
karl
Re: java.io.IOException: Cannot open filename
Posted by Karl Wettin <ka...@gmail.com>.
Raghu Angadi skrev:
> Karl Wettin wrote:
>> Raghu Angadi skrev:
>>> You could try cat-ing the file or fsck. something like
>>>
>>> $ ./haddop fs -cat /hier/results
>>> $ ./hadoop fsck /hier/results
>>
>> Note that /hier/results is a directory and not a file. Am I supposed
>> to point ant the part-0000 file?
>
> Yes. You can not open a directory for reading (files under it).
Oh right, I was thinking part-0000 was a part of some composite file
format to handle vary large sequence files (compare BDB). Now I recall
something about it beeing one file per combined reductions or so.
Where can I read more about this?
Thanks for the help!
karl
Re: java.io.IOException: Cannot open filename
Posted by Raghu Angadi <ra...@yahoo-inc.com>.
Karl Wettin wrote:
> Raghu Angadi skrev:
>> You could try cat-ing the file or fsck. something like
>>
>> $ ./haddop fs -cat /hier/results
>> $ ./hadoop fsck /hier/results
>
> Note that /hier/results is a directory and not a file. Am I supposed to
> point ant the part-0000 file?
Yes. You can not open a directory for reading (files under it).
Raghu.
Re: java.io.IOException: Cannot open filename
Posted by Karl Wettin <ka...@gmail.com>.
Raghu Angadi skrev:
> You could try cat-ing the file or fsck. something like
>
> $ ./haddop fs -cat /hier/results
> $ ./hadoop fsck /hier/results
Note that /hier/results is a directory and not a file. Am I supposed to
point ant the part-0000 file?
> kodapan:bin kalle$ ./hadoop fs -cat /hier/results
> cat: Source must be a file.
> kodapan:bin kalle$ ./hadoop fs -cat /hier/results/part-00000
> SEQ#org.apache.hadoop.io.DoubleWritable!org.apache.hadoop.io.LongWritablef?U?!{???????)@r?O? kodapan:bin kalle$
> kodapan:bin kalle$
> Total size: 120 B
> Total dirs: 0
> Total files: 1
> Total blocks: 1 (avg. block size 120 B)
> Minimally replicated blocks: 1 (100.0 %)
karl
Re: java.io.IOException: Cannot open filename
Posted by Raghu Angadi <ra...@yahoo-inc.com>.
You could try cat-ing the file or fsck. something like
$ ./haddop fs -cat /hier/results
$ ./hadoop fsck /hier/results
Raghu.
Karl Wettin wrote:
>> kodapan:bin kalle$ ./hadoop fs -ls /hier/results
>> Found 1 items
>> /hier/results/part-00000 <r 3> 120 2008-04-16 18:08
>> rw-r--r-- kalle supergroup
>> kodapan:bin kalle$
>
>
>
> DFSClient.java:
>
>> /**
>> * Grab the open-file info from namenode
>> */
>> synchronized void openInfo() throws IOException {
>> LocatedBlocks newInfo = callGetBlockLocations(src, 0,
>> prefetchSize);
>> if (newInfo == null) {
>> throw new IOException("Cannot open filename " + src);
>> }
>>
>
> It tries to load the block from the fs, but the block is not there? But
> I'm on a single node, should not the block already be there? Is there
> something that I forgot to flush or wait for? In deperation I also tried
> to sleep for a minute between executing job and reading the file.
>
> The complete source code can be made available on request.
>
>
>
> karl