You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by oddaniel <od...@msn.com> on 2008/04/13 06:46:23 UTC
java.io.IOException: No input paths specified in input
I am trying to merge two crawl results.
1. Merge linkdbs - WORKS FINE.
2. Merge crawldbs - WORKS FINE.
For some reason I keep getting java.io.IOException: No input paths specified
in input whern trying to Merge Segments. Can Anyone please tell me what
could be causing this and how to fix it. Thanks.
2008-04-13 14:39:58,109 INFO SegmentMerger - Merging 2 segments to
c:/TEMP/mergedindex/segments/20080413143958
2008-04-13 14:39:58,125 INFO SegmentMerger - SegmentMerger: adding
c:/TEMP/nutchindex/segments
2008-04-13 14:39:58,125 INFO SegmentMerger - SegmentMerger: adding
c:/TEMP/nutchindex2/segments
2008-04-13 14:39:58,125 INFO SegmentMerger - SegmentMerger: using segment
data from:
java.io.IOException: No input paths specified in input
at
org.apache.hadoop.mapred.InputFormatBase.validateInput(InputFormatBase.java:99)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:326)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:543)
at org.apache.nutch.segment.SegmentMerger.merge(SegmentMerger.java:590)
at org.apache.nutch.segment.SegmentMerger.main(SegmentMerger.java:638)
at net.blazer.nutch.test.TestRun.mergeIndex(TestRun.java:105)
at net.blazer.nutch.test.TestRun.main(TestRun.java:152)
--
View this message in context: http://www.nabble.com/java.io.IOException%3A-No-input-paths-specified-in-input-tp16656710p16656710.html
Sent from the Nutch - User mailing list archive at Nabble.com.
Re: java.io.IOException: No input paths specified in input
Posted by oddaniel <od...@msn.com>.
Finally solved the problem with java.io.IOException: No input paths specified
in input
This might be useful to someone else
JobConf job = new NutchJob(conf);
for(final Path segmentsDir:segmentsList){
System.out.println(segmentsDir);
Path[] files = fs.listPaths(segmentsDir, new PathFilter() {
public boolean accept(Path f) {
try {
if (fs.isDirectory(f)) return true;
} catch (IOException ioe) {};
return false;
}
});
for(Path segment:files){
job.addInputPath(new Path(segment, CrawlDatum.GENERATE_DIR_NAME));
job.addInputPath(new Path(segment, CrawlDatum.FETCH_DIR_NAME));
job.addInputPath(new Path(segment, CrawlDatum.PARSE_DIR_NAME));
job.addInputPath(new Path(segment, Content.DIR_NAME));
job.addInputPath(new Path(segment, ParseData.DIR_NAME));
job.addInputPath(new Path(segment, ParseText.DIR_NAME));
job.addInputPath(new Path(segment, ParseData.DIR_NAME));
}
}
SegmentMerger segmentMerger = new SegmentMerger(conf);
segmentMerger.configure(job);
//Merge Segments
segmentMerger.merge(segments, segmentsList.toArray(new Path[0]), false, 0L);
.
.
.
.
oddaniel wrote:
>
> I am trying to merge two crawl results.
>
> 1. Merge linkdbs - WORKS FINE.
> 2. Merge crawldbs - WORKS FINE.
>
> For some reason I keep getting java.io.IOException: No input paths
> specified in input whern trying to Merge Segments. Can Anyone please tell
> me what could be causing this and how to fix it. Thanks.
>
> 2008-04-13 14:39:58,109 INFO SegmentMerger - Merging 2 segments to
> c:/TEMP/mergedindex/segments/20080413143958
> 2008-04-13 14:39:58,125 INFO SegmentMerger - SegmentMerger: adding
> c:/TEMP/nutchindex/segments
> 2008-04-13 14:39:58,125 INFO SegmentMerger - SegmentMerger: adding
> c:/TEMP/nutchindex2/segments
> 2008-04-13 14:39:58,125 INFO SegmentMerger - SegmentMerger: using segment
> data from:
> java.io.IOException: No input paths specified in input
> at
> org.apache.hadoop.mapred.InputFormatBase.validateInput(InputFormatBase.java:99)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:326)
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:543)
> at org.apache.nutch.segment.SegmentMerger.merge(SegmentMerger.java:590)
> at org.apache.nutch.segment.SegmentMerger.main(SegmentMerger.java:638)
> at net.blazer.nutch.test.TestRun.mergeIndex(TestRun.java:105)
> at net.blazer.nutch.test.TestRun.main(TestRun.java:152)
>
--
View this message in context: http://www.nabble.com/java.io.IOException%3A-No-input-paths-specified-in-input-tp16656710p16700599.html
Sent from the Nutch - User mailing list archive at Nabble.com.