You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2018/06/02 12:19:00 UTC
[jira] [Commented] (NUTCH-2590) SegmentReader -get fails
[ https://issues.apache.org/jira/browse/NUTCH-2590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16499036#comment-16499036 ]
ASF GitHub Bot commented on NUTCH-2590:
---------------------------------------
sebastian-nagel closed pull request #337: NUTCH-2590 SegmentReader -get fails
URL: https://github.com/apache/nutch/pull/337
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/src/java/org/apache/nutch/segment/SegmentReader.java b/src/java/org/apache/nutch/segment/SegmentReader.java
index 3b3e706a2..2b9943561 100644
--- a/src/java/org/apache/nutch/segment/SegmentReader.java
+++ b/src/java/org/apache/nutch/segment/SegmentReader.java
@@ -59,7 +59,6 @@
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;
import org.apache.nutch.crawl.CrawlDatum;
-import org.apache.nutch.crawl.LinkDbReader;
import org.apache.nutch.crawl.NutchWritable;
import org.apache.nutch.parse.ParseData;
import org.apache.nutch.parse.ParseText;
@@ -440,8 +439,8 @@ public void run() {
}
private List<Writable> getSeqRecords(Path dir, Text key) throws Exception {
- MapFile.Reader[] readers = MapFileOutputFormat.getReaders(
- dir, getConf());
+ SequenceFile.Reader[] readers = org.apache.hadoop.mapred.SequenceFileOutputFormat
+ .getReaders(getConf(), dir);
ArrayList<Writable> res = new ArrayList<>();
Class<?> keyClass = readers[0].getKeyClass();
Class<?> valueClass = readers[0].getValueClass();
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
> SegmentReader -get fails
> ------------------------
>
> Key: NUTCH-2590
> URL: https://issues.apache.org/jira/browse/NUTCH-2590
> Project: Nutch
> Issue Type: Bug
> Components: segment
> Affects Versions: 1.15
> Reporter: Sebastian Nagel
> Priority: Major
> Fix For: 1.15
>
>
> SegmentReader {{\-get}} fails in local and (pseudo-)distributed mode:
> {noformat}
> % bin/nutch readseg -get crawl/segments/20180531124348/ "https://nutch.apache.org/"
> SegmentReader: get 'https://nutch.apache.org/'
> Exception:
> java.io.FileNotFoundException: File file:.../crawl/segments/20180531124348/crawl_generate/part-r-00000/data does not exist
> at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:611)
> at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:824)
> at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:601)
> at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:428)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1820)
> at org.apache.hadoop.io.MapFile$Reader.createDataFileReader(MapFile.java:456)
> at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:429)
> at org.apache.hadoop.io.MapFile$Reader.<init>(MapFile.java:399)
> at org.apache.hadoop.io.MapFile$Reader.<init>(MapFile.java:408)
> at org.apache.hadoop.mapreduce.lib.output.MapFileOutputFormat.getReaders(MapFileOutputFormat.java:98)
> at org.apache.nutch.segment.SegmentReader.getSeqRecords(SegmentReader.java:443)
> at org.apache.nutch.segment.SegmentReader.access$200(SegmentReader.java:72)
> at org.apache.nutch.segment.SegmentReader$3.run(SegmentReader.java:341)
> Exception:
> java.io.FileNotFoundException: File file:.../crawl/segments/20180531124348/crawl_parse/part-r-00000/data does not exist
> at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:611)
> at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:824)
> at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:601)
> at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:428)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1820)
> at org.apache.hadoop.io.MapFile$Reader.createDataFileReader(MapFile.java:456)
> at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:429)
> at org.apache.hadoop.io.MapFile$Reader.<init>(MapFile.java:399)
> at org.apache.hadoop.io.MapFile$Reader.<init>(MapFile.java:408)
> at org.apache.hadoop.mapreduce.lib.output.MapFileOutputFormat.getReaders(MapFileOutputFormat.java:98)
> at org.apache.nutch.segment.SegmentReader.getSeqRecords(SegmentReader.java:443)
> at org.apache.nutch.segment.SegmentReader.access$200(SegmentReader.java:72)
> at org.apache.nutch.segment.SegmentReader$4.run(SegmentReader.java:353)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)