You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Chong Li (JIRA)" <ji...@apache.org> on 2015/03/31 08:11:52 UTC

[jira] [Created] (NUTCH-1978) solrindex will fail when indexing corrupted segments

Chong Li created NUTCH-1978:
-------------------------------

             Summary: solrindex will fail when indexing corrupted segments
                 Key: NUTCH-1978
                 URL: https://issues.apache.org/jira/browse/NUTCH-1978
             Project: Nutch
          Issue Type: Bug
          Components: indexer
    Affects Versions: 1.10
            Reporter: Chong Li
            Priority: Minor
             Fix For: 1.10


The same issue from NUTCH-1771 but seems like this bug will appear in most of the versions since they all don't have the code to handle the corrupted segments.

Form NUTCH-1771, people pointed out that it will be very hard to handle this in the hadoop layer, and the program should skip the corrupted segments instead of end the program. By corrupted segments I mean that the segment may be just generated and doesn't have the content.

So my initial idea is to check if the segment folder is valid before putting the segment into the hadoop job. If the segment is not valid, we can simply just skip that segment. We can check if the segment folder contains exactly 6 sub directories as there should be. The other  approach will be to check all the six sub directories and see if they are exactly the six dir that should appear.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)