[jira] [Commented] (NUTCH-1559) parse-metatags duplicates extracted metatags in combination with parse-tika

["Jeff Cocking (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NUTCH-1559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14522115#comment-14522115 ] 

Jeff Cocking commented on NUTCH-1559:
-------------------------------------

In investigating this issue, it appears the MetaTagsParser.java is loading the info twice.  There is code placed in MetaTagsParser to handle metatags not handled by Tika. 

The Tika plugin copies all the Tika metadata into the nutch metadata. 
TikaParser.java (around line 184):
        // populate Nutch metadata with Tika metadata
        String[] TikaMDNames = tikamd.names();
        for (String tikaMDName : TikaMDNames) {
            if (tikaMDName.equalsIgnoreCase(Metadata.TITLE))
                continue;
            // TODO what if multivalued?
            nutchMetadata.add(tikaMDName, tikamd.get(tikaMDName));
        }

The MetaTagsParser is setup to parse both Tika metadata and Nutch metadata.  This is the reason for the duplicate values.
MetaTagsParser.java (around line 104)
    // check in the metadata first : the tika-parser
    // might have stored the values there already
    for (String mdName : metadata.names()) {
      addIndexedMetatags(metadata, mdName, metadata.getValues(mdName));
    }

    Metadata generalMetaTags = metaTags.getGeneralTags();
    for (String tagName : generalMetaTags.names()) {
      addIndexedMetatags(metadata, tagName, generalMetaTags.getValues(tagName));
    }

> parse-metatags duplicates extracted metatags in combination with parse-tika
> ---------------------------------------------------------------------------
>
>                 Key: NUTCH-1559
>                 URL: https://issues.apache.org/jira/browse/NUTCH-1559
>             Project: Nutch
>          Issue Type: Bug
>          Components: parser
>    Affects Versions: 1.6
>            Reporter: Sebastian Nagel
>            Priority: Minor
>             Fix For: 1.11
>
>
> If the plugin parse-metatags is used in combination with parse-tika, the extracted metatags (the pairs <name, value>) are duplicated.
> The metatags are found twice in parse.getData().getParseMeta() and in metaTags.getGeneralTags(). Is this necessary? Maybe we should fix parse-tika in this point?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)