[jira] [Commented] (NUTCH-585) [PARSE-HTML plugin] Block certain parts of HTML code from being indexed

["Bojan Tomic (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NUTCH-585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14595016#comment-14595016 ] 

Bojan Tomic commented on NUTCH-585:
-----------------------------------

@[~lara_cardozo]
It's been a while since I last used that plugin, but from what I remember, there was no need to modify gora-mysql-mapping.xml. Just create a new column in the DB and configure its name in the plugin configs. I now see the README on Github is out of date as I've added a lot better CSS selector support in the meantime. It could be also that I introduced a new bug :(
If you could tell me exactly what you did (maybe paste your configs), I will try to replicate. What version of Nutch are you using? As far as I know, relational DB support got dropped from Gora (and thus Nutch) soon after 2.1, which was the version I used.
Please put future question directly on my Github issue tracker: https://github.com/kaqqao/nutch-element-selector/issues It makes it easier for me to spot it (it took me over a month to see it here... sorry for that).
I gave up on making a patch as I don't know how to test that it works with other storages and the only storage I cared about at that time (relational DB) got dropped... Also, I figured a feature like this is maybe better kept in a plugin anyway.

> [PARSE-HTML plugin] Block certain parts of HTML code from being indexed
> -----------------------------------------------------------------------
>
>                 Key: NUTCH-585
>                 URL: https://issues.apache.org/jira/browse/NUTCH-585
>             Project: Nutch
>          Issue Type: Improvement
>    Affects Versions: 0.9.0
>         Environment: All operating systems
>            Reporter: Andrea Spinelli
>            Assignee: Markus Jelsma
>            Priority: Minor
>             Fix For: 1.11
>
>         Attachments: blacklist_whitelist_plugin.patch, nutch-585-excludeNodes.patch, nutch-585-jostens-excludeDIVs.patch
>
>
> We are using nutch to index our own web sites; we would like not to index certain parts of our pages, because we know they are not relevant (for instance, there are several links to change the background color) and generate spurious matches.
> We have modified the plugin so that it ignores HTML code between certain HTML comments, like
> <!-- START-IGNORE -->
> ... ignored part ...
> <!-- STOP-IGNORE -->
> We feel this might be useful to someone else, maybe factorizing the comment strings as constants in the configuration files (say parser.html.ignore.start and parser.html.ignore.stop in nutch-site.xml).
> We are almost ready to contribute our code snippet.  Looking forward for any expression of  interest - or for an explanation why waht we are doing is plain wrong!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)