[jira] [Resolved] (NUTCH-2231) Jexl support in generator job

["Markus Jelsma (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/NUTCH-2231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Markus Jelsma resolved NUTCH-2231.
----------------------------------
    Resolution: Fixed

I forgot to resolve this issue after fixing and committing the failure Hudson reported.

> Jexl support in generator job
> -----------------------------
>
>                 Key: NUTCH-2231
>                 URL: https://issues.apache.org/jira/browse/NUTCH-2231
>             Project: Nutch
>          Issue Type: Improvement
>    Affects Versions: 1.11
>            Reporter: Markus Jelsma
>            Assignee: Markus Jelsma
>             Fix For: 1.12
>
>         Attachments: NUTCH-2231.patch, NUTCH-2231.patch
>
>
> Generator should support Jexl expressions. This would make it much easier to implement focussing crawlers that rely on information stored in the CrawlDB. With the HostDB it is possible to restrict the generator to select only interesting records but it is very cumbersome and involves domainblacklist-urlfiltering.
> With Jexl support, it is no hassle!
> Crawl only english records:
> {code}
> bin/nutch generate crawl/crawldb/ crawl/segments/ -expr "(lang == 'en'')"
> {code}
> Crawl only HTML records:
> {code}
> bin/nutch generate crawl/crawldb/ crawl/segments/ -expr "(Content_Type == 'text/html' || Content_Type == 'application/xhtml+xml')"
> {code}
> Keep in mind:
> * Jexl doesn't allow a hyphen/minus in field identifier, they are transformed to underscores
> * string literals must be in quotes, only surrounding qoute needs to be escaped by backslash



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)