[jira] [Updated] (NUTCH-2229) Allow Jexl expressions on CrawlDatum's fixed attributes

["Markus Jelsma (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/NUTCH-2229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Markus Jelsma updated NUTCH-2229:
---------------------------------
    Fix Version/s:     (was: 1.12)
                   1.13

> Allow Jexl expressions on CrawlDatum's fixed attributes
> -------------------------------------------------------
>
>                 Key: NUTCH-2229
>                 URL: https://issues.apache.org/jira/browse/NUTCH-2229
>             Project: Nutch
>          Issue Type: Improvement
>          Components: crawldb
>    Affects Versions: 1.11
>            Reporter: Markus Jelsma
>            Assignee: Markus Jelsma
>             Fix For: 1.13
>
>         Attachments: NUTCH-2229.patch
>
>
> CrawlDatum allows Jexl expressions on its metadata fields nicely, but it lacks the opportunity to select on attributes like fetchTime and modifiedTime.
> This includes a rudimentary date parser only supporting the yyyy-MM-dd'T'HH:mm:ss'Z' format:
> Dump everything with a modifiedTime higher than 2016-03-20T00:00:00Z
> {code}
> bin/nutch readdb crawl/crawldb/ -dump out -format csv -expr "(modifiedTime > 2016-03-20T00:00:00Z)"
> {code}
> Dump everything that is an HTML file
> {code}
> bin/nutch readdb crawl/crawldb/ -dump out -format csv -expr "(Content_Type == 'text/html' || Content_Type == 'application/xhtml+xml')"
> {code}
> Keep in mind:
> * Jexl doesn't allow a hyphen/minus in field identifier, they are transformed to underscores
> * string literals must be in quotes, only surrounding qoute needs to be escaped by backslash



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)