[jira] [Assigned] (OPENNLP-1921) Add Paragraph Normalizer for Gutenberg-style and Markdown text

["Martin Wiesner (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/OPENNLP-1921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Martin Wiesner reassigned OPENNLP-1921:
---------------------------------------

    Assignee: Martin Wiesner  (was: Kristian Rickert)

> Add Paragraph Normalizer for Gutenberg-style and Markdown text
> --------------------------------------------------------------
>
>                 Key: OPENNLP-1921
>                 URL: https://issues.apache.org/jira/browse/OPENNLP-1921
>             Project: OpenNLP
>          Issue Type: Improvement
>    Affects Versions: 3.0.0-M5
>            Reporter: Kristian Rickert
>            Assignee: Martin Wiesner
>            Priority: Major
>             Fix For: 3.0.0-M6
>
>          Time Spent: 2h
>  Remaining Estimate: 0h
>
> I'm working a lot of gutenberg and markdown text.  Both have a habit of putting CRs and paragraph separation is to CR/LF (or variations thereof).  To better parse that text without splitting sentences, I made the paragraph normalizer.
> I ran the real class against the Alice in Wonderland opening (Project Gutenberg hard-wrap). Usage is one builder call:
> {code:java}
>      CharSequenceNormalizer normalizer = TextNormalizer.builder()
>          .whitespacePreservingParagraphs()
>          .build();
>      String unwrapped = normalizer.normalize(text).toString();
> {code}
>    Real output from the run:
> {noformat}
> === INPUT ===
> Alice was beginning to get very tired of sitting by her sister on the
> bank, and of having nothing to do: once or twice she had peeped into
> the book her sister was reading, but it had no pictures or
> conversations in it, "and what is the use of a book," thought Alice
> "without pictures or conversations?"
> So she was considering in her own mind (as well as she could, for the
> === NORMALIZED ===
> Alice was beginning to get very tired of sitting by her sister on the bank, and of having nothing to do: once or twice she had peeped into the book her sister was reading, but it had no pictures or conversations in it, "and what is the use of a book," thought Alice "without pictures or conversations?" 
> So she was considering in her own mind (as well as she could, for the hot day made her feel very sleepy and stupid), ...
> {noformat}
> The rule: a whitespace run with at most one line break becomes a single space (intra-paragraph wrap is layout, not meaning), a run with two or more becomes one newline (the blank line is the paragraph break), and CRLF counts as one break. Because it implements OffsetAwareNormalizer, token offsets  after unwrapping still map back into the original wrapped text, which is what makes it safe to put in front of a sentence detector.
> Markdown: it works for prose markdown (paragraphs stay intact, blank-line separated headers survive), but it will flatten anything single-newline-separated that you care about structurally, tight lists, code blocks. For markdown you'd typically want it only on extracted prose, not the raw file.  I'm thinking if I should make an option to normalize wiki/markdown style around this as an extension, but off the table for now.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)