[jira] [Updated] (FLUME-2779) MultiLine Deserializer for Spooling DIrectory Source

["Siva Kotipalli (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLUME-2779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Siva Kotipalli updated FLUME-2779:
----------------------------------
    Attachment: multilineSpool.patch

1. MultilineDeSerilizer.java 
    -- deserializer.logstartregexkey -- regxKey which is expected to log line start
    -- deserializer.maxLogLineLength -- this is the max log length that is considered for transfer to sink, the remaining  portion is trucated.
2.  The details and options are provided in TestMultiLineDeserializer.java

> MultiLine Deserializer for Spooling DIrectory Source
> ----------------------------------------------------
>
>                 Key: FLUME-2779
>                 URL: https://issues.apache.org/jira/browse/FLUME-2779
>             Project: Flume
>          Issue Type: Improvement
>          Components: Sinks+Sources
>    Affects Versions: 1.7
>            Reporter: Siva Kotipalli
>            Priority: Minor
>         Attachments: multilineSpool.patch
>
>
> 1. Spool Directory source defaults to LINE, this has issue when multiline log events like stack traces and have request/responses.
> 2. Proposal to have log line start regex Key to aggregate all the log lines till the next regex key is found.
> 3. As there is aggregation needed based on startKey, maxLineLength is dropped on deserializer and included maxLogLineLength as alternative, this basically truncates the event based on the length and remaining part is never transferred to sink. 
> 4. logstartregexkey is introduced on Deserializer to include the start regex key.
> 5. Patch available with unit tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)