[jira] [Reopened] (FLUME-717) WAL data grows forever even though data is delivered in E2E

["Alexander Lorenz-Alten (Reopened) (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLUME-717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alexander Lorenz-Alten reopened FLUME-717:
------------------------------------------

    
> WAL data grows forever even though data is delivered in E2E
> -----------------------------------------------------------
>
>                 Key: FLUME-717
>                 URL: https://issues.apache.org/jira/browse/FLUME-717
>             Project: Flume
>          Issue Type: Bug
>          Components: Master, Node, Sinks+Sources
>    Affects Versions: v0.9.5
>            Reporter: E. Sammer
>            Priority: Blocker
>
> With a heavy enough write load, it appears that the E2E agent WAL will get into a state where data just gets constantly shuffled around between the various directories / states (e.g. writing, logged, sending, sent). When this happens, the WAL directories grow indefinitely until the disk is exhausted, no matter how much data caused the problem.
> To reproduce:
> * Use the supplied config (or something similar).
> * Write to the agent source at a rate of > 1MB/s for a short burst (using something like the provided generator below).
> * Note that data is delivered to the collectorSink but the agent WAL manager constantly grows the data.
> The config:
> {code}
> n1 : execStream("tail -F datafile") | agentE2ESink("host", 12345);
> n2 : collectorSource(12345) | collectorSink("file://...", "n2-");
> {code}
> Generator:
> {code}
> perl -e 'while (1) { print $i++, "\n"; }' >> datafile
> {code}
> This looks and smells just like FLUME-430. I haven't yet examined the WAL or destination data for duplicates / missing events.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira