[jira] [Updated] (METRON-2005) Batch Writer writes 0-byte files to HDFS on rotation

["Justin Leet (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/METRON-2005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Justin Leet updated METRON-2005:
--------------------------------
    Fix Version/s: 0.7.1

> Batch Writer writes 0-byte files to HDFS on rotation
> ----------------------------------------------------
>
>                 Key: METRON-2005
>                 URL: https://issues.apache.org/jira/browse/METRON-2005
>             Project: Metron
>          Issue Type: Bug
>            Reporter: Justin Leet
>            Assignee: Justin Leet
>            Priority: Major
>             Fix For: 0.7.1
>
>          Time Spent: 3h
>  Remaining Estimate: 0h
>
> This results from https://github.com/apache/metron/pull/505
> That PR breaks the standard convention of just choose a file name and rotate the file repeatedly, because now any message could get routed to a different file based on a Stellar statement.  This break was noted in the PR, because we didn't care about the rotation number anyway.
> This works fine for the 0th rotation (a new file is opened, data is written, file is closed), but on the first rotation we signal to the HdfsWriter that the file has been closed in order to limit the maximum number of open files, but still create a new file with rotation 1.  This file never receives any data (because we no longer maintain an open file reference to it), and the SourceHandler for it stays open with the Timer still attempting further (pointless rotations). Note that no data is lost, any data that would go into this file just instead goes into a new 0 rotation file.
> This becomes more obvious the longer the cluster is running or the shorter the timeout on a file is.  As each open file attempts rotations, eventually large numbers of 0-byte files are created. 
> An easy fix for this is to remove the creation of new files during rotations (but still perform RotationActions). This means that every file will have a 0 rotation (which we don't actually use for anything anyway).  More complicated things could be done (e.g. evict oldest file from a cache), but it seems heavy handed for maintaining a rotation count we don't care about anyway.  Additionally, the Timer should be cancelled when the reference is removed from HdfsWriter.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)