[jira] [Created] (APEXCORE-343) Add property to AbstractFileInputOperator to trim processedFiles and ignoredFiles

["Munagala V. Ramanath (JIRA)" <ji...@apache.org>]

Munagala V. Ramanath created APEXCORE-343:
---------------------------------------------

             Summary: Add property to AbstractFileInputOperator to trim processedFiles and ignoredFiles
                 Key: APEXCORE-343
                 URL: https://issues.apache.org/jira/browse/APEXCORE-343
             Project: Apache Apex Core
          Issue Type: Improvement
    Affects Versions: 3.3.0
            Reporter: Munagala V. Ramanath
            Assignee: Munagala V. Ramanath


In AbstractFileInputOperator, the processedFiles and DirectoryScanner.ignoredFiles sets can continue to grow without bound as new files are added to the monitored directory. There are scenarios where an input file is deleted by the application once it is processed; here, it is useful to provide a property that, if true, will cause the scanner to remove deleted files from these sets to prevent unbounded growth.

The only way to do this currently is to extend DirectoryScanner and provide implementations of the scan() and createPartition() methods which forces the operator writer to unnecessarily grapple with the internals of the base class.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)