[jira] [Updated] (METRON-329) Add new functionality to BulkMessageWriter and related classes for batchTimeout flushing

["Matt Foley (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/METRON-329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matt Foley updated METRON-329:
------------------------------
    Summary: Add new functionality to BulkMessageWriter and related classes for batchTimeout flushing  (was: Add new functionality to BulkMessageWriter and its related classes for batchTimeout flushing)

> Add new functionality to BulkMessageWriter and related classes for batchTimeout flushing
> ----------------------------------------------------------------------------------------
>
>                 Key: METRON-329
>                 URL: https://issues.apache.org/jira/browse/METRON-329
>             Project: Metron
>          Issue Type: Sub-task
>            Reporter: Ajay Yadav
>            Assignee: Matt Foley
>
> Add new functionality to BulkMessageWriter and its related classes for batchTimeout flushing.
> This primarily affects:
> * BulkMessageWriterBolt
> * BulkWriterComponent
> * IndexingConfigurations
> * and newly added utilities in BatchTimeoutHelper
> All of the other changes (from METRON-516) are basically boilerplate, just laying "getBatchTimeout()" facade methods alongside existing "getBatchSize()" facade methods. (There's a lot of layers!)
> The flush-on-timeout logic is fairly straightforward. It was implemented by a refactoring of BulkWriterComponent and basic Tick Tuple reception in BulkMessageWriterBolt.
> The tricky part was figuring out the appropriate setting for 'topology.tick.tuple.freq.secs' if the administrator configures non-default batchTimeouts. It is necessary to enumerate the batchTimeout settings for all configured sensorNames, which is implemented in IndexingConfigurations$getAllConfiguredTimeouts(). Then multiple other factors must be taken into account to determine the allowed and recommended settings, which is implemented in BatchTimeoutHelper. If there are better ways to accomplish these things, please share your ideas.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)