[jira] [Updated] (METRON-516) Add batchTimeout parameters with every use of batchSize parameters, and set topology.tick.tuple.freq.secs appropriately

["Matt Foley (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/METRON-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matt Foley updated METRON-516:
------------------------------
    Description: 
This is the control mechanism for METRON-322, timeout-based flushing of batch queues.  The behavior of the batchTimeout parameter is as follows:
* The default value will be 1/2 the value of "topology.message.timeout.secs", minus delta, except in the Enrichment topology where the default value will be 1/4 the value of "topology.message.timeout.secs" (due to the possibility of having an enricher bolt and a threat intel bolt, both with batch queues, daisy-chained one after the other, in the same Enrichment topology).
** The default value of "topology.message.timeout.secs" is 30 sec, so the default batchTimeout for all topologies will be 14 sec, or 7 sec for Enrichment topologies.
* Because there is a good default setting, the user does not need to set batchTimeout unless they wish to force more frequent flushing.
* If the user sets a batchTimeout interval larger than the default based on "topology.message.timeout.secs", it will be ignored.  You can flush more frequently than the default, but not less frequently.  If you want less frequent flushing, you must set a larger "topology.message.timeout.secs" at the Storm level.
** Question: should we provide a Metron-level ZK parameter to override the Storm-level "topology.message.timeout.secs"?
* The value of batchTimeout is not a guaranteed flush interval:
** The bolt will attempt to set "topology.tick.tuple.freq.secs" to the smallest interval used by any of that bolt's queues' batchTimeout.  
** The actual interval of Tick Tuple receipt by all bolts within a topology will be the smallest "topology.tick.tuple.freq.secs" interval specified for any bolt in the topology.  (This is a Storm behavior.)
** Each bolt, when it receives a Tick Tuple, will decide whether its queue(s) need flushing, based on whether each queue is actually older then that queue's batchTimeout.
** Altogether, this means that each queue will flush at an indeterminate time when its age is between \(i) the batchTimeout interval for that queue and (ii) the sum of that batchTimeout interval plus the actual Tick Tuple receipt interval -- unless it flushes first due to batchSize.

Bolts that use batch queues will be modified to capture the value of "topology.message.timeout.secs" and set the value of "topology.tick.tuple.freq.secs" in the call to getComponentConfiguration() method.  Because this apparently only happens at topology startup:
* If the Metron administrator wishes to change "topology.message.timeout.secs" from the default value of 30 sec, she must do it in storm.yaml or in the CLI, and *must not attempt to change it* via custom java code in topology submission.  Attempted changes at topology submission time (except in the CLI) will not be seen in time to correctly set "topology.tick.tuple.freq.secs" in response.  
* Authors of custom bolts may change "topology.message.timeout.secs" in the getComponentConfiguration() method at the same time as setting "topology.tick.tuple.freq.secs".
* "topology.tick.tuple.freq.secs" cannot be changed during runtime; it requires restarting the topology (unless there are Storm GUI actions to change it in runtime, which has not yet been investigated).  As a result, attempts during runtime to change batchTimeout to a *smaller* value than set at topology startup, will have this result:  The affected queue will flush at an indeterminate time when its age is between \(i) the new smaller batchTimeout interval, and (ii) the sum of the new smaller batchTimeout plus the unchanged Tick Tuple receipt interval -- noting that the unchanged Tick Tuple receipt interval may be much larger than the new smaller batchTimeout, if the revised batchTimeout was set very small.

  was:
This is the control mechanism for METRON-322, timeout-based flushing of batch queues.  The behavior of the batchTimeout parameter is as follows:
* The default value will be 1/2 the value of "topology.message.timeout.secs", minus delta, except in the Enrichment topology where the default value will be 1/4 the value of "topology.message.timeout.secs" (due to the possibility of having an enricher and a threat intel Writer bolt, daisy-chained one after the other, in the same Enrichment topology).
* If the user sets a batchTimeout interval larger than the default based on "topology.message.timeout.secs", it will be ignored.  You can flush more frequently than the default, but not less frequently.
* The value of batchTimeout is not a guaranteed flush interval:
** The bolt will attempt to set "topology.tick.tuple.freq.secs" to the smallest interval used by any of that bolt's queues' batchTimeout.  
** The actual interval of Tick Tuple receipt by all bolts within a topology will be the smallest "topology.tick.tuple.freq.secs" interval specified for any bolt in the topology.  (This is a Storm behavior.)
** Each bolt, when it receives a Tick Tuple, will decide whether its queue(s) need flushing, based on whether each queue is actually older then that queue's batchTimeout.
** Altogether, this means that each queue will flush at an indeterminate time when its age is between 1 and 2 times the batchTimeout interval for that queue -- approximately.

Bolts that use batch queues must be sure to capture the value of "topology.message.timeout.secs" when the Storm config is passed in to the prepare() method at topology startup, so they can correctly calculate the default value of batchTimeout.


> Add batchTimeout parameters with every use of batchSize parameters, and set topology.tick.tuple.freq.secs appropriately
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: METRON-516
>                 URL: https://issues.apache.org/jira/browse/METRON-516
>             Project: Metron
>          Issue Type: Sub-task
>            Reporter: Matt Foley
>            Assignee: Matt Foley
>
> This is the control mechanism for METRON-322, timeout-based flushing of batch queues.  The behavior of the batchTimeout parameter is as follows:
> * The default value will be 1/2 the value of "topology.message.timeout.secs", minus delta, except in the Enrichment topology where the default value will be 1/4 the value of "topology.message.timeout.secs" (due to the possibility of having an enricher bolt and a threat intel bolt, both with batch queues, daisy-chained one after the other, in the same Enrichment topology).
> ** The default value of "topology.message.timeout.secs" is 30 sec, so the default batchTimeout for all topologies will be 14 sec, or 7 sec for Enrichment topologies.
> * Because there is a good default setting, the user does not need to set batchTimeout unless they wish to force more frequent flushing.
> * If the user sets a batchTimeout interval larger than the default based on "topology.message.timeout.secs", it will be ignored.  You can flush more frequently than the default, but not less frequently.  If you want less frequent flushing, you must set a larger "topology.message.timeout.secs" at the Storm level.
> ** Question: should we provide a Metron-level ZK parameter to override the Storm-level "topology.message.timeout.secs"?
> * The value of batchTimeout is not a guaranteed flush interval:
> ** The bolt will attempt to set "topology.tick.tuple.freq.secs" to the smallest interval used by any of that bolt's queues' batchTimeout.  
> ** The actual interval of Tick Tuple receipt by all bolts within a topology will be the smallest "topology.tick.tuple.freq.secs" interval specified for any bolt in the topology.  (This is a Storm behavior.)
> ** Each bolt, when it receives a Tick Tuple, will decide whether its queue(s) need flushing, based on whether each queue is actually older then that queue's batchTimeout.
> ** Altogether, this means that each queue will flush at an indeterminate time when its age is between \(i) the batchTimeout interval for that queue and (ii) the sum of that batchTimeout interval plus the actual Tick Tuple receipt interval -- unless it flushes first due to batchSize.
> Bolts that use batch queues will be modified to capture the value of "topology.message.timeout.secs" and set the value of "topology.tick.tuple.freq.secs" in the call to getComponentConfiguration() method.  Because this apparently only happens at topology startup:
> * If the Metron administrator wishes to change "topology.message.timeout.secs" from the default value of 30 sec, she must do it in storm.yaml or in the CLI, and *must not attempt to change it* via custom java code in topology submission.  Attempted changes at topology submission time (except in the CLI) will not be seen in time to correctly set "topology.tick.tuple.freq.secs" in response.  
> * Authors of custom bolts may change "topology.message.timeout.secs" in the getComponentConfiguration() method at the same time as setting "topology.tick.tuple.freq.secs".
> * "topology.tick.tuple.freq.secs" cannot be changed during runtime; it requires restarting the topology (unless there are Storm GUI actions to change it in runtime, which has not yet been investigated).  As a result, attempts during runtime to change batchTimeout to a *smaller* value than set at topology startup, will have this result:  The affected queue will flush at an indeterminate time when its age is between \(i) the new smaller batchTimeout interval, and (ii) the sum of the new smaller batchTimeout plus the unchanged Tick Tuple receipt interval -- noting that the unchanged Tick Tuple receipt interval may be much larger than the new smaller batchTimeout, if the revised batchTimeout was set very small.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)