[jira] [Commented] (CHUKWA-589) Functionality allowing collectors to close .chukwa files at a fixed offset inside the rotateInterval

["Shweta Shah (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CHUKWA-589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051373#comment-13051373 ] 

Shweta Shah commented on CHUKWA-589:
------------------------------------

Himanshu and I are working on this feature and we would be submitting version 1 of the patch. Two new properties have been defined in chukwa-collectors-conf.xml.template as below:

- To set the rotator scheme (boolean):
{noformat} 
<property>
    <name>chukwaCollector.isFixedTimeRotatorScheme</name>
    <value>false</value>
    <description>A flag to indicate that the collector should close at a fixed
    offset after every rotateInterval. If set to true then specify
    chukwaCollector.fixedTimeIntervalOffset value. A false value will use the
    default scheme where collectors close after regular rotateIntervals.
    e.g., if isFixedTimeRotatorScheme is true and fixedTimeIntervalOffset is
    set to 10000 and rotateInterval is set to 300000, then the collector will
    close its files at 10 seconds past the 5 minute mark, if
    isFixedTimeRotatorScheme is false, collectors will rotate approximately
    once every 5 minutes
    </description>
</property>
{noformat} 

- To set the offset interval in milliseconds:
{noformat} 
<property>
    <name>chukwaCollector.fixedTimeIntervalOffset</name>
    <value>@TODO-COLLECTORS-FIXEDTIMEINTERVALOFFSET@</value>
    <description>Chukwa fixed time interval offset value (ms)</description>
</property>
{noformat}

In SeqFileWriter class, the scheduleNextRotation() method calculates the delay in scheduling the next rotate() TimerTask as per the chosen scheme and the interval offset (if isFixedTimeRotatorScheme is true).

We have included two tests:
- testSeqWriterFixedCloseInterval() - this test checks the timestamps between two successive sample .chukwa files and compares the time difference with the pre-calculated delay.

- testFixedIntervalOffsetCalculation() - this test verifies the delay calculation using different combinations of rotateInterval, offsetInterval, and startTime (currentTime as used in the actual method).

> Functionality allowing collectors to close .chukwa files at a fixed offset inside the rotateInterval
> ----------------------------------------------------------------------------------------------------
>
>                 Key: CHUKWA-589
>                 URL: https://issues.apache.org/jira/browse/CHUKWA-589
>             Project: Chukwa
>          Issue Type: New Feature
>          Components: data collection
>    Affects Versions: 0.4.0
>            Reporter: Himanshu Gahlot
>
> We have a system built on top of Chukwa that introduces the need to have all .chukwa files for a given rotateInterval closed and available at a predictable time.  Currently, we are experience some drift in the times that the sequence files are closed due to the way the TimerTask is scheduled in the SeqFileWriter class.  We would like to submit a solution that will allow people to configure the time all collectors should close their files for processing in a given interval, while still supporting the default functionality. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira