You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@chukwa.apache.org by "Himanshu Gahlot (JIRA)" <ji...@apache.org> on 2011/06/13 21:58:51 UTC
[jira] [Created] (CHUKWA-589) Functionality allowing collectors to
close .chukwa files at a fixed offset inside the rotateInterval
Functionality allowing collectors to close .chukwa files at a fixed offset inside the rotateInterval
----------------------------------------------------------------------------------------------------
Key: CHUKWA-589
URL: https://issues.apache.org/jira/browse/CHUKWA-589
Project: Chukwa
Issue Type: New Feature
Components: data collection
Affects Versions: 0.4.0
Reporter: Himanshu Gahlot
We have a system built on top of Chukwa that introduces the need to have all .chukwa files for a given rotateInterval closed and available at a predictable time. Currently, we are experience some drift in the times that the sequence files are closed due to the way the TimerTask is scheduled in the SeqFileWriter class. We would like to submit a solution that will allow people to configure the time all collectors should close their files for processing in a given interval, while still supporting the default functionality.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CHUKWA-589) Functionality allowing collectors
to close .chukwa files at a fixed offset inside the rotateInterval
Posted by "Bill Graham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052269#comment-13052269 ]
Bill Graham commented on CHUKWA-589:
------------------------------------
+1. Reviewed code, ran unit tests. Only 1 failed (TestHBaseWriter). I'll commit unless anyone else has comments.
> Functionality allowing collectors to close .chukwa files at a fixed offset inside the rotateInterval
> ----------------------------------------------------------------------------------------------------
>
> Key: CHUKWA-589
> URL: https://issues.apache.org/jira/browse/CHUKWA-589
> Project: Chukwa
> Issue Type: New Feature
> Components: data collection
> Affects Versions: 0.5.0
> Reporter: Himanshu Gahlot
> Attachments: CHUKWA-589_1.patch, CHUKWA-589_2.patch, CHUKWA-589_3.patch
>
>
> We have a system built on top of Chukwa that introduces the need to have all .chukwa files for a given rotateInterval closed and available at a predictable time. Currently, we are experience some drift in the times that the sequence files are closed due to the way the TimerTask is scheduled in the SeqFileWriter class. We would like to submit a solution that will allow people to configure the time all collectors should close their files for processing in a given interval, while still supporting the default functionality.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CHUKWA-589) Functionality allowing collectors to
close .chukwa files at a fixed offset inside the rotateInterval
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang reassigned CHUKWA-589:
--------------------------------
Assignee: Himanshu Gahlot
> Functionality allowing collectors to close .chukwa files at a fixed offset inside the rotateInterval
> ----------------------------------------------------------------------------------------------------
>
> Key: CHUKWA-589
> URL: https://issues.apache.org/jira/browse/CHUKWA-589
> Project: Chukwa
> Issue Type: New Feature
> Components: data collection
> Affects Versions: 0.5.0
> Reporter: Himanshu Gahlot
> Assignee: Himanshu Gahlot
> Attachments: CHUKWA-589_1.patch, CHUKWA-589_2.patch, CHUKWA-589_3.patch
>
>
> We have a system built on top of Chukwa that introduces the need to have all .chukwa files for a given rotateInterval closed and available at a predictable time. Currently, we are experience some drift in the times that the sequence files are closed due to the way the TimerTask is scheduled in the SeqFileWriter class. We would like to submit a solution that will allow people to configure the time all collectors should close their files for processing in a given interval, while still supporting the default functionality.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CHUKWA-589) Functionality allowing collectors
to close .chukwa files at a fixed offset inside the rotateInterval
Posted by "Shweta Shah (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051373#comment-13051373 ]
Shweta Shah commented on CHUKWA-589:
------------------------------------
Himanshu and I are working on this feature and we would be submitting version 1 of the patch. Two new properties have been defined in chukwa-collectors-conf.xml.template as below:
- To set the rotator scheme (boolean):
{noformat}
<property>
<name>chukwaCollector.isFixedTimeRotatorScheme</name>
<value>false</value>
<description>A flag to indicate that the collector should close at a fixed
offset after every rotateInterval. If set to true then specify
chukwaCollector.fixedTimeIntervalOffset value. A false value will use the
default scheme where collectors close after regular rotateIntervals.
e.g., if isFixedTimeRotatorScheme is true and fixedTimeIntervalOffset is
set to 10000 and rotateInterval is set to 300000, then the collector will
close its files at 10 seconds past the 5 minute mark, if
isFixedTimeRotatorScheme is false, collectors will rotate approximately
once every 5 minutes
</description>
</property>
{noformat}
- To set the offset interval in milliseconds:
{noformat}
<property>
<name>chukwaCollector.fixedTimeIntervalOffset</name>
<value>@TODO-COLLECTORS-FIXEDTIMEINTERVALOFFSET@</value>
<description>Chukwa fixed time interval offset value (ms)</description>
</property>
{noformat}
In SeqFileWriter class, the scheduleNextRotation() method calculates the delay in scheduling the next rotate() TimerTask as per the chosen scheme and the interval offset (if isFixedTimeRotatorScheme is true).
We have included two tests:
- testSeqWriterFixedCloseInterval() - this test checks the timestamps between two successive sample .chukwa files and compares the time difference with the pre-calculated delay.
- testFixedIntervalOffsetCalculation() - this test verifies the delay calculation using different combinations of rotateInterval, offsetInterval, and startTime (currentTime as used in the actual method).
> Functionality allowing collectors to close .chukwa files at a fixed offset inside the rotateInterval
> ----------------------------------------------------------------------------------------------------
>
> Key: CHUKWA-589
> URL: https://issues.apache.org/jira/browse/CHUKWA-589
> Project: Chukwa
> Issue Type: New Feature
> Components: data collection
> Affects Versions: 0.4.0
> Reporter: Himanshu Gahlot
>
> We have a system built on top of Chukwa that introduces the need to have all .chukwa files for a given rotateInterval closed and available at a predictable time. Currently, we are experience some drift in the times that the sequence files are closed due to the way the TimerTask is scheduled in the SeqFileWriter class. We would like to submit a solution that will allow people to configure the time all collectors should close their files for processing in a given interval, while still supporting the default functionality.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CHUKWA-589) Functionality allowing collectors to
close .chukwa files at a fixed offset inside the rotateInterval
Posted by "Himanshu Gahlot (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Himanshu Gahlot updated CHUKWA-589:
-----------------------------------
Affects Version/s: (was: 0.4.0)
0.5.0
> Functionality allowing collectors to close .chukwa files at a fixed offset inside the rotateInterval
> ----------------------------------------------------------------------------------------------------
>
> Key: CHUKWA-589
> URL: https://issues.apache.org/jira/browse/CHUKWA-589
> Project: Chukwa
> Issue Type: New Feature
> Components: data collection
> Affects Versions: 0.5.0
> Reporter: Himanshu Gahlot
> Attachments: CHUKWA-589_1.patch
>
>
> We have a system built on top of Chukwa that introduces the need to have all .chukwa files for a given rotateInterval closed and available at a predictable time. Currently, we are experience some drift in the times that the sequence files are closed due to the way the TimerTask is scheduled in the SeqFileWriter class. We would like to submit a solution that will allow people to configure the time all collectors should close their files for processing in a given interval, while still supporting the default functionality.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CHUKWA-589) Functionality allowing collectors
to close .chukwa files at a fixed offset inside the rotateInterval
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051632#comment-13051632 ]
Eric Yang commented on CHUKWA-589:
----------------------------------
Overall, the patch looks good. One minor enhancement, can you add TODO-COLLECTORS-FIXEDTIMEINTERVALOFFSET to default.properties, and modify build.xml to replace the value for chukwa-config target?
> Functionality allowing collectors to close .chukwa files at a fixed offset inside the rotateInterval
> ----------------------------------------------------------------------------------------------------
>
> Key: CHUKWA-589
> URL: https://issues.apache.org/jira/browse/CHUKWA-589
> Project: Chukwa
> Issue Type: New Feature
> Components: data collection
> Affects Versions: 0.5.0
> Reporter: Himanshu Gahlot
> Attachments: CHUKWA-589_1.patch
>
>
> We have a system built on top of Chukwa that introduces the need to have all .chukwa files for a given rotateInterval closed and available at a predictable time. Currently, we are experience some drift in the times that the sequence files are closed due to the way the TimerTask is scheduled in the SeqFileWriter class. We would like to submit a solution that will allow people to configure the time all collectors should close their files for processing in a given interval, while still supporting the default functionality.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CHUKWA-589) Functionality allowing collectors
to close .chukwa files at a fixed offset inside the rotateInterval
Posted by "Himanshu Gahlot (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054154#comment-13054154 ]
Himanshu Gahlot commented on CHUKWA-589:
----------------------------------------
Thanks Bill !
> Functionality allowing collectors to close .chukwa files at a fixed offset inside the rotateInterval
> ----------------------------------------------------------------------------------------------------
>
> Key: CHUKWA-589
> URL: https://issues.apache.org/jira/browse/CHUKWA-589
> Project: Chukwa
> Issue Type: New Feature
> Components: data collection
> Affects Versions: 0.5.0
> Reporter: Himanshu Gahlot
> Assignee: Himanshu Gahlot
> Fix For: 0.5.0
>
> Attachments: CHUKWA-589_1.patch, CHUKWA-589_2.patch, CHUKWA-589_3.patch
>
>
> We have a system built on top of Chukwa that introduces the need to have all .chukwa files for a given rotateInterval closed and available at a predictable time. Currently, we are experience some drift in the times that the sequence files are closed due to the way the TimerTask is scheduled in the SeqFileWriter class. We would like to submit a solution that will allow people to configure the time all collectors should close their files for processing in a given interval, while still supporting the default functionality.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CHUKWA-589) Functionality allowing collectors to
close .chukwa files at a fixed offset inside the rotateInterval
Posted by "Shweta Shah (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shweta Shah updated CHUKWA-589:
-------------------------------
Attachment: CHUKWA-589_2.patch
We have made the required changes to the default.properties and build.xml files. Additionally, the default value (false) for property chukwaCollector.isFixedTimeRotatorScheme is also included in default.properties now.
{noformat}
<property>
<name>chukwaCollector.isFixedTimeRotatorScheme</name>
<value>@TODO-COLLECTORS-ISFIXEDTIMEROTATORSCHEME</value>
<description>A flag to indicate that the collector should close at a fixed
offset after every rotateInterval. The default value is false which uses
the default scheme where collectors close after regular rotateIntervals.
If set to true then specify chukwaCollector.fixedTimeIntervalOffset value.
e.g., if isFixedTimeRotatorScheme is true and fixedTimeIntervalOffset is
set to 10000 and rotateInterval is set to 300000, then the collector will
close its files at 10 seconds past the 5 minute mark, if
isFixedTimeRotatorScheme is false, collectors will rotate approximately
once every 5 minutes
</description>
</property>
{noformat}
Attached please find the version 2 of the patch - CHUKWA-589_2.patch with all changes.
> Functionality allowing collectors to close .chukwa files at a fixed offset inside the rotateInterval
> ----------------------------------------------------------------------------------------------------
>
> Key: CHUKWA-589
> URL: https://issues.apache.org/jira/browse/CHUKWA-589
> Project: Chukwa
> Issue Type: New Feature
> Components: data collection
> Affects Versions: 0.5.0
> Reporter: Himanshu Gahlot
> Attachments: CHUKWA-589_1.patch, CHUKWA-589_2.patch
>
>
> We have a system built on top of Chukwa that introduces the need to have all .chukwa files for a given rotateInterval closed and available at a predictable time. Currently, we are experience some drift in the times that the sequence files are closed due to the way the TimerTask is scheduled in the SeqFileWriter class. We would like to submit a solution that will allow people to configure the time all collectors should close their files for processing in a given interval, while still supporting the default functionality.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CHUKWA-589) Functionality allowing collectors
to close .chukwa files at a fixed offset inside the rotateInterval
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052301#comment-13052301 ]
Eric Yang commented on CHUKWA-589:
----------------------------------
+1 looks good. I will look at TestHBaseWriter.
> Functionality allowing collectors to close .chukwa files at a fixed offset inside the rotateInterval
> ----------------------------------------------------------------------------------------------------
>
> Key: CHUKWA-589
> URL: https://issues.apache.org/jira/browse/CHUKWA-589
> Project: Chukwa
> Issue Type: New Feature
> Components: data collection
> Affects Versions: 0.5.0
> Reporter: Himanshu Gahlot
> Attachments: CHUKWA-589_1.patch, CHUKWA-589_2.patch, CHUKWA-589_3.patch
>
>
> We have a system built on top of Chukwa that introduces the need to have all .chukwa files for a given rotateInterval closed and available at a predictable time. Currently, we are experience some drift in the times that the sequence files are closed due to the way the TimerTask is scheduled in the SeqFileWriter class. We would like to submit a solution that will allow people to configure the time all collectors should close their files for processing in a given interval, while still supporting the default functionality.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CHUKWA-589) Functionality allowing collectors to
close .chukwa files at a fixed offset inside the rotateInterval
Posted by "Himanshu Gahlot (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Himanshu Gahlot updated CHUKWA-589:
-----------------------------------
Attachment: CHUKWA-589_1.patch
Submitting CHUKWA-589_1.patch
> Functionality allowing collectors to close .chukwa files at a fixed offset inside the rotateInterval
> ----------------------------------------------------------------------------------------------------
>
> Key: CHUKWA-589
> URL: https://issues.apache.org/jira/browse/CHUKWA-589
> Project: Chukwa
> Issue Type: New Feature
> Components: data collection
> Affects Versions: 0.5.0
> Reporter: Himanshu Gahlot
> Attachments: CHUKWA-589_1.patch
>
>
> We have a system built on top of Chukwa that introduces the need to have all .chukwa files for a given rotateInterval closed and available at a predictable time. Currently, we are experience some drift in the times that the sequence files are closed due to the way the TimerTask is scheduled in the SeqFileWriter class. We would like to submit a solution that will allow people to configure the time all collectors should close their files for processing in a given interval, while still supporting the default functionality.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CHUKWA-589) Functionality allowing collectors to
close .chukwa files at a fixed offset inside the rotateInterval
Posted by "Shweta Shah (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shweta Shah updated CHUKWA-589:
-------------------------------
Attachment: CHUKWA-589_3.patch
In TestChukwaWriters, made changes to the outputDirectory paths in testWriters as "/testChukwaWriters_testWriters_" and in testSeqWriterFixedCloseInterval as "/testChukwaWriters_testSeqWriterFixedCloseInterval_"
Attaching the patch version 3 with these changes.
> Functionality allowing collectors to close .chukwa files at a fixed offset inside the rotateInterval
> ----------------------------------------------------------------------------------------------------
>
> Key: CHUKWA-589
> URL: https://issues.apache.org/jira/browse/CHUKWA-589
> Project: Chukwa
> Issue Type: New Feature
> Components: data collection
> Affects Versions: 0.5.0
> Reporter: Himanshu Gahlot
> Attachments: CHUKWA-589_1.patch, CHUKWA-589_2.patch, CHUKWA-589_3.patch
>
>
> We have a system built on top of Chukwa that introduces the need to have all .chukwa files for a given rotateInterval closed and available at a predictable time. Currently, we are experience some drift in the times that the sequence files are closed due to the way the TimerTask is scheduled in the SeqFileWriter class. We would like to submit a solution that will allow people to configure the time all collectors should close their files for processing in a given interval, while still supporting the default functionality.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CHUKWA-589) Functionality allowing collectors to
close .chukwa files at a fixed offset inside the rotateInterval
Posted by "Bill Graham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bill Graham resolved CHUKWA-589.
--------------------------------
Resolution: Fixed
Fix Version/s: 0.5.0
Release Note: Allow collectors to close .chukwa files at fixed offset.
Committed, thanks Himanshu and Shweta!
> Functionality allowing collectors to close .chukwa files at a fixed offset inside the rotateInterval
> ----------------------------------------------------------------------------------------------------
>
> Key: CHUKWA-589
> URL: https://issues.apache.org/jira/browse/CHUKWA-589
> Project: Chukwa
> Issue Type: New Feature
> Components: data collection
> Affects Versions: 0.5.0
> Reporter: Himanshu Gahlot
> Assignee: Himanshu Gahlot
> Fix For: 0.5.0
>
> Attachments: CHUKWA-589_1.patch, CHUKWA-589_2.patch, CHUKWA-589_3.patch
>
>
> We have a system built on top of Chukwa that introduces the need to have all .chukwa files for a given rotateInterval closed and available at a predictable time. Currently, we are experience some drift in the times that the sequence files are closed due to the way the TimerTask is scheduled in the SeqFileWriter class. We would like to submit a solution that will allow people to configure the time all collectors should close their files for processing in a given interval, while still supporting the default functionality.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [jira] [Commented] (CHUKWA-589) Functionality allowing collectors
to close .chukwa files at a fixed offset inside the rotateInterval
Posted by Bill Graham <bi...@gmail.com>.
That was my guess. :)
Thanks for clarifying. We'll change that.
On Mon, Jun 20, 2011 at 3:15 PM, Jerome Boulon <jb...@netflix.com> wrote:
> Yea, _JB_ should be changed to something else ;-)
> That's something I may have done as a quick workaround ...
> /Jerome
>
> On 6/20/11 2:52 PM, "Bill Graham (JIRA)" <ji...@apache.org> wrote:
>
> >
> > [
> >
> https://issues.apache.org/jira/browse/CHUKWA-589?page=com.atlassian.jira.p
> >lugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052241#com
> >ment-13052241 ]
> >
> >Bill Graham commented on CHUKWA-589:
> >------------------------------------
> >
> >Looks good to me too although I'd like to request one more small change.
> >In TestChukwaWriters you re-use the "testChukwaWriters_JB_" directory
> >prefix used in the test data. Can you change that to something that
> >implies the test name, so its content can be mapped to the test.
> >Something like "testChukwaWriters_testSeqWriterFixedCloseInterval_".
> >
> >I'm not sure how the "_JB_" convention was introduced in the pre-existing
> >test in that class (maybe Jerome's initials?), but you might a well
> >correct that in the other test as well (i.e.
> >"testChukwaWriters_testWriter_").
> >
> >> Functionality allowing collectors to close .chukwa files at a fixed
> >>offset inside the rotateInterval
> >>
> >>-------------------------------------------------------------------------
> >>---------------------------
> >>
> >> Key: CHUKWA-589
> >> URL: https://issues.apache.org/jira/browse/CHUKWA-589
> >> Project: Chukwa
> >> Issue Type: New Feature
> >> Components: data collection
> >> Affects Versions: 0.5.0
> >> Reporter: Himanshu Gahlot
> >> Attachments: CHUKWA-589_1.patch, CHUKWA-589_2.patch
> >>
> >>
> >> We have a system built on top of Chukwa that introduces the need to
> >>have all .chukwa files for a given rotateInterval closed and available
> >>at a predictable time. Currently, we are experience some drift in the
> >>times that the sequence files are closed due to the way the TimerTask is
> >>scheduled in the SeqFileWriter class. We would like to submit a
> >>solution that will allow people to configure the time all collectors
> >>should close their files for processing in a given interval, while still
> >>supporting the default functionality.
> >
> >--
> >This message is automatically generated by JIRA.
> >For more information on JIRA, see: http://www.atlassian.com/software/jira
> >
> >
> >
>
>
Re: [jira] [Commented] (CHUKWA-589) Functionality allowing
collectors to close .chukwa files at a fixed offset inside the rotateInterval
Posted by Jerome Boulon <jb...@netflix.com>.
Yea, _JB_ should be changed to something else ;-)
That's something I may have done as a quick workaround ...
/Jerome
On 6/20/11 2:52 PM, "Bill Graham (JIRA)" <ji...@apache.org> wrote:
>
> [
>https://issues.apache.org/jira/browse/CHUKWA-589?page=com.atlassian.jira.p
>lugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052241#com
>ment-13052241 ]
>
>Bill Graham commented on CHUKWA-589:
>------------------------------------
>
>Looks good to me too although I'd like to request one more small change.
>In TestChukwaWriters you re-use the "testChukwaWriters_JB_" directory
>prefix used in the test data. Can you change that to something that
>implies the test name, so its content can be mapped to the test.
>Something like "testChukwaWriters_testSeqWriterFixedCloseInterval_".
>
>I'm not sure how the "_JB_" convention was introduced in the pre-existing
>test in that class (maybe Jerome's initials?), but you might a well
>correct that in the other test as well (i.e.
>"testChukwaWriters_testWriter_").
>
>> Functionality allowing collectors to close .chukwa files at a fixed
>>offset inside the rotateInterval
>>
>>-------------------------------------------------------------------------
>>---------------------------
>>
>> Key: CHUKWA-589
>> URL: https://issues.apache.org/jira/browse/CHUKWA-589
>> Project: Chukwa
>> Issue Type: New Feature
>> Components: data collection
>> Affects Versions: 0.5.0
>> Reporter: Himanshu Gahlot
>> Attachments: CHUKWA-589_1.patch, CHUKWA-589_2.patch
>>
>>
>> We have a system built on top of Chukwa that introduces the need to
>>have all .chukwa files for a given rotateInterval closed and available
>>at a predictable time. Currently, we are experience some drift in the
>>times that the sequence files are closed due to the way the TimerTask is
>>scheduled in the SeqFileWriter class. We would like to submit a
>>solution that will allow people to configure the time all collectors
>>should close their files for processing in a given interval, while still
>>supporting the default functionality.
>
>--
>This message is automatically generated by JIRA.
>For more information on JIRA, see: http://www.atlassian.com/software/jira
>
>
>
[jira] [Commented] (CHUKWA-589) Functionality allowing collectors
to close .chukwa files at a fixed offset inside the rotateInterval
Posted by "Bill Graham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052241#comment-13052241 ]
Bill Graham commented on CHUKWA-589:
------------------------------------
Looks good to me too although I'd like to request one more small change. In TestChukwaWriters you re-use the "testChukwaWriters_JB_" directory prefix used in the test data. Can you change that to something that implies the test name, so its content can be mapped to the test. Something like "testChukwaWriters_testSeqWriterFixedCloseInterval_".
I'm not sure how the "_JB_" convention was introduced in the pre-existing test in that class (maybe Jerome's initials?), but you might a well correct that in the other test as well (i.e. "testChukwaWriters_testWriter_").
> Functionality allowing collectors to close .chukwa files at a fixed offset inside the rotateInterval
> ----------------------------------------------------------------------------------------------------
>
> Key: CHUKWA-589
> URL: https://issues.apache.org/jira/browse/CHUKWA-589
> Project: Chukwa
> Issue Type: New Feature
> Components: data collection
> Affects Versions: 0.5.0
> Reporter: Himanshu Gahlot
> Attachments: CHUKWA-589_1.patch, CHUKWA-589_2.patch
>
>
> We have a system built on top of Chukwa that introduces the need to have all .chukwa files for a given rotateInterval closed and available at a predictable time. Currently, we are experience some drift in the times that the sequence files are closed due to the way the TimerTask is scheduled in the SeqFileWriter class. We would like to submit a solution that will allow people to configure the time all collectors should close their files for processing in a given interval, while still supporting the default functionality.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira