You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@chukwa.apache.org by "Ari Rabkin (JIRA)" <ji...@apache.org> on 2009/04/06 23:14:13 UTC
[jira] Created: (CHUKWA-97) FileTailingAdaptor.tailFile should be
refactored
FileTailingAdaptor.tailFile should be refactored
------------------------------------------------
Key: CHUKWA-97
URL: https://issues.apache.org/jira/browse/CHUKWA-97
Project: Hadoop Chukwa
Issue Type: Improvement
Reporter: Ari Rabkin
Priority: Minor
FileTailingAdaptor.tailFile() is 150 lines long, and scary. An awful lot lof policy, for things like log file rotation, is hardcoded and shouldn't be.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-97) FileTailingAdaptor.tailFile should be
refactored and made more flexible
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-97?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756287#action_12756287 ]
Ari Rabkin commented on CHUKWA-97:
----------------------------------
The refactoring I did shouldn't have changed the code path in FileTailingAdaptor; it just refactored some code into a new base class.
We do have a test case for rotation. More tests and documentation are still welcome, of course.
> FileTailingAdaptor.tailFile should be refactored and made more flexible
> -----------------------------------------------------------------------
>
> Key: CHUKWA-97
> URL: https://issues.apache.org/jira/browse/CHUKWA-97
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Priority: Minor
> Attachments: CHUKWA-97.patch
>
>
> FileTailingAdaptor.tailFile() is 150 lines long, and scary. An awful lot lof policy, for things like log file rotation, is hardcoded and shouldn't be.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-97) FileTailingAdaptor.tailFile should be
refactored and made more flexible
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-97?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12758296#action_12758296 ]
Hudson commented on CHUKWA-97:
------------------------------
Integrated in Chukwa-trunk #145 (See [http://hudson.zones.apache.org/hudson/job/Chukwa-trunk/145/])
. Refactored FileTailers, added LWFTAdaptor.
> FileTailingAdaptor.tailFile should be refactored and made more flexible
> -----------------------------------------------------------------------
>
> Key: CHUKWA-97
> URL: https://issues.apache.org/jira/browse/CHUKWA-97
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Priority: Minor
> Attachments: CHUKWA-97.patch
>
>
> FileTailingAdaptor.tailFile() is 150 lines long, and scary. An awful lot lof policy, for things like log file rotation, is hardcoded and shouldn't be.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-97) FileTailingAdaptor.tailFile should be
refactored and made more flexible
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-97?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ari Rabkin updated CHUKWA-97:
-----------------------------
Status: Patch Available (was: Open)
Really, done to get at CHUKWA-361.
> FileTailingAdaptor.tailFile should be refactored and made more flexible
> -----------------------------------------------------------------------
>
> Key: CHUKWA-97
> URL: https://issues.apache.org/jira/browse/CHUKWA-97
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: data collection
> Reporter: Ari Rabkin
> Priority: Minor
> Attachments: CHUKWA-97.patch
>
>
> FileTailingAdaptor.tailFile() is 150 lines long, and scary. An awful lot lof policy, for things like log file rotation, is hardcoded and shouldn't be.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-97) FileTailingAdaptor.tailFile should be
refactored and made more flexible
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-97?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756727#action_12756727 ]
Ari Rabkin commented on CHUKWA-97:
----------------------------------
I want to commit this. Jerome, are you OK with this, or are there revisions you want to see first?
> FileTailingAdaptor.tailFile should be refactored and made more flexible
> -----------------------------------------------------------------------
>
> Key: CHUKWA-97
> URL: https://issues.apache.org/jira/browse/CHUKWA-97
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Priority: Minor
> Attachments: CHUKWA-97.patch
>
>
> FileTailingAdaptor.tailFile() is 150 lines long, and scary. An awful lot lof policy, for things like log file rotation, is hardcoded and shouldn't be.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-97) FileTailingAdaptor.tailFile should be
refactored and made more flexible
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-97?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ari Rabkin updated CHUKWA-97:
-----------------------------
Summary: FileTailingAdaptor.tailFile should be refactored and made more flexible (was: FileTailingAdaptor.tailFile should be refactored)
> FileTailingAdaptor.tailFile should be refactored and made more flexible
> -----------------------------------------------------------------------
>
> Key: CHUKWA-97
> URL: https://issues.apache.org/jira/browse/CHUKWA-97
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: data collection
> Reporter: Ari Rabkin
> Priority: Minor
>
> FileTailingAdaptor.tailFile() is 150 lines long, and scary. An awful lot lof policy, for things like log file rotation, is hardcoded and shouldn't be.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-97) FileTailingAdaptor.tailFile should be
refactored
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-97?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ari Rabkin updated CHUKWA-97:
-----------------------------
Component/s: data collection
> FileTailingAdaptor.tailFile should be refactored
> ------------------------------------------------
>
> Key: CHUKWA-97
> URL: https://issues.apache.org/jira/browse/CHUKWA-97
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: data collection
> Reporter: Ari Rabkin
> Priority: Minor
>
> FileTailingAdaptor.tailFile() is 150 lines long, and scary. An awful lot lof policy, for things like log file rotation, is hardcoded and shouldn't be.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-97) FileTailingAdaptor.tailFile should be
refactored and made more flexible
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-97?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12754828#action_12754828 ]
Ari Rabkin commented on CHUKWA-97:
----------------------------------
Will commit within a few days barring objection, and after testing. It's a refactoring, and passes unit tests.
> FileTailingAdaptor.tailFile should be refactored and made more flexible
> -----------------------------------------------------------------------
>
> Key: CHUKWA-97
> URL: https://issues.apache.org/jira/browse/CHUKWA-97
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Priority: Minor
> Attachments: CHUKWA-97.patch
>
>
> FileTailingAdaptor.tailFile() is 150 lines long, and scary. An awful lot lof policy, for things like log file rotation, is hardcoded and shouldn't be.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-97) FileTailingAdaptor.tailFile should be
refactored and made more flexible
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-97?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ari Rabkin updated CHUKWA-97:
-----------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thanks all, for the review.
> FileTailingAdaptor.tailFile should be refactored and made more flexible
> -----------------------------------------------------------------------
>
> Key: CHUKWA-97
> URL: https://issues.apache.org/jira/browse/CHUKWA-97
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Priority: Minor
> Attachments: CHUKWA-97.patch
>
>
> FileTailingAdaptor.tailFile() is 150 lines long, and scary. An awful lot lof policy, for things like log file rotation, is hardcoded and shouldn't be.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-97) FileTailingAdaptor.tailFile should be
refactored and made more flexible
Posted by "Jerome Boulon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-97?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756286#action_12756286 ]
Jerome Boulon commented on CHUKWA-97:
-------------------------------------
But do we have junit on log rotation ??
> FileTailingAdaptor.tailFile should be refactored and made more flexible
> -----------------------------------------------------------------------
>
> Key: CHUKWA-97
> URL: https://issues.apache.org/jira/browse/CHUKWA-97
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Priority: Minor
> Attachments: CHUKWA-97.patch
>
>
> FileTailingAdaptor.tailFile() is 150 lines long, and scary. An awful lot lof policy, for things like log file rotation, is hardcoded and shouldn't be.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-97) FileTailingAdaptor.tailFile should be
refactored and made more flexible
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-97?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ari Rabkin updated CHUKWA-97:
-----------------------------
Attachment: CHUKWA-97.patch
> FileTailingAdaptor.tailFile should be refactored and made more flexible
> -----------------------------------------------------------------------
>
> Key: CHUKWA-97
> URL: https://issues.apache.org/jira/browse/CHUKWA-97
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: data collection
> Reporter: Ari Rabkin
> Priority: Minor
> Attachments: CHUKWA-97.patch
>
>
> FileTailingAdaptor.tailFile() is 150 lines long, and scary. An awful lot lof policy, for things like log file rotation, is hardcoded and shouldn't be.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-97) FileTailingAdaptor.tailFile should be
refactored and made more flexible
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-97?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755233#action_12755233 ]
Ari Rabkin commented on CHUKWA-97:
----------------------------------
Able to tail thousands of files per second, using a small fraction of CPU, negligible memory. Intended to be used in combination with DirTailingAdaptor.
> FileTailingAdaptor.tailFile should be refactored and made more flexible
> -----------------------------------------------------------------------
>
> Key: CHUKWA-97
> URL: https://issues.apache.org/jira/browse/CHUKWA-97
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Priority: Minor
> Attachments: CHUKWA-97.patch
>
>
> FileTailingAdaptor.tailFile() is 150 lines long, and scary. An awful lot lof policy, for things like log file rotation, is hardcoded and shouldn't be.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (CHUKWA-97) FileTailingAdaptor.tailFile should be
refactored and made more flexible
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-97?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ari Rabkin reassigned CHUKWA-97:
--------------------------------
Assignee: Ari Rabkin
> FileTailingAdaptor.tailFile should be refactored and made more flexible
> -----------------------------------------------------------------------
>
> Key: CHUKWA-97
> URL: https://issues.apache.org/jira/browse/CHUKWA-97
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Priority: Minor
> Attachments: CHUKWA-97.patch
>
>
> FileTailingAdaptor.tailFile() is 150 lines long, and scary. An awful lot lof policy, for things like log file rotation, is hardcoded and shouldn't be.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-97) FileTailingAdaptor.tailFile should be
refactored and made more flexible
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-97?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756285#action_12756285 ]
Ari Rabkin commented on CHUKWA-97:
----------------------------------
This patch very carefully does not replace the current file tailing adaptor. It does two things.
1) It makes it easier to write file tailing adaptors that behave differently in the presence of files that have been rotated, deleted, not yet created, etc.
2) It offers a simple lightweight adaptor to handle the use case where there are thousands of files being watched, and the behavior when files are deleted isn't particularly crucial.
And yes, unit tests still pass.
> FileTailingAdaptor.tailFile should be refactored and made more flexible
> -----------------------------------------------------------------------
>
> Key: CHUKWA-97
> URL: https://issues.apache.org/jira/browse/CHUKWA-97
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Priority: Minor
> Attachments: CHUKWA-97.patch
>
>
> FileTailingAdaptor.tailFile() is 150 lines long, and scary. An awful lot lof policy, for things like log file rotation, is hardcoded and shouldn't be.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-97) FileTailingAdaptor.tailFile should be
refactored and made more flexible
Posted by "Jerome Boulon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-97?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756282#action_12756282 ]
Jerome Boulon commented on CHUKWA-97:
-------------------------------------
Ari, is this patch suppose to replace the current fileTailing adaptor? If yes, do you have tested fileRotation, register done but file creation has been delayed. etc with that patch?
> FileTailingAdaptor.tailFile should be refactored and made more flexible
> -----------------------------------------------------------------------
>
> Key: CHUKWA-97
> URL: https://issues.apache.org/jira/browse/CHUKWA-97
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Priority: Minor
> Attachments: CHUKWA-97.patch
>
>
> FileTailingAdaptor.tailFile() is 150 lines long, and scary. An awful lot lof policy, for things like log file rotation, is hardcoded and shouldn't be.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-97) FileTailingAdaptor.tailFile should be
refactored and made more flexible
Posted by "Jerome Boulon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-97?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12757955#action_12757955 ]
Jerome Boulon commented on CHUKWA-97:
-------------------------------------
+1
> FileTailingAdaptor.tailFile should be refactored and made more flexible
> -----------------------------------------------------------------------
>
> Key: CHUKWA-97
> URL: https://issues.apache.org/jira/browse/CHUKWA-97
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Priority: Minor
> Attachments: CHUKWA-97.patch
>
>
> FileTailingAdaptor.tailFile() is 150 lines long, and scary. An awful lot lof policy, for things like log file rotation, is hardcoded and shouldn't be.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.