You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@chukwa.apache.org by "Ari Rabkin (JIRA)" <ji...@apache.org> on 2009/12/03 21:27:20 UTC
[jira] Created: (CHUKWA-421) new logfile rotation detection scheme
new logfile rotation detection scheme
-------------------------------------
Key: CHUKWA-421
URL: https://issues.apache.org/jira/browse/CHUKWA-421
Project: Hadoop Chukwa
Issue Type: Improvement
Components: data collection
Reporter: Ari Rabkin
Currently, we keep files open in order to detect rotation. This is wasteful of file handles, doesn't work correctly across reboots, and I think we can do better.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-421) new logfile rotation detection
scheme
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12792046#action_12792046 ]
Eric Yang commented on CHUKWA-421:
----------------------------------
Give me a day to test this on my end.
> new logfile rotation detection scheme
> -------------------------------------
>
> Key: CHUKWA-421
> URL: https://issues.apache.org/jira/browse/CHUKWA-421
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Attachments: CHUKWA-421.patch
>
>
> Currently, we keep files open in order to detect rotation. This is wasteful of file handles, doesn't work correctly across reboots, and I think we can do better.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-421) new logfile rotation detection scheme
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ari Rabkin updated CHUKWA-421:
------------------------------
Attachment: (was: CHUKWA-421.patch)
> new logfile rotation detection scheme
> -------------------------------------
>
> Key: CHUKWA-421
> URL: https://issues.apache.org/jira/browse/CHUKWA-421
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Attachments: CHUKWA-421.patch
>
>
> Currently, we keep files open in order to detect rotation. This is wasteful of file handles, doesn't work correctly across reboots, and I think we can do better.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-421) new logfile rotation detection
scheme
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12791716#action_12791716 ]
Eric Yang commented on CHUKWA-421:
----------------------------------
-bash-3.1$ telnet localhost 9093
Trying 127.0.0.1...
Connected to localhost.localdomain (127.0.0.1).
Escape character is '^]'.
list
adaptor_1b0f8e15eedd2cb346aa8c1cfd4f35d9) org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped mapred 0 /usr/local/chukwa/current/var/log/metrics/chukwa-mapred-mapred-1260413773627.log 5332706
-bash-3.1$ ls -l chukwa-mapred-mapred-1260413773627.log
-rw-r--r-- 1 mapred users 168050 Dec 17 01:00 chukwa-mapred-mapred-1260413773627.log
-bash-3.1$ ls -l chukwa-mapred-mapred-1260413773627.log.2009-12-16
-rw-r--r-- 1 mapred users 5332706 Dec 16 23:59 chukwa-mapred-mapred-1260413773627.log.2009-12-16
It still doesn't rotate properly.
> new logfile rotation detection scheme
> -------------------------------------
>
> Key: CHUKWA-421
> URL: https://issues.apache.org/jira/browse/CHUKWA-421
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Attachments: CHUKWA-421.patch
>
>
> Currently, we keep files open in order to detect rotation. This is wasteful of file handles, doesn't work correctly across reboots, and I think we can do better.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Reopened: (CHUKWA-421) new logfile rotation detection scheme
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ari Rabkin reopened CHUKWA-421:
-------------------------------
There's a subtle bug in how the new adaptor writes checkpoints. The last-mod date in the checkpoint reflects data SENT, not data WRITTEN. So after a collector crash, this adaptor will resume at the wrong place. Fix is for the adaptor to keep a mapping from send offset to last-mod-date, listen to commit reports, and write accordingly.
If other adaptors start needing to do this, we might need to pull this functionality out into a utility class.
> new logfile rotation detection scheme
> -------------------------------------
>
> Key: CHUKWA-421
> URL: https://issues.apache.org/jira/browse/CHUKWA-421
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Attachments: CHUKWA-421.patch
>
>
> Currently, we keep files open in order to detect rotation. This is wasteful of file handles, doesn't work correctly across reboots, and I think we can do better.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-421) new logfile rotation detection
scheme
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12791145#action_12791145 ]
Eric Yang commented on CHUKWA-421:
----------------------------------
I deployed this patch on my cluster. Let me run it for 2 days to be sure.
> new logfile rotation detection scheme
> -------------------------------------
>
> Key: CHUKWA-421
> URL: https://issues.apache.org/jira/browse/CHUKWA-421
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Attachments: CHUKWA-421.patch
>
>
> Currently, we keep files open in order to detect rotation. This is wasteful of file handles, doesn't work correctly across reboots, and I think we can do better.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-421) new logfile rotation detection scheme
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ari Rabkin updated CHUKWA-421:
------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Eric, do you mean that the new adaptor worked well, or just that it didn't break anything?
> new logfile rotation detection scheme
> -------------------------------------
>
> Key: CHUKWA-421
> URL: https://issues.apache.org/jira/browse/CHUKWA-421
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Attachments: CHUKWA-421.patch
>
>
> Currently, we keep files open in order to detect rotation. This is wasteful of file handles, doesn't work correctly across reboots, and I think we can do better.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-421) new logfile rotation detection
scheme
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12785503#action_12785503 ]
Ari Rabkin commented on CHUKWA-421:
-----------------------------------
My idea is as follows:
1) Adaptors should keep track of the last-modified date of the file they're tailing.
2) Each time through, get the list of files whose names match the file in question, ignoring trailing stuff after a final dot.
(Possible rotation results)
3) Process them, oldest-first. Break ties using longest-first, or some name-based heuristic.
The end goal of this is that if file Foo is being tailed, and then when we look next, Foo is shorter and there's a Foo.1, we'll finish tailing Foo.1, and then start tailing Foo.
> new logfile rotation detection scheme
> -------------------------------------
>
> Key: CHUKWA-421
> URL: https://issues.apache.org/jira/browse/CHUKWA-421
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: data collection
> Reporter: Ari Rabkin
>
> Currently, we keep files open in order to detect rotation. This is wasteful of file handles, doesn't work correctly across reboots, and I think we can do better.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-421) new logfile rotation detection
scheme
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793649#action_12793649 ]
Hudson commented on CHUKWA-421:
-------------------------------
Integrated in Chukwa-trunk #235 (See [http://hudson.zones.apache.org/hudson/job/Chukwa-trunk/235/])
wrong version of
. Use modification time to detect rotation.
> new logfile rotation detection scheme
> -------------------------------------
>
> Key: CHUKWA-421
> URL: https://issues.apache.org/jira/browse/CHUKWA-421
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Attachments: CHUKWA-421.patch
>
>
> Currently, we keep files open in order to detect rotation. This is wasteful of file handles, doesn't work correctly across reboots, and I think we can do better.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-421) new logfile rotation detection scheme
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ari Rabkin updated CHUKWA-421:
------------------------------
Attachment: CHUKWA-421.patch
> new logfile rotation detection scheme
> -------------------------------------
>
> Key: CHUKWA-421
> URL: https://issues.apache.org/jira/browse/CHUKWA-421
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Attachments: CHUKWA-421.patch
>
>
> Currently, we keep files open in order to detect rotation. This is wasteful of file handles, doesn't work correctly across reboots, and I think we can do better.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-421) new logfile rotation detection
scheme
Posted by "Jerome Boulon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12785510#action_12785510 ]
Jerome Boulon commented on CHUKWA-421:
--------------------------------------
@Ari, incrementing a seqId on the file name will work but then you need control over the file creation
> new logfile rotation detection scheme
> -------------------------------------
>
> Key: CHUKWA-421
> URL: https://issues.apache.org/jira/browse/CHUKWA-421
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: data collection
> Reporter: Ari Rabkin
>
> Currently, we keep files open in order to detect rotation. This is wasteful of file handles, doesn't work correctly across reboots, and I think we can do better.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-421) new logfile rotation detection
scheme
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12792023#action_12792023 ]
Ari Rabkin commented on CHUKWA-421:
-----------------------------------
I am unhappy with JNI-based approaches, since I think it'll be unpleasant to maintain and debug, particularly across platforms. So I would rather stick with modification dates if at all possible.
> new logfile rotation detection scheme
> -------------------------------------
>
> Key: CHUKWA-421
> URL: https://issues.apache.org/jira/browse/CHUKWA-421
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Attachments: CHUKWA-421.patch
>
>
> Currently, we keep files open in order to detect rotation. This is wasteful of file handles, doesn't work correctly across reboots, and I think we can do better.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-421) new logfile rotation detection scheme
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ari Rabkin updated CHUKWA-421:
------------------------------
Attachment: (was: CHUKWA-421.patch)
> new logfile rotation detection scheme
> -------------------------------------
>
> Key: CHUKWA-421
> URL: https://issues.apache.org/jira/browse/CHUKWA-421
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Attachments: CHUKWA-421.patch
>
>
> Currently, we keep files open in order to detect rotation. This is wasteful of file handles, doesn't work correctly across reboots, and I think we can do better.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-421) new logfile rotation detection scheme
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ari Rabkin updated CHUKWA-421:
------------------------------
Attachment: CHUKWA-421.patch
Inadvertently broke FileTailingAdaptor. Fixed.
> new logfile rotation detection scheme
> -------------------------------------
>
> Key: CHUKWA-421
> URL: https://issues.apache.org/jira/browse/CHUKWA-421
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Attachments: CHUKWA-421.patch
>
>
> Currently, we keep files open in order to detect rotation. This is wasteful of file handles, doesn't work correctly across reboots, and I think we can do better.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-421) new logfile rotation detection
scheme
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12792040#action_12792040 ]
Ari Rabkin commented on CHUKWA-421:
-----------------------------------
How often do log files get touched after they're rotated out?
> new logfile rotation detection scheme
> -------------------------------------
>
> Key: CHUKWA-421
> URL: https://issues.apache.org/jira/browse/CHUKWA-421
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Attachments: CHUKWA-421.patch
>
>
> Currently, we keep files open in order to detect rotation. This is wasteful of file handles, doesn't work correctly across reboots, and I think we can do better.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-421) new logfile rotation detection
scheme
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12792029#action_12792029 ]
Eric Yang commented on CHUKWA-421:
----------------------------------
My only concern is modification time is easily changed by doing touch on the file. It is a fragile approach.
> new logfile rotation detection scheme
> -------------------------------------
>
> Key: CHUKWA-421
> URL: https://issues.apache.org/jira/browse/CHUKWA-421
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Attachments: CHUKWA-421.patch
>
>
> Currently, we keep files open in order to detect rotation. This is wasteful of file handles, doesn't work correctly across reboots, and I think we can do better.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-421) new logfile rotation detection
scheme
Posted by "Jerome Boulon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12785505#action_12785505 ]
Jerome Boulon commented on CHUKWA-421:
--------------------------------------
We can keep track of current offset and the last record and compare it.
This should work assuming that the date should not be the same and if it is and the record is the same then for us it will be a duplicate anyway.
> new logfile rotation detection scheme
> -------------------------------------
>
> Key: CHUKWA-421
> URL: https://issues.apache.org/jira/browse/CHUKWA-421
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: data collection
> Reporter: Ari Rabkin
>
> Currently, we keep files open in order to detect rotation. This is wasteful of file handles, doesn't work correctly across reboots, and I think we can do better.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-421) new logfile rotation detection
scheme
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793346#action_12793346 ]
Eric Yang commented on CHUKWA-421:
----------------------------------
+1 The current patch worked well on my machines for the last 3 days.
> new logfile rotation detection scheme
> -------------------------------------
>
> Key: CHUKWA-421
> URL: https://issues.apache.org/jira/browse/CHUKWA-421
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Attachments: CHUKWA-421.patch
>
>
> Currently, we keep files open in order to detect rotation. This is wasteful of file handles, doesn't work correctly across reboots, and I think we can do better.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-421) new logfile rotation detection
scheme
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12791905#action_12791905 ]
Ari Rabkin commented on CHUKWA-421:
-----------------------------------
I had originally wanted to use creation dates, but I don't think the java APIs offer a way to do that.
> new logfile rotation detection scheme
> -------------------------------------
>
> Key: CHUKWA-421
> URL: https://issues.apache.org/jira/browse/CHUKWA-421
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Attachments: CHUKWA-421.patch
>
>
> Currently, we keep files open in order to detect rotation. This is wasteful of file handles, doesn't work correctly across reboots, and I think we can do better.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-421) new logfile rotation detection
scheme
Posted by "Jerome Boulon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12792043#action_12792043 ]
Jerome Boulon commented on CHUKWA-421:
--------------------------------------
@Eric: I don't recommend to go down that road but if you want something safe, use the inode but again this will not be portable.
> new logfile rotation detection scheme
> -------------------------------------
>
> Key: CHUKWA-421
> URL: https://issues.apache.org/jira/browse/CHUKWA-421
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Attachments: CHUKWA-421.patch
>
>
> Currently, we keep files open in order to detect rotation. This is wasteful of file handles, doesn't work correctly across reboots, and I think we can do better.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-421) new logfile rotation detection scheme
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ari Rabkin updated CHUKWA-421:
------------------------------
Attachment: CHUKWA-421.patch
> new logfile rotation detection scheme
> -------------------------------------
>
> Key: CHUKWA-421
> URL: https://issues.apache.org/jira/browse/CHUKWA-421
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Attachments: CHUKWA-421.patch
>
>
> Currently, we keep files open in order to detect rotation. This is wasteful of file handles, doesn't work correctly across reboots, and I think we can do better.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-421) new logfile rotation detection
scheme
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12791820#action_12791820 ]
Eric Yang commented on CHUKWA-421:
----------------------------------
Instead of modification dates, it makes more sense to keep track of creation time. If the creation time has changed, it means there was a rotation, and we can sort out where to start tailing base on creation time. When log rotation happens, the new filename does not change the creation time of the rotated log file.
> new logfile rotation detection scheme
> -------------------------------------
>
> Key: CHUKWA-421
> URL: https://issues.apache.org/jira/browse/CHUKWA-421
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Attachments: CHUKWA-421.patch
>
>
> Currently, we keep files open in order to detect rotation. This is wasteful of file handles, doesn't work correctly across reboots, and I think we can do better.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-421) new logfile rotation detection
scheme
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12791763#action_12791763 ]
Ari Rabkin commented on CHUKWA-421:
-----------------------------------
Note that that's not actually exercising the new code. The new code very carefully does not change the behavior of any existing adaptors. Not yet.
> new logfile rotation detection scheme
> -------------------------------------
>
> Key: CHUKWA-421
> URL: https://issues.apache.org/jira/browse/CHUKWA-421
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Attachments: CHUKWA-421.patch
>
>
> Currently, we keep files open in order to detect rotation. This is wasteful of file handles, doesn't work correctly across reboots, and I think we can do better.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-421) new logfile rotation detection
scheme
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12785521#action_12785521 ]
Ari Rabkin commented on CHUKWA-421:
-----------------------------------
I am NOT proposing to increment a sequence id on the file names. I'm proposing to use the file mod date plus length to figure out which rotated files are how old. Hash of last record (or last 200 bytes or whatever) is also an option. My intent is to make the code as modular as possible so that we can try out a couple different approaches.
> new logfile rotation detection scheme
> -------------------------------------
>
> Key: CHUKWA-421
> URL: https://issues.apache.org/jira/browse/CHUKWA-421
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
>
> Currently, we keep files open in order to detect rotation. This is wasteful of file handles, doesn't work correctly across reboots, and I think we can do better.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-421) new logfile rotation detection
scheme
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12792012#action_12792012 ]
Eric Yang commented on CHUKWA-421:
----------------------------------
There is a package called jtux which provides posix system calls. It has UFile.s_stat.st_ctime. It's BSD license. This should fit our requirement.
http://www.basepath.com/aup/jtux
> new logfile rotation detection scheme
> -------------------------------------
>
> Key: CHUKWA-421
> URL: https://issues.apache.org/jira/browse/CHUKWA-421
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Attachments: CHUKWA-421.patch
>
>
> Currently, we keep files open in order to detect rotation. This is wasteful of file handles, doesn't work correctly across reboots, and I think we can do better.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-421) new logfile rotation detection
scheme
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12791375#action_12791375 ]
Ari Rabkin commented on CHUKWA-421:
-----------------------------------
*nod* My sense is that this patch is not quite ready to commit but that the overall approach of using modification dates to detect rotation is workable.
> new logfile rotation detection scheme
> -------------------------------------
>
> Key: CHUKWA-421
> URL: https://issues.apache.org/jira/browse/CHUKWA-421
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Attachments: CHUKWA-421.patch
>
>
> Currently, we keep files open in order to detect rotation. This is wasteful of file handles, doesn't work correctly across reboots, and I think we can do better.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-421) new logfile rotation detection
scheme
Posted by "Jerome Boulon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12792028#action_12792028 ]
Jerome Boulon commented on CHUKWA-421:
--------------------------------------
+1 on @Ari comment
> new logfile rotation detection scheme
> -------------------------------------
>
> Key: CHUKWA-421
> URL: https://issues.apache.org/jira/browse/CHUKWA-421
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Attachments: CHUKWA-421.patch
>
>
> Currently, we keep files open in order to detect rotation. This is wasteful of file handles, doesn't work correctly across reboots, and I think we can do better.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-421) new logfile rotation detection
scheme
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12792042#action_12792042 ]
Ari Rabkin commented on CHUKWA-421:
-----------------------------------
By the way. I now have added, more intrusive tests. I want to commit this.
> new logfile rotation detection scheme
> -------------------------------------
>
> Key: CHUKWA-421
> URL: https://issues.apache.org/jira/browse/CHUKWA-421
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Attachments: CHUKWA-421.patch
>
>
> Currently, we keep files open in order to detect rotation. This is wasteful of file handles, doesn't work correctly across reboots, and I think we can do better.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-421) new logfile rotation detection
scheme
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793376#action_12793376 ]
Eric Yang commented on CHUKWA-421:
----------------------------------
The new adaptor works well. Thank you Ari.
> new logfile rotation detection scheme
> -------------------------------------
>
> Key: CHUKWA-421
> URL: https://issues.apache.org/jira/browse/CHUKWA-421
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Attachments: CHUKWA-421.patch
>
>
> Currently, we keep files open in order to detect rotation. This is wasteful of file handles, doesn't work correctly across reboots, and I think we can do better.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-421) new logfile rotation detection scheme
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ari Rabkin updated CHUKWA-421:
------------------------------
Status: Patch Available (was: Open)
Various refactorings elsewhere. Unit tests pass.
> new logfile rotation detection scheme
> -------------------------------------
>
> Key: CHUKWA-421
> URL: https://issues.apache.org/jira/browse/CHUKWA-421
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Attachments: CHUKWA-421.patch
>
>
> Currently, we keep files open in order to detect rotation. This is wasteful of file handles, doesn't work correctly across reboots, and I think we can do better.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (CHUKWA-421) new logfile rotation detection scheme
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ari Rabkin reassigned CHUKWA-421:
---------------------------------
Assignee: Ari Rabkin
> new logfile rotation detection scheme
> -------------------------------------
>
> Key: CHUKWA-421
> URL: https://issues.apache.org/jira/browse/CHUKWA-421
> Project: Hadoop Chukwa
> Issue Type: Improvement
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
>
> Currently, we keep files open in order to detect rotation. This is wasteful of file handles, doesn't work correctly across reboots, and I think we can do better.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.