You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Patrick Wendell (JIRA)" <ji...@apache.org> on 2012/07/28 02:19:35 UTC
[jira] [Created] (FLUME-1400) Understand and document behavior of
exec source when tailing rolling logs
Patrick Wendell created FLUME-1400:
--------------------------------------
Summary: Understand and document behavior of exec source when tailing rolling logs
Key: FLUME-1400
URL: https://issues.apache.org/jira/browse/FLUME-1400
Project: Flume
Issue Type: Improvement
Reporter: Patrick Wendell
Assignee: Patrick Wendell
There has been some discussion recently around the behavior of exec("tail -f") when tailing logs, if the log files are themselves being rolled and renamed by a writer process.
The primary concern is whether there is a race condition or correctness problem related to tailing a log which is renamed.
I am fairly certain that the tail binary, when passed -f, will deal with all of the ugliness of recognizing when a file is rolled and open the new "current" file as expected. However, we should both verify and document that behavior.
Flume OG, which implemented its own tailing functionality, dealt with this ugliness by introducing increasing break-fix complexity to the tail source. I am hoping that by delegating to the "tail" binary we have avoided having to do this (though at the cost of OS portability). Nonetheless, I should confirm this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (FLUME-1400) Understand and document behavior of
exec source when tailing rolling logs
Posted by "Patrick Wendell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Wendell resolved FLUME-1400.
------------------------------------
Resolution: Not A Problem
> Understand and document behavior of exec source when tailing rolling logs
> -------------------------------------------------------------------------
>
> Key: FLUME-1400
> URL: https://issues.apache.org/jira/browse/FLUME-1400
> Project: Flume
> Issue Type: Improvement
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
>
> There has been some discussion recently around the behavior of exec("tail -f") when tailing logs, if the log files are themselves being rolled and renamed by a writer process.
> The primary concern is whether there is a race condition or correctness problem related to tailing a log which is renamed.
> I am fairly certain that the tail binary, when passed -f, will deal with all of the ugliness of recognizing when a file is rolled and open the new "current" file as expected. However, we should both verify and document that behavior.
> Flume OG, which implemented its own tailing functionality, dealt with this ugliness by introducing increasing break-fix complexity to the tail source. I am hoping that by delegating to the "tail" binary we have avoided having to do this (though at the cost of OS portability). Nonetheless, I should confirm this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1400) Understand and document behavior of
exec source when tailing rolling logs
Posted by "Jarek Jarcec Cecho (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424570#comment-13424570 ]
Jarek Jarcec Cecho commented on FLUME-1400:
-------------------------------------------
Hi Patrik,
there is already available some weak documentation for the tail in the user guide http://flume.apache.org/FlumeUserGuide.html#exec-source
I'm definitely open to the idea of extending it, so please do your investigation and put together notes. I'll more than happy to review them :-)
Jarcec
> Understand and document behavior of exec source when tailing rolling logs
> -------------------------------------------------------------------------
>
> Key: FLUME-1400
> URL: https://issues.apache.org/jira/browse/FLUME-1400
> Project: Flume
> Issue Type: Improvement
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
>
> There has been some discussion recently around the behavior of exec("tail -f") when tailing logs, if the log files are themselves being rolled and renamed by a writer process.
> The primary concern is whether there is a race condition or correctness problem related to tailing a log which is renamed.
> I am fairly certain that the tail binary, when passed -f, will deal with all of the ugliness of recognizing when a file is rolled and open the new "current" file as expected. However, we should both verify and document that behavior.
> Flume OG, which implemented its own tailing functionality, dealt with this ugliness by introducing increasing break-fix complexity to the tail source. I am hoping that by delegating to the "tail" binary we have avoided having to do this (though at the cost of OS portability). Nonetheless, I should confirm this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1400) Understand and document behavior of
exec source when tailing rolling logs
Posted by "Patrick Wendell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426254#comment-13426254 ]
Patrick Wendell commented on FLUME-1400:
----------------------------------------
I've looked closely at both the tail C source code and the flume user guide. I also talked with Mike extensively about this.
The race condition: the tail binary doesn't have this race condition, because it uses kernel event notification mechanisms to track file movements precisely. It is the case that a user will need to use "tail -F" to correctly track rolling logs, but that is consistent with our documentation.
IMHO, a much bigger issue that Mike and I identified is that if a Flume agent fails or restarts using the execSource(tail), there is near certainty of data loss. At a minimum I figured this should be made explicit in the User Guide.
I went to add this to the user guide and I found a big red warning box stating this rather explicitly that was already there.
With that in mind, I am closing as "not an issue". The only documentation merited by my exploration we already have :)
> Understand and document behavior of exec source when tailing rolling logs
> -------------------------------------------------------------------------
>
> Key: FLUME-1400
> URL: https://issues.apache.org/jira/browse/FLUME-1400
> Project: Flume
> Issue Type: Improvement
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
>
> There has been some discussion recently around the behavior of exec("tail -f") when tailing logs, if the log files are themselves being rolled and renamed by a writer process.
> The primary concern is whether there is a race condition or correctness problem related to tailing a log which is renamed.
> I am fairly certain that the tail binary, when passed -f, will deal with all of the ugliness of recognizing when a file is rolled and open the new "current" file as expected. However, we should both verify and document that behavior.
> Flume OG, which implemented its own tailing functionality, dealt with this ugliness by introducing increasing break-fix complexity to the tail source. I am hoping that by delegating to the "tail" binary we have avoided having to do this (though at the cost of OS portability). Nonetheless, I should confirm this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1400) Understand and document behavior of
exec source when tailing rolling logs
Posted by "Patrick Wendell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424259#comment-13424259 ]
Patrick Wendell commented on FLUME-1400:
----------------------------------------
Hey Jarek,
I definitely think this is something we should include in documentation with the exec source since it's a "gotcha".
Why don't I do some investigation and come up with something and then you can review? That way we have two independent looks at this.
> Understand and document behavior of exec source when tailing rolling logs
> -------------------------------------------------------------------------
>
> Key: FLUME-1400
> URL: https://issues.apache.org/jira/browse/FLUME-1400
> Project: Flume
> Issue Type: Improvement
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
>
> There has been some discussion recently around the behavior of exec("tail -f") when tailing logs, if the log files are themselves being rolled and renamed by a writer process.
> The primary concern is whether there is a race condition or correctness problem related to tailing a log which is renamed.
> I am fairly certain that the tail binary, when passed -f, will deal with all of the ugliness of recognizing when a file is rolled and open the new "current" file as expected. However, we should both verify and document that behavior.
> Flume OG, which implemented its own tailing functionality, dealt with this ugliness by introducing increasing break-fix complexity to the tail source. I am hoping that by delegating to the "tail" binary we have avoided having to do this (though at the cost of OS portability). Nonetheless, I should confirm this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1400) Understand and document behavior of
exec source when tailing rolling logs
Posted by "Mike Percy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424253#comment-13424253 ]
Mike Percy commented on FLUME-1400:
-----------------------------------
Someone also suggested at some point that the correct way to call tail via exec source would be "tail -F". But I haven't verified this personally.
> Understand and document behavior of exec source when tailing rolling logs
> -------------------------------------------------------------------------
>
> Key: FLUME-1400
> URL: https://issues.apache.org/jira/browse/FLUME-1400
> Project: Flume
> Issue Type: Improvement
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
>
> There has been some discussion recently around the behavior of exec("tail -f") when tailing logs, if the log files are themselves being rolled and renamed by a writer process.
> The primary concern is whether there is a race condition or correctness problem related to tailing a log which is renamed.
> I am fairly certain that the tail binary, when passed -f, will deal with all of the ugliness of recognizing when a file is rolled and open the new "current" file as expected. However, we should both verify and document that behavior.
> Flume OG, which implemented its own tailing functionality, dealt with this ugliness by introducing increasing break-fix complexity to the tail source. I am hoping that by delegating to the "tail" binary we have avoided having to do this (though at the cost of OS portability). Nonetheless, I should confirm this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1400) Understand and document behavior of
exec source when tailing rolling logs
Posted by "Jarek Jarcec Cecho (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426258#comment-13426258 ]
Jarek Jarcec Cecho commented on FLUME-1400:
-------------------------------------------
I told you that there is already something there :-)
> Understand and document behavior of exec source when tailing rolling logs
> -------------------------------------------------------------------------
>
> Key: FLUME-1400
> URL: https://issues.apache.org/jira/browse/FLUME-1400
> Project: Flume
> Issue Type: Improvement
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
>
> There has been some discussion recently around the behavior of exec("tail -f") when tailing logs, if the log files are themselves being rolled and renamed by a writer process.
> The primary concern is whether there is a race condition or correctness problem related to tailing a log which is renamed.
> I am fairly certain that the tail binary, when passed -f, will deal with all of the ugliness of recognizing when a file is rolled and open the new "current" file as expected. However, we should both verify and document that behavior.
> Flume OG, which implemented its own tailing functionality, dealt with this ugliness by introducing increasing break-fix complexity to the tail source. I am hoping that by delegating to the "tail" binary we have avoided having to do this (though at the cost of OS portability). Nonetheless, I should confirm this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1400) Understand and document behavior of
exec source when tailing rolling logs
Posted by "Jarek Jarcec Cecho (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424255#comment-13424255 ]
Jarek Jarcec Cecho commented on FLUME-1400:
-------------------------------------------
That someone was probably me. Small "f" (-f) will just follow one file - e.g. if someone will rotate the file, tail will still remain tailing original (moved) file. Big "f" (-F) will detect file move or rotation and will start tailing the new file (with name as was specified on the command line). Does that make sense or should I put together some examples?
Jarcec
> Understand and document behavior of exec source when tailing rolling logs
> -------------------------------------------------------------------------
>
> Key: FLUME-1400
> URL: https://issues.apache.org/jira/browse/FLUME-1400
> Project: Flume
> Issue Type: Improvement
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
>
> There has been some discussion recently around the behavior of exec("tail -f") when tailing logs, if the log files are themselves being rolled and renamed by a writer process.
> The primary concern is whether there is a race condition or correctness problem related to tailing a log which is renamed.
> I am fairly certain that the tail binary, when passed -f, will deal with all of the ugliness of recognizing when a file is rolled and open the new "current" file as expected. However, we should both verify and document that behavior.
> Flume OG, which implemented its own tailing functionality, dealt with this ugliness by introducing increasing break-fix complexity to the tail source. I am hoping that by delegating to the "tail" binary we have avoided having to do this (though at the cost of OS portability). Nonetheless, I should confirm this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira