You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Joey Echeverria (JIRA)" <ji...@apache.org> on 2012/05/02 23:36:50 UTC
[jira] [Created] (FLUME-1173) HDFSEventSink can leave orphaned .tmp
files
Joey Echeverria created FLUME-1173:
--------------------------------------
Summary: HDFSEventSink can leave orphaned .tmp files
Key: FLUME-1173
URL: https://issues.apache.org/jira/browse/FLUME-1173
Project: Flume
Issue Type: Bug
Components: Sinks+Sources
Affects Versions: v1.1.0
Reporter: Joey Echeverria
Currently HDFSEventSink only renames a .tmp file under the following conditions:
1) An attempt to write an event to the file coupled with hitting one of the three roll criteria
2) Stopping the HDFSEventSink closes all writers and thus renames all currently open .tmp files
3) If the number of max open files is hit, oler writers are closed, and thus their .tmp files get renamed
The problem that I see is if events are being routed by a path by timestamp, say day or hour, you should stop seeing any events written to that path after that timestamp has been hit. If this last event comes at an inopportune time, say 5 minutes after the last roll and you're rolling once an hour, then you could be left with an orphan .tmp file that won't get rolled until (2) or (3) hit. Unless you set the max number of open files low, that could be quite a long time.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (FLUME-1173) HDFSEventSink can leave orphaned
.tmp files
Posted by "Leslin (Hong Xiang Lin) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Leslin (Hong Xiang Lin) reassigned FLUME-1173:
----------------------------------------------
Assignee: (was: Leslin (Hong Xiang Lin))
Sorry for misoperation without any discussion. I will help to investigate this defect if no one have time for it. Thanks.
> HDFSEventSink can leave orphaned .tmp files
> -------------------------------------------
>
> Key: FLUME-1173
> URL: https://issues.apache.org/jira/browse/FLUME-1173
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.1.0
> Reporter: Joey Echeverria
>
> Currently HDFSEventSink only renames a .tmp file under the following conditions:
> 1) An attempt to write an event to the file coupled with hitting one of the three roll criteria
> 2) Stopping the HDFSEventSink closes all writers and thus renames all currently open .tmp files
> 3) If the number of max open files is hit, oler writers are closed, and thus their .tmp files get renamed
> The problem that I see is if events are being routed by a path by timestamp, say day or hour, you should stop seeing any events written to that path after that timestamp has been hit. If this last event comes at an inopportune time, say 5 minutes after the last roll and you're rolling once an hour, then you could be left with an orphan .tmp file that won't get rolled until (2) or (3) hit. Unless you set the max number of open files low, that could be quite a long time.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1173) HDFSEventSink can leave orphaned
.tmp files
Posted by "Joey Echeverria (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13266956#comment-13266956 ]
Joey Echeverria commented on FLUME-1173:
----------------------------------------
Yeah, that's a slightly different issue, but it's related. I added a related issue link so they can track each other.
> HDFSEventSink can leave orphaned .tmp files
> -------------------------------------------
>
> Key: FLUME-1173
> URL: https://issues.apache.org/jira/browse/FLUME-1173
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.1.0
> Reporter: Joey Echeverria
>
> Currently HDFSEventSink only renames a .tmp file under the following conditions:
> 1) An attempt to write an event to the file coupled with hitting one of the three roll criteria
> 2) Stopping the HDFSEventSink closes all writers and thus renames all currently open .tmp files
> 3) If the number of max open files is hit, oler writers are closed, and thus their .tmp files get renamed
> The problem that I see is if events are being routed by a path by timestamp, say day or hour, you should stop seeing any events written to that path after that timestamp has been hit. If this last event comes at an inopportune time, say 5 minutes after the last roll and you're rolling once an hour, then you could be left with an orphan .tmp file that won't get rolled until (2) or (3) hit. Unless you set the max number of open files low, that could be quite a long time.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1173) HDFSEventSink can leave orphaned
.tmp files
Posted by "Mike Percy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271186#comment-13271186 ]
Mike Percy commented on FLUME-1173:
-----------------------------------
Leslin, if you would like to investigate this issue and supply a patch please feel free. :) It's been somewhere in the middle of my TODO list for a while.
You can also check-in on #flume (Freenode IRC) or the flume-dev@ list if you run into anything you need help with.
> HDFSEventSink can leave orphaned .tmp files
> -------------------------------------------
>
> Key: FLUME-1173
> URL: https://issues.apache.org/jira/browse/FLUME-1173
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.1.0
> Reporter: Joey Echeverria
>
> Currently HDFSEventSink only renames a .tmp file under the following conditions:
> 1) An attempt to write an event to the file coupled with hitting one of the three roll criteria
> 2) Stopping the HDFSEventSink closes all writers and thus renames all currently open .tmp files
> 3) If the number of max open files is hit, oler writers are closed, and thus their .tmp files get renamed
> The problem that I see is if events are being routed by a path by timestamp, say day or hour, you should stop seeing any events written to that path after that timestamp has been hit. If this last event comes at an inopportune time, say 5 minutes after the last roll and you're rolling once an hour, then you could be left with an orphan .tmp file that won't get rolled until (2) or (3) hit. Unless you set the max number of open files low, that could be quite a long time.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (FLUME-1173) HDFSEventSink can leave orphaned
.tmp files
Posted by "Leslin (Hong Xiang Lin) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Leslin (Hong Xiang Lin) reassigned FLUME-1173:
----------------------------------------------
Assignee: Leslin (Hong Xiang Lin)
> HDFSEventSink can leave orphaned .tmp files
> -------------------------------------------
>
> Key: FLUME-1173
> URL: https://issues.apache.org/jira/browse/FLUME-1173
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.1.0
> Reporter: Joey Echeverria
> Assignee: Leslin (Hong Xiang Lin)
>
> Currently HDFSEventSink only renames a .tmp file under the following conditions:
> 1) An attempt to write an event to the file coupled with hitting one of the three roll criteria
> 2) Stopping the HDFSEventSink closes all writers and thus renames all currently open .tmp files
> 3) If the number of max open files is hit, oler writers are closed, and thus their .tmp files get renamed
> The problem that I see is if events are being routed by a path by timestamp, say day or hour, you should stop seeing any events written to that path after that timestamp has been hit. If this last event comes at an inopportune time, say 5 minutes after the last roll and you're rolling once an hour, then you could be left with an orphan .tmp file that won't get rolled until (2) or (3) hit. Unless you set the max number of open files low, that could be quite a long time.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1173) HDFSEventSink can leave orphaned
.tmp files
Posted by "Hari Shreedharan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13266953#comment-13266953 ]
Hari Shreedharan commented on FLUME-1173:
-----------------------------------------
Not sure if case (1) is covered by this, but case (2) is: FLUME-1163.
> HDFSEventSink can leave orphaned .tmp files
> -------------------------------------------
>
> Key: FLUME-1173
> URL: https://issues.apache.org/jira/browse/FLUME-1173
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.1.0
> Reporter: Joey Echeverria
>
> Currently HDFSEventSink only renames a .tmp file under the following conditions:
> 1) An attempt to write an event to the file coupled with hitting one of the three roll criteria
> 2) Stopping the HDFSEventSink closes all writers and thus renames all currently open .tmp files
> 3) If the number of max open files is hit, oler writers are closed, and thus their .tmp files get renamed
> The problem that I see is if events are being routed by a path by timestamp, say day or hour, you should stop seeing any events written to that path after that timestamp has been hit. If this last event comes at an inopportune time, say 5 minutes after the last roll and you're rolling once an hour, then you could be left with an orphan .tmp file that won't get rolled until (2) or (3) hit. Unless you set the max number of open files low, that could be quite a long time.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (FLUME-1173) HDFSEventSink can leave orphaned
.tmp files
Posted by "Mike Percy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mike Percy resolved FLUME-1173.
-------------------------------
Resolution: Duplicate
Fix Version/s: v1.2.0
Assignee: Mike Percy
> HDFSEventSink can leave orphaned .tmp files
> -------------------------------------------
>
> Key: FLUME-1173
> URL: https://issues.apache.org/jira/browse/FLUME-1173
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.1.0
> Reporter: Joey Echeverria
> Assignee: Mike Percy
> Fix For: v1.2.0
>
>
> Currently HDFSEventSink only renames a .tmp file under the following conditions:
> 1) An attempt to write an event to the file coupled with hitting one of the three roll criteria
> 2) Stopping the HDFSEventSink closes all writers and thus renames all currently open .tmp files
> 3) If the number of max open files is hit, oler writers are closed, and thus their .tmp files get renamed
> The problem that I see is if events are being routed by a path by timestamp, say day or hour, you should stop seeing any events written to that path after that timestamp has been hit. If this last event comes at an inopportune time, say 5 minutes after the last roll and you're rolling once an hour, then you could be left with an orphan .tmp file that won't get rolled until (2) or (3) hit. Unless you set the max number of open files low, that could be quite a long time.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1173) HDFSEventSink can leave orphaned
.tmp files
Posted by "Joey Echeverria (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13266933#comment-13266933 ]
Joey Echeverria commented on FLUME-1173:
----------------------------------------
I'm willing to let someone claim 'won't fix' if we update the documentation to tell you to set max number of open files to a low value if you're partitioning events by time. A better solution, in my book, would be to have a timer task run once every roll interval seconds to see if one of the roll criteria has been met. This means that you should see rolls after the last event has come but before you have to close the writer due to the number of open files.
> HDFSEventSink can leave orphaned .tmp files
> -------------------------------------------
>
> Key: FLUME-1173
> URL: https://issues.apache.org/jira/browse/FLUME-1173
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.1.0
> Reporter: Joey Echeverria
>
> Currently HDFSEventSink only renames a .tmp file under the following conditions:
> 1) An attempt to write an event to the file coupled with hitting one of the three roll criteria
> 2) Stopping the HDFSEventSink closes all writers and thus renames all currently open .tmp files
> 3) If the number of max open files is hit, oler writers are closed, and thus their .tmp files get renamed
> The problem that I see is if events are being routed by a path by timestamp, say day or hour, you should stop seeing any events written to that path after that timestamp has been hit. If this last event comes at an inopportune time, say 5 minutes after the last roll and you're rolling once an hour, then you could be left with an orphan .tmp file that won't get rolled until (2) or (3) hit. Unless you set the max number of open files low, that could be quite a long time.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (FLUME-1173) HDFSEventSink can leave orphaned .tmp
files
Posted by "Mike Percy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mike Percy closed FLUME-1173.
-----------------------------
> HDFSEventSink can leave orphaned .tmp files
> -------------------------------------------
>
> Key: FLUME-1173
> URL: https://issues.apache.org/jira/browse/FLUME-1173
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.1.0
> Reporter: Joey Echeverria
> Assignee: Mike Percy
> Fix For: v1.2.0
>
>
> Currently HDFSEventSink only renames a .tmp file under the following conditions:
> 1) An attempt to write an event to the file coupled with hitting one of the three roll criteria
> 2) Stopping the HDFSEventSink closes all writers and thus renames all currently open .tmp files
> 3) If the number of max open files is hit, oler writers are closed, and thus their .tmp files get renamed
> The problem that I see is if events are being routed by a path by timestamp, say day or hour, you should stop seeing any events written to that path after that timestamp has been hit. If this last event comes at an inopportune time, say 5 minutes after the last roll and you're rolling once an hour, then you could be left with an orphan .tmp file that won't get rolled until (2) or (3) hit. Unless you set the max number of open files low, that could be quite a long time.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira