You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Juhani Connolly (JIRA)" <ji...@apache.org> on 2012/10/19 05:50:05 UTC
[jira] [Created] (FLUME-1660) Close "idle" hdfs handles
Juhani Connolly created FLUME-1660:
--------------------------------------
Summary: Close "idle" hdfs handles
Key: FLUME-1660
URL: https://issues.apache.org/jira/browse/FLUME-1660
Project: Flume
Issue Type: Improvement
Affects Versions: v1.4.0
Reporter: Juhani Connolly
Assignee: Juhani Connolly
Priority: Minor
Fix For: v1.4.0
When hdfs paths are date dependent, many handles can get left open for a long time without anything happening to them.
Idea here is to watch the last update of each bucketWriter, and track idle writers, closing them once they pass a configured timeout
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1660) Close "idle" hdfs handles
Posted by "Brock Noland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brock Noland updated FLUME-1660:
--------------------------------
Fix Version/s: (was: v1.4.0)
v1.3.0
> Close "idle" hdfs handles
> -------------------------
>
> Key: FLUME-1660
> URL: https://issues.apache.org/jira/browse/FLUME-1660
> Project: Flume
> Issue Type: Improvement
> Affects Versions: v1.4.0
> Reporter: Juhani Connolly
> Assignee: Juhani Connolly
> Priority: Minor
> Fix For: v1.3.0
>
>
> When hdfs paths are date dependent, many handles can get left open for a long time without anything happening to them.
> Idea here is to watch the last update of each bucketWriter, and track idle writers, closing them once they pass a configured timeout
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1660) Close "idle" hdfs handles
Posted by "Juhani Connolly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Juhani Connolly updated FLUME-1660:
-----------------------------------
Attachment: (was: FLUME-1660-4.patch)
> Close "idle" hdfs handles
> -------------------------
>
> Key: FLUME-1660
> URL: https://issues.apache.org/jira/browse/FLUME-1660
> Project: Flume
> Issue Type: Improvement
> Affects Versions: v1.4.0
> Reporter: Juhani Connolly
> Assignee: Juhani Connolly
> Priority: Minor
> Fix For: v1.4.0
>
> Attachments: FLUME-1660-7.patch
>
>
> When hdfs paths are date dependent, many handles can get left open for a long time without anything happening to them.
> Idea here is to watch the last update of each bucketWriter, and track idle writers, closing them once they pass a configured timeout
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1660) Close "idle" hdfs handles
Posted by "Juhani Connolly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Juhani Connolly updated FLUME-1660:
-----------------------------------
Attachment: FLUME-1660-7.patch
Latest patch with unit tests and corner case fix
> Close "idle" hdfs handles
> -------------------------
>
> Key: FLUME-1660
> URL: https://issues.apache.org/jira/browse/FLUME-1660
> Project: Flume
> Issue Type: Improvement
> Affects Versions: v1.4.0
> Reporter: Juhani Connolly
> Assignee: Juhani Connolly
> Priority: Minor
> Fix For: v1.4.0
>
> Attachments: FLUME-1660-7.patch
>
>
> When hdfs paths are date dependent, many handles can get left open for a long time without anything happening to them.
> Idea here is to watch the last update of each bucketWriter, and track idle writers, closing them once they pass a configured timeout
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1660) Close "idle" hdfs handles
Posted by "Brock Noland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brock Noland updated FLUME-1660:
--------------------------------
Fix Version/s: (was: v1.3.0)
v1.4.0
> Close "idle" hdfs handles
> -------------------------
>
> Key: FLUME-1660
> URL: https://issues.apache.org/jira/browse/FLUME-1660
> Project: Flume
> Issue Type: Improvement
> Affects Versions: v1.4.0
> Reporter: Juhani Connolly
> Assignee: Juhani Connolly
> Priority: Minor
> Fix For: v1.4.0
>
> Attachments: FLUME-1660-4.patch
>
>
> When hdfs paths are date dependent, many handles can get left open for a long time without anything happening to them.
> Idea here is to watch the last update of each bucketWriter, and track idle writers, closing them once they pass a configured timeout
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1660) Close "idle" hdfs handles
Posted by "Juhani Connolly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Juhani Connolly updated FLUME-1660:
-----------------------------------
Attachment: FLUME-1660-4.patch
Adding patch from RB
> Close "idle" hdfs handles
> -------------------------
>
> Key: FLUME-1660
> URL: https://issues.apache.org/jira/browse/FLUME-1660
> Project: Flume
> Issue Type: Improvement
> Affects Versions: v1.4.0
> Reporter: Juhani Connolly
> Assignee: Juhani Connolly
> Priority: Minor
> Fix For: v1.3.0
>
> Attachments: FLUME-1660-4.patch
>
>
> When hdfs paths are date dependent, many handles can get left open for a long time without anything happening to them.
> Idea here is to watch the last update of each bucketWriter, and track idle writers, closing them once they pass a configured timeout
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1660) Close "idle" hdfs handles
Posted by "Juhani Connolly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13479604#comment-13479604 ]
Juhani Connolly commented on FLUME-1660:
----------------------------------------
We use roll interval, however it is not a fix applicable in all situations, and it is clumsy. For example, if people use size based rolling along with filenames that are named by header variables, they may or may not be open for an arbitrary amount of time.
Closing idle files on a timer seems to me a far more elegant solution than forcing users to guess an arbitrary time after which to roll or to limit open files(which close in order of oldest, even if the oldest is still active)
I don't really think having the extra option hurts so long as documentation is good. I'm setting the default setting to inactive
> Close "idle" hdfs handles
> -------------------------
>
> Key: FLUME-1660
> URL: https://issues.apache.org/jira/browse/FLUME-1660
> Project: Flume
> Issue Type: Improvement
> Affects Versions: v1.4.0
> Reporter: Juhani Connolly
> Assignee: Juhani Connolly
> Priority: Minor
> Fix For: v1.4.0
>
>
> When hdfs paths are date dependent, many handles can get left open for a long time without anything happening to them.
> Idea here is to watch the last update of each bucketWriter, and track idle writers, closing them once they pass a configured timeout
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1660) Close "idle" hdfs handles
Posted by "Brock Noland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13500319#comment-13500319 ]
Brock Noland commented on FLUME-1660:
-------------------------------------
Committed to 1.3.0
> Close "idle" hdfs handles
> -------------------------
>
> Key: FLUME-1660
> URL: https://issues.apache.org/jira/browse/FLUME-1660
> Project: Flume
> Issue Type: Improvement
> Affects Versions: v1.4.0
> Reporter: Juhani Connolly
> Assignee: Juhani Connolly
> Priority: Minor
> Fix For: v1.3.0
>
> Attachments: FLUME-1660-7.patch
>
>
> When hdfs paths are date dependent, many handles can get left open for a long time without anything happening to them.
> Idea here is to watch the last update of each bucketWriter, and track idle writers, closing them once they pass a configured timeout
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1660) Close "idle" hdfs handles
Posted by "Brock Noland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brock Noland updated FLUME-1660:
--------------------------------
Fix Version/s: (was: v1.4.0)
v1.3.0
> Close "idle" hdfs handles
> -------------------------
>
> Key: FLUME-1660
> URL: https://issues.apache.org/jira/browse/FLUME-1660
> Project: Flume
> Issue Type: Improvement
> Affects Versions: v1.4.0
> Reporter: Juhani Connolly
> Assignee: Juhani Connolly
> Priority: Minor
> Fix For: v1.3.0
>
> Attachments: FLUME-1660-7.patch
>
>
> When hdfs paths are date dependent, many handles can get left open for a long time without anything happening to them.
> Idea here is to watch the last update of each bucketWriter, and track idle writers, closing them once they pass a configured timeout
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1660) Close "idle" hdfs handles
Posted by "Hari Shreedharan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13479611#comment-13479611 ]
Hari Shreedharan commented on FLUME-1660:
-----------------------------------------
Thanks for the explanation, Juhani. Makes sense. Thanks!
> Close "idle" hdfs handles
> -------------------------
>
> Key: FLUME-1660
> URL: https://issues.apache.org/jira/browse/FLUME-1660
> Project: Flume
> Issue Type: Improvement
> Affects Versions: v1.4.0
> Reporter: Juhani Connolly
> Assignee: Juhani Connolly
> Priority: Minor
> Fix For: v1.4.0
>
>
> When hdfs paths are date dependent, many handles can get left open for a long time without anything happening to them.
> Idea here is to watch the last update of each bucketWriter, and track idle writers, closing them once they pass a configured timeout
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1660) Close "idle" hdfs handles
Posted by "Hari Shreedharan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13479599#comment-13479599 ]
Hari Shreedharan commented on FLUME-1660:
-----------------------------------------
Juhani,
I must admit I did not follow much of the discussion that happened on FLUME-1350 - and now there is too much to read. Why can't we really use the rollInterval for this? Once the rollInterval passes, don't we close the file? I am just concerned that we really end up having multiple configs which do almost the same thing. Or is this so that we can keep the rollInterval separate from the interval to consider a directory as "no longer written to?"
> Close "idle" hdfs handles
> -------------------------
>
> Key: FLUME-1660
> URL: https://issues.apache.org/jira/browse/FLUME-1660
> Project: Flume
> Issue Type: Improvement
> Affects Versions: v1.4.0
> Reporter: Juhani Connolly
> Assignee: Juhani Connolly
> Priority: Minor
> Fix For: v1.4.0
>
>
> When hdfs paths are date dependent, many handles can get left open for a long time without anything happening to them.
> Idea here is to watch the last update of each bucketWriter, and track idle writers, closing them once they pass a configured timeout
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira