You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by Mike Percy <mp...@apache.org> on 2013/08/10 04:33:18 UTC
Review Request 13463: FLUME-2157. Spool directory source does not shut down
correctly when Flume is reconfigured.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/13463/
-----------------------------------------------------------
Review request for Flume.
Bugs: FLUME-2157
https://issues.apache.org/jira/browse/FLUME-2157
Repository: flume-git
Description
-------
Spool directory source does not shut down correctly when Flume is reconfigured. This can lead to all sorts of problems stemming from the fact that we have threads running which should not be. In the logs, it looks like multiple spool directory sources are running on the same directory, even though the config may say otherwise.
Diffs
-----
flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java f82fe1f
flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java 7145580
flume-ng-core/src/test/java/org/apache/flume/source/TestSpoolDirectorySource.java 652d2a2
flume-ng-tests/src/test/java/org/apache/flume/test/agent/TestSpooldirSource.java PRE-CREATION
flume-ng-tests/src/test/java/org/apache/flume/test/util/StagedInstall.java bc58340
Diff: https://reviews.apache.org/r/13463/diff/
Testing
-------
Added a unit test for this specific case and also an integration test that simply runs an agent with many concurrent spool directory sources.
Thanks,
Mike Percy
Re: Review Request 13463: FLUME-2157. Spool directory source does not shut
down correctly when Flume is reconfigured.
Posted by Roshan Naik <ro...@hortonworks.com>.
> On Aug. 12, 2013, 6:18 p.m., Roshan Naik wrote:
> > In SpoolDirectorySource.shitdown(), it might be useful to add INFO or DEBUG level log indicating that it is waiting ... before entering a potentially 10 second long wait for the executor to wind up.
> >
> > Looks good overall.
of course i meant SpoolDirectorySource.shutdown() :-)
- Roshan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/13463/#review25018
-----------------------------------------------------------
On Aug. 10, 2013, 2:33 a.m., Mike Percy wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/13463/
> -----------------------------------------------------------
>
> (Updated Aug. 10, 2013, 2:33 a.m.)
>
>
> Review request for Flume.
>
>
> Bugs: FLUME-2157
> https://issues.apache.org/jira/browse/FLUME-2157
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> Spool directory source does not shut down correctly when Flume is reconfigured. This can lead to all sorts of problems stemming from the fact that we have threads running which should not be. In the logs, it looks like multiple spool directory sources are running on the same directory, even though the config may say otherwise.
>
>
> Diffs
> -----
>
> flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java f82fe1f
> flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java 7145580
> flume-ng-core/src/test/java/org/apache/flume/source/TestSpoolDirectorySource.java 652d2a2
> flume-ng-tests/src/test/java/org/apache/flume/test/agent/TestSpooldirSource.java PRE-CREATION
> flume-ng-tests/src/test/java/org/apache/flume/test/util/StagedInstall.java bc58340
>
> Diff: https://reviews.apache.org/r/13463/diff/
>
>
> Testing
> -------
>
> Added a unit test for this specific case and also an integration test that simply runs an agent with many concurrent spool directory sources.
>
>
> Thanks,
>
> Mike Percy
>
>
Re: Review Request 13463: FLUME-2157. Spool directory source does not shut
down correctly when Flume is reconfigured.
Posted by Roshan Naik <ro...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/13463/#review25018
-----------------------------------------------------------
In SpoolDirectorySource.shitdown(), it might be useful to add INFO or DEBUG level log indicating that it is waiting ... before entering a potentially 10 second long wait for the executor to wind up.
Looks good overall.
- Roshan Naik
On Aug. 10, 2013, 2:33 a.m., Mike Percy wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/13463/
> -----------------------------------------------------------
>
> (Updated Aug. 10, 2013, 2:33 a.m.)
>
>
> Review request for Flume.
>
>
> Bugs: FLUME-2157
> https://issues.apache.org/jira/browse/FLUME-2157
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> Spool directory source does not shut down correctly when Flume is reconfigured. This can lead to all sorts of problems stemming from the fact that we have threads running which should not be. In the logs, it looks like multiple spool directory sources are running on the same directory, even though the config may say otherwise.
>
>
> Diffs
> -----
>
> flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java f82fe1f
> flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java 7145580
> flume-ng-core/src/test/java/org/apache/flume/source/TestSpoolDirectorySource.java 652d2a2
> flume-ng-tests/src/test/java/org/apache/flume/test/agent/TestSpooldirSource.java PRE-CREATION
> flume-ng-tests/src/test/java/org/apache/flume/test/util/StagedInstall.java bc58340
>
> Diff: https://reviews.apache.org/r/13463/diff/
>
>
> Testing
> -------
>
> Added a unit test for this specific case and also an integration test that simply runs an agent with many concurrent spool directory sources.
>
>
> Thanks,
>
> Mike Percy
>
>