You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@flume.apache.org by Muhammad Ehsan ul Haque <m....@gmail.com> on 2014/02/08 12:42:30 UTC

Re: Review Request 17867: FLUME-2309 Adds consume order feature in the spooling directory source to allow files to be consumed in the order of oldest, youngest or random.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17867/
-----------------------------------------------------------

(Updated Feb. 8, 2014, 11:42 a.m.)

Review request for Flume and Hari Shreedharan.

Changes
-------

Updated summary to include JIRA Issue ID.
Updated diff to become visible in the show diff.

Summary (updated)
-----------------

FLUME-2309 Adds consume order feature in the spooling directory source to allow files to be consumed in the order of oldest, youngest or random.

Repository: flume-git

Description
-------

This patch provides.
* A consume order feature in the Spooling directory source, which will allow the users to explicitly state in which order; oldest, youngest or randomly files should be consumed from the spooling directory.
* Fixes the old implementation of selecting the file from spooling directory. Previously, each file to be consumed was selected by sorting, which might become extremly time consuming if there are many files (of the order of 10K or more). The new implementation instead do a linear scan in case when the consume order is oldest or youngest.
* Updates the Flume user guide accordingly.

Diffs (updated)
-----

flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java a88ed6e
flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java f42ed2d
flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySourceConfigurationConstants.java 83522c0
flume-ng-core/src/test/java/org/apache/flume/client/avro/TestReliableSpoolingFileEventReader.java 9d708c1
flume-ng-core/src/test/java/org/apache/flume/source/TestSpoolDirectorySource.java 503ab4d
flume-ng-doc/sphinx/FlumeUserGuide.rst d120a74

Diff: https://reviews.apache.org/r/17867/diff/

Testing
-------

Tested without specifying a consume order, by default oldest was used.
Tested with specifying oldest, youngest and random consume order.
Tested with large number of files, random order is fastest.

Thanks,

Muhammad Ehsan ul Haque