You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by Jeff jlord <jl...@cloudera.com> on 2013/11/28 03:15:43 UTC
Review Request 15899: FLUME-2056 Allow SpoolDir to pass just the filename
that is the source of an event
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15899/
-----------------------------------------------------------
Review request for Flume, Hari Shreedharan and Mike Percy.
Repository: flume-git
Description
-------
Pretty simple patch which adds a boolean config basenameHeader to SpoolDirSource.
This will allow for loading the basename of the file that the event was created from in the headers.
Diffs
-----
flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java bd684ed
flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java 72c4059
flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySourceConfigurationConstants.java 7bfb0ee
flume-ng-core/src/test/java/org/apache/flume/source/TestSpoolDirectorySource.java 837cf15
flume-ng-doc/sphinx/FlumeUserGuide.rst 0f12427
Diff: https://reviews.apache.org/r/15899/diff/
Testing
-------
Added Unit Test.
Thanks,
Jeff jlord
Re: Review Request 15899: FLUME-2056 Allow SpoolDir to pass just the
filename that is the source of an event
Posted by Bruno MAHE <bm...@apache.org>.
> On Dec. 13, 2013, 3:32 a.m., Mike Percy wrote:
> > flume-ng-core/src/test/java/org/apache/flume/source/TestSpoolDirectorySource.java, line 126
> > <https://reviews.apache.org/r/15899/diff/1/?file=392258#file392258line126>
> >
> > This is racy and on a slow build machine this test might spuriously fail once in a while.
> >
> > Let's just watch the SourceCounter for the expected # of events taken before continuing.
> >
> > Something like:
> >
> > while (source.getSourceCounter().getEventAcceptedCount() < numExpectedEvents) {
> > Thread.sleep(10);
> > }
What about using a CountDownLatch instead of sleep?
- Bruno
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15899/#review30331
-----------------------------------------------------------
On Nov. 28, 2013, 2:15 a.m., Jeff jlord wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15899/
> -----------------------------------------------------------
>
> (Updated Nov. 28, 2013, 2:15 a.m.)
>
>
> Review request for Flume, Hari Shreedharan and Mike Percy.
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> Pretty simple patch which adds a boolean config basenameHeader to SpoolDirSource.
> This will allow for loading the basename of the file that the event was created from in the headers.
>
>
> Diffs
> -----
>
> flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java bd684ed
> flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java 72c4059
> flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySourceConfigurationConstants.java 7bfb0ee
> flume-ng-core/src/test/java/org/apache/flume/source/TestSpoolDirectorySource.java 837cf15
> flume-ng-doc/sphinx/FlumeUserGuide.rst 0f12427
>
> Diff: https://reviews.apache.org/r/15899/diff/
>
>
> Testing
> -------
>
> Added Unit Test.
>
>
> Thanks,
>
> Jeff jlord
>
>
Re: Review Request 15899: FLUME-2056 Allow SpoolDir to pass just the
filename that is the source of an event
Posted by Mike Percy <mp...@apache.org>.
> On Dec. 13, 2013, 3:32 a.m., Mike Percy wrote:
> > flume-ng-core/src/test/java/org/apache/flume/source/TestSpoolDirectorySource.java, line 126
> > <https://reviews.apache.org/r/15899/diff/1/?file=392258#file392258line126>
> >
> > This is racy and on a slow build machine this test might spuriously fail once in a while.
> >
> > Let's just watch the SourceCounter for the expected # of events taken before continuing.
> >
> > Something like:
> >
> > while (source.getSourceCounter().getEventAcceptedCount() < numExpectedEvents) {
> > Thread.sleep(10);
> > }
>
> Bruno MAHE wrote:
> What about using a CountDownLatch instead of sleep?
Not worth the complication IMHO, especially for a unit test, this is a single threaded test and a 10ms sleep is no big deal.
- Mike
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15899/#review30331
-----------------------------------------------------------
On Nov. 28, 2013, 2:15 a.m., Jeff jlord wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15899/
> -----------------------------------------------------------
>
> (Updated Nov. 28, 2013, 2:15 a.m.)
>
>
> Review request for Flume, Hari Shreedharan and Mike Percy.
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> Pretty simple patch which adds a boolean config basenameHeader to SpoolDirSource.
> This will allow for loading the basename of the file that the event was created from in the headers.
>
>
> Diffs
> -----
>
> flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java bd684ed
> flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java 72c4059
> flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySourceConfigurationConstants.java 7bfb0ee
> flume-ng-core/src/test/java/org/apache/flume/source/TestSpoolDirectorySource.java 837cf15
> flume-ng-doc/sphinx/FlumeUserGuide.rst 0f12427
>
> Diff: https://reviews.apache.org/r/15899/diff/
>
>
> Testing
> -------
>
> Added Unit Test.
>
>
> Thanks,
>
> Jeff jlord
>
>
Re: Review Request 15899: FLUME-2056 Allow SpoolDir to pass just the
filename that is the source of an event
Posted by Mike Percy <mp...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15899/#review30331
-----------------------------------------------------------
flume-ng-core/src/test/java/org/apache/flume/source/TestSpoolDirectorySource.java
<https://reviews.apache.org/r/15899/#comment58059>
This is racy and on a slow build machine this test might spuriously fail once in a while.
Let's just watch the SourceCounter for the expected # of events taken before continuing.
Something like:
while (source.getSourceCounter().getEventAcceptedCount() < numExpectedEvents) {
Thread.sleep(10);
}
- Mike Percy
On Nov. 28, 2013, 2:15 a.m., Jeff jlord wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15899/
> -----------------------------------------------------------
>
> (Updated Nov. 28, 2013, 2:15 a.m.)
>
>
> Review request for Flume, Hari Shreedharan and Mike Percy.
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> Pretty simple patch which adds a boolean config basenameHeader to SpoolDirSource.
> This will allow for loading the basename of the file that the event was created from in the headers.
>
>
> Diffs
> -----
>
> flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java bd684ed
> flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java 72c4059
> flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySourceConfigurationConstants.java 7bfb0ee
> flume-ng-core/src/test/java/org/apache/flume/source/TestSpoolDirectorySource.java 837cf15
> flume-ng-doc/sphinx/FlumeUserGuide.rst 0f12427
>
> Diff: https://reviews.apache.org/r/15899/diff/
>
>
> Testing
> -------
>
> Added Unit Test.
>
>
> Thanks,
>
> Jeff jlord
>
>
Re: Review Request 15899: FLUME-2056 Allow SpoolDir to pass just the
filename that is the source of an event
Posted by Jeff jlord <jl...@cloudera.com>.
> On Dec. 14, 2013, 10:01 a.m., Mike Percy wrote:
> > +1 LGTM. Please attach the patch to the JIRA.
Done.
- Jeff
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15899/#review30419
-----------------------------------------------------------
On Dec. 14, 2013, 12:57 a.m., Jeff jlord wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15899/
> -----------------------------------------------------------
>
> (Updated Dec. 14, 2013, 12:57 a.m.)
>
>
> Review request for Flume, Hari Shreedharan and Mike Percy.
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> Pretty simple patch which adds a boolean config basenameHeader to SpoolDirSource.
> This will allow for loading the basename of the file that the event was created from in the headers.
>
>
> Diffs
> -----
>
> flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java bd684ed
> flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java 0160215
> flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySourceConfigurationConstants.java a2befe8
> flume-ng-core/src/test/java/org/apache/flume/source/TestSpoolDirectorySource.java 9a546a5
> flume-ng-doc/sphinx/FlumeUserGuide.rst 335a991
>
> Diff: https://reviews.apache.org/r/15899/diff/
>
>
> Testing
> -------
>
> Added Unit Test.
>
>
> Thanks,
>
> Jeff jlord
>
>
Re: Review Request 15899: FLUME-2056 Allow SpoolDir to pass just the
filename that is the source of an event
Posted by Mike Percy <mp...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15899/#review30419
-----------------------------------------------------------
Ship it!
+1 LGTM. Please attach the patch to the JIRA.
- Mike Percy
On Dec. 14, 2013, 12:57 a.m., Jeff jlord wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15899/
> -----------------------------------------------------------
>
> (Updated Dec. 14, 2013, 12:57 a.m.)
>
>
> Review request for Flume, Hari Shreedharan and Mike Percy.
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> Pretty simple patch which adds a boolean config basenameHeader to SpoolDirSource.
> This will allow for loading the basename of the file that the event was created from in the headers.
>
>
> Diffs
> -----
>
> flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java bd684ed
> flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java 0160215
> flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySourceConfigurationConstants.java a2befe8
> flume-ng-core/src/test/java/org/apache/flume/source/TestSpoolDirectorySource.java 9a546a5
> flume-ng-doc/sphinx/FlumeUserGuide.rst 335a991
>
> Diff: https://reviews.apache.org/r/15899/diff/
>
>
> Testing
> -------
>
> Added Unit Test.
>
>
> Thanks,
>
> Jeff jlord
>
>
Re: Review Request 15899: FLUME-2056 Allow SpoolDir to pass just the
filename that is the source of an event
Posted by Jeff jlord <jl...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15899/
-----------------------------------------------------------
(Updated Dec. 14, 2013, 12:57 a.m.)
Review request for Flume, Hari Shreedharan and Mike Percy.
Changes
-------
Modified Test:
while (source.getSourceCounter().getEventAcceptedCount() < 8 ) {
Thread.sleep(10);
}
Repository: flume-git
Description
-------
Pretty simple patch which adds a boolean config basenameHeader to SpoolDirSource.
This will allow for loading the basename of the file that the event was created from in the headers.
Diffs (updated)
-----
flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java bd684ed
flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java 0160215
flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySourceConfigurationConstants.java a2befe8
flume-ng-core/src/test/java/org/apache/flume/source/TestSpoolDirectorySource.java 9a546a5
flume-ng-doc/sphinx/FlumeUserGuide.rst 335a991
Diff: https://reviews.apache.org/r/15899/diff/
Testing
-------
Added Unit Test.
Thanks,
Jeff jlord