You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by Balázs Donát Bessenyei <be...@apache.org> on 2016/11/14 09:19:25 UTC
Re: Review Request 51244: FLUME-2171: Add Interceptor to remove
headers from event
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51244/
-----------------------------------------------------------
(Updated Nov. 14, 2016, 9:19 a.m.)
Review request for Flume.
Bugs: FLUME-2171
https://issues.apache.org/jira/browse/FLUME-2171
Repository: flume-git
Description
-------
I found Flume OG's decorators to handle event headers useful and some to be missing from Flume NG. More specifically, we could have an interceptor to remove headers from an event.
Diffs (updated)
-----
flume-ng-core/src/main/java/org/apache/flume/interceptor/InterceptorType.java fe341e9
flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java PRE-CREATION
flume-ng-core/src/test/java/org/apache/flume/interceptor/RemoveHeaderInterceptorTest.java PRE-CREATION
flume-ng-doc/sphinx/FlumeUserGuide.rst cd76bb3
Diff: https://reviews.apache.org/r/51244/diff/
Testing
-------
All tests (besides the FLUME-2974-related ones) in flume-ng-core run successfully
I've used this config for manual testing:
a1.sources = r1
a1.sources.r1.type = netcat
a1.sources.r1.bind = 0.0.0.0
a1.sources.r1.port = 6666
a1.sources.r1.channels = c1
a1.channels = c1
a1.channels.c1.type = memory
a1.channels.c1.capacity = 10000
a1.channels.c1.transactionCapacity = 10000
a1.channels.c1.byteCapacityBufferPercentage = 20
a1.channels.c1.byteCapacity = 800000
a1.channels = c1
a1.sinks = k1
a1.sinks.k1.type = logger
a1.sinks.k1.channel = c1
a1.sources.r1.interceptors = i1 i2 i3
a1.sources.r1.interceptors.i1.type = timestamp
a1.sources.r1.interceptors.i2.type = host
a1.sources.r1.interceptors.i2.hostHeader = hostname
a1.sources.r1.interceptors.i3.type = remove_header
a1.sources.r1.interceptors.i3.with.name = timestamp
Thanks,
Bal�zs Don�t Bessenyei
Re: Review Request 51244: FLUME-2171: Add Interceptor to remove
headers from event
Posted by Balázs Donát Bessenyei <be...@apache.org>.
> On Nov. 14, 2016, 1:58 p.m., Jeff Holoman wrote:
> > flume-ng-doc/sphinx/FlumeUserGuide.rst, line 3928
> > <https://reviews.apache.org/r/51244/diff/8/?file=1562953#file1562953line3928>
> >
> > Do you think you could add an example of a couple of common match patterns? Might be helpful to some people.
I could not really come up with any common match patterns. If anyone has suggestions, I'd be very happy to include them.
- Bal�zs Don�t
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51244/#review155805
-----------------------------------------------------------
On Nov. 14, 2016, 9:19 a.m., Bal�zs Don�t Bessenyei wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51244/
> -----------------------------------------------------------
>
> (Updated Nov. 14, 2016, 9:19 a.m.)
>
>
> Review request for Flume.
>
>
> Bugs: FLUME-2171
> https://issues.apache.org/jira/browse/FLUME-2171
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> I found Flume OG's decorators to handle event headers useful and some to be missing from Flume NG. More specifically, we could have an interceptor to remove headers from an event.
>
>
> Diffs
> -----
>
> flume-ng-core/src/main/java/org/apache/flume/interceptor/InterceptorType.java fe341e9
> flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java PRE-CREATION
> flume-ng-core/src/test/java/org/apache/flume/interceptor/RemoveHeaderInterceptorTest.java PRE-CREATION
> flume-ng-doc/sphinx/FlumeUserGuide.rst cd76bb3
>
> Diff: https://reviews.apache.org/r/51244/diff/
>
>
> Testing
> -------
>
> All tests (besides the FLUME-2974-related ones) in flume-ng-core run successfully
>
> I've used this config for manual testing:
>
> a1.sources = r1
> a1.sources.r1.type = netcat
> a1.sources.r1.bind = 0.0.0.0
> a1.sources.r1.port = 6666
> a1.sources.r1.channels = c1
>
> a1.channels = c1
> a1.channels.c1.type = memory
> a1.channels.c1.capacity = 10000
> a1.channels.c1.transactionCapacity = 10000
> a1.channels.c1.byteCapacityBufferPercentage = 20
> a1.channels.c1.byteCapacity = 800000
>
> a1.channels = c1
> a1.sinks = k1
> a1.sinks.k1.type = logger
> a1.sinks.k1.channel = c1
>
>
> a1.sources.r1.interceptors = i1 i2 i3
> a1.sources.r1.interceptors.i1.type = timestamp
>
> a1.sources.r1.interceptors.i2.type = host
> a1.sources.r1.interceptors.i2.hostHeader = hostname
>
> a1.sources.r1.interceptors.i3.type = remove_header
> a1.sources.r1.interceptors.i3.with.name = timestamp
>
>
> Thanks,
>
> Bal�zs Don�t Bessenyei
>
>
Re: Review Request 51244: FLUME-2171: Add Interceptor to remove
headers from event
Posted by Jeff Holoman <je...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51244/#review155805
-----------------------------------------------------------
Fix it, then Ship it!
flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java (line 122)
<https://reviews.apache.org/r/51244/#comment225898>
nit: I prefer variable names that aren't one character, also on line 127
flume-ng-doc/sphinx/FlumeUserGuide.rst (line 3928)
<https://reviews.apache.org/r/51244/#comment225897>
Do you think you could add an example of a couple of common match patterns? Might be helpful to some people.
- Jeff Holoman
On Nov. 14, 2016, 9:19 a.m., Bal�zs Don�t Bessenyei wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51244/
> -----------------------------------------------------------
>
> (Updated Nov. 14, 2016, 9:19 a.m.)
>
>
> Review request for Flume.
>
>
> Bugs: FLUME-2171
> https://issues.apache.org/jira/browse/FLUME-2171
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> I found Flume OG's decorators to handle event headers useful and some to be missing from Flume NG. More specifically, we could have an interceptor to remove headers from an event.
>
>
> Diffs
> -----
>
> flume-ng-core/src/main/java/org/apache/flume/interceptor/InterceptorType.java fe341e9
> flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java PRE-CREATION
> flume-ng-core/src/test/java/org/apache/flume/interceptor/RemoveHeaderInterceptorTest.java PRE-CREATION
> flume-ng-doc/sphinx/FlumeUserGuide.rst cd76bb3
>
> Diff: https://reviews.apache.org/r/51244/diff/
>
>
> Testing
> -------
>
> All tests (besides the FLUME-2974-related ones) in flume-ng-core run successfully
>
> I've used this config for manual testing:
>
> a1.sources = r1
> a1.sources.r1.type = netcat
> a1.sources.r1.bind = 0.0.0.0
> a1.sources.r1.port = 6666
> a1.sources.r1.channels = c1
>
> a1.channels = c1
> a1.channels.c1.type = memory
> a1.channels.c1.capacity = 10000
> a1.channels.c1.transactionCapacity = 10000
> a1.channels.c1.byteCapacityBufferPercentage = 20
> a1.channels.c1.byteCapacity = 800000
>
> a1.channels = c1
> a1.sinks = k1
> a1.sinks.k1.type = logger
> a1.sinks.k1.channel = c1
>
>
> a1.sources.r1.interceptors = i1 i2 i3
> a1.sources.r1.interceptors.i1.type = timestamp
>
> a1.sources.r1.interceptors.i2.type = host
> a1.sources.r1.interceptors.i2.hostHeader = hostname
>
> a1.sources.r1.interceptors.i3.type = remove_header
> a1.sources.r1.interceptors.i3.with.name = timestamp
>
>
> Thanks,
>
> Bal�zs Don�t Bessenyei
>
>
Re: Review Request 51244: FLUME-2171: Add Interceptor to remove
headers from event
Posted by Balázs Donát Bessenyei <be...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51244/
-----------------------------------------------------------
(Updated Nov. 14, 2016, 7:32 p.m.)
Review request for Flume.
Bugs: FLUME-2171
https://issues.apache.org/jira/browse/FLUME-2171
Repository: flume-git
Description
-------
I found Flume OG's decorators to handle event headers useful and some to be missing from Flume NG. More specifically, we could have an interceptor to remove headers from an event.
Diffs (updated)
-----
flume-ng-core/src/main/java/org/apache/flume/interceptor/InterceptorType.java fe341e9
flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java PRE-CREATION
flume-ng-core/src/test/java/org/apache/flume/interceptor/RemoveHeaderInterceptorTest.java PRE-CREATION
flume-ng-doc/sphinx/FlumeUserGuide.rst cd76bb3
Diff: https://reviews.apache.org/r/51244/diff/
Testing
-------
All tests (besides the FLUME-2974-related ones) in flume-ng-core run successfully
I've used this config for manual testing:
a1.sources = r1
a1.sources.r1.type = netcat
a1.sources.r1.bind = 0.0.0.0
a1.sources.r1.port = 6666
a1.sources.r1.channels = c1
a1.channels = c1
a1.channels.c1.type = memory
a1.channels.c1.capacity = 10000
a1.channels.c1.transactionCapacity = 10000
a1.channels.c1.byteCapacityBufferPercentage = 20
a1.channels.c1.byteCapacity = 800000
a1.channels = c1
a1.sinks = k1
a1.sinks.k1.type = logger
a1.sinks.k1.channel = c1
a1.sources.r1.interceptors = i1 i2 i3
a1.sources.r1.interceptors.i1.type = timestamp
a1.sources.r1.interceptors.i2.type = host
a1.sources.r1.interceptors.i2.hostHeader = hostname
a1.sources.r1.interceptors.i3.type = remove_header
a1.sources.r1.interceptors.i3.with.name = timestamp
Thanks,
Bal�zs Don�t Bessenyei