You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by Balázs Donát Bessenyei <be...@cloudera.com> on 2016/08/19 16:42:50 UTC
Review Request 51244: FLUME-2171: Add Interceptor to remove headers
from event
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51244/
-----------------------------------------------------------
Review request for Flume.
Repository: flume-git
Description
-------
I found Flume OG's decorators to handle event headers useful and some to be missing from Flume NG. More specifically, we could have an interceptor to remove headers from an event.
Diffs
-----
c/flume-ng-core/src/main/java/org/apache/flume/interceptor/InterceptorType.java fe341e9
c/flume-ng-doc/sphinx/FlumeUserGuide.rst 1c15f1e
w/flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java PRE-CREATION
w/flume-ng-core/src/test/java/org/apache/flume/interceptor/RemoveHeaderInterceptorTest.java PRE-CREATION
Diff: https://reviews.apache.org/r/51244/diff/
Testing
-------
All tests (besides the FLUME-2974-related ones) in flume-ng-core run successfully
I've used this config for manual testing:
a1.sources = r1
a1.sources.r1.type = netcat
a1.sources.r1.bind = 0.0.0.0
a1.sources.r1.port = 6666
a1.sources.r1.channels = c1
a1.channels = c1
a1.channels.c1.type = memory
a1.channels.c1.capacity = 10000
a1.channels.c1.transactionCapacity = 10000
a1.channels.c1.byteCapacityBufferPercentage = 20
a1.channels.c1.byteCapacity = 800000
a1.channels = c1
a1.sinks = k1
a1.sinks.k1.type = logger
a1.sinks.k1.channel = c1
a1.sources.r1.interceptors = i1 i2 i3
a1.sources.r1.interceptors.i1.type = timestamp
a1.sources.r1.interceptors.i2.type = host
a1.sources.r1.interceptors.i2.hostHeader = hostname
a1.sources.r1.interceptors.i3.type = remove_header
a1.sources.r1.interceptors.i3.with.name = timestamp
Thanks,
Bal�zs Don�t Bessenyei
Re: Review Request 51244: FLUME-2171: Add Interceptor to remove
headers from event
Posted by Balázs Donát Bessenyei <be...@cloudera.com>.
> On Aug. 23, 2016, 11:05 a.m., Denes Arvay wrote:
> > flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java, lines 63-65
> > <https://reviews.apache.org/r/51244/diff/5/?file=1481261#file1481261line63>
> >
> > why `protected`? `private` should work too.
This (and a few similar ones) were actually fixed previously in #1. I have uploaded an outdated revision by mistake.
> On Aug. 23, 2016, 11:05 a.m., Denes Arvay wrote:
> > flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java, line 121
> > <https://reviews.apache.org/r/51244/diff/5/?file=1481261#file1481261line121>
> >
> > Please keep an eye on FLUME-2954 as logging the event itself might be considered not safe according to that issue.
There is an ongoing discussion about this on the mailing list. I'll change these lines when there is an agreement.
> On Aug. 23, 2016, 11:05 a.m., Denes Arvay wrote:
> > flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java, line 141
> > <https://reviews.apache.org/r/51244/diff/5/?file=1481261#file1481261line141>
> >
> > Please keep an eye on FLUME-2954 as logging the event itself might be considered not safe according to that issue.
There is an ongoing discussion about this on the mailing list. I'll change these lines when there is an agreement.
- Bal�zs Don�t
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51244/#review146503
-----------------------------------------------------------
On Aug. 23, 2016, 2:43 p.m., Bal�zs Don�t Bessenyei wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51244/
> -----------------------------------------------------------
>
> (Updated Aug. 23, 2016, 2:43 p.m.)
>
>
> Review request for Flume.
>
>
> Bugs: FLUME-2171
> https://issues.apache.org/jira/browse/FLUME-2171
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> I found Flume OG's decorators to handle event headers useful and some to be missing from Flume NG. More specifically, we could have an interceptor to remove headers from an event.
>
>
> Diffs
> -----
>
> c/flume-ng-core/src/main/java/org/apache/flume/interceptor/InterceptorType.java fe341e9
> c/flume-ng-doc/sphinx/FlumeUserGuide.rst 5e677c6
> w/flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java PRE-CREATION
> w/flume-ng-core/src/test/java/org/apache/flume/interceptor/RemoveHeaderInterceptorTest.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/51244/diff/
>
>
> Testing
> -------
>
> All tests (besides the FLUME-2974-related ones) in flume-ng-core run successfully
>
> I've used this config for manual testing:
>
> a1.sources = r1
> a1.sources.r1.type = netcat
> a1.sources.r1.bind = 0.0.0.0
> a1.sources.r1.port = 6666
> a1.sources.r1.channels = c1
>
> a1.channels = c1
> a1.channels.c1.type = memory
> a1.channels.c1.capacity = 10000
> a1.channels.c1.transactionCapacity = 10000
> a1.channels.c1.byteCapacityBufferPercentage = 20
> a1.channels.c1.byteCapacity = 800000
>
> a1.channels = c1
> a1.sinks = k1
> a1.sinks.k1.type = logger
> a1.sinks.k1.channel = c1
>
>
> a1.sources.r1.interceptors = i1 i2 i3
> a1.sources.r1.interceptors.i1.type = timestamp
>
> a1.sources.r1.interceptors.i2.type = host
> a1.sources.r1.interceptors.i2.hostHeader = hostname
>
> a1.sources.r1.interceptors.i3.type = remove_header
> a1.sources.r1.interceptors.i3.with.name = timestamp
>
>
> Thanks,
>
> Bal�zs Don�t Bessenyei
>
>
Re: Review Request 51244: FLUME-2171: Add Interceptor to remove
headers from event
Posted by Denes Arvay <de...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51244/#review146503
-----------------------------------------------------------
flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java (line 49)
<https://reviews.apache.org/r/51244/#comment212999>
`matching` missing
flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java (line 53)
<https://reviews.apache.org/r/51244/#comment213005>
This pattern (defining interfaces to hold constants only) is considered bad practice. In this particular case it'd be better to put all the constants to the outer class as package private members (maybe prefixed with `CONFIG_`).
I don't know whether we have any general pattern in flume for this, but looking at the other interceptor implementations I don't think so.
* https://books.google.hu/books?id=ka2VUBqHiWkC&lpg=PA98&ots=yZJgRiv5R2&pg=PA98#v=onepage&q&f=false
flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java (lines 63 - 65)
<https://reviews.apache.org/r/51244/#comment213004>
why `protected`? `private` should work too.
flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java (line 121)
<https://reviews.apache.org/r/51244/#comment213006>
Please keep an eye on FLUME-2954 as logging the event itself might be considered not safe according to that issue.
flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java (line 123)
<https://reviews.apache.org/r/51244/#comment213000>
Misleading comment: not *otherwise* but *also*
flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java (line 136)
<https://reviews.apache.org/r/51244/#comment213003>
If the current key was in the `fromList` and also matches the regex the subsequent `it.remove()` call will throw an `IllegalStateException`.
Changing the `if` in line 133 to `else if` should solve this issue.
flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java (line 141)
<https://reviews.apache.org/r/51244/#comment213007>
Please keep an eye on FLUME-2954 as logging the event itself might be considered not safe according to that issue.
- Denes Arvay
On Aug. 22, 2016, 11:13 p.m., Bal�zs Don�t Bessenyei wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51244/
> -----------------------------------------------------------
>
> (Updated Aug. 22, 2016, 11:13 p.m.)
>
>
> Review request for Flume.
>
>
> Bugs: FLUME-2171
> https://issues.apache.org/jira/browse/FLUME-2171
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> I found Flume OG's decorators to handle event headers useful and some to be missing from Flume NG. More specifically, we could have an interceptor to remove headers from an event.
>
>
> Diffs
> -----
>
> flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java PRE-CREATION
> flume-ng-core/src/test/java/org/apache/flume/interceptor/RemoveHeaderInterceptorTest.java PRE-CREATION
> flume-ng-doc/sphinx/FlumeUserGuide.rst 5e677c6
>
> Diff: https://reviews.apache.org/r/51244/diff/
>
>
> Testing
> -------
>
> All tests (besides the FLUME-2974-related ones) in flume-ng-core run successfully
>
> I've used this config for manual testing:
>
> a1.sources = r1
> a1.sources.r1.type = netcat
> a1.sources.r1.bind = 0.0.0.0
> a1.sources.r1.port = 6666
> a1.sources.r1.channels = c1
>
> a1.channels = c1
> a1.channels.c1.type = memory
> a1.channels.c1.capacity = 10000
> a1.channels.c1.transactionCapacity = 10000
> a1.channels.c1.byteCapacityBufferPercentage = 20
> a1.channels.c1.byteCapacity = 800000
>
> a1.channels = c1
> a1.sinks = k1
> a1.sinks.k1.type = logger
> a1.sinks.k1.channel = c1
>
>
> a1.sources.r1.interceptors = i1 i2 i3
> a1.sources.r1.interceptors.i1.type = timestamp
>
> a1.sources.r1.interceptors.i2.type = host
> a1.sources.r1.interceptors.i2.hostHeader = hostname
>
> a1.sources.r1.interceptors.i3.type = remove_header
> a1.sources.r1.interceptors.i3.with.name = timestamp
>
>
> Thanks,
>
> Bal�zs Don�t Bessenyei
>
>
Re: Review Request 51244: FLUME-2171: Add Interceptor to remove
headers from event
Posted by Balázs Donát Bessenyei <be...@cloudera.com>.
> On Aug. 24, 2016, 4:30 p.m., Attila Simon wrote:
> > w/flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java, lines 52-56
> > <https://reviews.apache.org/r/51244/diff/6/?file=1482094#file1482094line52>
> >
> > Have you considered moving these out to a RemoveHeaderInterceptorConstants final class (not interface as pointed out earlier)? Additional javadoc to them wouldn't uglify this class then.
With only these 5 fields, I think this way makes it easier to navigate code. Is there any other benefit to moving them?
> On Aug. 24, 2016, 4:30 p.m., Attila Simon wrote:
> > w/flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java, lines 66-73
> > <https://reviews.apache.org/r/51244/diff/6/?file=1482094#file1482094line66>
> >
> > nit: there is not anon-inner class involved here so final keyword usage on function arguments is not encouraged. (I have no hard opinion on this but received this feedback earlier)
According to http://stackoverflow.com/a/502252 , it's a *nice* thing, however not necessary. I can remove them, although I think it's fine if they remain.
> On Aug. 24, 2016, 4:30 p.m., Attila Simon wrote:
> > w/flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java, line 120
> > <https://reviews.apache.org/r/51244/diff/6/?file=1482094#file1482094line120>
> >
> > I know this has been already pointed out, I would vote for not printing out the event only the removed headers. (Or protect with LogRawDataUtil)
LogRawDataUtil is unfortunately not yet in the codebase, however we can change this if required in the future.
On trace level it could be a useful thing to have the event logged for now.
> On Aug. 24, 2016, 4:30 p.m., Attila Simon wrote:
> > w/flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java, lines 164-168
> > <https://reviews.apache.org/r/51244/diff/6/?file=1482094#file1482094line164>
> >
> > nit: LOG.debug("Creating RemoveHeaderInterceptor with: withName={}, fromList={}, " +
> > "listSeparator={}, matchRegex={}", new String[]{
> > withName, fromList, listSeparator, matchRegex.toString()})
Fixed.
> On Aug. 24, 2016, 4:30 p.m., Attila Simon wrote:
> > w/flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java, line 139
> > <https://reviews.apache.org/r/51244/diff/6/?file=1482094#file1482094line139>
> >
> > Please don't log out the whole Event for each removed header. Also please collect all the headers which were removed and print them out as a single log entry per Event.
I think there is a benefit to keeping the event in the log, but I changed the code to collect the headers for logging
If you think the whole event logging is a blocking issue, please re-open
> On Aug. 24, 2016, 4:30 p.m., Attila Simon wrote:
> > w/flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java, lines 124-130
> > <https://reviews.apache.org/r/51244/diff/6/?file=1482094#file1482094line124>
> >
> > Iterating through the whole header keyset is inevitable if matching regexp was required. But if hasn't been set then this implementation looses the benefit coming from HashMap O(1) lookup complexity. Instead of iterating over the fromList and for each item headermap is checked, it iterates over each key from the map and then check fromList whether it contains it (fromList is backed by a sorted array as pointed out by javadocs of com.google.common.collect.ImmutableSortedSet which has poorer lookup performance than HashMap)
Changed the implementation for fromList to HashSet for better performance
- Bal�zs Don�t
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51244/#review146665
-----------------------------------------------------------
On Aug. 23, 2016, 2:43 p.m., Bal�zs Don�t Bessenyei wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51244/
> -----------------------------------------------------------
>
> (Updated Aug. 23, 2016, 2:43 p.m.)
>
>
> Review request for Flume.
>
>
> Bugs: FLUME-2171
> https://issues.apache.org/jira/browse/FLUME-2171
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> I found Flume OG's decorators to handle event headers useful and some to be missing from Flume NG. More specifically, we could have an interceptor to remove headers from an event.
>
>
> Diffs
> -----
>
> c/flume-ng-core/src/main/java/org/apache/flume/interceptor/InterceptorType.java fe341e9
> c/flume-ng-doc/sphinx/FlumeUserGuide.rst 5e677c6
> w/flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java PRE-CREATION
> w/flume-ng-core/src/test/java/org/apache/flume/interceptor/RemoveHeaderInterceptorTest.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/51244/diff/
>
>
> Testing
> -------
>
> All tests (besides the FLUME-2974-related ones) in flume-ng-core run successfully
>
> I've used this config for manual testing:
>
> a1.sources = r1
> a1.sources.r1.type = netcat
> a1.sources.r1.bind = 0.0.0.0
> a1.sources.r1.port = 6666
> a1.sources.r1.channels = c1
>
> a1.channels = c1
> a1.channels.c1.type = memory
> a1.channels.c1.capacity = 10000
> a1.channels.c1.transactionCapacity = 10000
> a1.channels.c1.byteCapacityBufferPercentage = 20
> a1.channels.c1.byteCapacity = 800000
>
> a1.channels = c1
> a1.sinks = k1
> a1.sinks.k1.type = logger
> a1.sinks.k1.channel = c1
>
>
> a1.sources.r1.interceptors = i1 i2 i3
> a1.sources.r1.interceptors.i1.type = timestamp
>
> a1.sources.r1.interceptors.i2.type = host
> a1.sources.r1.interceptors.i2.hostHeader = hostname
>
> a1.sources.r1.interceptors.i3.type = remove_header
> a1.sources.r1.interceptors.i3.with.name = timestamp
>
>
> Thanks,
>
> Bal�zs Don�t Bessenyei
>
>
Re: Review Request 51244: FLUME-2171: Add Interceptor to remove
headers from event
Posted by Attila Simon <sa...@cloudera.com>.
> On Aug. 24, 2016, 4:30 p.m., Attila Simon wrote:
> > w/flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java, line 139
> > <https://reviews.apache.org/r/51244/diff/6/?file=1482094#file1482094line139>
> >
> > Please don't log out the whole Event for each removed header. Also please collect all the headers which were removed and print them out as a single log entry per Event.
>
> Bal�zs Don�t Bessenyei wrote:
> I think there is a benefit to keeping the event in the log, but I changed the code to collect the headers for logging
>
> If you think the whole event logging is a blocking issue, please re-open
Fair enough. Then could you please consider wrapping the logging of event with a org.apache.flume.conf.LogPrivacyUtil#allowLogRawData() check?
- Attila
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51244/#review146665
-----------------------------------------------------------
On Aug. 24, 2016, 9:08 p.m., Bal�zs Don�t Bessenyei wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51244/
> -----------------------------------------------------------
>
> (Updated Aug. 24, 2016, 9:08 p.m.)
>
>
> Review request for Flume.
>
>
> Bugs: FLUME-2171
> https://issues.apache.org/jira/browse/FLUME-2171
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> I found Flume OG's decorators to handle event headers useful and some to be missing from Flume NG. More specifically, we could have an interceptor to remove headers from an event.
>
>
> Diffs
> -----
>
> c/flume-ng-core/src/main/java/org/apache/flume/interceptor/InterceptorType.java fe341e9
> c/flume-ng-doc/sphinx/FlumeUserGuide.rst 5e677c6
> w/flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java PRE-CREATION
> w/flume-ng-core/src/test/java/org/apache/flume/interceptor/RemoveHeaderInterceptorTest.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/51244/diff/
>
>
> Testing
> -------
>
> All tests (besides the FLUME-2974-related ones) in flume-ng-core run successfully
>
> I've used this config for manual testing:
>
> a1.sources = r1
> a1.sources.r1.type = netcat
> a1.sources.r1.bind = 0.0.0.0
> a1.sources.r1.port = 6666
> a1.sources.r1.channels = c1
>
> a1.channels = c1
> a1.channels.c1.type = memory
> a1.channels.c1.capacity = 10000
> a1.channels.c1.transactionCapacity = 10000
> a1.channels.c1.byteCapacityBufferPercentage = 20
> a1.channels.c1.byteCapacity = 800000
>
> a1.channels = c1
> a1.sinks = k1
> a1.sinks.k1.type = logger
> a1.sinks.k1.channel = c1
>
>
> a1.sources.r1.interceptors = i1 i2 i3
> a1.sources.r1.interceptors.i1.type = timestamp
>
> a1.sources.r1.interceptors.i2.type = host
> a1.sources.r1.interceptors.i2.hostHeader = hostname
>
> a1.sources.r1.interceptors.i3.type = remove_header
> a1.sources.r1.interceptors.i3.with.name = timestamp
>
>
> Thanks,
>
> Bal�zs Don�t Bessenyei
>
>
Re: Review Request 51244: FLUME-2171: Add Interceptor to remove
headers from event
Posted by Attila Simon <sa...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51244/#review146665
-----------------------------------------------------------
c/flume-ng-doc/sphinx/FlumeUserGuide.rst (lines 3474 - 3477)
<https://reviews.apache.org/r/51244/#comment213222>
Were there any particular reason why these aren't named like:
withName
matching
fromList
fromListSeparator
w/flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java (lines 52 - 56)
<https://reviews.apache.org/r/51244/#comment213223>
Have you considered moving these out to a RemoveHeaderInterceptorConstants final class (not interface as pointed out earlier)? Additional javadoc to them wouldn't uglify this class then.
w/flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java (lines 66 - 73)
<https://reviews.apache.org/r/51244/#comment213237>
nit: there is not anon-inner class involved here so final keyword usage on function arguments is not encouraged. (I have no hard opinion on this but received this feedback earlier)
w/flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java (line 120)
<https://reviews.apache.org/r/51244/#comment213231>
I know this has been already pointed out, I would vote for not printing out the event only the removed headers. (Or protect with LogRawDataUtil)
w/flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java (lines 124 - 130)
<https://reviews.apache.org/r/51244/#comment213238>
Iterating through the whole header keyset is inevitable if matching regexp was required. But if hasn't been set then this implementation looses the benefit coming from HashMap O(1) lookup complexity. Instead of iterating over the fromList and for each item headermap is checked, it iterates over each key from the map and then check fromList whether it contains it (fromList is backed by a sorted array as pointed out by javadocs of com.google.common.collect.ImmutableSortedSet which has poorer lookup performance than HashMap)
w/flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java (line 139)
<https://reviews.apache.org/r/51244/#comment213233>
Please don't log out the whole Event for each removed header. Also please collect all the headers which were removed and print them out as a single log entry per Event.
w/flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java (lines 164 - 168)
<https://reviews.apache.org/r/51244/#comment213235>
nit: LOG.debug("Creating RemoveHeaderInterceptor with: withName={}, fromList={}, " +
"listSeparator={}, matchRegex={}", new String[]{
withName, fromList, listSeparator, matchRegex.toString()})
- Attila Simon
On Aug. 23, 2016, 2:43 p.m., Bal�zs Don�t Bessenyei wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51244/
> -----------------------------------------------------------
>
> (Updated Aug. 23, 2016, 2:43 p.m.)
>
>
> Review request for Flume.
>
>
> Bugs: FLUME-2171
> https://issues.apache.org/jira/browse/FLUME-2171
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> I found Flume OG's decorators to handle event headers useful and some to be missing from Flume NG. More specifically, we could have an interceptor to remove headers from an event.
>
>
> Diffs
> -----
>
> c/flume-ng-core/src/main/java/org/apache/flume/interceptor/InterceptorType.java fe341e9
> c/flume-ng-doc/sphinx/FlumeUserGuide.rst 5e677c6
> w/flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java PRE-CREATION
> w/flume-ng-core/src/test/java/org/apache/flume/interceptor/RemoveHeaderInterceptorTest.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/51244/diff/
>
>
> Testing
> -------
>
> All tests (besides the FLUME-2974-related ones) in flume-ng-core run successfully
>
> I've used this config for manual testing:
>
> a1.sources = r1
> a1.sources.r1.type = netcat
> a1.sources.r1.bind = 0.0.0.0
> a1.sources.r1.port = 6666
> a1.sources.r1.channels = c1
>
> a1.channels = c1
> a1.channels.c1.type = memory
> a1.channels.c1.capacity = 10000
> a1.channels.c1.transactionCapacity = 10000
> a1.channels.c1.byteCapacityBufferPercentage = 20
> a1.channels.c1.byteCapacity = 800000
>
> a1.channels = c1
> a1.sinks = k1
> a1.sinks.k1.type = logger
> a1.sinks.k1.channel = c1
>
>
> a1.sources.r1.interceptors = i1 i2 i3
> a1.sources.r1.interceptors.i1.type = timestamp
>
> a1.sources.r1.interceptors.i2.type = host
> a1.sources.r1.interceptors.i2.hostHeader = hostname
>
> a1.sources.r1.interceptors.i3.type = remove_header
> a1.sources.r1.interceptors.i3.with.name = timestamp
>
>
> Thanks,
>
> Bal�zs Don�t Bessenyei
>
>
Re: Review Request 51244: FLUME-2171: Add Interceptor to remove
headers from event
Posted by Denes Arvay <de...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51244/#review148902
-----------------------------------------------------------
Ship it!
Ship It!
- Denes Arvay
On Aug. 24, 2016, 9:08 p.m., Bal�zs Don�t Bessenyei wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51244/
> -----------------------------------------------------------
>
> (Updated Aug. 24, 2016, 9:08 p.m.)
>
>
> Review request for Flume.
>
>
> Bugs: FLUME-2171
> https://issues.apache.org/jira/browse/FLUME-2171
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> I found Flume OG's decorators to handle event headers useful and some to be missing from Flume NG. More specifically, we could have an interceptor to remove headers from an event.
>
>
> Diffs
> -----
>
> c/flume-ng-core/src/main/java/org/apache/flume/interceptor/InterceptorType.java fe341e9
> c/flume-ng-doc/sphinx/FlumeUserGuide.rst 5e677c6
> w/flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java PRE-CREATION
> w/flume-ng-core/src/test/java/org/apache/flume/interceptor/RemoveHeaderInterceptorTest.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/51244/diff/
>
>
> Testing
> -------
>
> All tests (besides the FLUME-2974-related ones) in flume-ng-core run successfully
>
> I've used this config for manual testing:
>
> a1.sources = r1
> a1.sources.r1.type = netcat
> a1.sources.r1.bind = 0.0.0.0
> a1.sources.r1.port = 6666
> a1.sources.r1.channels = c1
>
> a1.channels = c1
> a1.channels.c1.type = memory
> a1.channels.c1.capacity = 10000
> a1.channels.c1.transactionCapacity = 10000
> a1.channels.c1.byteCapacityBufferPercentage = 20
> a1.channels.c1.byteCapacity = 800000
>
> a1.channels = c1
> a1.sinks = k1
> a1.sinks.k1.type = logger
> a1.sinks.k1.channel = c1
>
>
> a1.sources.r1.interceptors = i1 i2 i3
> a1.sources.r1.interceptors.i1.type = timestamp
>
> a1.sources.r1.interceptors.i2.type = host
> a1.sources.r1.interceptors.i2.hostHeader = hostname
>
> a1.sources.r1.interceptors.i3.type = remove_header
> a1.sources.r1.interceptors.i3.with.name = timestamp
>
>
> Thanks,
>
> Bal�zs Don�t Bessenyei
>
>
Re: Review Request 51244: FLUME-2171: Add Interceptor to remove
headers from event
Posted by Attila Simon <sa...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51244/#review153273
-----------------------------------------------------------
w/flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java (lines 119 - 138)
<https://reviews.apache.org/r/51244/#comment222554>
wasRemoved is not needed here. if removedHeaders list in not empty then there were a removal.
Otherwise it looks good to me.
- Attila Simon
On Aug. 24, 2016, 9:08 p.m., Bal�zs Don�t Bessenyei wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51244/
> -----------------------------------------------------------
>
> (Updated Aug. 24, 2016, 9:08 p.m.)
>
>
> Review request for Flume.
>
>
> Bugs: FLUME-2171
> https://issues.apache.org/jira/browse/FLUME-2171
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> I found Flume OG's decorators to handle event headers useful and some to be missing from Flume NG. More specifically, we could have an interceptor to remove headers from an event.
>
>
> Diffs
> -----
>
> c/flume-ng-core/src/main/java/org/apache/flume/interceptor/InterceptorType.java fe341e9
> c/flume-ng-doc/sphinx/FlumeUserGuide.rst 5e677c6
> w/flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java PRE-CREATION
> w/flume-ng-core/src/test/java/org/apache/flume/interceptor/RemoveHeaderInterceptorTest.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/51244/diff/
>
>
> Testing
> -------
>
> All tests (besides the FLUME-2974-related ones) in flume-ng-core run successfully
>
> I've used this config for manual testing:
>
> a1.sources = r1
> a1.sources.r1.type = netcat
> a1.sources.r1.bind = 0.0.0.0
> a1.sources.r1.port = 6666
> a1.sources.r1.channels = c1
>
> a1.channels = c1
> a1.channels.c1.type = memory
> a1.channels.c1.capacity = 10000
> a1.channels.c1.transactionCapacity = 10000
> a1.channels.c1.byteCapacityBufferPercentage = 20
> a1.channels.c1.byteCapacity = 800000
>
> a1.channels = c1
> a1.sinks = k1
> a1.sinks.k1.type = logger
> a1.sinks.k1.channel = c1
>
>
> a1.sources.r1.interceptors = i1 i2 i3
> a1.sources.r1.interceptors.i1.type = timestamp
>
> a1.sources.r1.interceptors.i2.type = host
> a1.sources.r1.interceptors.i2.hostHeader = hostname
>
> a1.sources.r1.interceptors.i3.type = remove_header
> a1.sources.r1.interceptors.i3.with.name = timestamp
>
>
> Thanks,
>
> Bal�zs Don�t Bessenyei
>
>
Re: Review Request 51244: FLUME-2171: Add Interceptor to remove
headers from event
Posted by Balázs Donát Bessenyei <be...@apache.org>.
> On Nov. 14, 2016, 1:58 p.m., Jeff Holoman wrote:
> > flume-ng-doc/sphinx/FlumeUserGuide.rst, line 3928
> > <https://reviews.apache.org/r/51244/diff/8/?file=1562953#file1562953line3928>
> >
> > Do you think you could add an example of a couple of common match patterns? Might be helpful to some people.
I could not really come up with any common match patterns. If anyone has suggestions, I'd be very happy to include them.
- Bal�zs Don�t
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51244/#review155805
-----------------------------------------------------------
On Nov. 14, 2016, 9:19 a.m., Bal�zs Don�t Bessenyei wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51244/
> -----------------------------------------------------------
>
> (Updated Nov. 14, 2016, 9:19 a.m.)
>
>
> Review request for Flume.
>
>
> Bugs: FLUME-2171
> https://issues.apache.org/jira/browse/FLUME-2171
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> I found Flume OG's decorators to handle event headers useful and some to be missing from Flume NG. More specifically, we could have an interceptor to remove headers from an event.
>
>
> Diffs
> -----
>
> flume-ng-core/src/main/java/org/apache/flume/interceptor/InterceptorType.java fe341e9
> flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java PRE-CREATION
> flume-ng-core/src/test/java/org/apache/flume/interceptor/RemoveHeaderInterceptorTest.java PRE-CREATION
> flume-ng-doc/sphinx/FlumeUserGuide.rst cd76bb3
>
> Diff: https://reviews.apache.org/r/51244/diff/
>
>
> Testing
> -------
>
> All tests (besides the FLUME-2974-related ones) in flume-ng-core run successfully
>
> I've used this config for manual testing:
>
> a1.sources = r1
> a1.sources.r1.type = netcat
> a1.sources.r1.bind = 0.0.0.0
> a1.sources.r1.port = 6666
> a1.sources.r1.channels = c1
>
> a1.channels = c1
> a1.channels.c1.type = memory
> a1.channels.c1.capacity = 10000
> a1.channels.c1.transactionCapacity = 10000
> a1.channels.c1.byteCapacityBufferPercentage = 20
> a1.channels.c1.byteCapacity = 800000
>
> a1.channels = c1
> a1.sinks = k1
> a1.sinks.k1.type = logger
> a1.sinks.k1.channel = c1
>
>
> a1.sources.r1.interceptors = i1 i2 i3
> a1.sources.r1.interceptors.i1.type = timestamp
>
> a1.sources.r1.interceptors.i2.type = host
> a1.sources.r1.interceptors.i2.hostHeader = hostname
>
> a1.sources.r1.interceptors.i3.type = remove_header
> a1.sources.r1.interceptors.i3.with.name = timestamp
>
>
> Thanks,
>
> Bal�zs Don�t Bessenyei
>
>
Re: Review Request 51244: FLUME-2171: Add Interceptor to remove
headers from event
Posted by Jeff Holoman <je...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51244/#review155805
-----------------------------------------------------------
Fix it, then Ship it!
flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java (line 122)
<https://reviews.apache.org/r/51244/#comment225898>
nit: I prefer variable names that aren't one character, also on line 127
flume-ng-doc/sphinx/FlumeUserGuide.rst (line 3928)
<https://reviews.apache.org/r/51244/#comment225897>
Do you think you could add an example of a couple of common match patterns? Might be helpful to some people.
- Jeff Holoman
On Nov. 14, 2016, 9:19 a.m., Bal�zs Don�t Bessenyei wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51244/
> -----------------------------------------------------------
>
> (Updated Nov. 14, 2016, 9:19 a.m.)
>
>
> Review request for Flume.
>
>
> Bugs: FLUME-2171
> https://issues.apache.org/jira/browse/FLUME-2171
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> I found Flume OG's decorators to handle event headers useful and some to be missing from Flume NG. More specifically, we could have an interceptor to remove headers from an event.
>
>
> Diffs
> -----
>
> flume-ng-core/src/main/java/org/apache/flume/interceptor/InterceptorType.java fe341e9
> flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java PRE-CREATION
> flume-ng-core/src/test/java/org/apache/flume/interceptor/RemoveHeaderInterceptorTest.java PRE-CREATION
> flume-ng-doc/sphinx/FlumeUserGuide.rst cd76bb3
>
> Diff: https://reviews.apache.org/r/51244/diff/
>
>
> Testing
> -------
>
> All tests (besides the FLUME-2974-related ones) in flume-ng-core run successfully
>
> I've used this config for manual testing:
>
> a1.sources = r1
> a1.sources.r1.type = netcat
> a1.sources.r1.bind = 0.0.0.0
> a1.sources.r1.port = 6666
> a1.sources.r1.channels = c1
>
> a1.channels = c1
> a1.channels.c1.type = memory
> a1.channels.c1.capacity = 10000
> a1.channels.c1.transactionCapacity = 10000
> a1.channels.c1.byteCapacityBufferPercentage = 20
> a1.channels.c1.byteCapacity = 800000
>
> a1.channels = c1
> a1.sinks = k1
> a1.sinks.k1.type = logger
> a1.sinks.k1.channel = c1
>
>
> a1.sources.r1.interceptors = i1 i2 i3
> a1.sources.r1.interceptors.i1.type = timestamp
>
> a1.sources.r1.interceptors.i2.type = host
> a1.sources.r1.interceptors.i2.hostHeader = hostname
>
> a1.sources.r1.interceptors.i3.type = remove_header
> a1.sources.r1.interceptors.i3.with.name = timestamp
>
>
> Thanks,
>
> Bal�zs Don�t Bessenyei
>
>
Re: Review Request 51244: FLUME-2171: Add Interceptor to remove
headers from event
Posted by Balázs Donát Bessenyei <be...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51244/
-----------------------------------------------------------
(Updated Nov. 14, 2016, 7:32 p.m.)
Review request for Flume.
Bugs: FLUME-2171
https://issues.apache.org/jira/browse/FLUME-2171
Repository: flume-git
Description
-------
I found Flume OG's decorators to handle event headers useful and some to be missing from Flume NG. More specifically, we could have an interceptor to remove headers from an event.
Diffs (updated)
-----
flume-ng-core/src/main/java/org/apache/flume/interceptor/InterceptorType.java fe341e9
flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java PRE-CREATION
flume-ng-core/src/test/java/org/apache/flume/interceptor/RemoveHeaderInterceptorTest.java PRE-CREATION
flume-ng-doc/sphinx/FlumeUserGuide.rst cd76bb3
Diff: https://reviews.apache.org/r/51244/diff/
Testing
-------
All tests (besides the FLUME-2974-related ones) in flume-ng-core run successfully
I've used this config for manual testing:
a1.sources = r1
a1.sources.r1.type = netcat
a1.sources.r1.bind = 0.0.0.0
a1.sources.r1.port = 6666
a1.sources.r1.channels = c1
a1.channels = c1
a1.channels.c1.type = memory
a1.channels.c1.capacity = 10000
a1.channels.c1.transactionCapacity = 10000
a1.channels.c1.byteCapacityBufferPercentage = 20
a1.channels.c1.byteCapacity = 800000
a1.channels = c1
a1.sinks = k1
a1.sinks.k1.type = logger
a1.sinks.k1.channel = c1
a1.sources.r1.interceptors = i1 i2 i3
a1.sources.r1.interceptors.i1.type = timestamp
a1.sources.r1.interceptors.i2.type = host
a1.sources.r1.interceptors.i2.hostHeader = hostname
a1.sources.r1.interceptors.i3.type = remove_header
a1.sources.r1.interceptors.i3.with.name = timestamp
Thanks,
Bal�zs Don�t Bessenyei
Re: Review Request 51244: FLUME-2171: Add Interceptor to remove
headers from event
Posted by Balázs Donát Bessenyei <be...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51244/
-----------------------------------------------------------
(Updated Nov. 14, 2016, 9:19 a.m.)
Review request for Flume.
Bugs: FLUME-2171
https://issues.apache.org/jira/browse/FLUME-2171
Repository: flume-git
Description
-------
I found Flume OG's decorators to handle event headers useful and some to be missing from Flume NG. More specifically, we could have an interceptor to remove headers from an event.
Diffs (updated)
-----
flume-ng-core/src/main/java/org/apache/flume/interceptor/InterceptorType.java fe341e9
flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java PRE-CREATION
flume-ng-core/src/test/java/org/apache/flume/interceptor/RemoveHeaderInterceptorTest.java PRE-CREATION
flume-ng-doc/sphinx/FlumeUserGuide.rst cd76bb3
Diff: https://reviews.apache.org/r/51244/diff/
Testing
-------
All tests (besides the FLUME-2974-related ones) in flume-ng-core run successfully
I've used this config for manual testing:
a1.sources = r1
a1.sources.r1.type = netcat
a1.sources.r1.bind = 0.0.0.0
a1.sources.r1.port = 6666
a1.sources.r1.channels = c1
a1.channels = c1
a1.channels.c1.type = memory
a1.channels.c1.capacity = 10000
a1.channels.c1.transactionCapacity = 10000
a1.channels.c1.byteCapacityBufferPercentage = 20
a1.channels.c1.byteCapacity = 800000
a1.channels = c1
a1.sinks = k1
a1.sinks.k1.type = logger
a1.sinks.k1.channel = c1
a1.sources.r1.interceptors = i1 i2 i3
a1.sources.r1.interceptors.i1.type = timestamp
a1.sources.r1.interceptors.i2.type = host
a1.sources.r1.interceptors.i2.hostHeader = hostname
a1.sources.r1.interceptors.i3.type = remove_header
a1.sources.r1.interceptors.i3.with.name = timestamp
Thanks,
Bal�zs Don�t Bessenyei
Re: Review Request 51244: FLUME-2171: Add Interceptor to remove
headers from event
Posted by Balázs Donát Bessenyei <be...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51244/
-----------------------------------------------------------
(Updated Aug. 24, 2016, 9:08 p.m.)
Review request for Flume.
Bugs: FLUME-2171
https://issues.apache.org/jira/browse/FLUME-2171
Repository: flume-git
Description
-------
I found Flume OG's decorators to handle event headers useful and some to be missing from Flume NG. More specifically, we could have an interceptor to remove headers from an event.
Diffs (updated)
-----
c/flume-ng-core/src/main/java/org/apache/flume/interceptor/InterceptorType.java fe341e9
c/flume-ng-doc/sphinx/FlumeUserGuide.rst 5e677c6
w/flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java PRE-CREATION
w/flume-ng-core/src/test/java/org/apache/flume/interceptor/RemoveHeaderInterceptorTest.java PRE-CREATION
Diff: https://reviews.apache.org/r/51244/diff/
Testing
-------
All tests (besides the FLUME-2974-related ones) in flume-ng-core run successfully
I've used this config for manual testing:
a1.sources = r1
a1.sources.r1.type = netcat
a1.sources.r1.bind = 0.0.0.0
a1.sources.r1.port = 6666
a1.sources.r1.channels = c1
a1.channels = c1
a1.channels.c1.type = memory
a1.channels.c1.capacity = 10000
a1.channels.c1.transactionCapacity = 10000
a1.channels.c1.byteCapacityBufferPercentage = 20
a1.channels.c1.byteCapacity = 800000
a1.channels = c1
a1.sinks = k1
a1.sinks.k1.type = logger
a1.sinks.k1.channel = c1
a1.sources.r1.interceptors = i1 i2 i3
a1.sources.r1.interceptors.i1.type = timestamp
a1.sources.r1.interceptors.i2.type = host
a1.sources.r1.interceptors.i2.hostHeader = hostname
a1.sources.r1.interceptors.i3.type = remove_header
a1.sources.r1.interceptors.i3.with.name = timestamp
Thanks,
Bal�zs Don�t Bessenyei
Re: Review Request 51244: FLUME-2171: Add Interceptor to remove
headers from event
Posted by Balázs Donát Bessenyei <be...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51244/
-----------------------------------------------------------
(Updated Aug. 23, 2016, 2:43 p.m.)
Review request for Flume.
Bugs: FLUME-2171
https://issues.apache.org/jira/browse/FLUME-2171
Repository: flume-git
Description
-------
I found Flume OG's decorators to handle event headers useful and some to be missing from Flume NG. More specifically, we could have an interceptor to remove headers from an event.
Diffs (updated)
-----
c/flume-ng-core/src/main/java/org/apache/flume/interceptor/InterceptorType.java fe341e9
c/flume-ng-doc/sphinx/FlumeUserGuide.rst 5e677c6
w/flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java PRE-CREATION
w/flume-ng-core/src/test/java/org/apache/flume/interceptor/RemoveHeaderInterceptorTest.java PRE-CREATION
Diff: https://reviews.apache.org/r/51244/diff/
Testing
-------
All tests (besides the FLUME-2974-related ones) in flume-ng-core run successfully
I've used this config for manual testing:
a1.sources = r1
a1.sources.r1.type = netcat
a1.sources.r1.bind = 0.0.0.0
a1.sources.r1.port = 6666
a1.sources.r1.channels = c1
a1.channels = c1
a1.channels.c1.type = memory
a1.channels.c1.capacity = 10000
a1.channels.c1.transactionCapacity = 10000
a1.channels.c1.byteCapacityBufferPercentage = 20
a1.channels.c1.byteCapacity = 800000
a1.channels = c1
a1.sinks = k1
a1.sinks.k1.type = logger
a1.sinks.k1.channel = c1
a1.sources.r1.interceptors = i1 i2 i3
a1.sources.r1.interceptors.i1.type = timestamp
a1.sources.r1.interceptors.i2.type = host
a1.sources.r1.interceptors.i2.hostHeader = hostname
a1.sources.r1.interceptors.i3.type = remove_header
a1.sources.r1.interceptors.i3.with.name = timestamp
Thanks,
Bal�zs Don�t Bessenyei
Re: Review Request 51244: FLUME-2171: Add Interceptor to remove
headers from event
Posted by Balázs Donát Bessenyei <be...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51244/
-----------------------------------------------------------
(Updated Aug. 22, 2016, 11:13 p.m.)
Review request for Flume.
Changes
-------
git diff --no-prefix
so that changes would show
Bugs: FLUME-2171
https://issues.apache.org/jira/browse/FLUME-2171
Repository: flume-git
Description
-------
I found Flume OG's decorators to handle event headers useful and some to be missing from Flume NG. More specifically, we could have an interceptor to remove headers from an event.
Diffs (updated)
-----
flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java PRE-CREATION
flume-ng-core/src/test/java/org/apache/flume/interceptor/RemoveHeaderInterceptorTest.java PRE-CREATION
flume-ng-doc/sphinx/FlumeUserGuide.rst 5e677c6
Diff: https://reviews.apache.org/r/51244/diff/
Testing
-------
All tests (besides the FLUME-2974-related ones) in flume-ng-core run successfully
I've used this config for manual testing:
a1.sources = r1
a1.sources.r1.type = netcat
a1.sources.r1.bind = 0.0.0.0
a1.sources.r1.port = 6666
a1.sources.r1.channels = c1
a1.channels = c1
a1.channels.c1.type = memory
a1.channels.c1.capacity = 10000
a1.channels.c1.transactionCapacity = 10000
a1.channels.c1.byteCapacityBufferPercentage = 20
a1.channels.c1.byteCapacity = 800000
a1.channels = c1
a1.sinks = k1
a1.sinks.k1.type = logger
a1.sinks.k1.channel = c1
a1.sources.r1.interceptors = i1 i2 i3
a1.sources.r1.interceptors.i1.type = timestamp
a1.sources.r1.interceptors.i2.type = host
a1.sources.r1.interceptors.i2.hostHeader = hostname
a1.sources.r1.interceptors.i3.type = remove_header
a1.sources.r1.interceptors.i3.with.name = timestamp
Thanks,
Bal�zs Don�t Bessenyei
Re: Review Request 51244: FLUME-2171: Add Interceptor to remove
headers from event
Posted by Balázs Donát Bessenyei <be...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51244/
-----------------------------------------------------------
(Updated Aug. 22, 2016, 11:10 p.m.)
Review request for Flume.
Changes
-------
git diff --no-prefix
so that changes would show
Bugs: FLUME-2171
https://issues.apache.org/jira/browse/FLUME-2171
Repository: flume-git
Description
-------
I found Flume OG's decorators to handle event headers useful and some to be missing from Flume NG. More specifically, we could have an interceptor to remove headers from an event.
Diffs (updated)
-----
flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java PRE-CREATION
flume-ng-core/src/test/java/org/apache/flume/interceptor/RemoveHeaderInterceptorTest.java PRE-CREATION
flume-ng-doc/sphinx/FlumeUserGuide.rst c614991
Diff: https://reviews.apache.org/r/51244/diff/
Testing
-------
All tests (besides the FLUME-2974-related ones) in flume-ng-core run successfully
I've used this config for manual testing:
a1.sources = r1
a1.sources.r1.type = netcat
a1.sources.r1.bind = 0.0.0.0
a1.sources.r1.port = 6666
a1.sources.r1.channels = c1
a1.channels = c1
a1.channels.c1.type = memory
a1.channels.c1.capacity = 10000
a1.channels.c1.transactionCapacity = 10000
a1.channels.c1.byteCapacityBufferPercentage = 20
a1.channels.c1.byteCapacity = 800000
a1.channels = c1
a1.sinks = k1
a1.sinks.k1.type = logger
a1.sinks.k1.channel = c1
a1.sources.r1.interceptors = i1 i2 i3
a1.sources.r1.interceptors.i1.type = timestamp
a1.sources.r1.interceptors.i2.type = host
a1.sources.r1.interceptors.i2.hostHeader = hostname
a1.sources.r1.interceptors.i3.type = remove_header
a1.sources.r1.interceptors.i3.with.name = timestamp
Thanks,
Bal�zs Don�t Bessenyei
Re: Review Request 51244: FLUME-2171: Add Interceptor to remove
headers from event
Posted by Balázs Donát Bessenyei <be...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51244/
-----------------------------------------------------------
(Updated Aug. 22, 2016, 10:51 p.m.)
Review request for Flume.
Changes
-------
Posting changes
Bugs: FLUME-2171
https://issues.apache.org/jira/browse/FLUME-2171
Repository: flume-git
Description
-------
I found Flume OG's decorators to handle event headers useful and some to be missing from Flume NG. More specifically, we could have an interceptor to remove headers from an event.
Diffs (updated)
-----
c/flume-ng-core/src/main/java/org/apache/flume/interceptor/InterceptorType.java fe341e9
c/flume-ng-doc/sphinx/FlumeUserGuide.rst 1c15f1e
w/flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java PRE-CREATION
w/flume-ng-core/src/test/java/org/apache/flume/interceptor/RemoveHeaderInterceptorTest.java PRE-CREATION
Diff: https://reviews.apache.org/r/51244/diff/
Testing
-------
All tests (besides the FLUME-2974-related ones) in flume-ng-core run successfully
I've used this config for manual testing:
a1.sources = r1
a1.sources.r1.type = netcat
a1.sources.r1.bind = 0.0.0.0
a1.sources.r1.port = 6666
a1.sources.r1.channels = c1
a1.channels = c1
a1.channels.c1.type = memory
a1.channels.c1.capacity = 10000
a1.channels.c1.transactionCapacity = 10000
a1.channels.c1.byteCapacityBufferPercentage = 20
a1.channels.c1.byteCapacity = 800000
a1.channels = c1
a1.sinks = k1
a1.sinks.k1.type = logger
a1.sinks.k1.channel = c1
a1.sources.r1.interceptors = i1 i2 i3
a1.sources.r1.interceptors.i1.type = timestamp
a1.sources.r1.interceptors.i2.type = host
a1.sources.r1.interceptors.i2.hostHeader = hostname
a1.sources.r1.interceptors.i3.type = remove_header
a1.sources.r1.interceptors.i3.with.name = timestamp
Thanks,
Bal�zs Don�t Bessenyei
Re: Review Request 51244: FLUME-2171: Add Interceptor to remove
headers from event
Posted by Balázs Donát Bessenyei <be...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51244/
-----------------------------------------------------------
(Updated Aug. 22, 2016, 10:49 p.m.)
Review request for Flume.
Changes
-------
Posting original patch
Bugs: FLUME-2171
https://issues.apache.org/jira/browse/FLUME-2171
Repository: flume-git
Description
-------
I found Flume OG's decorators to handle event headers useful and some to be missing from Flume NG. More specifically, we could have an interceptor to remove headers from an event.
Diffs (updated)
-----
flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java PRE-CREATION
flume-ng-core/src/test/java/org/apache/flume/interceptor/RemoveHeaderInterceptorTest.java PRE-CREATION
flume-ng-doc/sphinx/FlumeUserGuide.rst c614991
Diff: https://reviews.apache.org/r/51244/diff/
Testing
-------
All tests (besides the FLUME-2974-related ones) in flume-ng-core run successfully
I've used this config for manual testing:
a1.sources = r1
a1.sources.r1.type = netcat
a1.sources.r1.bind = 0.0.0.0
a1.sources.r1.port = 6666
a1.sources.r1.channels = c1
a1.channels = c1
a1.channels.c1.type = memory
a1.channels.c1.capacity = 10000
a1.channels.c1.transactionCapacity = 10000
a1.channels.c1.byteCapacityBufferPercentage = 20
a1.channels.c1.byteCapacity = 800000
a1.channels = c1
a1.sinks = k1
a1.sinks.k1.type = logger
a1.sinks.k1.channel = c1
a1.sources.r1.interceptors = i1 i2 i3
a1.sources.r1.interceptors.i1.type = timestamp
a1.sources.r1.interceptors.i2.type = host
a1.sources.r1.interceptors.i2.hostHeader = hostname
a1.sources.r1.interceptors.i3.type = remove_header
a1.sources.r1.interceptors.i3.with.name = timestamp
Thanks,
Bal�zs Don�t Bessenyei
Re: Review Request 51244: FLUME-2171: Add Interceptor to remove
headers from event
Posted by Balázs Donát Bessenyei <be...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51244/
-----------------------------------------------------------
(Updated Aug. 22, 2016, 10:02 a.m.)
Review request for Flume.
Bugs: FLUME-2171
https://issues.apache.org/jira/browse/FLUME-2171
Repository: flume-git
Description
-------
I found Flume OG's decorators to handle event headers useful and some to be missing from Flume NG. More specifically, we could have an interceptor to remove headers from an event.
Diffs
-----
c/flume-ng-core/src/main/java/org/apache/flume/interceptor/InterceptorType.java fe341e9
c/flume-ng-doc/sphinx/FlumeUserGuide.rst 1c15f1e
w/flume-ng-core/src/main/java/org/apache/flume/interceptor/RemoveHeaderInterceptor.java PRE-CREATION
w/flume-ng-core/src/test/java/org/apache/flume/interceptor/RemoveHeaderInterceptorTest.java PRE-CREATION
Diff: https://reviews.apache.org/r/51244/diff/
Testing
-------
All tests (besides the FLUME-2974-related ones) in flume-ng-core run successfully
I've used this config for manual testing:
a1.sources = r1
a1.sources.r1.type = netcat
a1.sources.r1.bind = 0.0.0.0
a1.sources.r1.port = 6666
a1.sources.r1.channels = c1
a1.channels = c1
a1.channels.c1.type = memory
a1.channels.c1.capacity = 10000
a1.channels.c1.transactionCapacity = 10000
a1.channels.c1.byteCapacityBufferPercentage = 20
a1.channels.c1.byteCapacity = 800000
a1.channels = c1
a1.sinks = k1
a1.sinks.k1.type = logger
a1.sinks.k1.channel = c1
a1.sources.r1.interceptors = i1 i2 i3
a1.sources.r1.interceptors.i1.type = timestamp
a1.sources.r1.interceptors.i2.type = host
a1.sources.r1.interceptors.i2.hostHeader = hostname
a1.sources.r1.interceptors.i3.type = remove_header
a1.sources.r1.interceptors.i3.with.name = timestamp
Thanks,
Bal�zs Don�t Bessenyei