You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by neerja khattar <ne...@cloudera.com> on 2016/05/08 19:48:17 UTC
Review Request 47098: FLUME-2620 File channel throws
NullPointerException if a header value is null
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47098/
-----------------------------------------------------------
Review request for Flume.
Repository: flume-git
Description
-------
The issue is when the header value is null it throws null pointer exception and flume stops processing further events.
For example:
[{
"headers" : {
"timestamp" : "434324343",
"host" : null
},
"body" : "random_body"
}]
The solution to fix this is:
1. If the header has a null value in the json, flume will replace it with a replacement string.
2. The default value for a replacement string is an empty string.
3. To overwrite default string, set "handler.nullReplacementHeader" property in flume config.
Diffs
-----
flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSource.java b520b03
flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSourceConfigurationConstants.java 86caf7d
flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSourceHandler.java 726bf0c
flume-ng-core/src/main/java/org/apache/flume/source/http/JSONHandler.java 197f66a
flume-ng-core/src/test/java/org/apache/flume/source/http/TestJSONHandler.java 455781c
Diff: https://reviews.apache.org/r/47098/diff/
Testing
-------
The following are the test cases:
1. Header has null value in json and handler.nullReplacementHeader is not set in flume config. The default value will be used to replace null.
[{
"headers" : {
"timestamp" : "434324343",
"host" : null
},
"body" : "random_body"
}]
Output in hdfs : {timestamp=434324343, host=} random_body
2. Header is not null in json and handler.nullReplacementHeader is not set in flume config. The replacement implementation doesnt come in to consideration.
[{
"headers" : {
"timestamp" : "434324343",
"host" : 1
},
"body" : "random_body"
}]
Output in hdfs : {timestamp=434324343, host=1} random_body
3. Header has null value in json and handler.nullReplacementHeader=abc is set in flume config. The null value in header will be replaced by abc.
[{
"headers" : {
"timestamp" : "434324343",
"host" : null
},
"body" : "random_body"
}]
Output in hdfs {timestamp=434324343, host=abc} random_body
4. Header has null value in json and handler.nullReplacementHeader=1 is set in flume config. The null value in header will be replaced by 1 as a string .
[{
"headers" : {
"timestamp" : "434324343",
"host" : null
},
"body" : "random_body"
}]
Output in hdfs: {timestamp=434324343, host=1} random_body
5. Header is not null in json and handler.nullReplacementHeader is also set in flume config. The replacement implementation doesnt come in to consideration.
[{
"headers" : {
"timestamp" : "434324343",
"host" : 1
},
"body" : "random_body"
}]
Output in hdfs : {timestamp=434324343, host=1} random_body
Thanks,
neerja khattar
Re: Review Request 47098: FLUME-2620 File channel throws
NullPointerException if a header value is null
Posted by neerja khattar <ne...@cloudera.com>.
> On May 8, 2016, 9:16 p.m., Jarek Cecho wrote:
> > flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSourceHandler.java, line 44
> > <https://reviews.apache.org/r/47098/diff/1/?file=1375930#file1375930line44>
> >
> > This will be a backward in-compatible change. Can we perhaps convert this interface to abstract class, move handling of the nullReplacement here, so that we will break the contract only once?
> >
> > E.g. that we won't have to break it somewhere in the future when we will need to add yet another argument.
>
> Hari Shreedharan wrote:
> Unfortunately, changing this to an abstract class does not solve the issue here - since that is a bytecode change and breaks compat (code too - implements vs extends). One way of fixing this is to add an abstract class that inherits this one, and have an instance of check in the Source itself, and call this method only if the handler is an instance of the abstract class.
>
> neerja khattar wrote:
> ahh i was just compiling flume-ng-core didnt realize had to compile whole project for dependency check:) working on it now.. sorry about that.
Thanks hari. I am making the change
- neerja
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47098/#review132183
-----------------------------------------------------------
On May 9, 2016, 4:57 a.m., neerja khattar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47098/
> -----------------------------------------------------------
>
> (Updated May 9, 2016, 4:57 a.m.)
>
>
> Review request for Flume.
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> The issue is when the header value is null it throws null pointer exception and flume stops processing further events.
> For example:
> [{
> "headers" : {
> "timestamp" : "434324343",
> "host" : null
> },
> "body" : "random_body"
> }]
>
> The solution to fix this is:
>
> 1. If the header has a null value in the json, flume will replace it with a replacement string.
> 2. The default value for a replacement string is an empty string.
> 3. To overwrite default string, set "handler.nullReplacementHeader" property in flume config.
>
>
> Diffs
> -----
>
> flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSource.java b520b03
> flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSourceConfigurationConstants.java 86caf7d
> flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSourceHandler.java 726bf0c
> flume-ng-core/src/main/java/org/apache/flume/source/http/JSONHandler.java 197f66a
> flume-ng-core/src/test/java/org/apache/flume/source/http/TestJSONHandler.java 455781c
> flume-ng-sinks/flume-ng-morphline-solr-sink/src/main/java/org/apache/flume/sink/solr/morphline/BlobHandler.java e84dec1
>
> Diff: https://reviews.apache.org/r/47098/diff/
>
>
> Testing
> -------
>
> The following are the test cases:
>
> 1. Header has null value in json and handler.nullReplacementHeader is not set in flume config. The default value will be used to replace null.
>
> [{
> "headers" : {
> "timestamp" : "434324343",
> "host" : null
> },
> "body" : "random_body"
> }]
>
> Output in hdfs : {timestamp=434324343, host=} random_body
>
> 2. Header is not null in json and handler.nullReplacementHeader is not set in flume config. The replacement implementation doesnt come in to consideration.
>
> [{
> "headers" : {
> "timestamp" : "434324343",
> "host" : 1
> },
> "body" : "random_body"
> }]
> Output in hdfs : {timestamp=434324343, host=1} random_body
>
> 3. Header has null value in json and handler.nullReplacementHeader=abc is set in flume config. The null value in header will be replaced by abc.
>
>
> [{
> "headers" : {
> "timestamp" : "434324343",
> "host" : null
> },
> "body" : "random_body"
> }]
>
>
> Output in hdfs {timestamp=434324343, host=abc} random_body
>
> 4. Header has null value in json and handler.nullReplacementHeader=1 is set in flume config. The null value in header will be replaced by 1 as a string .
>
> [{
> "headers" : {
> "timestamp" : "434324343",
> "host" : null
> },
> "body" : "random_body"
> }]
>
>
> Output in hdfs: {timestamp=434324343, host=1} random_body
>
> 5. Header is not null in json and handler.nullReplacementHeader is also set in flume config. The replacement implementation doesnt come in to consideration.
>
> [{
> "headers" : {
> "timestamp" : "434324343",
> "host" : 1
> },
> "body" : "random_body"
> }]
> Output in hdfs : {timestamp=434324343, host=1} random_body
>
>
> File Attachments
> ----------------
>
> flume-2620
> https://reviews.apache.org/media/uploaded/files/2016/05/09/0eff1d56-caf3-4d36-bb45-6b9e7fd6a1ff__FLUME-2620-1.patch
>
>
> Thanks,
>
> neerja khattar
>
>
Re: Review Request 47098: FLUME-2620 File channel throws
NullPointerException if a header value is null
Posted by Hari Shreedharan <hs...@cloudera.com>.
> On May 8, 2016, 9:16 p.m., Jarek Cecho wrote:
> > flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSourceHandler.java, line 44
> > <https://reviews.apache.org/r/47098/diff/1/?file=1375930#file1375930line44>
> >
> > This will be a backward in-compatible change. Can we perhaps convert this interface to abstract class, move handling of the nullReplacement here, so that we will break the contract only once?
> >
> > E.g. that we won't have to break it somewhere in the future when we will need to add yet another argument.
Unfortunately, changing this to an abstract class does not solve the issue here - since that is a bytecode change and breaks compat (code too - implements vs extends). One way of fixing this is to add an abstract class that inherits this one, and have an instance of check in the Source itself, and call this method only if the handler is an instance of the abstract class.
- Hari
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47098/#review132183
-----------------------------------------------------------
On May 9, 2016, 4:57 a.m., neerja khattar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47098/
> -----------------------------------------------------------
>
> (Updated May 9, 2016, 4:57 a.m.)
>
>
> Review request for Flume.
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> The issue is when the header value is null it throws null pointer exception and flume stops processing further events.
> For example:
> [{
> "headers" : {
> "timestamp" : "434324343",
> "host" : null
> },
> "body" : "random_body"
> }]
>
> The solution to fix this is:
>
> 1. If the header has a null value in the json, flume will replace it with a replacement string.
> 2. The default value for a replacement string is an empty string.
> 3. To overwrite default string, set "handler.nullReplacementHeader" property in flume config.
>
>
> Diffs
> -----
>
> flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSource.java b520b03
> flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSourceConfigurationConstants.java 86caf7d
> flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSourceHandler.java 726bf0c
> flume-ng-core/src/main/java/org/apache/flume/source/http/JSONHandler.java 197f66a
> flume-ng-core/src/test/java/org/apache/flume/source/http/TestJSONHandler.java 455781c
> flume-ng-sinks/flume-ng-morphline-solr-sink/src/main/java/org/apache/flume/sink/solr/morphline/BlobHandler.java e84dec1
>
> Diff: https://reviews.apache.org/r/47098/diff/
>
>
> Testing
> -------
>
> The following are the test cases:
>
> 1. Header has null value in json and handler.nullReplacementHeader is not set in flume config. The default value will be used to replace null.
>
> [{
> "headers" : {
> "timestamp" : "434324343",
> "host" : null
> },
> "body" : "random_body"
> }]
>
> Output in hdfs : {timestamp=434324343, host=} random_body
>
> 2. Header is not null in json and handler.nullReplacementHeader is not set in flume config. The replacement implementation doesnt come in to consideration.
>
> [{
> "headers" : {
> "timestamp" : "434324343",
> "host" : 1
> },
> "body" : "random_body"
> }]
> Output in hdfs : {timestamp=434324343, host=1} random_body
>
> 3. Header has null value in json and handler.nullReplacementHeader=abc is set in flume config. The null value in header will be replaced by abc.
>
>
> [{
> "headers" : {
> "timestamp" : "434324343",
> "host" : null
> },
> "body" : "random_body"
> }]
>
>
> Output in hdfs {timestamp=434324343, host=abc} random_body
>
> 4. Header has null value in json and handler.nullReplacementHeader=1 is set in flume config. The null value in header will be replaced by 1 as a string .
>
> [{
> "headers" : {
> "timestamp" : "434324343",
> "host" : null
> },
> "body" : "random_body"
> }]
>
>
> Output in hdfs: {timestamp=434324343, host=1} random_body
>
> 5. Header is not null in json and handler.nullReplacementHeader is also set in flume config. The replacement implementation doesnt come in to consideration.
>
> [{
> "headers" : {
> "timestamp" : "434324343",
> "host" : 1
> },
> "body" : "random_body"
> }]
> Output in hdfs : {timestamp=434324343, host=1} random_body
>
>
> File Attachments
> ----------------
>
> flume-2620
> https://reviews.apache.org/media/uploaded/files/2016/05/09/0eff1d56-caf3-4d36-bb45-6b9e7fd6a1ff__FLUME-2620-1.patch
>
>
> Thanks,
>
> neerja khattar
>
>
Re: Review Request 47098: FLUME-2620 File channel throws
NullPointerException if a header value is null
Posted by neerja khattar <ne...@cloudera.com>.
> On May 8, 2016, 9:16 p.m., Jarek Cecho wrote:
> > I'm getting the following when I tried to compile flume after applying the patch:
> >
> > [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.3.2:compile (default-compile) on project flume-ng-core: Compilation failure
> > [ERROR] /Users/jarcec/apache/flume/flume-ng-core/src/main/java/org/apache/flume/source/http/BLOBHandler.java:[43,7] error: BLOBHandler is not abstract and does not override abstract method setNullHeaderReplacement(String) in HTTPSourceHandler
> > [ERROR] -> [Help 1]
> >
> > Which points to:
Jarec, I have made the following change and attached the patch.
we perhaps convert this interface to abstract class, move handling of the nullReplacement here, so that we will break the contract only once
Let me know whats the next step.
> On May 8, 2016, 9:16 p.m., Jarek Cecho wrote:
> > flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSourceHandler.java, line 44
> > <https://reviews.apache.org/r/47098/diff/1/?file=1375930#file1375930line44>
> >
> > This will be a backward in-compatible change. Can we perhaps convert this interface to abstract class, move handling of the nullReplacement here, so that we will break the contract only once?
> >
> > E.g. that we won't have to break it somewhere in the future when we will need to add yet another argument.
>
> Hari Shreedharan wrote:
> Unfortunately, changing this to an abstract class does not solve the issue here - since that is a bytecode change and breaks compat (code too - implements vs extends). One way of fixing this is to add an abstract class that inherits this one, and have an instance of check in the Source itself, and call this method only if the handler is an instance of the abstract class.
ahh i was just compiling flume-ng-core didnt realize had to compile whole project for dependency check:) working on it now.. sorry about that.
- neerja
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47098/#review132183
-----------------------------------------------------------
On May 9, 2016, 4:57 a.m., neerja khattar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47098/
> -----------------------------------------------------------
>
> (Updated May 9, 2016, 4:57 a.m.)
>
>
> Review request for Flume.
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> The issue is when the header value is null it throws null pointer exception and flume stops processing further events.
> For example:
> [{
> "headers" : {
> "timestamp" : "434324343",
> "host" : null
> },
> "body" : "random_body"
> }]
>
> The solution to fix this is:
>
> 1. If the header has a null value in the json, flume will replace it with a replacement string.
> 2. The default value for a replacement string is an empty string.
> 3. To overwrite default string, set "handler.nullReplacementHeader" property in flume config.
>
>
> Diffs
> -----
>
> flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSource.java b520b03
> flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSourceConfigurationConstants.java 86caf7d
> flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSourceHandler.java 726bf0c
> flume-ng-core/src/main/java/org/apache/flume/source/http/JSONHandler.java 197f66a
> flume-ng-core/src/test/java/org/apache/flume/source/http/TestJSONHandler.java 455781c
> flume-ng-sinks/flume-ng-morphline-solr-sink/src/main/java/org/apache/flume/sink/solr/morphline/BlobHandler.java e84dec1
>
> Diff: https://reviews.apache.org/r/47098/diff/
>
>
> Testing
> -------
>
> The following are the test cases:
>
> 1. Header has null value in json and handler.nullReplacementHeader is not set in flume config. The default value will be used to replace null.
>
> [{
> "headers" : {
> "timestamp" : "434324343",
> "host" : null
> },
> "body" : "random_body"
> }]
>
> Output in hdfs : {timestamp=434324343, host=} random_body
>
> 2. Header is not null in json and handler.nullReplacementHeader is not set in flume config. The replacement implementation doesnt come in to consideration.
>
> [{
> "headers" : {
> "timestamp" : "434324343",
> "host" : 1
> },
> "body" : "random_body"
> }]
> Output in hdfs : {timestamp=434324343, host=1} random_body
>
> 3. Header has null value in json and handler.nullReplacementHeader=abc is set in flume config. The null value in header will be replaced by abc.
>
>
> [{
> "headers" : {
> "timestamp" : "434324343",
> "host" : null
> },
> "body" : "random_body"
> }]
>
>
> Output in hdfs {timestamp=434324343, host=abc} random_body
>
> 4. Header has null value in json and handler.nullReplacementHeader=1 is set in flume config. The null value in header will be replaced by 1 as a string .
>
> [{
> "headers" : {
> "timestamp" : "434324343",
> "host" : null
> },
> "body" : "random_body"
> }]
>
>
> Output in hdfs: {timestamp=434324343, host=1} random_body
>
> 5. Header is not null in json and handler.nullReplacementHeader is also set in flume config. The replacement implementation doesnt come in to consideration.
>
> [{
> "headers" : {
> "timestamp" : "434324343",
> "host" : 1
> },
> "body" : "random_body"
> }]
> Output in hdfs : {timestamp=434324343, host=1} random_body
>
>
> File Attachments
> ----------------
>
> flume-2620
> https://reviews.apache.org/media/uploaded/files/2016/05/09/0eff1d56-caf3-4d36-bb45-6b9e7fd6a1ff__FLUME-2620-1.patch
>
>
> Thanks,
>
> neerja khattar
>
>
Re: Review Request 47098: FLUME-2620 File channel throws
NullPointerException if a header value is null
Posted by Jarek Cecho <ja...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47098/#review132183
-----------------------------------------------------------
I'm getting the following when I tried to compile flume after applying the patch:
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.3.2:compile (default-compile) on project flume-ng-core: Compilation failure
[ERROR] /Users/jarcec/apache/flume/flume-ng-core/src/main/java/org/apache/flume/source/http/BLOBHandler.java:[43,7] error: BLOBHandler is not abstract and does not override abstract method setNullHeaderReplacement(String) in HTTPSourceHandler
[ERROR] -> [Help 1]
Which points to:
flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSourceHandler.java (line 44)
<https://reviews.apache.org/r/47098/#comment196350>
This will be a backward in-compatible change. Can we perhaps convert this interface to abstract class, move handling of the nullReplacement here, so that we will break the contract only once?
E.g. that we won't have to break it somewhere in the future when we will need to add yet another argument.
- Jarek Cecho
On May 8, 2016, 7:48 p.m., neerja khattar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47098/
> -----------------------------------------------------------
>
> (Updated May 8, 2016, 7:48 p.m.)
>
>
> Review request for Flume.
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> The issue is when the header value is null it throws null pointer exception and flume stops processing further events.
> For example:
> [{
> "headers" : {
> "timestamp" : "434324343",
> "host" : null
> },
> "body" : "random_body"
> }]
>
> The solution to fix this is:
>
> 1. If the header has a null value in the json, flume will replace it with a replacement string.
> 2. The default value for a replacement string is an empty string.
> 3. To overwrite default string, set "handler.nullReplacementHeader" property in flume config.
>
>
> Diffs
> -----
>
> flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSource.java b520b03
> flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSourceConfigurationConstants.java 86caf7d
> flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSourceHandler.java 726bf0c
> flume-ng-core/src/main/java/org/apache/flume/source/http/JSONHandler.java 197f66a
> flume-ng-core/src/test/java/org/apache/flume/source/http/TestJSONHandler.java 455781c
>
> Diff: https://reviews.apache.org/r/47098/diff/
>
>
> Testing
> -------
>
> The following are the test cases:
>
> 1. Header has null value in json and handler.nullReplacementHeader is not set in flume config. The default value will be used to replace null.
>
> [{
> "headers" : {
> "timestamp" : "434324343",
> "host" : null
> },
> "body" : "random_body"
> }]
>
> Output in hdfs : {timestamp=434324343, host=} random_body
>
> 2. Header is not null in json and handler.nullReplacementHeader is not set in flume config. The replacement implementation doesnt come in to consideration.
>
> [{
> "headers" : {
> "timestamp" : "434324343",
> "host" : 1
> },
> "body" : "random_body"
> }]
> Output in hdfs : {timestamp=434324343, host=1} random_body
>
> 3. Header has null value in json and handler.nullReplacementHeader=abc is set in flume config. The null value in header will be replaced by abc.
>
>
> [{
> "headers" : {
> "timestamp" : "434324343",
> "host" : null
> },
> "body" : "random_body"
> }]
>
>
> Output in hdfs {timestamp=434324343, host=abc} random_body
>
> 4. Header has null value in json and handler.nullReplacementHeader=1 is set in flume config. The null value in header will be replaced by 1 as a string .
>
> [{
> "headers" : {
> "timestamp" : "434324343",
> "host" : null
> },
> "body" : "random_body"
> }]
>
>
> Output in hdfs: {timestamp=434324343, host=1} random_body
>
> 5. Header is not null in json and handler.nullReplacementHeader is also set in flume config. The replacement implementation doesnt come in to consideration.
>
> [{
> "headers" : {
> "timestamp" : "434324343",
> "host" : 1
> },
> "body" : "random_body"
> }]
> Output in hdfs : {timestamp=434324343, host=1} random_body
>
>
> Thanks,
>
> neerja khattar
>
>
Re: Review Request 47098: FLUME-2620 File channel throws
NullPointerException if a header value is null
Posted by Jarek Cecho <ja...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47098/#review132968
-----------------------------------------------------------
flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSource.java (line 153)
<https://reviews.apache.org/r/47098/#comment197265>
if(handler isinstanceof NullHeaderReplacement), right?
flume-ng-core/src/main/java/org/apache/flume/source/http/JSONHandler.java (line 98)
<https://reviews.apache.org/r/47098/#comment197260>
Nit: Trailing whitespace.
flume-ng-core/src/main/java/org/apache/flume/source/http/NullHeaderReplacement.java (line 1)
<https://reviews.apache.org/r/47098/#comment197261>
Missing licence header.
flume-ng-core/src/test/java/org/apache/flume/source/http/TestJSONHandler.java (lines 161 - 163)
<https://reviews.apache.org/r/47098/#comment197262>
I believe that in the test we can be sure that we have JSONHandler, right?
flume-ng-core/src/test/java/org/apache/flume/source/http/TestJSONHandler.java (lines 174 - 176)
<https://reviews.apache.org/r/47098/#comment197263>
I believe that in the test we can be sure that we have JSONHandler, right?
flume-ng-core/src/test/java/org/apache/flume/source/http/TestJSONHandler.java (lines 214 - 216)
<https://reviews.apache.org/r/47098/#comment197264>
I believe that in the test we can be sure that we have JSONHandler, right?
- Jarek Cecho
On May 12, 2016, 1:03 a.m., neerja khattar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47098/
> -----------------------------------------------------------
>
> (Updated May 12, 2016, 1:03 a.m.)
>
>
> Review request for Flume.
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> The issue is when the header value is null it throws null pointer exception and flume stops processing further events.
> For example:
> [{
> "headers" : {
> "timestamp" : "434324343",
> "host" : null
> },
> "body" : "random_body"
> }]
>
> The solution to fix this is:
>
> 1. If the header has a null value in the json, flume will replace it with a replacement string.
> 2. The default value for a replacement string is an empty string.
> 3. To overwrite default string, set "handler.nullReplacementHeader" property in flume config.
>
>
> Diffs
> -----
>
> flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSource.java b520b03
> flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSourceConfigurationConstants.java 86caf7d
> flume-ng-core/src/main/java/org/apache/flume/source/http/JSONHandler.java 197f66a
> flume-ng-core/src/main/java/org/apache/flume/source/http/NullHeaderReplacement.java PRE-CREATION
> flume-ng-core/src/test/java/org/apache/flume/source/http/TestJSONHandler.java 455781c
>
> Diff: https://reviews.apache.org/r/47098/diff/
>
>
> Testing
> -------
>
> The following are the test cases:
>
> 1. Header has null value in json and handler.nullReplacementHeader is not set in flume config. The default value will be used to replace null.
>
> [{
> "headers" : {
> "timestamp" : "434324343",
> "host" : null
> },
> "body" : "random_body"
> }]
>
> Output in hdfs : {timestamp=434324343, host=} random_body
>
> 2. Header is not null in json and handler.nullReplacementHeader is not set in flume config. The replacement implementation doesnt come in to consideration.
>
> [{
> "headers" : {
> "timestamp" : "434324343",
> "host" : 1
> },
> "body" : "random_body"
> }]
> Output in hdfs : {timestamp=434324343, host=1} random_body
>
> 3. Header has null value in json and handler.nullReplacementHeader=abc is set in flume config. The null value in header will be replaced by abc.
>
>
> [{
> "headers" : {
> "timestamp" : "434324343",
> "host" : null
> },
> "body" : "random_body"
> }]
>
>
> Output in hdfs {timestamp=434324343, host=abc} random_body
>
> 4. Header has null value in json and handler.nullReplacementHeader=1 is set in flume config. The null value in header will be replaced by 1 as a string .
>
> [{
> "headers" : {
> "timestamp" : "434324343",
> "host" : null
> },
> "body" : "random_body"
> }]
>
>
> Output in hdfs: {timestamp=434324343, host=1} random_body
>
> 5. Header is not null in json and handler.nullReplacementHeader is also set in flume config. The replacement implementation doesnt come in to consideration.
>
> [{
> "headers" : {
> "timestamp" : "434324343",
> "host" : 1
> },
> "body" : "random_body"
> }]
> Output in hdfs : {timestamp=434324343, host=1} random_body
>
>
> File Attachments
> ----------------
>
> flume-2620
> https://reviews.apache.org/media/uploaded/files/2016/05/09/0eff1d56-caf3-4d36-bb45-6b9e7fd6a1ff__FLUME-2620-1.patch
>
>
> Thanks,
>
> neerja khattar
>
>
Re: Review Request 47098: FLUME-2620 File channel throws
NullPointerException if a header value is null
Posted by Jarek Cecho <ja...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47098/#review133783
-----------------------------------------------------------
Generally I think that we're almost ready to push this in. My last request is to update the documentation with the new configuration option.
flume-ng-core/src/main/java/org/apache/flume/source/http/JSONHandler.java (line 98)
<https://reviews.apache.org/r/47098/#comment198376>
Nit: Trailing white space.
Jarcec
- Jarek Cecho
On May 13, 2016, 4:43 a.m., neerja khattar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47098/
> -----------------------------------------------------------
>
> (Updated May 13, 2016, 4:43 a.m.)
>
>
> Review request for Flume.
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> The issue is when the header value is null it throws null pointer exception and flume stops processing further events.
> For example:
> [{
> "headers" : {
> "timestamp" : "434324343",
> "host" : null
> },
> "body" : "random_body"
> }]
>
> The solution to fix this is:
>
> 1. If the header has a null value in the json, flume will replace it with a replacement string.
> 2. The default value for a replacement string is an empty string.
> 3. To overwrite default string, set "handler.nullReplacementHeader" property in flume config.
>
>
> Diffs
> -----
>
> flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSource.java b520b03
> flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSourceConfigurationConstants.java 86caf7d
> flume-ng-core/src/main/java/org/apache/flume/source/http/JSONHandler.java 197f66a
> flume-ng-core/src/main/java/org/apache/flume/source/http/NullHeaderReplacement.java PRE-CREATION
> flume-ng-core/src/test/java/org/apache/flume/source/http/TestJSONHandler.java 455781c
>
> Diff: https://reviews.apache.org/r/47098/diff/
>
>
> Testing
> -------
>
> The following are the test cases:
>
> 1. Header has null value in json and handler.nullReplacementHeader is not set in flume config. The default value will be used to replace null.
>
> [{
> "headers" : {
> "timestamp" : "434324343",
> "host" : null
> },
> "body" : "random_body"
> }]
>
> Output in hdfs : {timestamp=434324343, host=} random_body
>
> 2. Header is not null in json and handler.nullReplacementHeader is not set in flume config. The replacement implementation doesnt come in to consideration.
>
> [{
> "headers" : {
> "timestamp" : "434324343",
> "host" : 1
> },
> "body" : "random_body"
> }]
> Output in hdfs : {timestamp=434324343, host=1} random_body
>
> 3. Header has null value in json and handler.nullReplacementHeader=abc is set in flume config. The null value in header will be replaced by abc.
>
>
> [{
> "headers" : {
> "timestamp" : "434324343",
> "host" : null
> },
> "body" : "random_body"
> }]
>
>
> Output in hdfs {timestamp=434324343, host=abc} random_body
>
> 4. Header has null value in json and handler.nullReplacementHeader=1 is set in flume config. The null value in header will be replaced by 1 as a string .
>
> [{
> "headers" : {
> "timestamp" : "434324343",
> "host" : null
> },
> "body" : "random_body"
> }]
>
>
> Output in hdfs: {timestamp=434324343, host=1} random_body
>
> 5. Header is not null in json and handler.nullReplacementHeader is also set in flume config. The replacement implementation doesnt come in to consideration.
>
> [{
> "headers" : {
> "timestamp" : "434324343",
> "host" : 1
> },
> "body" : "random_body"
> }]
> Output in hdfs : {timestamp=434324343, host=1} random_body
>
>
> File Attachments
> ----------------
>
> flume-2620
> https://reviews.apache.org/media/uploaded/files/2016/05/09/0eff1d56-caf3-4d36-bb45-6b9e7fd6a1ff__FLUME-2620-1.patch
>
>
> Thanks,
>
> neerja khattar
>
>
Re: Review Request 47098: FLUME-2620 File channel throws
NullPointerException if a header value is null
Posted by Mike Percy <mp...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47098/#review140181
-----------------------------------------------------------
This doesn't look like it changes the file channel at all. Isn't that the underlying issue? Couldn't the FileChannel allow a null value in a header, or treat it as an empty string? Maybe that would be a cleaner solution.
- Mike Percy
On May 24, 2016, 4:11 p.m., neerja khattar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47098/
> -----------------------------------------------------------
>
> (Updated May 24, 2016, 4:11 p.m.)
>
>
> Review request for Flume.
>
>
> Repository: flume-git
>
>
> Description
> -------
>
> The issue is when the header value is null it throws null pointer exception and flume stops processing further events.
> For example:
> [{
> "headers" : {
> "timestamp" : "434324343",
> "host" : null
> },
> "body" : "random_body"
> }]
>
> The solution to fix this is:
>
> 1. If the header has a null value in the json, flume will replace it with a replacement string.
> 2. The default value for a replacement string is an empty string.
> 3. To overwrite default string, set "handler.nullReplacementHeader" property in flume config.
>
>
> Diffs
> -----
>
> flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSource.java b520b03
> flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSourceConfigurationConstants.java 86caf7d
> flume-ng-core/src/main/java/org/apache/flume/source/http/JSONHandler.java 197f66a
> flume-ng-core/src/main/java/org/apache/flume/source/http/NullHeaderReplacement.java PRE-CREATION
> flume-ng-core/src/test/java/org/apache/flume/source/http/TestJSONHandler.java 455781c
> flume-ng-doc/sphinx/FlumeUserGuide.rst e04fd50
>
> Diff: https://reviews.apache.org/r/47098/diff/
>
>
> Testing
> -------
>
> The following are the test cases:
>
> 1. Header has null value in json and handler.nullReplacementHeader is not set in flume config. The default value will be used to replace null.
>
> [{
> "headers" : {
> "timestamp" : "434324343",
> "host" : null
> },
> "body" : "random_body"
> }]
>
> Output in hdfs : {timestamp=434324343, host=} random_body
>
> 2. Header is not null in json and handler.nullReplacementHeader is not set in flume config. The replacement implementation doesnt come in to consideration.
>
> [{
> "headers" : {
> "timestamp" : "434324343",
> "host" : 1
> },
> "body" : "random_body"
> }]
> Output in hdfs : {timestamp=434324343, host=1} random_body
>
> 3. Header has null value in json and handler.nullReplacementHeader=abc is set in flume config. The null value in header will be replaced by abc.
>
>
> [{
> "headers" : {
> "timestamp" : "434324343",
> "host" : null
> },
> "body" : "random_body"
> }]
>
>
> Output in hdfs {timestamp=434324343, host=abc} random_body
>
> 4. Header has null value in json and handler.nullReplacementHeader=1 is set in flume config. The null value in header will be replaced by 1 as a string .
>
> [{
> "headers" : {
> "timestamp" : "434324343",
> "host" : null
> },
> "body" : "random_body"
> }]
>
>
> Output in hdfs: {timestamp=434324343, host=1} random_body
>
> 5. Header is not null in json and handler.nullReplacementHeader is also set in flume config. The replacement implementation doesnt come in to consideration.
>
> [{
> "headers" : {
> "timestamp" : "434324343",
> "host" : 1
> },
> "body" : "random_body"
> }]
> Output in hdfs : {timestamp=434324343, host=1} random_body
>
>
> File Attachments
> ----------------
>
> flume-2620
> https://reviews.apache.org/media/uploaded/files/2016/05/09/0eff1d56-caf3-4d36-bb45-6b9e7fd6a1ff__FLUME-2620-1.patch
>
>
> Thanks,
>
> neerja khattar
>
>
Re: Review Request 47098: FLUME-2620 File channel throws
NullPointerException if a header value is null
Posted by neerja khattar <ne...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47098/
-----------------------------------------------------------
(Updated May 24, 2016, 4:11 p.m.)
Review request for Flume.
Changes
-------
added the document change.
Repository: flume-git
Description
-------
The issue is when the header value is null it throws null pointer exception and flume stops processing further events.
For example:
[{
"headers" : {
"timestamp" : "434324343",
"host" : null
},
"body" : "random_body"
}]
The solution to fix this is:
1. If the header has a null value in the json, flume will replace it with a replacement string.
2. The default value for a replacement string is an empty string.
3. To overwrite default string, set "handler.nullReplacementHeader" property in flume config.
Diffs (updated)
-----
flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSource.java b520b03
flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSourceConfigurationConstants.java 86caf7d
flume-ng-core/src/main/java/org/apache/flume/source/http/JSONHandler.java 197f66a
flume-ng-core/src/main/java/org/apache/flume/source/http/NullHeaderReplacement.java PRE-CREATION
flume-ng-core/src/test/java/org/apache/flume/source/http/TestJSONHandler.java 455781c
flume-ng-doc/sphinx/FlumeUserGuide.rst e04fd50
Diff: https://reviews.apache.org/r/47098/diff/
Testing
-------
The following are the test cases:
1. Header has null value in json and handler.nullReplacementHeader is not set in flume config. The default value will be used to replace null.
[{
"headers" : {
"timestamp" : "434324343",
"host" : null
},
"body" : "random_body"
}]
Output in hdfs : {timestamp=434324343, host=} random_body
2. Header is not null in json and handler.nullReplacementHeader is not set in flume config. The replacement implementation doesnt come in to consideration.
[{
"headers" : {
"timestamp" : "434324343",
"host" : 1
},
"body" : "random_body"
}]
Output in hdfs : {timestamp=434324343, host=1} random_body
3. Header has null value in json and handler.nullReplacementHeader=abc is set in flume config. The null value in header will be replaced by abc.
[{
"headers" : {
"timestamp" : "434324343",
"host" : null
},
"body" : "random_body"
}]
Output in hdfs {timestamp=434324343, host=abc} random_body
4. Header has null value in json and handler.nullReplacementHeader=1 is set in flume config. The null value in header will be replaced by 1 as a string .
[{
"headers" : {
"timestamp" : "434324343",
"host" : null
},
"body" : "random_body"
}]
Output in hdfs: {timestamp=434324343, host=1} random_body
5. Header is not null in json and handler.nullReplacementHeader is also set in flume config. The replacement implementation doesnt come in to consideration.
[{
"headers" : {
"timestamp" : "434324343",
"host" : 1
},
"body" : "random_body"
}]
Output in hdfs : {timestamp=434324343, host=1} random_body
File Attachments
----------------
flume-2620
https://reviews.apache.org/media/uploaded/files/2016/05/09/0eff1d56-caf3-4d36-bb45-6b9e7fd6a1ff__FLUME-2620-1.patch
Thanks,
neerja khattar
Re: Review Request 47098: FLUME-2620 File channel throws
NullPointerException if a header value is null
Posted by neerja khattar <ne...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47098/
-----------------------------------------------------------
(Updated May 13, 2016, 4:43 a.m.)
Review request for Flume.
Changes
-------
fixed all the changes.
Repository: flume-git
Description
-------
The issue is when the header value is null it throws null pointer exception and flume stops processing further events.
For example:
[{
"headers" : {
"timestamp" : "434324343",
"host" : null
},
"body" : "random_body"
}]
The solution to fix this is:
1. If the header has a null value in the json, flume will replace it with a replacement string.
2. The default value for a replacement string is an empty string.
3. To overwrite default string, set "handler.nullReplacementHeader" property in flume config.
Diffs (updated)
-----
flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSource.java b520b03
flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSourceConfigurationConstants.java 86caf7d
flume-ng-core/src/main/java/org/apache/flume/source/http/JSONHandler.java 197f66a
flume-ng-core/src/main/java/org/apache/flume/source/http/NullHeaderReplacement.java PRE-CREATION
flume-ng-core/src/test/java/org/apache/flume/source/http/TestJSONHandler.java 455781c
Diff: https://reviews.apache.org/r/47098/diff/
Testing
-------
The following are the test cases:
1. Header has null value in json and handler.nullReplacementHeader is not set in flume config. The default value will be used to replace null.
[{
"headers" : {
"timestamp" : "434324343",
"host" : null
},
"body" : "random_body"
}]
Output in hdfs : {timestamp=434324343, host=} random_body
2. Header is not null in json and handler.nullReplacementHeader is not set in flume config. The replacement implementation doesnt come in to consideration.
[{
"headers" : {
"timestamp" : "434324343",
"host" : 1
},
"body" : "random_body"
}]
Output in hdfs : {timestamp=434324343, host=1} random_body
3. Header has null value in json and handler.nullReplacementHeader=abc is set in flume config. The null value in header will be replaced by abc.
[{
"headers" : {
"timestamp" : "434324343",
"host" : null
},
"body" : "random_body"
}]
Output in hdfs {timestamp=434324343, host=abc} random_body
4. Header has null value in json and handler.nullReplacementHeader=1 is set in flume config. The null value in header will be replaced by 1 as a string .
[{
"headers" : {
"timestamp" : "434324343",
"host" : null
},
"body" : "random_body"
}]
Output in hdfs: {timestamp=434324343, host=1} random_body
5. Header is not null in json and handler.nullReplacementHeader is also set in flume config. The replacement implementation doesnt come in to consideration.
[{
"headers" : {
"timestamp" : "434324343",
"host" : 1
},
"body" : "random_body"
}]
Output in hdfs : {timestamp=434324343, host=1} random_body
File Attachments
----------------
flume-2620
https://reviews.apache.org/media/uploaded/files/2016/05/09/0eff1d56-caf3-4d36-bb45-6b9e7fd6a1ff__FLUME-2620-1.patch
Thanks,
neerja khattar
Re: Review Request 47098: FLUME-2620 File channel throws
NullPointerException if a header value is null
Posted by neerja khattar <ne...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47098/
-----------------------------------------------------------
(Updated May 12, 2016, 1:03 a.m.)
Review request for Flume.
Changes
-------
Implemented the following change: add an abstract class that inherits this one, and have an instance of check in the Source itself, and call this method only if the handler is an instance of the abstract class
Repository: flume-git
Description
-------
The issue is when the header value is null it throws null pointer exception and flume stops processing further events.
For example:
[{
"headers" : {
"timestamp" : "434324343",
"host" : null
},
"body" : "random_body"
}]
The solution to fix this is:
1. If the header has a null value in the json, flume will replace it with a replacement string.
2. The default value for a replacement string is an empty string.
3. To overwrite default string, set "handler.nullReplacementHeader" property in flume config.
Diffs (updated)
-----
flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSource.java b520b03
flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSourceConfigurationConstants.java 86caf7d
flume-ng-core/src/main/java/org/apache/flume/source/http/JSONHandler.java 197f66a
flume-ng-core/src/main/java/org/apache/flume/source/http/NullHeaderReplacement.java PRE-CREATION
flume-ng-core/src/test/java/org/apache/flume/source/http/TestJSONHandler.java 455781c
Diff: https://reviews.apache.org/r/47098/diff/
Testing
-------
The following are the test cases:
1. Header has null value in json and handler.nullReplacementHeader is not set in flume config. The default value will be used to replace null.
[{
"headers" : {
"timestamp" : "434324343",
"host" : null
},
"body" : "random_body"
}]
Output in hdfs : {timestamp=434324343, host=} random_body
2. Header is not null in json and handler.nullReplacementHeader is not set in flume config. The replacement implementation doesnt come in to consideration.
[{
"headers" : {
"timestamp" : "434324343",
"host" : 1
},
"body" : "random_body"
}]
Output in hdfs : {timestamp=434324343, host=1} random_body
3. Header has null value in json and handler.nullReplacementHeader=abc is set in flume config. The null value in header will be replaced by abc.
[{
"headers" : {
"timestamp" : "434324343",
"host" : null
},
"body" : "random_body"
}]
Output in hdfs {timestamp=434324343, host=abc} random_body
4. Header has null value in json and handler.nullReplacementHeader=1 is set in flume config. The null value in header will be replaced by 1 as a string .
[{
"headers" : {
"timestamp" : "434324343",
"host" : null
},
"body" : "random_body"
}]
Output in hdfs: {timestamp=434324343, host=1} random_body
5. Header is not null in json and handler.nullReplacementHeader is also set in flume config. The replacement implementation doesnt come in to consideration.
[{
"headers" : {
"timestamp" : "434324343",
"host" : 1
},
"body" : "random_body"
}]
Output in hdfs : {timestamp=434324343, host=1} random_body
File Attachments
----------------
flume-2620
https://reviews.apache.org/media/uploaded/files/2016/05/09/0eff1d56-caf3-4d36-bb45-6b9e7fd6a1ff__FLUME-2620-1.patch
Thanks,
neerja khattar
Re: Review Request 47098: FLUME-2620 File channel throws
NullPointerException if a header value is null
Posted by neerja khattar <ne...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47098/
-----------------------------------------------------------
(Updated May 9, 2016, 4:57 a.m.)
Review request for Flume.
Changes
-------
Changed the HttpSourceHandler interface to abstract class so that BlobHandler class doesnt have to implement setNullHeaderReplacement Method.
Repository: flume-git
Description
-------
The issue is when the header value is null it throws null pointer exception and flume stops processing further events.
For example:
[{
"headers" : {
"timestamp" : "434324343",
"host" : null
},
"body" : "random_body"
}]
The solution to fix this is:
1. If the header has a null value in the json, flume will replace it with a replacement string.
2. The default value for a replacement string is an empty string.
3. To overwrite default string, set "handler.nullReplacementHeader" property in flume config.
Diffs (updated)
-----
flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSource.java b520b03
flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSourceConfigurationConstants.java 86caf7d
flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSourceHandler.java 726bf0c
flume-ng-core/src/main/java/org/apache/flume/source/http/JSONHandler.java 197f66a
flume-ng-core/src/test/java/org/apache/flume/source/http/TestJSONHandler.java 455781c
flume-ng-sinks/flume-ng-morphline-solr-sink/src/main/java/org/apache/flume/sink/solr/morphline/BlobHandler.java e84dec1
Diff: https://reviews.apache.org/r/47098/diff/
Testing
-------
The following are the test cases:
1. Header has null value in json and handler.nullReplacementHeader is not set in flume config. The default value will be used to replace null.
[{
"headers" : {
"timestamp" : "434324343",
"host" : null
},
"body" : "random_body"
}]
Output in hdfs : {timestamp=434324343, host=} random_body
2. Header is not null in json and handler.nullReplacementHeader is not set in flume config. The replacement implementation doesnt come in to consideration.
[{
"headers" : {
"timestamp" : "434324343",
"host" : 1
},
"body" : "random_body"
}]
Output in hdfs : {timestamp=434324343, host=1} random_body
3. Header has null value in json and handler.nullReplacementHeader=abc is set in flume config. The null value in header will be replaced by abc.
[{
"headers" : {
"timestamp" : "434324343",
"host" : null
},
"body" : "random_body"
}]
Output in hdfs {timestamp=434324343, host=abc} random_body
4. Header has null value in json and handler.nullReplacementHeader=1 is set in flume config. The null value in header will be replaced by 1 as a string .
[{
"headers" : {
"timestamp" : "434324343",
"host" : null
},
"body" : "random_body"
}]
Output in hdfs: {timestamp=434324343, host=1} random_body
5. Header is not null in json and handler.nullReplacementHeader is also set in flume config. The replacement implementation doesnt come in to consideration.
[{
"headers" : {
"timestamp" : "434324343",
"host" : 1
},
"body" : "random_body"
}]
Output in hdfs : {timestamp=434324343, host=1} random_body
File Attachments (updated)
----------------
flume-2620
https://reviews.apache.org/media/uploaded/files/2016/05/09/0eff1d56-caf3-4d36-bb45-6b9e7fd6a1ff__FLUME-2620-1.patch
Thanks,
neerja khattar