You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by Brock Noland <br...@cloudera.com> on 2012/10/30 20:08:42 UTC

Review Request: FLUME-1277: Error parsing Syslog rfc 3164 messages with null values

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7780/
-----------------------------------------------------------

Review request for Flume.


Description
-------

The SyslogUtils class doesn't properly parse rfc 3164 style messages containing a null (hyphen) value. e.g., 
<10>Apr 1 13:14:04 ubuntu-11.cloudera.com - rest_of_message

It tries to parse it as a 5424 style message, skips over the date information, and interprets the first hyphen as a null timestamp. Part of the problem is the use of a Scanner and regex. This skips over a properly formatted 3164 style message until it finds anything that matches the 5424 regex, including a hyphen.


This addresses bug FLUME-1227.
    https://issues.apache.org/jira/browse/FLUME-1227


Diffs
-----

  flume-ng-core/src/main/java/org/apache/flume/source/SyslogUtils.java 4cded11 
  flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUtils.java acfb29d 

Diff: https://reviews.apache.org/r/7780/diff/


Testing
-------

Unit test added and units pass


Thanks,

Brock Noland


Re: Review Request: FLUME-1277: Error parsing Syslog rfc 3164 messages with null values

Posted by Mike Percy <mp...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7780/#review13331
-----------------------------------------------------------

Ship it!


Ship It!

- Mike Percy


On Nov. 10, 2012, 4:24 p.m., Brock Noland wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/7780/
> -----------------------------------------------------------
> 
> (Updated Nov. 10, 2012, 4:24 p.m.)
> 
> 
> Review request for Flume.
> 
> 
> Description
> -------
> 
> The SyslogUtils class doesn't properly parse rfc 3164 style messages containing a null (hyphen) value. e.g., 
> <10>Apr 1 13:14:04 ubuntu-11.cloudera.com - rest_of_message
> 
> It tries to parse it as a 5424 style message, skips over the date information, and interprets the first hyphen as a null timestamp. Part of the problem is the use of a Scanner and regex. This skips over a properly formatted 3164 style message until it finds anything that matches the 5424 regex, including a hyphen.
> 
> 
> This addresses bug FLUME-1277.
>     https://issues.apache.org/jira/browse/FLUME-1277
> 
> 
> Diffs
> -----
> 
>   flume-ng-core/src/main/java/org/apache/flume/source/SyslogUtils.java 4cded11 
>   flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUtils.java acfb29d 
> 
> Diff: https://reviews.apache.org/r/7780/diff/
> 
> 
> Testing
> -------
> 
> Unit test added and units pass
> 
> 
> Thanks,
> 
> Brock Noland
> 
>


Re: Review Request: FLUME-1277: Error parsing Syslog rfc 3164 messages with null values

Posted by Brock Noland <br...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7780/
-----------------------------------------------------------

(Updated Nov. 10, 2012, 4:24 p.m.)


Review request for Flume.


Changes
-------

Updated patch based on review


Description
-------

The SyslogUtils class doesn't properly parse rfc 3164 style messages containing a null (hyphen) value. e.g., 
<10>Apr 1 13:14:04 ubuntu-11.cloudera.com - rest_of_message

It tries to parse it as a 5424 style message, skips over the date information, and interprets the first hyphen as a null timestamp. Part of the problem is the use of a Scanner and regex. This skips over a properly formatted 3164 style message until it finds anything that matches the 5424 regex, including a hyphen.


This addresses bug FLUME-1277.
    https://issues.apache.org/jira/browse/FLUME-1277


Diffs (updated)
-----

  flume-ng-core/src/main/java/org/apache/flume/source/SyslogUtils.java 4cded11 
  flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUtils.java acfb29d 

Diff: https://reviews.apache.org/r/7780/diff/


Testing
-------

Unit test added and units pass


Thanks,

Brock Noland


Re: Review Request: FLUME-1277: Error parsing Syslog rfc 3164 messages with null values

Posted by Mike Percy <mp...@apache.org>.

> On Nov. 10, 2012, 2:05 a.m., Mike Percy wrote:
> > flume-ng-core/src/main/java/org/apache/flume/source/SyslogUtils.java, line 247
> > <https://reviews.apache.org/r/7780/diff/1/?file=182869#file182869line247>
> >
> >     We should avoid calling baos.toString() inside the loop over the configured formats. Better to call it once at the top since this will likely have a performance impact.

(Otherwise, this patch looks fine to me)


- Mike


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7780/#review13315
-----------------------------------------------------------


On Oct. 30, 2012, 7:10 p.m., Brock Noland wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/7780/
> -----------------------------------------------------------
> 
> (Updated Oct. 30, 2012, 7:10 p.m.)
> 
> 
> Review request for Flume.
> 
> 
> Description
> -------
> 
> The SyslogUtils class doesn't properly parse rfc 3164 style messages containing a null (hyphen) value. e.g., 
> <10>Apr 1 13:14:04 ubuntu-11.cloudera.com - rest_of_message
> 
> It tries to parse it as a 5424 style message, skips over the date information, and interprets the first hyphen as a null timestamp. Part of the problem is the use of a Scanner and regex. This skips over a properly formatted 3164 style message until it finds anything that matches the 5424 regex, including a hyphen.
> 
> 
> This addresses bug FLUME-1277.
>     https://issues.apache.org/jira/browse/FLUME-1277
> 
> 
> Diffs
> -----
> 
>   flume-ng-core/src/main/java/org/apache/flume/source/SyslogUtils.java 4cded11 
>   flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUtils.java acfb29d 
> 
> Diff: https://reviews.apache.org/r/7780/diff/
> 
> 
> Testing
> -------
> 
> Unit test added and units pass
> 
> 
> Thanks,
> 
> Brock Noland
> 
>


Re: Review Request: FLUME-1277: Error parsing Syslog rfc 3164 messages with null values

Posted by Mike Percy <mp...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7780/#review13315
-----------------------------------------------------------



flume-ng-core/src/main/java/org/apache/flume/source/SyslogUtils.java
<https://reviews.apache.org/r/7780/#comment28581>

    We should avoid calling baos.toString() inside the loop over the configured formats. Better to call it once at the top since this will likely have a performance impact.


- Mike Percy


On Oct. 30, 2012, 7:10 p.m., Brock Noland wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/7780/
> -----------------------------------------------------------
> 
> (Updated Oct. 30, 2012, 7:10 p.m.)
> 
> 
> Review request for Flume.
> 
> 
> Description
> -------
> 
> The SyslogUtils class doesn't properly parse rfc 3164 style messages containing a null (hyphen) value. e.g., 
> <10>Apr 1 13:14:04 ubuntu-11.cloudera.com - rest_of_message
> 
> It tries to parse it as a 5424 style message, skips over the date information, and interprets the first hyphen as a null timestamp. Part of the problem is the use of a Scanner and regex. This skips over a properly formatted 3164 style message until it finds anything that matches the 5424 regex, including a hyphen.
> 
> 
> This addresses bug FLUME-1277.
>     https://issues.apache.org/jira/browse/FLUME-1277
> 
> 
> Diffs
> -----
> 
>   flume-ng-core/src/main/java/org/apache/flume/source/SyslogUtils.java 4cded11 
>   flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUtils.java acfb29d 
> 
> Diff: https://reviews.apache.org/r/7780/diff/
> 
> 
> Testing
> -------
> 
> Unit test added and units pass
> 
> 
> Thanks,
> 
> Brock Noland
> 
>


Re: Review Request: FLUME-1277: Error parsing Syslog rfc 3164 messages with null values

Posted by Alexander Alten-Lorenz <al...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7780/#review13150
-----------------------------------------------------------

Ship it!


Ship It!

- Alexander Alten-Lorenz


On Oct. 30, 2012, 7:10 p.m., Brock Noland wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/7780/
> -----------------------------------------------------------
> 
> (Updated Oct. 30, 2012, 7:10 p.m.)
> 
> 
> Review request for Flume.
> 
> 
> Description
> -------
> 
> The SyslogUtils class doesn't properly parse rfc 3164 style messages containing a null (hyphen) value. e.g., 
> <10>Apr 1 13:14:04 ubuntu-11.cloudera.com - rest_of_message
> 
> It tries to parse it as a 5424 style message, skips over the date information, and interprets the first hyphen as a null timestamp. Part of the problem is the use of a Scanner and regex. This skips over a properly formatted 3164 style message until it finds anything that matches the 5424 regex, including a hyphen.
> 
> 
> This addresses bug FLUME-1277.
>     https://issues.apache.org/jira/browse/FLUME-1277
> 
> 
> Diffs
> -----
> 
>   flume-ng-core/src/main/java/org/apache/flume/source/SyslogUtils.java 4cded11 
>   flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUtils.java acfb29d 
> 
> Diff: https://reviews.apache.org/r/7780/diff/
> 
> 
> Testing
> -------
> 
> Unit test added and units pass
> 
> 
> Thanks,
> 
> Brock Noland
> 
>


Re: Review Request: FLUME-1277: Error parsing Syslog rfc 3164 messages with null values

Posted by Brock Noland <br...@cloudera.com>.

> On Nov. 6, 2012, 8:42 a.m., Alexander Alten-Lorenz wrote:
> > flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUtils.java, line 162
> > <https://reviews.apache.org/r/7780/diff/1/?file=182870#file182870line162>
> >
> >     Can we change cloudera.com into something else?

Doesn't really matter does it? Just a place holder domain.


- Brock


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7780/#review13132
-----------------------------------------------------------


On Oct. 30, 2012, 7:10 p.m., Brock Noland wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/7780/
> -----------------------------------------------------------
> 
> (Updated Oct. 30, 2012, 7:10 p.m.)
> 
> 
> Review request for Flume.
> 
> 
> Description
> -------
> 
> The SyslogUtils class doesn't properly parse rfc 3164 style messages containing a null (hyphen) value. e.g., 
> <10>Apr 1 13:14:04 ubuntu-11.cloudera.com - rest_of_message
> 
> It tries to parse it as a 5424 style message, skips over the date information, and interprets the first hyphen as a null timestamp. Part of the problem is the use of a Scanner and regex. This skips over a properly formatted 3164 style message until it finds anything that matches the 5424 regex, including a hyphen.
> 
> 
> This addresses bug FLUME-1277.
>     https://issues.apache.org/jira/browse/FLUME-1277
> 
> 
> Diffs
> -----
> 
>   flume-ng-core/src/main/java/org/apache/flume/source/SyslogUtils.java 4cded11 
>   flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUtils.java acfb29d 
> 
> Diff: https://reviews.apache.org/r/7780/diff/
> 
> 
> Testing
> -------
> 
> Unit test added and units pass
> 
> 
> Thanks,
> 
> Brock Noland
> 
>


Re: Review Request: FLUME-1277: Error parsing Syslog rfc 3164 messages with null values

Posted by Alexander Alten-Lorenz <al...@cloudera.com>.

> On Nov. 6, 2012, 8:42 a.m., Alexander Alten-Lorenz wrote:
> > flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUtils.java, line 162
> > <https://reviews.apache.org/r/7780/diff/1/?file=182870#file182870line162>
> >
> >     Can we change cloudera.com into something else?
> 
> Brock Noland wrote:
>     Doesn't really matter does it? Just a place holder domain.

not really, but some kind of example.com|apache.org would look better, I think. Just my impression. Ship it


- Alexander


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7780/#review13132
-----------------------------------------------------------


On Oct. 30, 2012, 7:10 p.m., Brock Noland wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/7780/
> -----------------------------------------------------------
> 
> (Updated Oct. 30, 2012, 7:10 p.m.)
> 
> 
> Review request for Flume.
> 
> 
> Description
> -------
> 
> The SyslogUtils class doesn't properly parse rfc 3164 style messages containing a null (hyphen) value. e.g., 
> <10>Apr 1 13:14:04 ubuntu-11.cloudera.com - rest_of_message
> 
> It tries to parse it as a 5424 style message, skips over the date information, and interprets the first hyphen as a null timestamp. Part of the problem is the use of a Scanner and regex. This skips over a properly formatted 3164 style message until it finds anything that matches the 5424 regex, including a hyphen.
> 
> 
> This addresses bug FLUME-1277.
>     https://issues.apache.org/jira/browse/FLUME-1277
> 
> 
> Diffs
> -----
> 
>   flume-ng-core/src/main/java/org/apache/flume/source/SyslogUtils.java 4cded11 
>   flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUtils.java acfb29d 
> 
> Diff: https://reviews.apache.org/r/7780/diff/
> 
> 
> Testing
> -------
> 
> Unit test added and units pass
> 
> 
> Thanks,
> 
> Brock Noland
> 
>


Re: Review Request: FLUME-1277: Error parsing Syslog rfc 3164 messages with null values

Posted by Alexander Alten-Lorenz <al...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7780/#review13132
-----------------------------------------------------------



flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUtils.java
<https://reviews.apache.org/r/7780/#comment28291>

    Can we change cloudera.com into something else?


- Alexander Alten-Lorenz


On Oct. 30, 2012, 7:10 p.m., Brock Noland wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/7780/
> -----------------------------------------------------------
> 
> (Updated Oct. 30, 2012, 7:10 p.m.)
> 
> 
> Review request for Flume.
> 
> 
> Description
> -------
> 
> The SyslogUtils class doesn't properly parse rfc 3164 style messages containing a null (hyphen) value. e.g., 
> <10>Apr 1 13:14:04 ubuntu-11.cloudera.com - rest_of_message
> 
> It tries to parse it as a 5424 style message, skips over the date information, and interprets the first hyphen as a null timestamp. Part of the problem is the use of a Scanner and regex. This skips over a properly formatted 3164 style message until it finds anything that matches the 5424 regex, including a hyphen.
> 
> 
> This addresses bug FLUME-1277.
>     https://issues.apache.org/jira/browse/FLUME-1277
> 
> 
> Diffs
> -----
> 
>   flume-ng-core/src/main/java/org/apache/flume/source/SyslogUtils.java 4cded11 
>   flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUtils.java acfb29d 
> 
> Diff: https://reviews.apache.org/r/7780/diff/
> 
> 
> Testing
> -------
> 
> Unit test added and units pass
> 
> 
> Thanks,
> 
> Brock Noland
> 
>


Re: Review Request: FLUME-1277: Error parsing Syslog rfc 3164 messages with null values

Posted by Brock Noland <br...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7780/
-----------------------------------------------------------

(Updated Oct. 30, 2012, 7:10 p.m.)


Review request for Flume.


Changes
-------

Updated JIRA #


Description
-------

The SyslogUtils class doesn't properly parse rfc 3164 style messages containing a null (hyphen) value. e.g., 
<10>Apr 1 13:14:04 ubuntu-11.cloudera.com - rest_of_message

It tries to parse it as a 5424 style message, skips over the date information, and interprets the first hyphen as a null timestamp. Part of the problem is the use of a Scanner and regex. This skips over a properly formatted 3164 style message until it finds anything that matches the 5424 regex, including a hyphen.


This addresses bug FLUME-1277.
    https://issues.apache.org/jira/browse/FLUME-1277


Diffs
-----

  flume-ng-core/src/main/java/org/apache/flume/source/SyslogUtils.java 4cded11 
  flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUtils.java acfb29d 

Diff: https://reviews.apache.org/r/7780/diff/


Testing
-------

Unit test added and units pass


Thanks,

Brock Noland