You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Felix McPherson <lj...@yahoo.se> on 2019/01/03 20:13:01 UTC
ParseCEF processor fails when Msg field exceed 1023 characters
Hi,
I'm using the ParseCEF processor to parse CEF message to Json format. Unfortunately the ParseCEF processor fails for message/events that holds a string in the Msg field that has more than 1023 character. According to the CEF standard the Msg field in an event shall not exceed 1023 character. The PARSECEF fails with:
"Error
ParseCEF[id=...] Failed to parse...
...as a CEF message; it does not conform to the CEF standard; routing to failure.
Any ideas on a workaround for this problem? I would prefer not having to remove character in the Msg field string.
Regards,lj
Sv: Re: ParseCEF processor fails when Msg field exceed 1023
characters
Posted by Felix McPherson <lj...@yahoo.se>.
Hi,
I will test your idea to use RouteOnContent with a length check before the ParseCEF processor.
Since all message needs to be parsed, even if the msg field exceed 1023 character, I will probably have to save the content of the msg field temporarily, truncate the msg field in the message, parse the message and then reassign the original content to the msg field in the parsed message. Not sure that it is possible but that would probably be a suitable workaround.
Regards,
Lj
På tor, jan. 2019 klockan 22:58, Andy LoPresto<al...@apache.org> skrev: You may be able to do some light detection by using a regex in RouteOnContent (not RouteText, which evaluates lines individually) to capture the group and perform a length check before trying to parse the CEF format. However, if the spec states that the field cannot contain more than 1023 characters, there isn’t anything NiFi can do to change that though.
Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
On Jan 3, 2019, at 12:13 PM, Felix McPherson <lj...@yahoo.se> wrote:
Hi,
I'm using the ParseCEF processor to parse CEF message to Json format. Unfortunately the ParseCEF processor fails for message/events that holds a string in the Msg field that has more than 1023 character. According to the CEF standard the Msg field in an event shall not exceed 1023 character. The PARSECEF fails with:
"Error
ParseCEF[id=...] Failed to parse...
...as a CEF message; it does not conform to the CEF standard; routing to failure.
Any ideas on a workaround for this problem? I would prefer not having to remove character in the Msg field string.
Regards,lj
Re: ParseCEF processor fails when Msg field exceed 1023 characters
Posted by Andy LoPresto <al...@apache.org>.
You may be able to do some light detection by using a regex in RouteOnContent (not RouteText, which evaluates lines individually) to capture the group and perform a length check before trying to parse the CEF format. However, if the spec states that the field cannot contain more than 1023 characters, there isn’t anything NiFi can do to change that though.
Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Jan 3, 2019, at 12:13 PM, Felix McPherson <lj...@yahoo.se> wrote:
>
> Hi,
> I'm using the ParseCEF processor to parse CEF message to Json format. Unfortunately the ParseCEF processor fails for message/events that holds a string in the Msg field that has more than 1023 character. According to the CEF standard the Msg field in an event shall not exceed 1023 character. The PARSECEF fails with:
>
> "Error
> ParseCEF[id=...] Failed to parse...
> ...as a CEF message; it does not conform to the CEF standard; routing to failure.
>
> Any ideas on a workaround for this problem? I would prefer not having to remove character in the Msg field string.
> Regards,lj
Sv: Re: ParseCEF processor fails when Msg field exceed 1023
characters
Posted by Felix McPherson <lj...@yahoo.se>.
Hi, I`m quite sure, I have verified it in a small test Flow. A propert to set the do_validate flag to false would be helpful in this case. Regards, Lj
På fre, jan. 2019 klockan 0:05, Otto Fowler<ot...@gmail.com> skrev: #yiv4068205678 body{font-family:Helvetica, Arial;font-size:13px;}Can I ask how you are sure it is the message size that is causing the error? The parser returns null for any error parsing, so the processor doesn’t know what happened. It could be that the message didn’t validate, or something else.
If the issues _is_ with the validator, then we could allow a property to optionally call the parser with the do validate flag to false.
Maybe you can create a jira with a sanitized example line that causes the error?
On January 3, 2019 at 15:13:14, Felix McPherson (ljungpipare@yahoo.se) wrote:
Hi,
I'musing the ParseCEF processor to parse CEF message to Json format.Unfortunately the ParseCEF processor fails for message/events thatholds a string in the Msg field that has more than 1023 character.According to the CEF standard the Msg field in an event shall notexceed 1023 character. The PARSECEF fails with:
"Error
ParseCEF[id=...] Failed to parse...
...as a CEF message; it does not conform to the CEF standard;routing to failure.
Any ideas on a workaround for this problem? I would prefer nothaving to remove character in the Msg field string.
Regards,lj
Re: ParseCEF processor fails when Msg field exceed 1023 characters
Posted by Andre <an...@fucs.org>.
Otto,
Depends on what we choose to let go...
After all, one could be saving to HDFS as Parquet, ORC or anything
involving schemas where type mismatch or length (in case of JSON schemas
for example) it may be an issue...
This is illustrated by the sacrifices required in here:
https://github.com/fluenda/SecuritySchemas/blame/master/README.md#L11
Logs... logs... logs...
Again, I am not fully opposed to the idea of being more lenient, reason why
I allowed ParCEFone has a method allowing it:
https://github.com/fluenda/ParCEFone/blob/master/src/main/java/com/fluenda/parcefone/parser/CEFParser.java#L78
But I suspect we may need to polish the failure handling a bit more.
Cheers
PS-Sight... Sometimes I miss my old security data engineering life! :-)
On Mon, Jan 7, 2019 at 8:41 AM Otto Fowler <ot...@gmail.com> wrote:
> Well, in that case they would use strict mode and have to jump through
> hoops with other processors. If they are storing into HDFS, or mongo or ES
> they would not have that issue right
>
>
> On January 6, 2019 at 14:47:43, Andre (andre-lists@fucs.org) wrote:
>
> Otto,
>
> Yes we can, but what if receiving system relies on char(1023) for one of
> its columns?
>
> In a hindsight, I should probably been even more strict and made the
> processor to reject messages that are too large (I believe message limit is
> 4k chars) before feeding them to the parsers, avoiding "overflows" and
> failing fast.
>
> Having said that, I am not such a CEF zealot and I will be happy to
> consider the implementation. PRs welcome :-)
>
> Cheers
>
>
> On Sun., 6 Jan. 2019, 14:50 Otto Fowler <ottobackwards@gmail.com wrote:
>
>> Would you consider a message, fully formed and parsed, that only
>> invalidates on size constraints to not be allowable? Could there not be
>> some form of validation that is a compromise between those concerns and
>> some otherwise logically correct message who breaks the size constraint?
>> It is imaginable that a source formats a correct CEF message, but never
>> checks the size in the field.
>>
>>
>> On January 4, 2019 at 19:57:36, Andre (andre-lists@fucs.org) wrote:
>>
>> Otto,
>>
>> I considered that but never implemented it in NiFi as I had concerns
>> people would have the incorrect assumption the parser can help on those
>> cases.
>>
>> From the top of my head, I have seen type mismatch, length overflows,
>> malformed CEF headers (the pipe delimited section of the CEF message). I
>> wouldn't be surprised there are a lot of other poorly formed messages.
>>
>> Cheers
>>
>> On Fri., 4 Jan. 2019, 23:40 Otto Fowler <ottobackwards@gmail.com wrote:
>>
>>> Yeah, I went through the code after my mail last night. Would having
>>> validation optional not be a valid setting? How often would the output
>>> feeding this flow be invalid for some other reason?
>>>
>>>
>>>
>>> On January 4, 2019 at 04:33:54, Andre (andre-lists@fucs.org) wrote:
>>>
>>> Otto,
>>>
>>> The parser should fail in that case (and many others).
>>>
>>>
>>> https://github.com/fluenda/ParCEFone/blob/master/src/main/java/com/fluenda/parcefone/event/CefRev23.java#L327
>>>
>>> Cheers
>>>
>>> On Fri, Jan 4, 2019 at 10:05 AM Otto Fowler <ot...@gmail.com>
>>> wrote:
>>>
>>>> Can I ask how you are sure it is the message size that is causing the
>>>> error? The parser returns null for any error parsing, so the processor
>>>> doesn’t know what happened. It could be that the message didn’t validate,
>>>> or something else.
>>>>
>>>> If the issues _is_ with the validator, then we could allow a property
>>>> to optionally call the parser with the do validate flag to false.
>>>>
>>>> Maybe you can create a jira with a sanitized example line that causes
>>>> the error?
>>>>
>>>>
>>>>
>>>>
>>>> On January 3, 2019 at 15:13:14, Felix McPherson (ljungpipare@yahoo.se)
>>>> wrote:
>>>>
>>>> Hi,
>>>> I'm using the ParseCEF processor to parse CEF message to Json format.
>>>> Unfortunately the ParseCEF processor fails for message/events that holds a
>>>> string in the Msg field that has more than 1023 character. According to the
>>>> CEF standard the Msg field in an event shall not exceed 1023 character. The
>>>> PARSECEF fails with:
>>>>
>>>> "Error
>>>> ParseCEF[id=...] Failed to parse...
>>>> ...as a CEF message; it does not conform to the CEF standard; routing
>>>> to failure.
>>>>
>>>> Any ideas on a workaround for this problem? I would prefer not having
>>>> to remove character in the Msg field string.
>>>> Regards,lj
>>>>
>>>>
Re: ParseCEF processor fails when Msg field exceed 1023 characters
Posted by Otto Fowler <ot...@gmail.com>.
Well, in that case they would use strict mode and have to jump through
hoops with other processors. If they are storing into HDFS, or mongo or ES
they would not have that issue right
On January 6, 2019 at 14:47:43, Andre (andre-lists@fucs.org) wrote:
Otto,
Yes we can, but what if receiving system relies on char(1023) for one of
its columns?
In a hindsight, I should probably been even more strict and made the
processor to reject messages that are too large (I believe message limit is
4k chars) before feeding them to the parsers, avoiding "overflows" and
failing fast.
Having said that, I am not such a CEF zealot and I will be happy to
consider the implementation. PRs welcome :-)
Cheers
On Sun., 6 Jan. 2019, 14:50 Otto Fowler <ottobackwards@gmail.com wrote:
> Would you consider a message, fully formed and parsed, that only
> invalidates on size constraints to not be allowable? Could there not be
> some form of validation that is a compromise between those concerns and
> some otherwise logically correct message who breaks the size constraint?
> It is imaginable that a source formats a correct CEF message, but never
> checks the size in the field.
>
>
> On January 4, 2019 at 19:57:36, Andre (andre-lists@fucs.org) wrote:
>
> Otto,
>
> I considered that but never implemented it in NiFi as I had concerns
> people would have the incorrect assumption the parser can help on those
> cases.
>
> From the top of my head, I have seen type mismatch, length overflows,
> malformed CEF headers (the pipe delimited section of the CEF message). I
> wouldn't be surprised there are a lot of other poorly formed messages.
>
> Cheers
>
> On Fri., 4 Jan. 2019, 23:40 Otto Fowler <ottobackwards@gmail.com wrote:
>
>> Yeah, I went through the code after my mail last night. Would having
>> validation optional not be a valid setting? How often would the output
>> feeding this flow be invalid for some other reason?
>>
>>
>>
>> On January 4, 2019 at 04:33:54, Andre (andre-lists@fucs.org) wrote:
>>
>> Otto,
>>
>> The parser should fail in that case (and many others).
>>
>>
>> https://github.com/fluenda/ParCEFone/blob/master/src/main/java/com/fluenda/parcefone/event/CefRev23.java#L327
>>
>> Cheers
>>
>> On Fri, Jan 4, 2019 at 10:05 AM Otto Fowler <ot...@gmail.com>
>> wrote:
>>
>>> Can I ask how you are sure it is the message size that is causing the
>>> error? The parser returns null for any error parsing, so the processor
>>> doesn’t know what happened. It could be that the message didn’t validate,
>>> or something else.
>>>
>>> If the issues _is_ with the validator, then we could allow a property to
>>> optionally call the parser with the do validate flag to false.
>>>
>>> Maybe you can create a jira with a sanitized example line that causes
>>> the error?
>>>
>>>
>>>
>>>
>>> On January 3, 2019 at 15:13:14, Felix McPherson (ljungpipare@yahoo.se)
>>> wrote:
>>>
>>> Hi,
>>> I'm using the ParseCEF processor to parse CEF message to Json format.
>>> Unfortunately the ParseCEF processor fails for message/events that holds a
>>> string in the Msg field that has more than 1023 character. According to the
>>> CEF standard the Msg field in an event shall not exceed 1023 character. The
>>> PARSECEF fails with:
>>>
>>> "Error
>>> ParseCEF[id=...] Failed to parse...
>>> ...as a CEF message; it does not conform to the CEF standard; routing to
>>> failure.
>>>
>>> Any ideas on a workaround for this problem? I would prefer not having to
>>> remove character in the Msg field string.
>>> Regards,lj
>>>
>>>
Re: ParseCEF processor fails when Msg field exceed 1023 characters
Posted by Andre <an...@fucs.org>.
Otto,
Yes we can, but what if receiving system relies on char(1023) for one of
its columns?
In a hindsight, I should probably been even more strict and made the
processor to reject messages that are too large (I believe message limit is
4k chars) before feeding them to the parsers, avoiding "overflows" and
failing fast.
Having said that, I am not such a CEF zealot and I will be happy to
consider the implementation. PRs welcome :-)
Cheers
On Sun., 6 Jan. 2019, 14:50 Otto Fowler <ottobackwards@gmail.com wrote:
> Would you consider a message, fully formed and parsed, that only
> invalidates on size constraints to not be allowable? Could there not be
> some form of validation that is a compromise between those concerns and
> some otherwise logically correct message who breaks the size constraint?
> It is imaginable that a source formats a correct CEF message, but never
> checks the size in the field.
>
>
> On January 4, 2019 at 19:57:36, Andre (andre-lists@fucs.org) wrote:
>
> Otto,
>
> I considered that but never implemented it in NiFi as I had concerns
> people would have the incorrect assumption the parser can help on those
> cases.
>
> From the top of my head, I have seen type mismatch, length overflows,
> malformed CEF headers (the pipe delimited section of the CEF message). I
> wouldn't be surprised there are a lot of other poorly formed messages.
>
> Cheers
>
> On Fri., 4 Jan. 2019, 23:40 Otto Fowler <ottobackwards@gmail.com wrote:
>
>> Yeah, I went through the code after my mail last night. Would having
>> validation optional not be a valid setting? How often would the output
>> feeding this flow be invalid for some other reason?
>>
>>
>>
>> On January 4, 2019 at 04:33:54, Andre (andre-lists@fucs.org) wrote:
>>
>> Otto,
>>
>> The parser should fail in that case (and many others).
>>
>>
>> https://github.com/fluenda/ParCEFone/blob/master/src/main/java/com/fluenda/parcefone/event/CefRev23.java#L327
>>
>> Cheers
>>
>> On Fri, Jan 4, 2019 at 10:05 AM Otto Fowler <ot...@gmail.com>
>> wrote:
>>
>>> Can I ask how you are sure it is the message size that is causing the
>>> error? The parser returns null for any error parsing, so the processor
>>> doesn’t know what happened. It could be that the message didn’t validate,
>>> or something else.
>>>
>>> If the issues _is_ with the validator, then we could allow a property to
>>> optionally call the parser with the do validate flag to false.
>>>
>>> Maybe you can create a jira with a sanitized example line that causes
>>> the error?
>>>
>>>
>>>
>>>
>>> On January 3, 2019 at 15:13:14, Felix McPherson (ljungpipare@yahoo.se)
>>> wrote:
>>>
>>> Hi,
>>> I'm using the ParseCEF processor to parse CEF message to Json format.
>>> Unfortunately the ParseCEF processor fails for message/events that holds a
>>> string in the Msg field that has more than 1023 character. According to the
>>> CEF standard the Msg field in an event shall not exceed 1023 character. The
>>> PARSECEF fails with:
>>>
>>> "Error
>>> ParseCEF[id=...] Failed to parse...
>>> ...as a CEF message; it does not conform to the CEF standard; routing to
>>> failure.
>>>
>>> Any ideas on a workaround for this problem? I would prefer not having to
>>> remove character in the Msg field string.
>>> Regards,lj
>>>
>>>
Re: ParseCEF processor fails when Msg field exceed 1023 characters
Posted by Otto Fowler <ot...@gmail.com>.
Would you consider a message, fully formed and parsed, that only
invalidates on size constraints to not be allowable? Could there not be
some form of validation that is a compromise between those concerns and
some otherwise logically correct message who breaks the size constraint?
It is imaginable that a source formats a correct CEF message, but never
checks the size in the field.
On January 4, 2019 at 19:57:36, Andre (andre-lists@fucs.org) wrote:
Otto,
I considered that but never implemented it in NiFi as I had concerns people
would have the incorrect assumption the parser can help on those cases.
From the top of my head, I have seen type mismatch, length overflows,
malformed CEF headers (the pipe delimited section of the CEF message). I
wouldn't be surprised there are a lot of other poorly formed messages.
Cheers
On Fri., 4 Jan. 2019, 23:40 Otto Fowler <ottobackwards@gmail.com wrote:
> Yeah, I went through the code after my mail last night. Would having
> validation optional not be a valid setting? How often would the output
> feeding this flow be invalid for some other reason?
>
>
>
> On January 4, 2019 at 04:33:54, Andre (andre-lists@fucs.org) wrote:
>
> Otto,
>
> The parser should fail in that case (and many others).
>
>
> https://github.com/fluenda/ParCEFone/blob/master/src/main/java/com/fluenda/parcefone/event/CefRev23.java#L327
>
> Cheers
>
> On Fri, Jan 4, 2019 at 10:05 AM Otto Fowler <ot...@gmail.com>
> wrote:
>
>> Can I ask how you are sure it is the message size that is causing the
>> error? The parser returns null for any error parsing, so the processor
>> doesn’t know what happened. It could be that the message didn’t validate,
>> or something else.
>>
>> If the issues _is_ with the validator, then we could allow a property to
>> optionally call the parser with the do validate flag to false.
>>
>> Maybe you can create a jira with a sanitized example line that causes the
>> error?
>>
>>
>>
>>
>> On January 3, 2019 at 15:13:14, Felix McPherson (ljungpipare@yahoo.se)
>> wrote:
>>
>> Hi,
>> I'm using the ParseCEF processor to parse CEF message to Json format.
>> Unfortunately the ParseCEF processor fails for message/events that holds a
>> string in the Msg field that has more than 1023 character. According to the
>> CEF standard the Msg field in an event shall not exceed 1023 character. The
>> PARSECEF fails with:
>>
>> "Error
>> ParseCEF[id=...] Failed to parse...
>> ...as a CEF message; it does not conform to the CEF standard; routing to
>> failure.
>>
>> Any ideas on a workaround for this problem? I would prefer not having to
>> remove character in the Msg field string.
>> Regards,lj
>>
>>
Re: ParseCEF processor fails when Msg field exceed 1023 characters
Posted by Andre <an...@fucs.org>.
Otto,
I considered that but never implemented it in NiFi as I had concerns people
would have the incorrect assumption the parser can help on those cases.
From the top of my head, I have seen type mismatch, length overflows,
malformed CEF headers (the pipe delimited section of the CEF message). I
wouldn't be surprised there are a lot of other poorly formed messages.
Cheers
On Fri., 4 Jan. 2019, 23:40 Otto Fowler <ottobackwards@gmail.com wrote:
> Yeah, I went through the code after my mail last night. Would having
> validation optional not be a valid setting? How often would the output
> feeding this flow be invalid for some other reason?
>
>
>
> On January 4, 2019 at 04:33:54, Andre (andre-lists@fucs.org) wrote:
>
> Otto,
>
> The parser should fail in that case (and many others).
>
>
> https://github.com/fluenda/ParCEFone/blob/master/src/main/java/com/fluenda/parcefone/event/CefRev23.java#L327
>
> Cheers
>
> On Fri, Jan 4, 2019 at 10:05 AM Otto Fowler <ot...@gmail.com>
> wrote:
>
>> Can I ask how you are sure it is the message size that is causing the
>> error? The parser returns null for any error parsing, so the processor
>> doesn’t know what happened. It could be that the message didn’t validate,
>> or something else.
>>
>> If the issues _is_ with the validator, then we could allow a property to
>> optionally call the parser with the do validate flag to false.
>>
>> Maybe you can create a jira with a sanitized example line that causes the
>> error?
>>
>>
>>
>>
>> On January 3, 2019 at 15:13:14, Felix McPherson (ljungpipare@yahoo.se)
>> wrote:
>>
>> Hi,
>> I'm using the ParseCEF processor to parse CEF message to Json format.
>> Unfortunately the ParseCEF processor fails for message/events that holds a
>> string in the Msg field that has more than 1023 character. According to the
>> CEF standard the Msg field in an event shall not exceed 1023 character. The
>> PARSECEF fails with:
>>
>> "Error
>> ParseCEF[id=...] Failed to parse...
>> ...as a CEF message; it does not conform to the CEF standard; routing to
>> failure.
>>
>> Any ideas on a workaround for this problem? I would prefer not having to
>> remove character in the Msg field string.
>> Regards,lj
>>
>>
Re: ParseCEF processor fails when Msg field exceed 1023 characters
Posted by Otto Fowler <ot...@gmail.com>.
Yeah, I went through the code after my mail last night. Would having
validation optional not be a valid setting? How often would the output
feeding this flow be invalid for some other reason?
On January 4, 2019 at 04:33:54, Andre (andre-lists@fucs.org) wrote:
Otto,
The parser should fail in that case (and many others).
https://github.com/fluenda/ParCEFone/blob/master/src/main/java/com/fluenda/parcefone/event/CefRev23.java#L327
Cheers
On Fri, Jan 4, 2019 at 10:05 AM Otto Fowler <ot...@gmail.com> wrote:
> Can I ask how you are sure it is the message size that is causing the
> error? The parser returns null for any error parsing, so the processor
> doesn’t know what happened. It could be that the message didn’t validate,
> or something else.
>
> If the issues _is_ with the validator, then we could allow a property to
> optionally call the parser with the do validate flag to false.
>
> Maybe you can create a jira with a sanitized example line that causes the
> error?
>
>
>
>
> On January 3, 2019 at 15:13:14, Felix McPherson (ljungpipare@yahoo.se)
> wrote:
>
> Hi,
> I'm using the ParseCEF processor to parse CEF message to Json format.
> Unfortunately the ParseCEF processor fails for message/events that holds a
> string in the Msg field that has more than 1023 character. According to the
> CEF standard the Msg field in an event shall not exceed 1023 character. The
> PARSECEF fails with:
>
> "Error
> ParseCEF[id=...] Failed to parse...
> ...as a CEF message; it does not conform to the CEF standard; routing to
> failure.
>
> Any ideas on a workaround for this problem? I would prefer not having to
> remove character in the Msg field string.
> Regards,lj
>
>
Re: ParseCEF processor fails when Msg field exceed 1023 characters
Posted by Andre <an...@fucs.org>.
Otto,
The parser should fail in that case (and many others).
https://github.com/fluenda/ParCEFone/blob/master/src/main/java/com/fluenda/parcefone/event/CefRev23.java#L327
Cheers
On Fri, Jan 4, 2019 at 10:05 AM Otto Fowler <ot...@gmail.com> wrote:
> Can I ask how you are sure it is the message size that is causing the
> error? The parser returns null for any error parsing, so the processor
> doesn’t know what happened. It could be that the message didn’t validate,
> or something else.
>
> If the issues _is_ with the validator, then we could allow a property to
> optionally call the parser with the do validate flag to false.
>
> Maybe you can create a jira with a sanitized example line that causes the
> error?
>
>
>
>
> On January 3, 2019 at 15:13:14, Felix McPherson (ljungpipare@yahoo.se)
> wrote:
>
> Hi,
> I'm using the ParseCEF processor to parse CEF message to Json format.
> Unfortunately the ParseCEF processor fails for message/events that holds a
> string in the Msg field that has more than 1023 character. According to the
> CEF standard the Msg field in an event shall not exceed 1023 character. The
> PARSECEF fails with:
>
> "Error
> ParseCEF[id=...] Failed to parse...
> ...as a CEF message; it does not conform to the CEF standard; routing to
> failure.
>
> Any ideas on a workaround for this problem? I would prefer not having to
> remove character in the Msg field string.
> Regards,lj
>
>
Re: ParseCEF processor fails when Msg field exceed 1023 characters
Posted by Otto Fowler <ot...@gmail.com>.
Can I ask how you are sure it is the message size that is causing the
error? The parser returns null for any error parsing, so the processor
doesn’t know what happened. It could be that the message didn’t validate,
or something else.
If the issues _is_ with the validator, then we could allow a property to
optionally call the parser with the do validate flag to false.
Maybe you can create a jira with a sanitized example line that causes the
error?
On January 3, 2019 at 15:13:14, Felix McPherson (ljungpipare@yahoo.se)
wrote:
Hi,
I'm using the ParseCEF processor to parse CEF message to Json format.
Unfortunately the ParseCEF processor fails for message/events that holds a
string in the Msg field that has more than 1023 character. According to the
CEF standard the Msg field in an event shall not exceed 1023 character. The
PARSECEF fails with:
"Error
ParseCEF[id=...] Failed to parse...
...as a CEF message; it does not conform to the CEF standard; routing to
failure.
Any ideas on a workaround for this problem? I would prefer not having to
remove character in the Msg field string.
Regards,lj
Re: ParseCEF processor fails when Msg field exceed 1023 characters
Posted by Andre <an...@fucs.org>.
Hi Felix,
As you noticed, the CEF standard indeed states that 1023 characters is the
limit and I such when I wrote the underlying parser (ParCEFone) I enforced
a length check and failed the parsing in case of exceeding that value.
Could I just truncate? Possibly... But what to do when the vendor sends
text data on a field that the std defines as numeric? :-\
So the motto was: never try fixing messages during parsing, instead treat
individual cases one by one basis.
How to treat the odd bunches?
Historically I used a shlex based jython Parser running from
(ExecuteScript) to parse messages that failed CEF validation and then used
logic to address particular issues.
This approach poses a question: Why not use the jython shlex parser in the
first place?
1. Because shlex may fail on valid CEF
2. Because ExecuteScript was slower than the native processor.
Since the processor routes failures to a separate connector, I relied on
the ParseCEF processor for everything that worked out of the box, but used
quick and dirty jython parser where I could address the odd cases.
Sad reality of dealing with vendor logs is that they constantly fail to
follow the standards the "comply with" and honestly you can't blame them.
Some of the fields length defy logic (a perfect example being the Request
field that sets a limit well below the max URI length of most common
browsers...)
Hope this helps?
André
On Fri, Jan 4, 2019 at 7:13 AM Felix McPherson <lj...@yahoo.se> wrote:
> Hi,
> I'm using the ParseCEF processor to parse CEF message to Json format.
> Unfortunately the ParseCEF processor fails for message/events that holds a
> string in the Msg field that has more than 1023 character. According to the
> CEF standard the Msg field in an event shall not exceed 1023 character. The
> PARSECEF fails with:
>
> "Error
> ParseCEF[id=...] Failed to parse...
> ...as a CEF message; it does not conform to the CEF standard; routing to
> failure.
>
> Any ideas on a workaround for this problem? I would prefer not having to
> remove character in the Msg field string.
> Regards,lj
>