You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by "Hinojosa, Ozzie" <Oz...@chevron.com> on 2017/01/17 22:28:12 UTC
Regular Expression, or Similar Processor
Hi,
I am in the process of reading the Apache NiFi documentation and setting up a single instance to obtain basic working knowledge of the platform. But in the meantime I need an answer regarding Syslog messages and guidance on which educational resources would be best to leverage in order to understand NiFi in depth and in the shortest time possible.
My questions regarding Syslog. How can I parse the FlowFile contents produce by a ListenSyslog processor. By parse, I mean apply regular expressions and transform the entire contents to JSON. I know extensibility is an integral part of the platform, but again I am new to NiFi and I am looking to confirm the feasibility of parsing Syslog messages; to add additional complexity Netflows would have the same requirement as the Syslog messages. What is available out of the box for Netflows and JFlows?
Thanks, much appreciated.
Oziel Hinojosa
Security Technologies Analyst
Strategy, Service Delivery & Programs (SSDP)
oziel.hinojosa@chevron.com
CITC - IRSM
1600 Smith St. HOU160-23042B
Houston, TX 77002
Tel +1 713 754 4749
Use http://csocbehavior.chevron.com/ to give me or others CSOC Behavioral Feedback, my CAI is OZNH.
Re: Regular Expression, or Similar Processor
Posted by Matt Burgess <ma...@apache.org>.
Good call Bryan, thanks! That template and another
(SyslogExample.xml) are available at [1].
Regards,
Matt
https://cwiki.apache.org/confluence/display/NIFI/Example+Dataflow+Templates
On Tue, Jan 17, 2017 at 6:45 PM, Bryan Bende <bb...@gmail.com> wrote:
> One more resource to add to Matt's list:
>
> https://blogs.apache.org/nifi/entry/storing_syslog_events_in_hbase
>
> Thanks,
>
> Bryan
>
> On Tue, Jan 17, 2017 at 6:35 PM, Matt Burgess <ma...@apache.org> wrote:
>
>> Oziel,
>>
>> The ListenSyslog processor [1] will apply the regular expression and
>> extract each of the fields into flow file attributes. From there you
>> could use AttributesToJSON [2] with "Include Core Attributes" set to
>> false, that should give you fields named "syslog.hostname" for
>> example. You could use JoltTransformJSON [3] if you need to
>> rename/reorganize the fields, or if you need more complex logic and
>> are familiar with a scripting language such as Groovy, Javascript,
>> Python, Ruby, or Lua, you could use ExecuteScript [4] to build a
>> custom JSON output, I have examples of custom JSON transformations
>> using Groovy [5], Javascript [6], and Jython [7].
>>
>> If you are looking for the regular expression(s) to do the parsing
>> yourself (perhaps with ExtractText [8]), you can find them here [9].
>>
>> Regards,
>> Matt
>>
>> [1] https://nifi.apache.org/docs/nifi-docs/components/org.
>> apache.nifi.processors.standard.ListenSyslog/index.html
>> [2] https://nifi.apache.org/docs/nifi-docs/components/org.
>> apache.nifi.processors.standard.AttributesToJSON/index.html
>> [3] https://nifi.apache.org/docs/nifi-docs/components/org.
>> apache.nifi.processors.standard.JoltTransformJSON/index.html
>> [4] https://nifi.apache.org/docs/nifi-docs/components/org.
>> apache.nifi.processors.script.ExecuteScript/index.html
>> [5] http://funnifi.blogspot.com/2016/02/executescript-json-to-
>> json-conversion.html
>> [6] http://funnifi.blogspot.com/2016/03/executescript-json-to-
>> json-revisited.html
>> [7] http://funnifi.blogspot.com/2016/03/executescript-json-to-
>> json-revisited_14.html
>> [8] https://nifi.apache.org/docs/nifi-docs/components/org.
>> apache.nifi.processors.standard.ExtractText/index.html
>> [9] https://github.com/apache/nifi/blob/master/nifi-nar-
>> bundles/nifi-standard-bundle/nifi-standard-processors/src/
>> main/java/org/apache/nifi/processors/standard/syslog/SyslogParser.java#L36
>>
>> On Tue, Jan 17, 2017 at 5:28 PM, Hinojosa, Ozzie
>> <Oz...@chevron.com> wrote:
>> > Hi,
>> >
>> > I am in the process of reading the Apache NiFi documentation and setting
>> up a single instance to obtain basic working knowledge of the platform. But
>> in the meantime I need an answer regarding Syslog messages and guidance on
>> which educational resources would be best to leverage in order to
>> understand NiFi in depth and in the shortest time possible.
>> >
>> > My questions regarding Syslog. How can I parse the FlowFile contents
>> produce by a ListenSyslog processor. By parse, I mean apply regular
>> expressions and transform the entire contents to JSON. I know extensibility
>> is an integral part of the platform, but again I am new to NiFi and I am
>> looking to confirm the feasibility of parsing Syslog messages; to add
>> additional complexity Netflows would have the same requirement as the
>> Syslog messages. What is available out of the box for Netflows and JFlows?
>> >
>> > Thanks, much appreciated.
>> >
>> > Oziel Hinojosa
>> > Security Technologies Analyst
>> > Strategy, Service Delivery & Programs (SSDP)
>> > oziel.hinojosa@chevron.com
>> >
>> > CITC - IRSM
>> > 1600 Smith St. HOU160-23042B
>> > Houston, TX 77002
>> >
>> > Tel +1 713 754 4749
>> >
>> > Use http://csocbehavior.chevron.com/ to give me or others CSOC
>> Behavioral Feedback, my CAI is OZNH.
>> >
>>
Re: Regular Expression, or Similar Processor
Posted by Bryan Bende <bb...@gmail.com>.
One more resource to add to Matt's list:
https://blogs.apache.org/nifi/entry/storing_syslog_events_in_hbase
Thanks,
Bryan
On Tue, Jan 17, 2017 at 6:35 PM, Matt Burgess <ma...@apache.org> wrote:
> Oziel,
>
> The ListenSyslog processor [1] will apply the regular expression and
> extract each of the fields into flow file attributes. From there you
> could use AttributesToJSON [2] with "Include Core Attributes" set to
> false, that should give you fields named "syslog.hostname" for
> example. You could use JoltTransformJSON [3] if you need to
> rename/reorganize the fields, or if you need more complex logic and
> are familiar with a scripting language such as Groovy, Javascript,
> Python, Ruby, or Lua, you could use ExecuteScript [4] to build a
> custom JSON output, I have examples of custom JSON transformations
> using Groovy [5], Javascript [6], and Jython [7].
>
> If you are looking for the regular expression(s) to do the parsing
> yourself (perhaps with ExtractText [8]), you can find them here [9].
>
> Regards,
> Matt
>
> [1] https://nifi.apache.org/docs/nifi-docs/components/org.
> apache.nifi.processors.standard.ListenSyslog/index.html
> [2] https://nifi.apache.org/docs/nifi-docs/components/org.
> apache.nifi.processors.standard.AttributesToJSON/index.html
> [3] https://nifi.apache.org/docs/nifi-docs/components/org.
> apache.nifi.processors.standard.JoltTransformJSON/index.html
> [4] https://nifi.apache.org/docs/nifi-docs/components/org.
> apache.nifi.processors.script.ExecuteScript/index.html
> [5] http://funnifi.blogspot.com/2016/02/executescript-json-to-
> json-conversion.html
> [6] http://funnifi.blogspot.com/2016/03/executescript-json-to-
> json-revisited.html
> [7] http://funnifi.blogspot.com/2016/03/executescript-json-to-
> json-revisited_14.html
> [8] https://nifi.apache.org/docs/nifi-docs/components/org.
> apache.nifi.processors.standard.ExtractText/index.html
> [9] https://github.com/apache/nifi/blob/master/nifi-nar-
> bundles/nifi-standard-bundle/nifi-standard-processors/src/
> main/java/org/apache/nifi/processors/standard/syslog/SyslogParser.java#L36
>
> On Tue, Jan 17, 2017 at 5:28 PM, Hinojosa, Ozzie
> <Oz...@chevron.com> wrote:
> > Hi,
> >
> > I am in the process of reading the Apache NiFi documentation and setting
> up a single instance to obtain basic working knowledge of the platform. But
> in the meantime I need an answer regarding Syslog messages and guidance on
> which educational resources would be best to leverage in order to
> understand NiFi in depth and in the shortest time possible.
> >
> > My questions regarding Syslog. How can I parse the FlowFile contents
> produce by a ListenSyslog processor. By parse, I mean apply regular
> expressions and transform the entire contents to JSON. I know extensibility
> is an integral part of the platform, but again I am new to NiFi and I am
> looking to confirm the feasibility of parsing Syslog messages; to add
> additional complexity Netflows would have the same requirement as the
> Syslog messages. What is available out of the box for Netflows and JFlows?
> >
> > Thanks, much appreciated.
> >
> > Oziel Hinojosa
> > Security Technologies Analyst
> > Strategy, Service Delivery & Programs (SSDP)
> > oziel.hinojosa@chevron.com
> >
> > CITC - IRSM
> > 1600 Smith St. HOU160-23042B
> > Houston, TX 77002
> >
> > Tel +1 713 754 4749
> >
> > Use http://csocbehavior.chevron.com/ to give me or others CSOC
> Behavioral Feedback, my CAI is OZNH.
> >
>
Re: Regular Expression, or Similar Processor
Posted by Matt Burgess <ma...@apache.org>.
Oziel,
The ListenSyslog processor [1] will apply the regular expression and
extract each of the fields into flow file attributes. From there you
could use AttributesToJSON [2] with "Include Core Attributes" set to
false, that should give you fields named "syslog.hostname" for
example. You could use JoltTransformJSON [3] if you need to
rename/reorganize the fields, or if you need more complex logic and
are familiar with a scripting language such as Groovy, Javascript,
Python, Ruby, or Lua, you could use ExecuteScript [4] to build a
custom JSON output, I have examples of custom JSON transformations
using Groovy [5], Javascript [6], and Jython [7].
If you are looking for the regular expression(s) to do the parsing
yourself (perhaps with ExtractText [8]), you can find them here [9].
Regards,
Matt
[1] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.ListenSyslog/index.html
[2] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.AttributesToJSON/index.html
[3] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.JoltTransformJSON/index.html
[4] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.script.ExecuteScript/index.html
[5] http://funnifi.blogspot.com/2016/02/executescript-json-to-json-conversion.html
[6] http://funnifi.blogspot.com/2016/03/executescript-json-to-json-revisited.html
[7] http://funnifi.blogspot.com/2016/03/executescript-json-to-json-revisited_14.html
[8] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.ExtractText/index.html
[9] https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/syslog/SyslogParser.java#L36
On Tue, Jan 17, 2017 at 5:28 PM, Hinojosa, Ozzie
<Oz...@chevron.com> wrote:
> Hi,
>
> I am in the process of reading the Apache NiFi documentation and setting up a single instance to obtain basic working knowledge of the platform. But in the meantime I need an answer regarding Syslog messages and guidance on which educational resources would be best to leverage in order to understand NiFi in depth and in the shortest time possible.
>
> My questions regarding Syslog. How can I parse the FlowFile contents produce by a ListenSyslog processor. By parse, I mean apply regular expressions and transform the entire contents to JSON. I know extensibility is an integral part of the platform, but again I am new to NiFi and I am looking to confirm the feasibility of parsing Syslog messages; to add additional complexity Netflows would have the same requirement as the Syslog messages. What is available out of the box for Netflows and JFlows?
>
> Thanks, much appreciated.
>
> Oziel Hinojosa
> Security Technologies Analyst
> Strategy, Service Delivery & Programs (SSDP)
> oziel.hinojosa@chevron.com
>
> CITC - IRSM
> 1600 Smith St. HOU160-23042B
> Houston, TX 77002
>
> Tel +1 713 754 4749
>
> Use http://csocbehavior.chevron.com/ to give me or others CSOC Behavioral Feedback, my CAI is OZNH.
>