You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Madhukar Thota <ma...@gmail.com> on 2016/02/12 06:16:43 UTC
Log4j/logback parser via syslog
Hi
I am very new to Apache Nifi and just started learning about how to use it.
We have a requirement where we need to parse log4j/logback pattern messages
coming from SyslogAppenders via Syslog udp. I can read the standard syslog
messages, but how can i further extract log4j/logback messages from syslog
body.
Is there any log parsers( log4j/logback/Apache access log format) available
in apache nifi?
Any help on this much appreciated.
Thanks in Advance.
Re: Log4j/logback parser via syslog
Posted by Matthew Burgess <ma...@gmail.com>.
That is pretty much the list although native Python and Ruby are not supported, only Jython and JRuby. The link in Bryan’s email is to a single post but the blog itself has about 5 so far: http://funnifi.blogspot.com.
The two scripting processors are ExecuteScript and InvokeScriptedProcessor, the former is lighter-weight and lets you interact with incoming flow files, create new ones, add attributes, etc. The latter lets you script a full Processor implementation for when you need more control over the processor lifecycle, relationships, properties, etc.
A quick Google search didn’t turn up a good non-GUI open-source log4J parser, but if there is one out there (for the JVM), you can download it and use the Module Directory property to make it available to your script. There’s a post on my blog about that too: http://funnifi.blogspot.com/2016/02/executescript-using-modules.html
Regards,
Matt
From: Bryan Bende <bb...@gmail.com>
Reply-To: <us...@nifi.apache.org>
Date: Friday, February 12, 2016 at 10:55 AM
To: <us...@nifi.apache.org>
Subject: Re: Log4j/logback parser via syslog
I believe groovy, python, jython, jruby, ruby, javascript, and lua.
The associated JIRA is here:
https://issues.apache.org/jira/browse/NIFI-210
There are some cool blogs about them here:
http://funnifi.blogspot.com/2016/02/executescript-processor-hello-world.html
-Bryan
On Fri, Feb 12, 2016 at 10:48 AM, Madhukar Thota <ma...@gmail.com> wrote:
Thanks Bryan. I will look into ExtractText processor.
Do you know what scripting languages are supported with new processors?
-Madhu
On Fri, Feb 12, 2016 at 9:27 AM, Bryan Bende <bb...@gmail.com> wrote:
Hello,
Currently there are no built in processors to parse log formats, but have you taken a look at the ExtractText processor [1]?
If you can come up with a regular expression for whatever you are trying to extract, then you should be able to use ExtractText.
Other options...
You could write a custom processor, but this sounds like it might be overkill for your scenario.
In the next release (hopefully out in a few days) there will be two new processors that support scripting languages. It may be easier to use a scripting language to manipulate/parse the text.
Thanks,
Bryan
[1] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.ExtractText/index.html
On Fri, Feb 12, 2016 at 12:16 AM, Madhukar Thota <ma...@gmail.com> wrote:
Hi
I am very new to Apache Nifi and just started learning about how to use it.
We have a requirement where we need to parse log4j/logback pattern messages coming from SyslogAppenders via Syslog udp. I can read the standard syslog messages, but how can i further extract log4j/logback messages from syslog body.
Is there any log parsers( log4j/logback/Apache access log format) available in apache nifi?
Any help on this much appreciated.
Thanks in Advance.
Re: Log4j/logback parser via syslog
Posted by Bryan Bende <bb...@gmail.com>.
Also, are you using ListenSyslog?
If so you can see the attributes it produces in documentation [1] where it
says "Writes Attributes", or by looking in provenance. I believe it is
syslog.facility
[1]
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.ListenSyslog/index.html
On Fri, Feb 12, 2016 at 2:38 PM, Joe Percivall <jo...@yahoo.com>
wrote:
> Hello Madhu,
>
>
> If you're looking for a template to show how to create a dynamic property
> for RouteOnAttribute to use, I'd suggest checking out this template[1]. It
> is a simple template that checks to see if the an attribute matches 'NiFi'.
>
> Also provenance can be a very powerful debugging tool. If a flowfile gets
> routed to a relationship you don't expect, simply check the provenance for
> the destination of the relationship. You'll be able to see the exact
> attributes for any recent flowfile that was routed there.
> [1]
> https://github.com/hortonworks-gallery/nifi-templates/blob/master/templates/simple-httpget-route-flow.xml
>
>
> Hope that helps,
> Joe
>
> - - - - - -
> Joseph Percivall
> linkedin.com/in/Percivall
> e: joepercivall@yahoo.com
>
>
>
> On Friday, February 12, 2016 2:28 PM, Madhukar Thota <
> madhukar.thota@gmail.com> wrote:
>
>
>
> I am getting my log4j logs on facility value 23 ( LOCAL7) how can route
> only facility 23 logs for further extraction.
>
> I added RouteonAttribute processor and defined this property
> :${facility:contains(23)} but none of them messages getting matched. I am
> not sure my defined property is correct. How can i route messages based on
> the field value to different processors?
>
> -Madhu
>
>
> On Fri, Feb 12, 2016 at 11:33 AM, Madhukar Thota <ma...@gmail.com>
> wrote:
>
> Thanks Bryan. Looking forward for the release.
> >
> >
> >
> >
> >
> >On Fri, Feb 12, 2016 at 10:55 AM, Bryan Bende <bb...@gmail.com> wrote:
> >
> >I believe groovy, python, jython, jruby, ruby, javascript, and lua.
> >>
> >>
> >>The associated JIRA is here:
> >>https://issues.apache.org/jira/browse/NIFI-210
> >>
> >>
> >>
> >>There are some cool blogs about them here:
> >>
> http://funnifi.blogspot.com/2016/02/executescript-processor-hello-world.html
> >>
> >>
> >>
> >>-Bryan
> >>
> >>
> >>On Fri, Feb 12, 2016 at 10:48 AM, Madhukar Thota <
> madhukar.thota@gmail.com> wrote:
> >>
> >>Thanks Bryan. I will look into ExtractText processor.
> >>>
> >>>
> >>>Do you know what scripting languages are supported with new processors?
> >>>
> >>>
> >>>-Madhu
> >>>
> >>>
> >>>On Fri, Feb 12, 2016 at 9:27 AM, Bryan Bende <bb...@gmail.com> wrote:
> >>>
> >>>Hello,
> >>>>
> >>>>
> >>>>Currently there are no built in processors to parse log formats, but
> have you taken a look at the ExtractText processor [1]?
> >>>>
> >>>>
> >>>>If you can come up with a regular expression for whatever you are
> trying to extract, then you should be able to use ExtractText.
> >>>>
> >>>>
> >>>>Other options...
> >>>>
> >>>>
> >>>>You could write a custom processor, but this sounds like it might be
> overkill for your scenario.
> >>>>In the next release (hopefully out in a few days) there will be two
> new processors that support scripting languages. It may be easier to use a
> scripting language to manipulate/parse the text.
> >>>>
> >>>>
> >>>>Thanks,
> >>>>
> >>>>
> >>>>Bryan
> >>>>
> >>>>
> >>>>[1]
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.ExtractText/index.html
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>On Fri, Feb 12, 2016 at 12:16 AM, Madhukar Thota <
> madhukar.thota@gmail.com> wrote:
> >>>>
> >>>>Hi
> >>>>>
> >>>>>
> >>>>>I am very new to Apache Nifi and just started learning about how to
> use it.
> >>>>>
> >>>>>
> >>>>>We have a requirement where we need to parse log4j/logback pattern
> messages coming from SyslogAppenders via Syslog udp. I can read the
> standard syslog messages, but how can i further extract log4j/logback
> messages from syslog body.
> >>>>>
> >>>>>
> >>>>>Is there any log parsers( log4j/logback/Apache access log format)
> available in apache nifi?
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>Any help on this much appreciated.
> >>>>>
> >>>>>
> >>>>>Thanks in Advance.
> >>>>>
> >>>>>
> >>>>
> >>>
> >>
> >
>
Re: Log4j/logback parser via syslog
Posted by Joe Percivall <jo...@yahoo.com>.
Hello Madhu,
If you're looking for a template to show how to create a dynamic property for RouteOnAttribute to use, I'd suggest checking out this template[1]. It is a simple template that checks to see if the an attribute matches 'NiFi'.
Also provenance can be a very powerful debugging tool. If a flowfile gets routed to a relationship you don't expect, simply check the provenance for the destination of the relationship. You'll be able to see the exact attributes for any recent flowfile that was routed there.
[1] https://github.com/hortonworks-gallery/nifi-templates/blob/master/templates/simple-httpget-route-flow.xml
Hope that helps,
Joe
- - - - - -
Joseph Percivall
linkedin.com/in/Percivall
e: joepercivall@yahoo.com
On Friday, February 12, 2016 2:28 PM, Madhukar Thota <ma...@gmail.com> wrote:
I am getting my log4j logs on facility value 23 ( LOCAL7) how can route only facility 23 logs for further extraction.
I added RouteonAttribute processor and defined this property :${facility:contains(23)} but none of them messages getting matched. I am not sure my defined property is correct. How can i route messages based on the field value to different processors?
-Madhu
On Fri, Feb 12, 2016 at 11:33 AM, Madhukar Thota <ma...@gmail.com> wrote:
Thanks Bryan. Looking forward for the release.
>
>
>
>
>
>On Fri, Feb 12, 2016 at 10:55 AM, Bryan Bende <bb...@gmail.com> wrote:
>
>I believe groovy, python, jython, jruby, ruby, javascript, and lua.
>>
>>
>>The associated JIRA is here:
>>https://issues.apache.org/jira/browse/NIFI-210
>>
>>
>>
>>There are some cool blogs about them here:
>>http://funnifi.blogspot.com/2016/02/executescript-processor-hello-world.html
>>
>>
>>
>>-Bryan
>>
>>
>>On Fri, Feb 12, 2016 at 10:48 AM, Madhukar Thota <ma...@gmail.com> wrote:
>>
>>Thanks Bryan. I will look into ExtractText processor.
>>>
>>>
>>>Do you know what scripting languages are supported with new processors?
>>>
>>>
>>>-Madhu
>>>
>>>
>>>On Fri, Feb 12, 2016 at 9:27 AM, Bryan Bende <bb...@gmail.com> wrote:
>>>
>>>Hello,
>>>>
>>>>
>>>>Currently there are no built in processors to parse log formats, but have you taken a look at the ExtractText processor [1]?
>>>>
>>>>
>>>>If you can come up with a regular expression for whatever you are trying to extract, then you should be able to use ExtractText.
>>>>
>>>>
>>>>Other options...
>>>>
>>>>
>>>>You could write a custom processor, but this sounds like it might be overkill for your scenario.
>>>>In the next release (hopefully out in a few days) there will be two new processors that support scripting languages. It may be easier to use a scripting language to manipulate/parse the text.
>>>>
>>>>
>>>>Thanks,
>>>>
>>>>
>>>>Bryan
>>>>
>>>>
>>>>[1] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.ExtractText/index.html
>>>>
>>>>
>>>>
>>>>
>>>>On Fri, Feb 12, 2016 at 12:16 AM, Madhukar Thota <ma...@gmail.com> wrote:
>>>>
>>>>Hi
>>>>>
>>>>>
>>>>>I am very new to Apache Nifi and just started learning about how to use it.
>>>>>
>>>>>
>>>>>We have a requirement where we need to parse log4j/logback pattern messages coming from SyslogAppenders via Syslog udp. I can read the standard syslog messages, but how can i further extract log4j/logback messages from syslog body.
>>>>>
>>>>>
>>>>>Is there any log parsers( log4j/logback/Apache access log format) available in apache nifi?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>Any help on this much appreciated.
>>>>>
>>>>>
>>>>>Thanks in Advance.
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Log4j/logback parser via syslog
Posted by Madhukar Thota <ma...@gmail.com>.
I am getting my log4j logs on facility value 23 ( LOCAL7) how can route
only facility 23 logs for further extraction.
I added RouteonAttribute processor and defined this property :
${facility:contains(23)} but none of them messages getting matched. I am
not sure my defined property is correct. How can i route messages based on
the field value to different processors?
-Madhu
On Fri, Feb 12, 2016 at 11:33 AM, Madhukar Thota <ma...@gmail.com>
wrote:
> Thanks Bryan. Looking forward for the release.
>
>
>
> On Fri, Feb 12, 2016 at 10:55 AM, Bryan Bende <bb...@gmail.com> wrote:
>
>> I believe groovy, python, jython, jruby, ruby, javascript, and lua.
>>
>> The associated JIRA is here:
>> https://issues.apache.org/jira/browse/NIFI-210
>>
>> There are some cool blogs about them here:
>>
>> http://funnifi.blogspot.com/2016/02/executescript-processor-hello-world.html
>>
>> -Bryan
>>
>> On Fri, Feb 12, 2016 at 10:48 AM, Madhukar Thota <
>> madhukar.thota@gmail.com> wrote:
>>
>>> Thanks Bryan. I will look into ExtractText processor.
>>>
>>> Do you know what scripting languages are supported with new processors?
>>>
>>> -Madhu
>>>
>>> On Fri, Feb 12, 2016 at 9:27 AM, Bryan Bende <bb...@gmail.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> Currently there are no built in processors to parse log formats, but
>>>> have you taken a look at the ExtractText processor [1]?
>>>>
>>>> If you can come up with a regular expression for whatever you are
>>>> trying to extract, then you should be able to use ExtractText.
>>>>
>>>> Other options...
>>>>
>>>> You could write a custom processor, but this sounds like it might be
>>>> overkill for your scenario.
>>>> In the next release (hopefully out in a few days) there will be two new
>>>> processors that support scripting languages. It may be easier to use a
>>>> scripting language to manipulate/parse the text.
>>>>
>>>> Thanks,
>>>>
>>>> Bryan
>>>>
>>>> [1]
>>>> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.ExtractText/index.html
>>>>
>>>>
>>>> On Fri, Feb 12, 2016 at 12:16 AM, Madhukar Thota <
>>>> madhukar.thota@gmail.com> wrote:
>>>>
>>>>> Hi
>>>>>
>>>>> I am very new to Apache Nifi and just started learning about how to
>>>>> use it.
>>>>>
>>>>> We have a requirement where we need to parse log4j/logback pattern
>>>>> messages coming from SyslogAppenders via Syslog udp. I can read the
>>>>> standard syslog messages, but how can i further extract log4j/logback
>>>>> messages from syslog body.
>>>>>
>>>>> Is there any log parsers( log4j/logback/Apache access log format)
>>>>> available in apache nifi?
>>>>>
>>>>>
>>>>> Any help on this much appreciated.
>>>>>
>>>>> Thanks in Advance.
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Log4j/logback parser via syslog
Posted by Madhukar Thota <ma...@gmail.com>.
Thanks Bryan. Looking forward for the release.
On Fri, Feb 12, 2016 at 10:55 AM, Bryan Bende <bb...@gmail.com> wrote:
> I believe groovy, python, jython, jruby, ruby, javascript, and lua.
>
> The associated JIRA is here:
> https://issues.apache.org/jira/browse/NIFI-210
>
> There are some cool blogs about them here:
>
> http://funnifi.blogspot.com/2016/02/executescript-processor-hello-world.html
>
> -Bryan
>
> On Fri, Feb 12, 2016 at 10:48 AM, Madhukar Thota <madhukar.thota@gmail.com
> > wrote:
>
>> Thanks Bryan. I will look into ExtractText processor.
>>
>> Do you know what scripting languages are supported with new processors?
>>
>> -Madhu
>>
>> On Fri, Feb 12, 2016 at 9:27 AM, Bryan Bende <bb...@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> Currently there are no built in processors to parse log formats, but
>>> have you taken a look at the ExtractText processor [1]?
>>>
>>> If you can come up with a regular expression for whatever you are trying
>>> to extract, then you should be able to use ExtractText.
>>>
>>> Other options...
>>>
>>> You could write a custom processor, but this sounds like it might be
>>> overkill for your scenario.
>>> In the next release (hopefully out in a few days) there will be two new
>>> processors that support scripting languages. It may be easier to use a
>>> scripting language to manipulate/parse the text.
>>>
>>> Thanks,
>>>
>>> Bryan
>>>
>>> [1]
>>> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.ExtractText/index.html
>>>
>>>
>>> On Fri, Feb 12, 2016 at 12:16 AM, Madhukar Thota <
>>> madhukar.thota@gmail.com> wrote:
>>>
>>>> Hi
>>>>
>>>> I am very new to Apache Nifi and just started learning about how to use
>>>> it.
>>>>
>>>> We have a requirement where we need to parse log4j/logback pattern
>>>> messages coming from SyslogAppenders via Syslog udp. I can read the
>>>> standard syslog messages, but how can i further extract log4j/logback
>>>> messages from syslog body.
>>>>
>>>> Is there any log parsers( log4j/logback/Apache access log format)
>>>> available in apache nifi?
>>>>
>>>>
>>>> Any help on this much appreciated.
>>>>
>>>> Thanks in Advance.
>>>>
>>>>
>>>
>>
>
Re: Log4j/logback parser via syslog
Posted by Bryan Bende <bb...@gmail.com>.
I believe groovy, python, jython, jruby, ruby, javascript, and lua.
The associated JIRA is here:
https://issues.apache.org/jira/browse/NIFI-210
There are some cool blogs about them here:
http://funnifi.blogspot.com/2016/02/executescript-processor-hello-world.html
-Bryan
On Fri, Feb 12, 2016 at 10:48 AM, Madhukar Thota <ma...@gmail.com>
wrote:
> Thanks Bryan. I will look into ExtractText processor.
>
> Do you know what scripting languages are supported with new processors?
>
> -Madhu
>
> On Fri, Feb 12, 2016 at 9:27 AM, Bryan Bende <bb...@gmail.com> wrote:
>
>> Hello,
>>
>> Currently there are no built in processors to parse log formats, but have
>> you taken a look at the ExtractText processor [1]?
>>
>> If you can come up with a regular expression for whatever you are trying
>> to extract, then you should be able to use ExtractText.
>>
>> Other options...
>>
>> You could write a custom processor, but this sounds like it might be
>> overkill for your scenario.
>> In the next release (hopefully out in a few days) there will be two new
>> processors that support scripting languages. It may be easier to use a
>> scripting language to manipulate/parse the text.
>>
>> Thanks,
>>
>> Bryan
>>
>> [1]
>> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.ExtractText/index.html
>>
>>
>> On Fri, Feb 12, 2016 at 12:16 AM, Madhukar Thota <
>> madhukar.thota@gmail.com> wrote:
>>
>>> Hi
>>>
>>> I am very new to Apache Nifi and just started learning about how to use
>>> it.
>>>
>>> We have a requirement where we need to parse log4j/logback pattern
>>> messages coming from SyslogAppenders via Syslog udp. I can read the
>>> standard syslog messages, but how can i further extract log4j/logback
>>> messages from syslog body.
>>>
>>> Is there any log parsers( log4j/logback/Apache access log format)
>>> available in apache nifi?
>>>
>>>
>>> Any help on this much appreciated.
>>>
>>> Thanks in Advance.
>>>
>>>
>>
>
Re: Log4j/logback parser via syslog
Posted by Madhukar Thota <ma...@gmail.com>.
Thanks Bryan. I will look into ExtractText processor.
Do you know what scripting languages are supported with new processors?
-Madhu
On Fri, Feb 12, 2016 at 9:27 AM, Bryan Bende <bb...@gmail.com> wrote:
> Hello,
>
> Currently there are no built in processors to parse log formats, but have
> you taken a look at the ExtractText processor [1]?
>
> If you can come up with a regular expression for whatever you are trying
> to extract, then you should be able to use ExtractText.
>
> Other options...
>
> You could write a custom processor, but this sounds like it might be
> overkill for your scenario.
> In the next release (hopefully out in a few days) there will be two new
> processors that support scripting languages. It may be easier to use a
> scripting language to manipulate/parse the text.
>
> Thanks,
>
> Bryan
>
> [1]
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.ExtractText/index.html
>
>
> On Fri, Feb 12, 2016 at 12:16 AM, Madhukar Thota <madhukar.thota@gmail.com
> > wrote:
>
>> Hi
>>
>> I am very new to Apache Nifi and just started learning about how to use
>> it.
>>
>> We have a requirement where we need to parse log4j/logback pattern
>> messages coming from SyslogAppenders via Syslog udp. I can read the
>> standard syslog messages, but how can i further extract log4j/logback
>> messages from syslog body.
>>
>> Is there any log parsers( log4j/logback/Apache access log format)
>> available in apache nifi?
>>
>>
>> Any help on this much appreciated.
>>
>> Thanks in Advance.
>>
>>
>
Re: Log4j/logback parser via syslog
Posted by Bryan Bende <bb...@gmail.com>.
Hello,
Currently there are no built in processors to parse log formats, but have
you taken a look at the ExtractText processor [1]?
If you can come up with a regular expression for whatever you are trying to
extract, then you should be able to use ExtractText.
Other options...
You could write a custom processor, but this sounds like it might be
overkill for your scenario.
In the next release (hopefully out in a few days) there will be two new
processors that support scripting languages. It may be easier to use a
scripting language to manipulate/parse the text.
Thanks,
Bryan
[1]
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.ExtractText/index.html
On Fri, Feb 12, 2016 at 12:16 AM, Madhukar Thota <ma...@gmail.com>
wrote:
> Hi
>
> I am very new to Apache Nifi and just started learning about how to use it.
>
> We have a requirement where we need to parse log4j/logback pattern
> messages coming from SyslogAppenders via Syslog udp. I can read the
> standard syslog messages, but how can i further extract log4j/logback
> messages from syslog body.
>
> Is there any log parsers( log4j/logback/Apache access log format)
> available in apache nifi?
>
>
> Any help on this much appreciated.
>
> Thanks in Advance.
>
>