You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by sudeep mishra <su...@gmail.com> on 2016/02/11 04:54:11 UTC
NiFi logging
Hi,
What is the preferred practice for logging details for NiFi data flow? How
can I use my own logging using log4j to log custom details for NiFi data
flow.
Thanks & Regards,
Sudeep
Re: NiFi logging
Posted by Joe Witt <jo...@gmail.com>.
There is also the provenance data itself which captures this and there
is the LogAttributes processor.
Multiple options to get the information about the flow out.
Thanks
Joe
On Thu, Feb 11, 2016 at 10:34 AM, Andrew Grande <ag...@hortonworks.com> wrote:
> Take a look at ReportingTask, this sounds like a good fit. Much higher level
> than explicitly putting log steps in the flow or reconfiguring the logging
> system to forward a category elsewhere.
>
> HTH,
> Andrew
>
> [1]
> https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#reporting-tasks
>
> From: sudeep mishra <su...@gmail.com>
> Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
> Date: Wednesday, February 10, 2016 at 11:59 PM
> To: "users@nifi.apache.org" <us...@nifi.apache.org>
> Subject: Re: NiFi logging
>
> Thanks for your feedback Andy.
>
> I am looking to log the general events taking place in data flow to track my
> data at a later stage. Say I have below flow:
>
> Flow 1 - User sending XML data over HTTP POST --> NiFi HTTPListener --> NiFi
> PutKafka processor
> Flow 2 - NiFi GetKafka processor --> NiFi UpdateAttribute processor <include
> some metaadta> --> NiFi RouteOnAttribute processor --> NiFi PutHDFS
> processor
>
> For each such XML posted by a user I want to log the time and some
> attributes for the XML received at each flow. Such as
>
> <Time> <NiFi HTTPListener> Received <X> XML at Listener
> <Time> < NiFi PutKafka processor> <X> XML moved to topic <KafkaTopic>
> <Time> < NiFi GettKafka processor> Received <X> XML moved from topic
> <KafkaTopic>
> ......
> <Time> < NiFi PutHDFS processor> Moved <X> XML moved to HDFS
>
>
> On Thu, Feb 11, 2016 at 10:13 AM, Andy LoPresto <al...@gmail.com>
> wrote:
>>
>> Hi Sudeep.
>>
>> SLF4J Loggers are available throughout the code. Are you trying to log
>> details from inside a processor? Within the framework, there are various
>> implementations of `ComponentLog` [1], and `ProcessorLog` (just a marker
>> interface), which is implemented by `SimpleProcessLogger` [2]. From within a
>> processor which extends `AbstractProcessor` [3], you can call `getLogger()`
>> to return an instance of `ProcessorLog` and then write to it as you would
>> any SLF4J logger instance.
>>
>> ```java
>>
>> String key = "someKey";
>> String value = "someValue";
>> final ProcessorLog logger = getLogger();
>> logger.warn("This is a warning message");
>> logger.info("This is a message with some interpolation key={}, value={}",
>> new String[]{key, value});
>>
>> ```
>>
>> This will result in the output:
>>
>> ```
>> [pool-2-thread-1] WARN org.apache.nifi.processors.standard.EncryptContent
>> - EncryptContent[id=1c569c19-d37b-4d30-aec7-410b318c360d] This is a warning
>> message
>> [pool-2-thread-1] INFO org.apache.nifi.processors.standard.EncryptContent
>> - EncryptContent[id=1c569c19-d37b-4d30-aec7-410b318c360d] This is a message
>> with some interpolation key=someKey, value=someValue
>> ```
>>
>> We recommend using SLF4J, but if you have a requirement to use Log4j, you
>> can use an adapter as described here [4].
>>
>> [1]
>> https://github.com/apache/nifi/blob/3a7ddc6a35f143c1d4a735226fe7467a83d04583/nifi-api/src/main/java/org/apache/nifi/logging/ComponentLog.java
>> [2]
>> https://github.com/apache/nifi/blob/807eb8de6d7054cef1fac91cbf2e2a1eb37d30ed/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/processor/SimpleProcessLogger.java
>> [3]
>> https://github.com/apache/nifi/blob/31fba6b3332978ca2f6a1d693f6053d719fb9daa/nifi-api/src/main/java/org/apache/nifi/processor/AbstractProcessor.java
>> [4] http://www.slf4j.org/legacy.html
>>
>>
>> Andy LoPresto
>> alopresto.apache@gmail.com
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>>
>> On Feb 10, 2016, at 7:54 PM, sudeep mishra <su...@gmail.com>
>> wrote:
>>
>> Hi,
>>
>> What is the preferred practice for logging details for NiFi data flow? How
>> can I use my own logging using log4j to log custom details for NiFi data
>> flow.
>>
>>
>> Thanks & Regards,
>>
>> Sudeep
>>
>>
>
>
>
> --
> Thanks & Regards,
>
> Sudeep
Re: NiFi logging
Posted by Andrew Grande <ag...@hortonworks.com>.
Take a look at ReportingTask, this sounds like a good fit. Much higher level than explicitly putting log steps in the flow or reconfiguring the logging system to forward a category elsewhere.
HTH,
Andrew
[1] https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#reporting-tasks
From: sudeep mishra <su...@gmail.com>>
Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Date: Wednesday, February 10, 2016 at 11:59 PM
To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Subject: Re: NiFi logging
Thanks for your feedback Andy.
I am looking to log the general events taking place in data flow to track my data at a later stage. Say I have below flow:
Flow 1 - User sending XML data over HTTP POST --> NiFi HTTPListener --> NiFi PutKafka processor
Flow 2 - NiFi GetKafka processor --> NiFi UpdateAttribute processor <include some metaadta> --> NiFi RouteOnAttribute processor --> NiFi PutHDFS processor
For each such XML posted by a user I want to log the time and some attributes for the XML received at each flow. Such as
<Time> <NiFi HTTPListener> Received <X> XML at Listener
<Time> < NiFi PutKafka processor> <X> XML moved to topic <KafkaTopic>
<Time> < NiFi GettKafka processor> Received <X> XML moved from topic <KafkaTopic>
......
<Time> < NiFi PutHDFS processor> Moved <X> XML moved to HDFS
On Thu, Feb 11, 2016 at 10:13 AM, Andy LoPresto <al...@gmail.com>> wrote:
Hi Sudeep.
SLF4J Loggers are available throughout the code. Are you trying to log details from inside a processor? Within the framework, there are various implementations of `ComponentLog` [1], and `ProcessorLog` (just a marker interface), which is implemented by `SimpleProcessLogger` [2]. From within a processor which extends `AbstractProcessor` [3], you can call `getLogger()` to return an instance of `ProcessorLog` and then write to it as you would any SLF4J logger instance.
```java
String key = "someKey";
String value = "someValue";
final ProcessorLog logger = getLogger();
logger.warn("This is a warning message");
logger.info<http://logger.info>("This is a message with some interpolation key={}, value={}", new String[]{key, value});
```
This will result in the output:
```
[pool-2-thread-1] WARN org.apache.nifi.processors.standard.EncryptContent - EncryptContent[id=1c569c19-d37b-4d30-aec7-410b318c360d] This is a warning message
[pool-2-thread-1] INFO org.apache.nifi.processors.standard.EncryptContent - EncryptContent[id=1c569c19-d37b-4d30-aec7-410b318c360d] This is a message with some interpolation key=someKey, value=someValue
```
We recommend using SLF4J, but if you have a requirement to use Log4j, you can use an adapter as described here [4].
[1] https://github.com/apache/nifi/blob/3a7ddc6a35f143c1d4a735226fe7467a83d04583/nifi-api/src/main/java/org/apache/nifi/logging/ComponentLog.java
[2] https://github.com/apache/nifi/blob/807eb8de6d7054cef1fac91cbf2e2a1eb37d30ed/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/processor/SimpleProcessLogger.java
[3] https://github.com/apache/nifi/blob/31fba6b3332978ca2f6a1d693f6053d719fb9daa/nifi-api/src/main/java/org/apache/nifi/processor/AbstractProcessor.java
[4] http://www.slf4j.org/legacy.html
Andy LoPresto
alopresto.apache@gmail.com<ma...@gmail.com>
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
On Feb 10, 2016, at 7:54 PM, sudeep mishra <su...@gmail.com>> wrote:
Hi,
What is the preferred practice for logging details for NiFi data flow? How can I use my own logging using log4j to log custom details for NiFi data flow.
Thanks & Regards,
Sudeep
--
Thanks & Regards,
Sudeep
Re: NiFi logging
Posted by sudeep mishra <su...@gmail.com>.
Thanks for your feedback Andy.
I am looking to log the general events taking place in data flow to track
my data at a later stage. Say I have below flow:
Flow 1 - User sending XML data over HTTP POST --> NiFi HTTPListener -->
NiFi PutKafka processor
Flow 2 - NiFi GetKafka processor --> NiFi UpdateAttribute processor
<include some metaadta> --> NiFi RouteOnAttribute processor --> NiFi
PutHDFS processor
For each such XML posted by a user I want to log the time and some
attributes for the XML received at each flow. Such as
<Time> <NiFi HTTPListener> Received <X> XML at Listener
<Time> < NiFi PutKafka processor> <X> XML moved to topic <KafkaTopic>
<Time> < NiFi GettKafka processor> Received <X> XML moved from topic
<KafkaTopic>
......
<Time> < NiFi PutHDFS processor> Moved <X> XML moved to HDFS
On Thu, Feb 11, 2016 at 10:13 AM, Andy LoPresto <al...@gmail.com>
wrote:
> Hi Sudeep.
>
> SLF4J Loggers are available throughout the code. Are you trying to log
> details from inside a processor? Within the framework, there are various
> implementations of `ComponentLog` [1], and `ProcessorLog` (just a marker
> interface), which is implemented by `SimpleProcessLogger` [2]. From within
> a processor which extends `AbstractProcessor` [3], you can call
> `getLogger()` to return an instance of `ProcessorLog` and then write to it
> as you would any SLF4J logger instance.
>
> ```java
>
> String key = "someKey";
> String value = "someValue";
> final ProcessorLog logger = getLogger();
> logger.warn("This is a warning message");
> logger.info("This is a message with some interpolation key={}, value={}", new String[]{key, value});
>
> ```
>
> This will result in the output:
>
> ```
> [pool-2-thread-1] WARN org.apache.nifi.processors.standard.EncryptContent
> - EncryptContent[id=1c569c19-d37b-4d30-aec7-410b318c360d] This is a warning
> message
> [pool-2-thread-1] INFO org.apache.nifi.processors.standard.EncryptContent
> - EncryptContent[id=1c569c19-d37b-4d30-aec7-410b318c360d] This is a message
> with some interpolation key=someKey, value=someValue
> ```
>
> We recommend using SLF4J, but if you have a requirement to use Log4j, you
> can use an adapter as described here [4].
>
> [1]
> https://github.com/apache/nifi/blob/3a7ddc6a35f143c1d4a735226fe7467a83d04583/nifi-api/src/main/java/org/apache/nifi/logging/ComponentLog.java
> [2]
> https://github.com/apache/nifi/blob/807eb8de6d7054cef1fac91cbf2e2a1eb37d30ed/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/processor/SimpleProcessLogger.java
> [3]
> https://github.com/apache/nifi/blob/31fba6b3332978ca2f6a1d693f6053d719fb9daa/nifi-api/src/main/java/org/apache/nifi/processor/AbstractProcessor.java
> [4] http://www.slf4j.org/legacy.html
>
>
> Andy LoPresto
> alopresto.apache@gmail.com
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
> On Feb 10, 2016, at 7:54 PM, sudeep mishra <su...@gmail.com>
> wrote:
>
> Hi,
>
> What is the preferred practice for logging details for NiFi data flow? How
> can I use my own logging using log4j to log custom details for NiFi data
> flow.
>
>
> Thanks & Regards,
>
> Sudeep
>
>
>
--
Thanks & Regards,
Sudeep
Re: NiFi logging
Posted by Andy LoPresto <al...@gmail.com>.
Hi Sudeep.
SLF4J Loggers are available throughout the code. Are you trying to log details from inside a processor? Within the framework, there are various implementations of `ComponentLog` [1], and `ProcessorLog` (just a marker interface), which is implemented by `SimpleProcessLogger` [2]. From within a processor which extends `AbstractProcessor` [3], you can call `getLogger()` to return an instance of `ProcessorLog` and then write to it as you would any SLF4J logger instance.
```java
String key = "someKey";
String value = "someValue";
final ProcessorLog logger = getLogger();
logger.warn("This is a warning message");
logger.info("This is a message with some interpolation key={}, value={}", new String[]{key, value});
```
This will result in the output:
```
[pool-2-thread-1] WARN org.apache.nifi.processors.standard.EncryptContent - EncryptContent[id=1c569c19-d37b-4d30-aec7-410b318c360d] This is a warning message
[pool-2-thread-1] INFO org.apache.nifi.processors.standard.EncryptContent - EncryptContent[id=1c569c19-d37b-4d30-aec7-410b318c360d] This is a message with some interpolation key=someKey, value=someValue
```
We recommend using SLF4J, but if you have a requirement to use Log4j, you can use an adapter as described here [4].
[1] https://github.com/apache/nifi/blob/3a7ddc6a35f143c1d4a735226fe7467a83d04583/nifi-api/src/main/java/org/apache/nifi/logging/ComponentLog.java
[2] https://github.com/apache/nifi/blob/807eb8de6d7054cef1fac91cbf2e2a1eb37d30ed/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/processor/SimpleProcessLogger.java
[3] https://github.com/apache/nifi/blob/31fba6b3332978ca2f6a1d693f6053d719fb9daa/nifi-api/src/main/java/org/apache/nifi/processor/AbstractProcessor.java
[4] http://www.slf4j.org/legacy.html
Andy LoPresto
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Feb 10, 2016, at 7:54 PM, sudeep mishra <su...@gmail.com> wrote:
>
> Hi,
>
> What is the preferred practice for logging details for NiFi data flow? How can I use my own logging using log4j to log custom details for NiFi data flow.
>
>
> Thanks & Regards,
>
> Sudeep