You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Vibhath Ileperuma <vi...@gmail.com> on 2021/03/25 10:22:29 UTC
Writing custom logs
Hi All,
I'm developing a NIFI flow which fetches data from a set of database
tables. To log custom messages (no. of rows fetched, query execution time
etc.), I'm using a 'LogMessage' processor. By changing the logback.xml
file, I was able to write my custom messages in a separate log file. But
it's better if I can maintain log files separately for different tables.
Is there a way to do this in NIFI.
Thanks & Regards
*Vibhath Ileperuma*
Re: Writing custom logs
Posted by Etienne Jouvin <la...@gmail.com>.
If you know all your tables, and there is too much
you can duplicate this configuration for all messages.
Le jeu. 25 mars 2021 à 13:03, Vibhath Ileperuma <vi...@gmail.com>
a écrit :
> Hello Etienne,
>
> With the configuration you have mentioned, we can log messages related to
> only one table using the LogMessage processor, can't we?
> But what I really want is to log custom messages related to all the tables
> using the LogMessage processor(s). But the messages should be written to
> different log files based on the table name.
>
> Thanks & Regards
>
> *Vibhath Ileperuma*
>
>
>
> On Thu, Mar 25, 2021 at 4:18 PM Etienne Jouvin <la...@gmail.com>
> wrote:
>
>> Hello.
>>
>> What you can do it to prefix your log message, with something like
>> [MY_TABLE]
>> and then in the logback configuration use an EvaluatorFilter.
>>
>> Something like this :
>>
>> <appender name="CUSTOM_APPENDER"
>> class="ch.qos.logback.core.rolling.RollingFileAppender">
>>
>> <file>${org.apache.nifi.bootstrap.config.log.dir}/loreal-sync-dctm-process.log</file>
>> <rollingPolicy
>> class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
>> <!--
>> For daily rollover, use 'user_%d.log'.
>> For hourly rollover, use 'user_%d{yyyy-MM-dd_HH}.log'.
>> To GZIP rolled files, replace '.log' with '.log.gz'.
>> To ZIP rolled files, replace '.log' with '.log.zip'.
>> -->
>>
>> <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/mytable-logger_%d{yyyy-MM-dd_HH}.log</fileNamePattern>
>> <!-- keep 30 log files worth of history -->
>> <maxHistory>10</maxHistory>
>> <cleanHistoryOnStart>true</cleanHistoryOnStart>
>> </rollingPolicy>
>>
>> <filter class="ch.qos.logback.core.filter.EvaluatorFilter">
>> <evaluator>
>> <matcher>
>> <Name>syncNuxeo</Name>
>> <regex>\[MY_TABLE\] </regex>
>> </matcher>
>>
>>
>> <expression>syncNuxeo.matches(formattedMessage)</expression>
>> </evaluator>
>> <OnMismatch>DENY</OnMismatch>
>> <OnMatch>NEUTRAL</OnMatch>
>> </filter>
>>
>> <encoder
>> class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
>> <pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
>> </encoder>
>> </appender>
>>
>> Then use this new appender for LogMessage
>> <logger name="org.apache.nifi.processors.standard.LogMessage"
>> level="INFO">
>> <appender-ref ref=" CUSTOM_APPENDER "/>
>> </logger>
>>
>> But this means you have to deploy specific libraries in the NiFi lib
>> folder.
>> org.codehaus.janino.commons-compiler
>> org.codehaus.janino.janino
>>
>> Those libraries are potentially already used in standard processors /
>> services.
>> Depending on your NiFi version, check which version you need to deploy.
>>
>> Regards
>>
>> Etienne Jouvin
>>
>> Le jeu. 25 mars 2021 à 11:23, Vibhath Ileperuma <
>> vibhatharunapriya@gmail.com> a écrit :
>>
>>> Hi All,
>>>
>>> I'm developing a NIFI flow which fetches data from a set of database
>>> tables. To log custom messages (no. of rows fetched, query execution time
>>> etc.), I'm using a 'LogMessage' processor. By changing the logback.xml
>>> file, I was able to write my custom messages in a separate log file. But
>>> it's better if I can maintain log files separately for different tables.
>>> Is there a way to do this in NIFI.
>>>
>>> Thanks & Regards
>>>
>>> *Vibhath Ileperuma*
>>>
>>>
>>>
>>>
Re: Writing custom logs
Posted by Vibhath Ileperuma <vi...@gmail.com>.
Hello Etienne,
With the configuration you have mentioned, we can log messages related to
only one table using the LogMessage processor, can't we?
But what I really want is to log custom messages related to all the tables
using the LogMessage processor(s). But the messages should be written to
different log files based on the table name.
Thanks & Regards
*Vibhath Ileperuma*
On Thu, Mar 25, 2021 at 4:18 PM Etienne Jouvin <la...@gmail.com>
wrote:
> Hello.
>
> What you can do it to prefix your log message, with something like
> [MY_TABLE]
> and then in the logback configuration use an EvaluatorFilter.
>
> Something like this :
>
> <appender name="CUSTOM_APPENDER"
> class="ch.qos.logback.core.rolling.RollingFileAppender">
>
> <file>${org.apache.nifi.bootstrap.config.log.dir}/loreal-sync-dctm-process.log</file>
> <rollingPolicy
> class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
> <!--
> For daily rollover, use 'user_%d.log'.
> For hourly rollover, use 'user_%d{yyyy-MM-dd_HH}.log'.
> To GZIP rolled files, replace '.log' with '.log.gz'.
> To ZIP rolled files, replace '.log' with '.log.zip'.
> -->
>
> <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/mytable-logger_%d{yyyy-MM-dd_HH}.log</fileNamePattern>
> <!-- keep 30 log files worth of history -->
> <maxHistory>10</maxHistory>
> <cleanHistoryOnStart>true</cleanHistoryOnStart>
> </rollingPolicy>
>
> <filter class="ch.qos.logback.core.filter.EvaluatorFilter">
> <evaluator>
> <matcher>
> <Name>syncNuxeo</Name>
> <regex>\[MY_TABLE\] </regex>
> </matcher>
>
>
> <expression>syncNuxeo.matches(formattedMessage)</expression>
> </evaluator>
> <OnMismatch>DENY</OnMismatch>
> <OnMatch>NEUTRAL</OnMatch>
> </filter>
>
> <encoder
> class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
> <pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
> </encoder>
> </appender>
>
> Then use this new appender for LogMessage
> <logger name="org.apache.nifi.processors.standard.LogMessage"
> level="INFO">
> <appender-ref ref=" CUSTOM_APPENDER "/>
> </logger>
>
> But this means you have to deploy specific libraries in the NiFi lib
> folder.
> org.codehaus.janino.commons-compiler
> org.codehaus.janino.janino
>
> Those libraries are potentially already used in standard processors /
> services.
> Depending on your NiFi version, check which version you need to deploy.
>
> Regards
>
> Etienne Jouvin
>
> Le jeu. 25 mars 2021 à 11:23, Vibhath Ileperuma <
> vibhatharunapriya@gmail.com> a écrit :
>
>> Hi All,
>>
>> I'm developing a NIFI flow which fetches data from a set of database
>> tables. To log custom messages (no. of rows fetched, query execution time
>> etc.), I'm using a 'LogMessage' processor. By changing the logback.xml
>> file, I was able to write my custom messages in a separate log file. But
>> it's better if I can maintain log files separately for different tables.
>> Is there a way to do this in NIFI.
>>
>> Thanks & Regards
>>
>> *Vibhath Ileperuma*
>>
>>
>>
>>
Re: Writing custom logs
Posted by Etienne Jouvin <la...@gmail.com>.
Hello.
What you can do it to prefix your log message, with something like
[MY_TABLE]
and then in the logback configuration use an EvaluatorFilter.
Something like this :
<appender name="CUSTOM_APPENDER"
class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${org.apache.nifi.bootstrap.config.log.dir}/loreal-sync-dctm-process.log</file>
<rollingPolicy
class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<!--
For daily rollover, use 'user_%d.log'.
For hourly rollover, use 'user_%d{yyyy-MM-dd_HH}.log'.
To GZIP rolled files, replace '.log' with '.log.gz'.
To ZIP rolled files, replace '.log' with '.log.zip'.
-->
<fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/mytable-logger_%d{yyyy-MM-dd_HH}.log</fileNamePattern>
<!-- keep 30 log files worth of history -->
<maxHistory>10</maxHistory>
<cleanHistoryOnStart>true</cleanHistoryOnStart>
</rollingPolicy>
<filter class="ch.qos.logback.core.filter.EvaluatorFilter">
<evaluator>
<matcher>
<Name>syncNuxeo</Name>
<regex>\[MY_TABLE\] </regex>
</matcher>
<expression>syncNuxeo.matches(formattedMessage)</expression>
</evaluator>
<OnMismatch>DENY</OnMismatch>
<OnMatch>NEUTRAL</OnMatch>
</filter>
<encoder
class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
<pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
</encoder>
</appender>
Then use this new appender for LogMessage
<logger name="org.apache.nifi.processors.standard.LogMessage"
level="INFO">
<appender-ref ref=" CUSTOM_APPENDER "/>
</logger>
But this means you have to deploy specific libraries in the NiFi lib folder.
org.codehaus.janino.commons-compiler
org.codehaus.janino.janino
Those libraries are potentially already used in standard processors /
services.
Depending on your NiFi version, check which version you need to deploy.
Regards
Etienne Jouvin
Le jeu. 25 mars 2021 à 11:23, Vibhath Ileperuma <vi...@gmail.com>
a écrit :
> Hi All,
>
> I'm developing a NIFI flow which fetches data from a set of database
> tables. To log custom messages (no. of rows fetched, query execution time
> etc.), I'm using a 'LogMessage' processor. By changing the logback.xml
> file, I was able to write my custom messages in a separate log file. But
> it's better if I can maintain log files separately for different tables.
> Is there a way to do this in NIFI.
>
> Thanks & Regards
>
> *Vibhath Ileperuma*
>
>
>
>