You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Dylan Klomparens <dk...@foodallergy.org> on 2022/10/03 16:16:24 UTC
Re: Trouble configuring logging
Everyone, thank you for the feedback on this topic.
I have created a JIRA Improvement ticket about this topic. I proposed that there be a way to disable redirection of standard out and standard error to logback.
If there is a positive response from the community and Apache foundation members, I will implement this improvement - following the contribution guide<https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide> - and submit a pull request.
________________________________
From: Chris Sampson <ch...@naimuri.com>
Sent: Thursday, September 29, 2022 6:52 AM
To: users@nifi.apache.org <us...@nifi.apache.org>
Subject: Re: Trouble configuring logging
We had similar logging configuration issues when using the apache/nifi image in a Kubernetes environment, so had to adjust how we log things via the logback.xml to make formats a little more consistent, remove some logs we didn't (generally) find so useful and hide some things in log files within the container because it would cause policy/GDPR issues if it was output to StdOut where it could potentially be exposed to other tenants within our multi-tenant cloud.
A cutdown/sanitised version of this logback.xml file has been uploaded as a gist [1] in case it helps anyone in future - pay attention to the comments within the XML if you want to use the file as it's not complete (I've removed some duplicated appender definitions where the only difference was the appender name and output log file, for example).
We also customised the image with a different startup script that is effectively the same as the apache/nifi start.sh but doesn't include the "tail" of the "nifi-app.log" file (as we no longer have that file in our config and the logback.xml sends everything to StdOut anyway).
In terms of a future approach, I wonder whether taking a similar approach to some other applications, like Elasticsearch, would be sensible where they include different default configurations in the various distribution mechanisms they provide, e.g. RPM vs. Docker Image? Re-invigorating the work to make more things configurable via environment variables would also be worth considering if someone has time because Spring Boot/Logback can definitely handle that, allowing users to then configure their deployment with little more than environment variables, etc.
[1] https://gist.github.com/ChrisSamo632/813fdfec45f1e0e28c674b133f036811
---
Chris Sampson
IT Consultant
chris.sampson@naimuri.com<ma...@naimuri.com>
On Thu, 29 Sept 2022 at 01:21, Kevin Doran <kd...@apache.org>> wrote:
Yeah, if it helps at all, this is how we work around this in the Apache Dockerfile (I knew realize why this method was used rather than a standard console appender in logback):
# Continuously provide logs so that 'docker logs' can produce them
"${NIFI_HOME}/bin/nifi.sh" run &
nifi_pid="$!"
tail -F --pid=${nifi_pid} "${NIFI_HOME}/logs/nifi-app.log" &
https://github.com/apache/nifi/blob/main/nifi-docker/dockerhub/sh/start.sh#L126-L129
Something to look into. As more NiFi deployments become containerized, this may be an area for improvement over time.
Thanks,
Kevin
On Sep 28, 2022 at 19:40:35, Dylan Klomparens <dk...@foodallergy.org>> wrote:
Mark and Kevin, thank you for your insightful comments. That information allowed me to piece together the puzzle. Indeed, anything that is sent to standard out subsequently causes the log message to come from the the org.apache.nifi.StdOut logger in the RunNiFi class. Your description allowed me to find the exact line of code that does this<https://github.com/apache/nifi/blob/7823156606ca541ef9cae7192b092efd2cfe4e9a/nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/RunNiFi.java#L1485>.
From my perspective this is an unfortunate design choice for NiFi because it does not allow standard out to be redirected easily. (For example, flowing onward to Docker, and forwarded to AWS CloudWatch using Docker's built-in log driver, in my case). I suppose I'll have to find an alternative logback appender class to output the logs to, then find a way to forward them on from there.
Thanks again for the additional information that put the picture into view.
________________________________
From: Mark Payne <ma...@hotmail.com>>
Sent: Wednesday, September 28, 2022 10:14 AM
To: users <us...@nifi.apache.org>>
Subject: Re: Trouble configuring logging
[EXTERNAL]
This is because of how NiFi is run. When you startup nifi (bin/nifi.sh start) it doesn’t directly start the NiFi process.
Instead, it starts a different processor, which is called RunNiFi. This RunNiFi process is responsible for doing a lot of things, including monitoring the nifi process and if it dies restarting it.
Anything written to NiFi’s Standard Out goes to this processor, which then logs it.
So you’d probably need to update the logging for the appender writing to the bootstrap file:
<appender name="BOOTSTRAP_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender”>
And redirect that to standard out
Thanks
-Mark
On Sep 28, 2022, at 9:48 AM, Kevin Doran <kd...@apache.org>> wrote:
Dylan - I looked into this and am yet unable to offer an explaination. Perhaps others that are familiar with how org.apache.nifi.StdOut can shed some light, or else I will keep digging when I have a block of time. To help in my understanding: Which Docker image are you using? Is it the apace/nifi image or a custom one, and if custom, can you share the Dockerfile?
Thanks,
Kevin
On Sep 27, 2022 at 10:21:12, Dylan Klomparens <dk...@foodallergy.org>> wrote:
I am attempting to configure logging for NiFi. I have NiFi running in a Docker container, which sends all console logs to AWS CloudWatch. Therefore, I am configuring NiFi to send all logs to the console.
The problem is, for some reason all log messages are coming from the org.apache.nifi.StdOut logger. I cannot figure out why, since I would like messages to be printed directly from the logger that is receiving them.
It seems like messages are "passing through" loggers, which are ultimately printed out from the org.apache.nifi.StdOut logger. Here is an example of one log message:
2022-09-27 10:08:01,849 INFO [NiFi logging handler] org.apache.nifi.StdOut 2022-09-27 10:08:01,848 INFO [pool-6-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
Why would every single log message come from the StdOut logger? And how can I have logs delivered from the logger they're supposedly originally coming from?
My logback.xml configuration is below for reference.
<?xml version="1.0" encoding="UTF-8"?>
<configuration debug="true">
<shutdownHook class="ch.qos.logback.core.hook.DelayingShutdownHook" />
<contextListener class="ch.qos.logback.classic.jul.LevelChangePropagator">
<resetJUL>true</resetJUL>
</contextListener>
<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
<pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
</encoder>
</appender>
<logger name="org.apache.nifi" level="INFO"/>
<logger name="org.apache.nifi.processors" level="WARN"/>
<logger name="org.apache.nifi.processors.standard.LogAttribute" level="INFO"/>
<logger name="org.apache.nifi.processors.standard.LogMessage" level="INFO"/>
<logger name="org.apache.nifi.controller.repository.StandardProcessSession" level="WARN" />
<logger name="org.apache.zookeeper.ClientCnxn" level="ERROR" />
<logger name="org.apache.zookeeper.server.NIOServerCnxn" level="ERROR" />
<logger name="org.apache.zookeeper.server.NIOServerCnxnFactory" level="ERROR" />
<logger name="org.apache.zookeeper.server.NettyServerCnxnFactory" level="ERROR" />
<logger name="org.apache.zookeeper.server.quorum" level="ERROR" />
<logger name="org.apache.zookeeper.ZooKeeper" level="ERROR" />
<logger name="org.apache.zookeeper.server.PrepRequestProcessor" level="ERROR" />
<logger name="org.apache.nifi.controller.reporting.LogComponentStatuses" level="ERROR" />
<logger name="org.apache.calcite.runtime.CalciteException" level="OFF" />
<logger name="org.apache.curator.framework.recipes.leader.LeaderSelector" level="OFF" />
<logger name="org.apache.curator.ConnectionState" level="OFF" />
<!-- Logger for managing logging statements for nifi clusters. -->
<logger name="org.apache.nifi.cluster" level="INFO"/>
<!-- Logger for logging HTTP requests received by the web server. -->
<logger name="org.apache.nifi.server.JettyServer" level="INFO"/>
<!-- Logger for managing logging statements for jetty -->
<logger name="org.eclipse.jetty" level="INFO"/>
<!-- Suppress non-error messages due to excessive logging by class or library -->
<logger name="org.springframework" level="ERROR"/>
<!-- Suppress non-error messages due to known warning about redundant path annotation (NIFI-574) -->
<logger name="org.glassfish.jersey.internal.Errors" level="ERROR"/>
<!-- Suppress non-error messages due to Jetty AnnotationParser emitting a large amount of WARNS. Issue described in NIFI-5479. -->
<logger name="org.eclipse.jetty.annotations.AnnotationParser" level="ERROR"/>
<!-- Suppress non-error messages from SSHJ which was emitting large amounts of INFO logs by default -->
<logger name="net.schmizz.sshj" level="WARN" />
<logger name="com.hierynomus.sshj" level="WARN" />
<!-- Suppress non-error messages from SMBJ which was emitting large amounts of INFO logs by default -->
<logger name="com.hierynomus.smbj" level="WARN" />
<!-- Suppress non-error messages from AWS KCL which was emitting large amounts of INFO logs by default -->
<logger name="com.amazonaws.services.kinesis" level="WARN" />
<!-- Suppress non-error messages from Apache Atlas which was emitting large amounts of INFO logs by default -->
<logger name="org.apache.atlas" level="WARN" />
<logger name="org.apache.nifi.web.security.requests" level="INFO" />
<!--
Logger for capturing user events. We do not want to propagate these
log events to the root logger. These messages are only sent to the
user-log appender.
-->
<logger name="org.apache.nifi.web.security" level="INFO" />
<logger name="org.apache.nifi.web.api.config" level="INFO" />
<logger name="org.apache.nifi.authorization" level="INFO" />
<logger name="org.apache.nifi.cluster.authorization" level="INFO" />
<logger name="org.apache.nifi.web.api.AccessResource" level="INFO" />
<logger name="org.springframework.security.saml.log" level="WARN" />
<logger name="org.opensaml" level="WARN" />
<logger name="org.apache.nifi.web.server.RequestLog" level="INFO" />
<logger name="org.apache.nifi.bootstrap" level="INFO" />
<logger name="org.apache.nifi.bootstrap.Command" level="INFO" />
<logger name="org.apache.nifi.StdOut" level="INFO" />
<logger name="org.apache.nifi.StdErr" level="INFO" />
<root level="INFO">
<appender-ref ref="CONSOLE" />
</root>
</configuration>
Re: Trouble configuring logging
Posted by Kevin Doran <kd...@apache.org>.
Thanks Dylan - Thanks for your interest in contributing to a solution for
the issue! Let's collaborate on the Jira.
Cheers,
Kevin
On Oct 3, 2022 at 12:59:02, Dylan Klomparens <dk...@foodallergy.org>
wrote:
> Pardon me, I forgot to include the link to the JIRA ticket:
> https://issues.apache.org/jira/browse/NIFI-10579
> ------------------------------
> *From:* Dylan Klomparens <dk...@foodallergy.org>
> *Sent:* Monday, October 3, 2022 12:16 PM
> *To:* users@nifi.apache.org <us...@nifi.apache.org>
> *Subject:* Re: Trouble configuring logging
>
> Everyone, thank you for the feedback on this topic.
>
> I have created a JIRA Improvement ticket about this topic. I proposed that
> there be a way to disable redirection of standard out and standard error to
> logback.
>
> If there is a positive response from the community and Apache foundation
> members, I will implement this improvement - following the contribution
> guide <https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide>
> - and submit a pull request.
>
>
> ------------------------------
> *From:* Chris Sampson <ch...@naimuri.com>
> *Sent:* Thursday, September 29, 2022 6:52 AM
> *To:* users@nifi.apache.org <us...@nifi.apache.org>
> *Subject:* Re: Trouble configuring logging
>
> We had similar logging configuration issues when using the apache/nifi
> image in a Kubernetes environment, so had to adjust how we log things via
> the logback.xml to make formats a little more consistent, remove some logs
> we didn't (generally) find so useful and hide some things in log files
> within the container because it would cause policy/GDPR issues if it was
> output to StdOut where it could potentially be exposed to other tenants
> within our multi-tenant cloud.
>
> A cutdown/sanitised version of this logback.xml file has been uploaded as
> a gist [1] in case it helps anyone in future - pay attention to the
> comments within the XML if you want to use the file as it's not complete
> (I've removed some duplicated appender definitions where the only
> difference was the appender name and output log file, for example).
>
> We also customised the image with a different startup script that is
> effectively the same as the apache/nifi start.sh but doesn't include the
> "tail" of the "nifi-app.log" file (as we no longer have that file in our
> config and the logback.xml sends everything to StdOut anyway).
>
> In terms of a future approach, I wonder whether taking a similar approach
> to some other applications, like Elasticsearch, would be sensible where
> they include different default configurations in the various distribution
> mechanisms they provide, e.g. RPM vs. Docker Image? Re-invigorating the
> work to make more things configurable via environment variables would also
> be worth considering if someone has time because Spring Boot/Logback can
> definitely handle that, allowing users to then configure their deployment
> with little more than environment variables, etc.
>
>
> [1] https://gist.github.com/ChrisSamo632/813fdfec45f1e0e28c674b133f036811
>
> ---
> *Chris Sampson*
> IT Consultant
> chris.sampson@naimuri.com
>
>
> On Thu, 29 Sept 2022 at 01:21, Kevin Doran <kd...@apache.org> wrote:
>
> Yeah, if it helps at all, this is how we work around this in the Apache
> Dockerfile (I knew realize why this method was used rather than a standard
> console appender in logback):
>
> # Continuously provide logs so that 'docker logs' can produce them
> "${NIFI_HOME}/bin/nifi.sh" run &
> nifi_pid="$!"
> tail -F --pid=${nifi_pid} "${NIFI_HOME}/logs/nifi-app.log" &
>
>
> https://github.com/apache/nifi/blob/main/nifi-docker/dockerhub/sh/start.sh#L126-L129
>
>
> Something to look into. As more NiFi deployments become containerized,
> this may be an area for improvement over time.
>
> Thanks,
> Kevin
>
> On Sep 28, 2022 at 19:40:35, Dylan Klomparens <dk...@foodallergy.org>
> wrote:
>
> Mark and Kevin, thank you for your insightful comments. That information
> allowed me to piece together the puzzle. Indeed, anything that is sent to
> standard out subsequently *causes* the log message to come from the the
> org.apache.nifi.StdOut logger in the RunNiFi class. Your description
> allowed me to find the exact line of code that does this
> <https://github.com/apache/nifi/blob/7823156606ca541ef9cae7192b092efd2cfe4e9a/nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/RunNiFi.java#L1485>
> .
>
> From my perspective this is an unfortunate design choice for NiFi because
> it does not allow standard out to be redirected easily. (For example,
> flowing onward to Docker, and forwarded to AWS CloudWatch using Docker's
> built-in log driver, in my case). I suppose I'll have to find an
> alternative logback appender class to output the logs to, then find a way
> to forward them on from there.
>
> Thanks again for the additional information that put the picture into view.
> ------------------------------
> *From:* Mark Payne <ma...@hotmail.com>
> *Sent:* Wednesday, September 28, 2022 10:14 AM
> *To:* users <us...@nifi.apache.org>
> *Subject:* Re: Trouble configuring logging
>
>
> [EXTERNAL]
> This is because of how NiFi is run. When you startup nifi (bin/nifi.sh
> start) it doesn’t directly start the NiFi process.
> Instead, it starts a different processor, which is called RunNiFi. This
> RunNiFi process is responsible for doing a lot of things, including
> monitoring the nifi process and if it dies restarting it.
> Anything written to NiFi’s Standard Out goes to this processor, which then
> logs it.
> So you’d probably need to update the logging for the appender writing to
> the bootstrap file:
> <appender name="BOOTSTRAP_FILE"
> class="ch.qos.logback.core.rolling.RollingFileAppender”>
>
> And redirect that to standard out
>
> Thanks
> -Mark
>
>
> On Sep 28, 2022, at 9:48 AM, Kevin Doran <kd...@apache.org> wrote:
>
> Dylan - I looked into this and am yet unable to offer an explaination.
> Perhaps others that are familiar with how org.apache.nifi.StdOut can shed
> some light, or else I will keep digging when I have a block of time. To
> help in my understanding: Which Docker image are you using? Is it the
> apace/nifi image or a custom one, and if custom, can you share the
> Dockerfile?
>
> Thanks,
> Kevin
>
> On Sep 27, 2022 at 10:21:12, Dylan Klomparens <dk...@foodallergy.org>
> wrote:
>
> I am attempting to configure logging for NiFi. I have NiFi running in a
> Docker container, which sends all console logs to AWS CloudWatch.
> Therefore, I am configuring NiFi to send all logs to the console.
>
> The problem is, for some reason *all log messages are coming from the
> org.apache.nifi.StdOut logger*. I cannot figure out why, since I would
> like messages to be printed directly from the logger that is receiving them.
>
> It seems like messages are "passing through" loggers, which are ultimately
> printed out from the org.apache.nifi.StdOut logger. Here is an example of
> *one* log message:
> *2022-09-27 10:08:01,849 INFO [NiFi logging handler]
> org.apache.nifi.StdOut* *2022-09-27 10:08:01,848 INFO [pool-6-thread-1]
> o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile
> Repository*
>
> *Why would every single log message come from the StdOut logger? And how
> can I have logs delivered from the logger they're supposedly originally
> coming from?*
>
> My logback.xml configuration is below for reference.
>
> <?xml version="1.0" encoding="UTF-8"?>
> <configuration debug="true">
> <shutdownHook class="ch.qos.logback.core.hook.DelayingShutdownHook" />
>
> <contextListener class="ch.qos.logback.classic.jul.LevelChangePropagator">
> <resetJUL>true</resetJUL>
> </contextListener>
>
> <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
> <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
> <pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
> </encoder>
> </appender>
>
> <logger name="org.apache.nifi" level="INFO"/>
> <logger name="org.apache.nifi.processors" level="WARN"/>
> <logger name="org.apache.nifi.processors.standard.LogAttribute"
> level="INFO"/>
> <logger name="org.apache.nifi.processors.standard.LogMessage"
> level="INFO"/>
> <logger
> name="org.apache.nifi.controller.repository.StandardProcessSession"
> level="WARN" />
>
> <logger name="org.apache.zookeeper.ClientCnxn" level="ERROR" />
> <logger name="org.apache.zookeeper.server.NIOServerCnxn" level="ERROR" />
> <logger name="org.apache.zookeeper.server.NIOServerCnxnFactory"
> level="ERROR" />
> <logger name="org.apache.zookeeper.server.NettyServerCnxnFactory"
> level="ERROR" />
> <logger name="org.apache.zookeeper.server.quorum" level="ERROR" />
> <logger name="org.apache.zookeeper.ZooKeeper" level="ERROR" />
> <logger name="org.apache.zookeeper.server.PrepRequestProcessor"
> level="ERROR" />
> <logger name="org.apache.nifi.controller.reporting.LogComponentStatuses"
> level="ERROR" />
>
> <logger name="org.apache.calcite.runtime.CalciteException" level="OFF" />
>
> <logger name="org.apache.curator.framework.recipes.leader.LeaderSelector"
> level="OFF" />
> <logger name="org.apache.curator.ConnectionState" level="OFF" />
>
> <!-- Logger for managing logging statements for nifi clusters. -->
> <logger name="org.apache.nifi.cluster" level="INFO"/>
>
> <!-- Logger for logging HTTP requests received by the web server. -->
> <logger name="org.apache.nifi.server.JettyServer" level="INFO"/>
>
> <!-- Logger for managing logging statements for jetty -->
> <logger name="org.eclipse.jetty" level="INFO"/>
>
> <!-- Suppress non-error messages due to excessive logging by class or
> library -->
> <logger name="org.springframework" level="ERROR"/>
>
> <!-- Suppress non-error messages due to known warning about redundant path
> annotation (NIFI-574) -->
> <logger name="org.glassfish.jersey.internal.Errors" level="ERROR"/>
>
> <!-- Suppress non-error messages due to Jetty AnnotationParser emitting a
> large amount of WARNS. Issue described in NIFI-5479. -->
> <logger name="org.eclipse.jetty.annotations.AnnotationParser"
> level="ERROR"/>
>
> <!-- Suppress non-error messages from SSHJ which was emitting large
> amounts of INFO logs by default -->
> <logger name="net.schmizz.sshj" level="WARN" />
> <logger name="com.hierynomus.sshj" level="WARN" />
>
> <!-- Suppress non-error messages from SMBJ which was emitting large
> amounts of INFO logs by default -->
> <logger name="com.hierynomus.smbj" level="WARN" />
>
> <!-- Suppress non-error messages from AWS KCL which was emitting large
> amounts of INFO logs by default -->
> <logger name="com.amazonaws.services.kinesis" level="WARN" />
>
> <!-- Suppress non-error messages from Apache Atlas which was emitting
> large amounts of INFO logs by default -->
> <logger name="org.apache.atlas" level="WARN" />
>
> <logger name="org.apache.nifi.web.security.requests" level="INFO" />
>
> <!--
> Logger for capturing user events. We do not want to propagate these
> log events to the root logger. These messages are only sent to the
> user-log appender.
> -->
> <logger name="org.apache.nifi.web.security" level="INFO" />
> <logger name="org.apache.nifi.web.api.config" level="INFO" />
> <logger name="org.apache.nifi.authorization" level="INFO" />
> <logger name="org.apache.nifi.cluster.authorization" level="INFO" />
> <logger name="org.apache.nifi.web.api.AccessResource" level="INFO" />
> <logger name="org.springframework.security.saml.log" level="WARN" />
> <logger name="org.opensaml" level="WARN" />
> <logger name="org.apache.nifi.web.server.RequestLog" level="INFO" />
> <logger name="org.apache.nifi.bootstrap" level="INFO" />
> <logger name="org.apache.nifi.bootstrap.Command" level="INFO" />
>
> <logger name="org.apache.nifi.StdOut" level="INFO" />
> <logger name="org.apache.nifi.StdErr" level="INFO" />
>
> <root level="INFO">
> <appender-ref ref="CONSOLE" />
> </root>
>
> </configuration>
>
>
>
Re: Trouble configuring logging
Posted by Dylan Klomparens <dk...@foodallergy.org>.
Pardon me, I forgot to include the link to the JIRA ticket:
https://issues.apache.org/jira/browse/NIFI-10579
________________________________
From: Dylan Klomparens <dk...@foodallergy.org>
Sent: Monday, October 3, 2022 12:16 PM
To: users@nifi.apache.org <us...@nifi.apache.org>
Subject: Re: Trouble configuring logging
Everyone, thank you for the feedback on this topic.
I have created a JIRA Improvement ticket about this topic. I proposed that there be a way to disable redirection of standard out and standard error to logback.
If there is a positive response from the community and Apache foundation members, I will implement this improvement - following the contribution guide<https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide> - and submit a pull request.
________________________________
From: Chris Sampson <ch...@naimuri.com>
Sent: Thursday, September 29, 2022 6:52 AM
To: users@nifi.apache.org <us...@nifi.apache.org>
Subject: Re: Trouble configuring logging
We had similar logging configuration issues when using the apache/nifi image in a Kubernetes environment, so had to adjust how we log things via the logback.xml to make formats a little more consistent, remove some logs we didn't (generally) find so useful and hide some things in log files within the container because it would cause policy/GDPR issues if it was output to StdOut where it could potentially be exposed to other tenants within our multi-tenant cloud.
A cutdown/sanitised version of this logback.xml file has been uploaded as a gist [1] in case it helps anyone in future - pay attention to the comments within the XML if you want to use the file as it's not complete (I've removed some duplicated appender definitions where the only difference was the appender name and output log file, for example).
We also customised the image with a different startup script that is effectively the same as the apache/nifi start.sh but doesn't include the "tail" of the "nifi-app.log" file (as we no longer have that file in our config and the logback.xml sends everything to StdOut anyway).
In terms of a future approach, I wonder whether taking a similar approach to some other applications, like Elasticsearch, would be sensible where they include different default configurations in the various distribution mechanisms they provide, e.g. RPM vs. Docker Image? Re-invigorating the work to make more things configurable via environment variables would also be worth considering if someone has time because Spring Boot/Logback can definitely handle that, allowing users to then configure their deployment with little more than environment variables, etc.
[1] https://gist.github.com/ChrisSamo632/813fdfec45f1e0e28c674b133f036811
---
Chris Sampson
IT Consultant
chris.sampson@naimuri.com<ma...@naimuri.com>
On Thu, 29 Sept 2022 at 01:21, Kevin Doran <kd...@apache.org>> wrote:
Yeah, if it helps at all, this is how we work around this in the Apache Dockerfile (I knew realize why this method was used rather than a standard console appender in logback):
# Continuously provide logs so that 'docker logs' can produce them
"${NIFI_HOME}/bin/nifi.sh" run &
nifi_pid="$!"
tail -F --pid=${nifi_pid} "${NIFI_HOME}/logs/nifi-app.log" &
https://github.com/apache/nifi/blob/main/nifi-docker/dockerhub/sh/start.sh#L126-L129
Something to look into. As more NiFi deployments become containerized, this may be an area for improvement over time.
Thanks,
Kevin
On Sep 28, 2022 at 19:40:35, Dylan Klomparens <dk...@foodallergy.org>> wrote:
Mark and Kevin, thank you for your insightful comments. That information allowed me to piece together the puzzle. Indeed, anything that is sent to standard out subsequently causes the log message to come from the the org.apache.nifi.StdOut logger in the RunNiFi class. Your description allowed me to find the exact line of code that does this<https://github.com/apache/nifi/blob/7823156606ca541ef9cae7192b092efd2cfe4e9a/nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/RunNiFi.java#L1485>.
From my perspective this is an unfortunate design choice for NiFi because it does not allow standard out to be redirected easily. (For example, flowing onward to Docker, and forwarded to AWS CloudWatch using Docker's built-in log driver, in my case). I suppose I'll have to find an alternative logback appender class to output the logs to, then find a way to forward them on from there.
Thanks again for the additional information that put the picture into view.
________________________________
From: Mark Payne <ma...@hotmail.com>>
Sent: Wednesday, September 28, 2022 10:14 AM
To: users <us...@nifi.apache.org>>
Subject: Re: Trouble configuring logging
[EXTERNAL]
This is because of how NiFi is run. When you startup nifi (bin/nifi.sh start) it doesn’t directly start the NiFi process.
Instead, it starts a different processor, which is called RunNiFi. This RunNiFi process is responsible for doing a lot of things, including monitoring the nifi process and if it dies restarting it.
Anything written to NiFi’s Standard Out goes to this processor, which then logs it.
So you’d probably need to update the logging for the appender writing to the bootstrap file:
<appender name="BOOTSTRAP_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender”>
And redirect that to standard out
Thanks
-Mark
On Sep 28, 2022, at 9:48 AM, Kevin Doran <kd...@apache.org>> wrote:
Dylan - I looked into this and am yet unable to offer an explaination. Perhaps others that are familiar with how org.apache.nifi.StdOut can shed some light, or else I will keep digging when I have a block of time. To help in my understanding: Which Docker image are you using? Is it the apace/nifi image or a custom one, and if custom, can you share the Dockerfile?
Thanks,
Kevin
On Sep 27, 2022 at 10:21:12, Dylan Klomparens <dk...@foodallergy.org>> wrote:
I am attempting to configure logging for NiFi. I have NiFi running in a Docker container, which sends all console logs to AWS CloudWatch. Therefore, I am configuring NiFi to send all logs to the console.
The problem is, for some reason all log messages are coming from the org.apache.nifi.StdOut logger. I cannot figure out why, since I would like messages to be printed directly from the logger that is receiving them.
It seems like messages are "passing through" loggers, which are ultimately printed out from the org.apache.nifi.StdOut logger. Here is an example of one log message:
2022-09-27 10:08:01,849 INFO [NiFi logging handler] org.apache.nifi.StdOut 2022-09-27 10:08:01,848 INFO [pool-6-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
Why would every single log message come from the StdOut logger? And how can I have logs delivered from the logger they're supposedly originally coming from?
My logback.xml configuration is below for reference.
<?xml version="1.0" encoding="UTF-8"?>
<configuration debug="true">
<shutdownHook class="ch.qos.logback.core.hook.DelayingShutdownHook" />
<contextListener class="ch.qos.logback.classic.jul.LevelChangePropagator">
<resetJUL>true</resetJUL>
</contextListener>
<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
<pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
</encoder>
</appender>
<logger name="org.apache.nifi" level="INFO"/>
<logger name="org.apache.nifi.processors" level="WARN"/>
<logger name="org.apache.nifi.processors.standard.LogAttribute" level="INFO"/>
<logger name="org.apache.nifi.processors.standard.LogMessage" level="INFO"/>
<logger name="org.apache.nifi.controller.repository.StandardProcessSession" level="WARN" />
<logger name="org.apache.zookeeper.ClientCnxn" level="ERROR" />
<logger name="org.apache.zookeeper.server.NIOServerCnxn" level="ERROR" />
<logger name="org.apache.zookeeper.server.NIOServerCnxnFactory" level="ERROR" />
<logger name="org.apache.zookeeper.server.NettyServerCnxnFactory" level="ERROR" />
<logger name="org.apache.zookeeper.server.quorum" level="ERROR" />
<logger name="org.apache.zookeeper.ZooKeeper" level="ERROR" />
<logger name="org.apache.zookeeper.server.PrepRequestProcessor" level="ERROR" />
<logger name="org.apache.nifi.controller.reporting.LogComponentStatuses" level="ERROR" />
<logger name="org.apache.calcite.runtime.CalciteException" level="OFF" />
<logger name="org.apache.curator.framework.recipes.leader.LeaderSelector" level="OFF" />
<logger name="org.apache.curator.ConnectionState" level="OFF" />
<!-- Logger for managing logging statements for nifi clusters. -->
<logger name="org.apache.nifi.cluster" level="INFO"/>
<!-- Logger for logging HTTP requests received by the web server. -->
<logger name="org.apache.nifi.server.JettyServer" level="INFO"/>
<!-- Logger for managing logging statements for jetty -->
<logger name="org.eclipse.jetty" level="INFO"/>
<!-- Suppress non-error messages due to excessive logging by class or library -->
<logger name="org.springframework" level="ERROR"/>
<!-- Suppress non-error messages due to known warning about redundant path annotation (NIFI-574) -->
<logger name="org.glassfish.jersey.internal.Errors" level="ERROR"/>
<!-- Suppress non-error messages due to Jetty AnnotationParser emitting a large amount of WARNS. Issue described in NIFI-5479. -->
<logger name="org.eclipse.jetty.annotations.AnnotationParser" level="ERROR"/>
<!-- Suppress non-error messages from SSHJ which was emitting large amounts of INFO logs by default -->
<logger name="net.schmizz.sshj" level="WARN" />
<logger name="com.hierynomus.sshj" level="WARN" />
<!-- Suppress non-error messages from SMBJ which was emitting large amounts of INFO logs by default -->
<logger name="com.hierynomus.smbj" level="WARN" />
<!-- Suppress non-error messages from AWS KCL which was emitting large amounts of INFO logs by default -->
<logger name="com.amazonaws.services.kinesis" level="WARN" />
<!-- Suppress non-error messages from Apache Atlas which was emitting large amounts of INFO logs by default -->
<logger name="org.apache.atlas" level="WARN" />
<logger name="org.apache.nifi.web.security.requests" level="INFO" />
<!--
Logger for capturing user events. We do not want to propagate these
log events to the root logger. These messages are only sent to the
user-log appender.
-->
<logger name="org.apache.nifi.web.security" level="INFO" />
<logger name="org.apache.nifi.web.api.config" level="INFO" />
<logger name="org.apache.nifi.authorization" level="INFO" />
<logger name="org.apache.nifi.cluster.authorization" level="INFO" />
<logger name="org.apache.nifi.web.api.AccessResource" level="INFO" />
<logger name="org.springframework.security.saml.log" level="WARN" />
<logger name="org.opensaml" level="WARN" />
<logger name="org.apache.nifi.web.server.RequestLog" level="INFO" />
<logger name="org.apache.nifi.bootstrap" level="INFO" />
<logger name="org.apache.nifi.bootstrap.Command" level="INFO" />
<logger name="org.apache.nifi.StdOut" level="INFO" />
<logger name="org.apache.nifi.StdErr" level="INFO" />
<root level="INFO">
<appender-ref ref="CONSOLE" />
</root>
</configuration>