You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Stéphane Maarek <st...@gmail.com> on 2016/06/06 04:56:09 UTC
How to control NiFi logging
Hi,
I'm running NiFi on a t1.micro in AWS. I have attached a 100GB EBS volume
at /mnt/xvdf and all the NiFi databases are there. I have a problem though.
It seems that NiFi keeps an internal log and I can't change the position of
it, so it's quickly clogging my t1.micro disk. See below. How can I improve
/ change that? And what's the "work" folder for?
[ec2-user@ ~]$ du -s *
5271404 nifi
[ec2-user@ ~]$ cd nifi
[ec2-user@ nifi]$ du -s *
28 bin
756 conf
8412 docs
427468 lib
64 LICENSE
4331244 logs
44 NOTICE
8 README
144 state
503232 work
Thanks!
Stephane
Re: How to control NiFi logging
Posted by Andre <an...@fucs.org>.
Stephane,
Please upload both your bootstrap and logback configuration files to a gist
or pastebin so we can have a look.
Cheers
On Wed, Jun 8, 2016 at 11:50 AM, Stéphane Maarek <st...@gmail.com>
wrote:
> Hi Andrew,
>
> Thanks for the quick response. Sorry stuff isn't obvious just yet. Please
> bear with me
> I copied the file you indicated to my logback.xml and didn't change a thing
>
> In bootstrap.conf, I added the following:
> org.apache.nifi.bootstrap.config.log.dir=/mnt/xvdf/logs
>
> Now on start of NiFi, the logging still doesn't work, and I'm getting the
> following folder in my nifi root:
> org.apache.nifi.bootstrap.config.log.dir_IS_UNDEFINED
>
> Can you please help?
>
> regards,
> Stephane
>
> On Wed, Jun 8, 2016 at 11:07 AM Andre <an...@fucs.org> wrote:
>
>> Stephane,
>>
>> There are two settings controlling each individual log. The sections are
>> clearly commented within logback settings
>>
>> You can read the instance in your logback.xml or read the one available
>> online
>>
>>
>> https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/conf/logback.xml
>>
>> Cheers
>>
>> On Wed, Jun 8, 2016 at 10:22 AM, Stéphane Maarek <
>> stephane.maarek@gmail.com> wrote:
>>
>>> Hi Andrew,
>>>
>>> It looks like the settings don't work. only the main logs go to the
>>> indicated folder, the other dated files still go to my internal log
>>> folder...
>>> ls /mnt/xvdf/logs
>>> nifi-app.log nifi-bootstrap.log nifi-user.log
>>>
>>> ls ~/nifi/logs/
>>> nifi-app_2016-06-06_16.0.log nifi-app_2016-06-07_22.1.log
>>> nifi-app_2016-06-06_17.0.log nifi-app_2016-06-07_22.20.log
>>> nifi-app_2016-06-06_18.0.log nifi-app_2016-06-07_22.21.log
>>> nifi-app_2016-06-06_19.0.log nifi-app_2016-06-07_22.22.log
>>> nifi-app_2016-06-06_20.0.log nifi-app_2016-06-07_22.23.log
>>> nifi-app_2016-06-06_21.0.log nifi-app_2016-06-07_22.24.log
>>> nifi-app_2016-06-06_22.0.log nifi-app_2016-06-07_22.25.log
>>> nifi-app_2016-06-06_23.0.log nifi-app_2016-06-07_22.26.log
>>> nifi-app_2016-06-07_00.0.log nifi-app_2016-06-07_22.27.log
>>> nifi-app_2016-06-07_01.0.log nifi-app_2016-06-07_22.28.log
>>> nifi-app_2016-06-07_01.1.log nifi-app_2016-06-07_22.29.log
>>> nifi-app_2016-06-07_02.0.log nifi-app_2016-06-07_22.2.log
>>> nifi-app_2016-06-07_03.0.log nifi-app_2016-06-07_22.30.log
>>> nifi-app_2016-06-07_04.0.log nifi-app_2016-06-07_22.31.log
>>> nifi-app_2016-06-07_05.0.log nifi-app_2016-06-07_22.32.log
>>> ....
>>> nifi-app_2016-06-07_22.17.log nifi-bootstrap_2016-06-06.log
>>> nifi-app_2016-06-07_22.18.log nifi-user_2016-06-06.log
>>> nifi-app_2016-06-07_22.19.log
>>>
>>>
>>> Can you please check on your end and let me know what I need to change?
>>>
>>> Thanks,
>>> Stephane
>>>
>>> On Tue, Jun 7, 2016 at 9:49 AM Pat Trainor <pa...@gmail.com>
>>> wrote:
>>>
>>>> No wonder it looked like log4j! That's some pretty slick stuff, right
>>>> there!
>>>>
>>>> On Jun 6 2016, at 8:06 am, Andrew Psaltis <ps...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Pat,
>>>>> It is all standard logback, described here: http://logback.qos.ch/
>>>>>
>>>>
>>
Re: How to control NiFi logging
Posted by Stéphane Maarek <st...@gmail.com>.
Hi Andrew,
Thanks for the quick response. Sorry stuff isn't obvious just yet. Please
bear with me
I copied the file you indicated to my logback.xml and didn't change a thing
In bootstrap.conf, I added the following:
org.apache.nifi.bootstrap.config.log.dir=/mnt/xvdf/logs
Now on start of NiFi, the logging still doesn't work, and I'm getting the
following folder in my nifi root:
org.apache.nifi.bootstrap.config.log.dir_IS_UNDEFINED
Can you please help?
regards,
Stephane
On Wed, Jun 8, 2016 at 11:07 AM Andre <an...@fucs.org> wrote:
> Stephane,
>
> There are two settings controlling each individual log. The sections are
> clearly commented within logback settings
>
> You can read the instance in your logback.xml or read the one available
> online
>
>
> https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/conf/logback.xml
>
> Cheers
>
> On Wed, Jun 8, 2016 at 10:22 AM, Stéphane Maarek <
> stephane.maarek@gmail.com> wrote:
>
>> Hi Andrew,
>>
>> It looks like the settings don't work. only the main logs go to the
>> indicated folder, the other dated files still go to my internal log
>> folder...
>> ls /mnt/xvdf/logs
>> nifi-app.log nifi-bootstrap.log nifi-user.log
>>
>> ls ~/nifi/logs/
>> nifi-app_2016-06-06_16.0.log nifi-app_2016-06-07_22.1.log
>> nifi-app_2016-06-06_17.0.log nifi-app_2016-06-07_22.20.log
>> nifi-app_2016-06-06_18.0.log nifi-app_2016-06-07_22.21.log
>> nifi-app_2016-06-06_19.0.log nifi-app_2016-06-07_22.22.log
>> nifi-app_2016-06-06_20.0.log nifi-app_2016-06-07_22.23.log
>> nifi-app_2016-06-06_21.0.log nifi-app_2016-06-07_22.24.log
>> nifi-app_2016-06-06_22.0.log nifi-app_2016-06-07_22.25.log
>> nifi-app_2016-06-06_23.0.log nifi-app_2016-06-07_22.26.log
>> nifi-app_2016-06-07_00.0.log nifi-app_2016-06-07_22.27.log
>> nifi-app_2016-06-07_01.0.log nifi-app_2016-06-07_22.28.log
>> nifi-app_2016-06-07_01.1.log nifi-app_2016-06-07_22.29.log
>> nifi-app_2016-06-07_02.0.log nifi-app_2016-06-07_22.2.log
>> nifi-app_2016-06-07_03.0.log nifi-app_2016-06-07_22.30.log
>> nifi-app_2016-06-07_04.0.log nifi-app_2016-06-07_22.31.log
>> nifi-app_2016-06-07_05.0.log nifi-app_2016-06-07_22.32.log
>> ....
>> nifi-app_2016-06-07_22.17.log nifi-bootstrap_2016-06-06.log
>> nifi-app_2016-06-07_22.18.log nifi-user_2016-06-06.log
>> nifi-app_2016-06-07_22.19.log
>>
>>
>> Can you please check on your end and let me know what I need to change?
>>
>> Thanks,
>> Stephane
>>
>> On Tue, Jun 7, 2016 at 9:49 AM Pat Trainor <pa...@gmail.com> wrote:
>>
>>> No wonder it looked like log4j! That's some pretty slick stuff, right
>>> there!
>>>
>>> On Jun 6 2016, at 8:06 am, Andrew Psaltis <ps...@gmail.com>
>>> wrote:
>>>
>>>> Hi Pat,
>>>> It is all standard logback, described here: http://logback.qos.ch/
>>>>
>>>
>
Re: How to control NiFi logging
Posted by Andre <an...@fucs.org>.
Stephane,
There are two settings controlling each individual log. The sections are
clearly commented within logback settings
You can read the instance in your logback.xml or read the one available
online
https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/conf/logback.xml
Cheers
On Wed, Jun 8, 2016 at 10:22 AM, Stéphane Maarek <st...@gmail.com>
wrote:
> Hi Andrew,
>
> It looks like the settings don't work. only the main logs go to the
> indicated folder, the other dated files still go to my internal log
> folder...
> ls /mnt/xvdf/logs
> nifi-app.log nifi-bootstrap.log nifi-user.log
>
> ls ~/nifi/logs/
> nifi-app_2016-06-06_16.0.log nifi-app_2016-06-07_22.1.log
> nifi-app_2016-06-06_17.0.log nifi-app_2016-06-07_22.20.log
> nifi-app_2016-06-06_18.0.log nifi-app_2016-06-07_22.21.log
> nifi-app_2016-06-06_19.0.log nifi-app_2016-06-07_22.22.log
> nifi-app_2016-06-06_20.0.log nifi-app_2016-06-07_22.23.log
> nifi-app_2016-06-06_21.0.log nifi-app_2016-06-07_22.24.log
> nifi-app_2016-06-06_22.0.log nifi-app_2016-06-07_22.25.log
> nifi-app_2016-06-06_23.0.log nifi-app_2016-06-07_22.26.log
> nifi-app_2016-06-07_00.0.log nifi-app_2016-06-07_22.27.log
> nifi-app_2016-06-07_01.0.log nifi-app_2016-06-07_22.28.log
> nifi-app_2016-06-07_01.1.log nifi-app_2016-06-07_22.29.log
> nifi-app_2016-06-07_02.0.log nifi-app_2016-06-07_22.2.log
> nifi-app_2016-06-07_03.0.log nifi-app_2016-06-07_22.30.log
> nifi-app_2016-06-07_04.0.log nifi-app_2016-06-07_22.31.log
> nifi-app_2016-06-07_05.0.log nifi-app_2016-06-07_22.32.log
> ....
> nifi-app_2016-06-07_22.17.log nifi-bootstrap_2016-06-06.log
> nifi-app_2016-06-07_22.18.log nifi-user_2016-06-06.log
> nifi-app_2016-06-07_22.19.log
>
>
> Can you please check on your end and let me know what I need to change?
>
> Thanks,
> Stephane
>
> On Tue, Jun 7, 2016 at 9:49 AM Pat Trainor <pa...@gmail.com> wrote:
>
>> No wonder it looked like log4j! That's some pretty slick stuff, right
>> there!
>>
>> On Jun 6 2016, at 8:06 am, Andrew Psaltis <ps...@gmail.com>
>> wrote:
>>
>>> Hi Pat,
>>> It is all standard logback, described here: http://logback.qos.ch/
>>>
>>
Re: How to control NiFi logging
Posted by Stéphane Maarek <st...@gmail.com>.
Hi Andrew,
It looks like the settings don't work. only the main logs go to the
indicated folder, the other dated files still go to my internal log
folder...
ls /mnt/xvdf/logs
nifi-app.log nifi-bootstrap.log nifi-user.log
ls ~/nifi/logs/
nifi-app_2016-06-06_16.0.log nifi-app_2016-06-07_22.1.log
nifi-app_2016-06-06_17.0.log nifi-app_2016-06-07_22.20.log
nifi-app_2016-06-06_18.0.log nifi-app_2016-06-07_22.21.log
nifi-app_2016-06-06_19.0.log nifi-app_2016-06-07_22.22.log
nifi-app_2016-06-06_20.0.log nifi-app_2016-06-07_22.23.log
nifi-app_2016-06-06_21.0.log nifi-app_2016-06-07_22.24.log
nifi-app_2016-06-06_22.0.log nifi-app_2016-06-07_22.25.log
nifi-app_2016-06-06_23.0.log nifi-app_2016-06-07_22.26.log
nifi-app_2016-06-07_00.0.log nifi-app_2016-06-07_22.27.log
nifi-app_2016-06-07_01.0.log nifi-app_2016-06-07_22.28.log
nifi-app_2016-06-07_01.1.log nifi-app_2016-06-07_22.29.log
nifi-app_2016-06-07_02.0.log nifi-app_2016-06-07_22.2.log
nifi-app_2016-06-07_03.0.log nifi-app_2016-06-07_22.30.log
nifi-app_2016-06-07_04.0.log nifi-app_2016-06-07_22.31.log
nifi-app_2016-06-07_05.0.log nifi-app_2016-06-07_22.32.log
....
nifi-app_2016-06-07_22.17.log nifi-bootstrap_2016-06-06.log
nifi-app_2016-06-07_22.18.log nifi-user_2016-06-06.log
nifi-app_2016-06-07_22.19.log
Can you please check on your end and let me know what I need to change?
Thanks,
Stephane
On Tue, Jun 7, 2016 at 9:49 AM Pat Trainor <pa...@gmail.com> wrote:
> No wonder it looked like log4j! That's some pretty slick stuff, right
> there!
>
> On Jun 6 2016, at 8:06 am, Andrew Psaltis <ps...@gmail.com>
> wrote:
>
>> Hi Pat,
>> It is all standard logback, described here: http://logback.qos.ch/
>>
>
Re: How to control NiFi logging
Posted by Pat Trainor <pa...@gmail.com>.
No wonder it looked like log4j! That's some pretty slick stuff, right there!
On Jun 6 2016, at 8:06 am, Andrew Psaltis <psaltis.andrew@gmail.com>
wrote:
> Hi Pat,
>
> It is all standard logback, described here: <http://logback.qos.ch/>
Re: How to control NiFi logging
Posted by Andrew Psaltis <ps...@gmail.com>.
Hi Pat,
It is all standard logback, described here: http://logback.qos.ch/
On Mon, Jun 6, 2016 at 12:18 PM, Pat Trainor <pa...@gmail.com> wrote:
> Andrew,
> Is any of this not standard log4j, and nifi-specific?
> On Jun 6, 2016 2:34 AM, "Andrew Psaltis" <ps...@gmail.com> wrote:
>
>> You are correct, sorry about that Stephane!
>>
>> On Mon, Jun 6, 2016 at 8:23 AM, Stéphane Maarek <
>> stephane.maarek@gmail.com> wrote:
>>
>>> Actually, it seems I only needed to change one line (versus two)
>>>
>>> <appender name="BOOTSTRAP_FILE"
>>> class="ch.qos.logback.core.rolling.RollingFileAppender">
>>> <file>/mnt/xvdf/logs/nifi-bootstrap.log</file>
>>> <rollingPolicy
>>> class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
>>> <!--
>>> For daily rollover, use 'user_%d.log'.
>>> For hourly rollover, use 'user_%d{yyyy-MM-dd_HH}.log'.
>>> To GZIP rolled files, replace '.log' with '.log.gz'.
>>> To ZIP rolled files, replace '.log' with '.log.zip'.
>>> -->
>>>
>>> <fileNamePattern>./logs/nifi-bootstrap_%d.log</fileNamePattern>
>>>
>>> On Mon, Jun 6, 2016 at 4:20 PM Stéphane Maarek <
>>> stephane.maarek@gmail.com> wrote:
>>>
>>>> Hi Andrew,
>>>>
>>>> After changing the file I'm getting the following error. I changed the
>>>> lines you indicated.
>>>>
>>>> Failed to auto configure default logger context
>>>> Reported exception:
>>>> ch.qos.logback.core.joran.spi.JoranException: Problem parsing XML
>>>> document. See previously reported errors.
>>>> at
>>>> ch.qos.logback.core.joran.event.SaxEventRecorder.recordEvents(SaxEventRecorder.java:67)
>>>> at
>>>> ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:134)
>>>> at
>>>> ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:99)
>>>> at
>>>> ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:49)
>>>> at
>>>> ch.qos.logback.classic.util.ContextInitializer.configureByResource(ContextInitializer.java:77)
>>>> at
>>>> ch.qos.logback.classic.util.ContextInitializer.autoConfig(ContextInitializer.java:152)
>>>> at
>>>> org.slf4j.impl.StaticLoggerBinder.init(StaticLoggerBinder.java:85)
>>>> at
>>>> org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:55)
>>>> at org.slf4j.LoggerFactory.bind(LoggerFactory.java:141)
>>>> at
>>>> org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:120)
>>>> at
>>>> org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:331)
>>>> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:283)
>>>> at org.apache.nifi.bootstrap.RunNiFi.<init>(RunNiFi.java:117)
>>>> at org.apache.nifi.bootstrap.RunNiFi.main(RunNiFi.java:199)
>>>> Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber:
>>>> 1; Premature end of file.
>>>> at
>>>> com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1239)
>>>> at
>>>> com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:642)
>>>> at
>>>> com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:326)
>>>> at
>>>> ch.qos.logback.core.joran.event.SaxEventRecorder.recordEvents(SaxEventRecorder.java:61)
>>>> ... 13 more
>>>> 06:16:11,055 |-INFO in ch.qos.logback.classic.LoggerContext[default] -
>>>> Could NOT find resource [logback.groovy]
>>>> 06:16:11,055 |-INFO in ch.qos.logback.classic.LoggerContext[default] -
>>>> Could NOT find resource [logback-test.xml]
>>>> 06:16:11,055 |-INFO in ch.qos.logback.classic.LoggerContext[default] -
>>>> Found resource [logback.xml] at [file:/home/ec2-user/nifi/conf/logback.xml]
>>>> 06:16:11,127 |-ERROR in
>>>> ch.qos.logback.core.joran.event.SaxEventRecorder@682ea4b1 -
>>>> XML_PARSING - Parsing fatal error on line 1 and column 1
>>>> org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Premature
>>>> end of file.
>>>> at org.xml.sax.SAXParseException: Premature end of file.
>>>> at at
>>>> com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:198)
>>>> at at
>>>> com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
>>>> at at
>>>> com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:400)
>>>> at at
>>>> com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:327)
>>>> at at
>>>> com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1438)
>>>> at at
>>>> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:1019)
>>>> at at
>>>> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
>>>> at at
>>>> com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:118)
>>>> at at
>>>> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
>>>> at at
>>>> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
>>>> at at
>>>> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
>>>> at at
>>>> com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
>>>> at at
>>>> com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)
>>>> at at
>>>> com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:642)
>>>> at at
>>>> com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:326)
>>>> at at
>>>> ch.qos.logback.core.joran.event.SaxEventRecorder.recordEvents(SaxEventRecorder.java:61)
>>>> at at
>>>> ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:134)
>>>> at at
>>>> ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:99)
>>>> at at
>>>> ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:49)
>>>> at at
>>>> ch.qos.logback.classic.util.ContextInitializer.configureByResource(ContextInitializer.java:77)
>>>> at at
>>>> ch.qos.logback.classic.util.ContextInitializer.autoConfig(ContextInitializer.java:152)
>>>> at at
>>>> org.slf4j.impl.StaticLoggerBinder.init(StaticLoggerBinder.java:85)
>>>> at at
>>>> org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:55)
>>>> at at org.slf4j.LoggerFactory.bind(LoggerFactory.java:141)
>>>> at at
>>>> org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:120)
>>>> at at
>>>> org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:331)
>>>> at at
>>>> org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:283)
>>>> at at
>>>> org.apache.nifi.bootstrap.RunNiFi.<init>(RunNiFi.java:117)
>>>> at at
>>>> org.apache.nifi.bootstrap.RunNiFi.main(RunNiFi.java:199)
>>>>
>>>>
>>>> On Mon, Jun 6, 2016 at 4:10 PM Andrew Psaltis <ps...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Stephane,
>>>>> In the conf directory of your nifi install there is a logback.xml file
>>>>> that needs to be edited. This configuration file controls the internal
>>>>> logging done by nifi. For each of the appenders you will see an XML
>>>>> elements that reference the "logs" directory. For example here is the
>>>>> config for the app.log
>>>>>
>>>>> <appender name="APP_FILE"
>>>>> class="ch.qos.logback.core.rolling.RollingFileAppender">
>>>>> * <file>logs/nifi-app.log</file> <--- This should be changed*
>>>>> <rollingPolicy
>>>>> class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
>>>>> <!--
>>>>> For daily rollover, use 'app_%d.log'.
>>>>> For hourly rollover, use 'app_%d{yyyy-MM-dd_HH}.log'.
>>>>> To GZIP rolled files, replace '.log' with '.log.gz'.
>>>>> To ZIP rolled files, replace '.log' with '.log.zip'.
>>>>> -->
>>>>> *
>>>>> <fileNamePattern>./logs/nifi-app_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern>
>>>>> <--This should be changed*
>>>>> <timeBasedFileNamingAndTriggeringPolicy
>>>>> class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
>>>>> <maxFileSize>100MB</maxFileSize>
>>>>> </timeBasedFileNamingAndTriggeringPolicy>
>>>>> <!-- keep 30 log files worth of history -->
>>>>> <maxHistory>30</maxHistory>
>>>>> </rollingPolicy>
>>>>> <encoder
>>>>> class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
>>>>> <pattern>%date %level [%thread] %logger{40}
>>>>> %msg%n</pattern>
>>>>> <immediateFlush>true</immediateFlush>
>>>>> </encoder>
>>>>> </appender>
>>>>>
>>>>>
>>>>> You would want to change the references above in to your attached
>>>>> volume. I have marked them with bold above. In the logback config file you
>>>>> will see two other appenders as well, these same changes should be done
>>>>> there as well.
>>>>>
>>>>> Thanks,
>>>>> Andrew
>>>>>
>>>>>
>>>>> On Mon, Jun 6, 2016 at 6:56 AM, Stéphane Maarek <
>>>>> stephane.maarek@gmail.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I'm running NiFi on a t1.micro in AWS. I have attached a 100GB EBS
>>>>>> volume at /mnt/xvdf and all the NiFi databases are there. I have a problem
>>>>>> though. It seems that NiFi keeps an internal log and I can't change the
>>>>>> position of it, so it's quickly clogging my t1.micro disk. See below. How
>>>>>> can I improve / change that? And what's the "work" folder for?
>>>>>>
>>>>>> [ec2-user@ ~]$ du -s *
>>>>>> 5271404 nifi
>>>>>> [ec2-user@ ~]$ cd nifi
>>>>>> [ec2-user@ nifi]$ du -s *
>>>>>> 28 bin
>>>>>> 756 conf
>>>>>> 8412 docs
>>>>>> 427468 lib
>>>>>> 64 LICENSE
>>>>>> 4331244 logs
>>>>>> 44 NOTICE
>>>>>> 8 README
>>>>>> 144 state
>>>>>> 503232 work
>>>>>>
>>>>>>
>>>>>> Thanks!
>>>>>> Stephane
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Thanks,
>>>>> Andrew
>>>>>
>>>>> Subscribe to my book: Streaming Data <http://manning.com/psaltis>
>>>>> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306>
>>>>> twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata>
>>>>>
>>>>
>>
>>
>> --
>> Thanks,
>> Andrew
>>
>> Subscribe to my book: Streaming Data <http://manning.com/psaltis>
>> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306>
>> twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata>
>>
>
--
Thanks,
Andrew
Subscribe to my book: Streaming Data <http://manning.com/psaltis>
<https://www.linkedin.com/pub/andrew-psaltis/1/17b/306>
twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata>
Re: How to control NiFi logging
Posted by Pat Trainor <pa...@gmail.com>.
Andrew,
Is any of this not standard log4j, and nifi-specific?
On Jun 6, 2016 2:34 AM, "Andrew Psaltis" <ps...@gmail.com> wrote:
> You are correct, sorry about that Stephane!
>
> On Mon, Jun 6, 2016 at 8:23 AM, Stéphane Maarek <stephane.maarek@gmail.com
> > wrote:
>
>> Actually, it seems I only needed to change one line (versus two)
>>
>> <appender name="BOOTSTRAP_FILE"
>> class="ch.qos.logback.core.rolling.RollingFileAppender">
>> <file>/mnt/xvdf/logs/nifi-bootstrap.log</file>
>> <rollingPolicy
>> class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
>> <!--
>> For daily rollover, use 'user_%d.log'.
>> For hourly rollover, use 'user_%d{yyyy-MM-dd_HH}.log'.
>> To GZIP rolled files, replace '.log' with '.log.gz'.
>> To ZIP rolled files, replace '.log' with '.log.zip'.
>> -->
>>
>> <fileNamePattern>./logs/nifi-bootstrap_%d.log</fileNamePattern>
>>
>> On Mon, Jun 6, 2016 at 4:20 PM Stéphane Maarek <st...@gmail.com>
>> wrote:
>>
>>> Hi Andrew,
>>>
>>> After changing the file I'm getting the following error. I changed the
>>> lines you indicated.
>>>
>>> Failed to auto configure default logger context
>>> Reported exception:
>>> ch.qos.logback.core.joran.spi.JoranException: Problem parsing XML
>>> document. See previously reported errors.
>>> at
>>> ch.qos.logback.core.joran.event.SaxEventRecorder.recordEvents(SaxEventRecorder.java:67)
>>> at
>>> ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:134)
>>> at
>>> ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:99)
>>> at
>>> ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:49)
>>> at
>>> ch.qos.logback.classic.util.ContextInitializer.configureByResource(ContextInitializer.java:77)
>>> at
>>> ch.qos.logback.classic.util.ContextInitializer.autoConfig(ContextInitializer.java:152)
>>> at
>>> org.slf4j.impl.StaticLoggerBinder.init(StaticLoggerBinder.java:85)
>>> at
>>> org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:55)
>>> at org.slf4j.LoggerFactory.bind(LoggerFactory.java:141)
>>> at
>>> org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:120)
>>> at
>>> org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:331)
>>> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:283)
>>> at org.apache.nifi.bootstrap.RunNiFi.<init>(RunNiFi.java:117)
>>> at org.apache.nifi.bootstrap.RunNiFi.main(RunNiFi.java:199)
>>> Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber:
>>> 1; Premature end of file.
>>> at
>>> com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1239)
>>> at
>>> com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:642)
>>> at
>>> com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:326)
>>> at
>>> ch.qos.logback.core.joran.event.SaxEventRecorder.recordEvents(SaxEventRecorder.java:61)
>>> ... 13 more
>>> 06:16:11,055 |-INFO in ch.qos.logback.classic.LoggerContext[default] -
>>> Could NOT find resource [logback.groovy]
>>> 06:16:11,055 |-INFO in ch.qos.logback.classic.LoggerContext[default] -
>>> Could NOT find resource [logback-test.xml]
>>> 06:16:11,055 |-INFO in ch.qos.logback.classic.LoggerContext[default] -
>>> Found resource [logback.xml] at [file:/home/ec2-user/nifi/conf/logback.xml]
>>> 06:16:11,127 |-ERROR in
>>> ch.qos.logback.core.joran.event.SaxEventRecorder@682ea4b1 - XML_PARSING
>>> - Parsing fatal error on line 1 and column 1 org.xml.sax.SAXParseException;
>>> lineNumber: 1; columnNumber: 1; Premature end of file.
>>> at org.xml.sax.SAXParseException: Premature end of file.
>>> at at
>>> com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:198)
>>> at at
>>> com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
>>> at at
>>> com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:400)
>>> at at
>>> com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:327)
>>> at at
>>> com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1438)
>>> at at
>>> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:1019)
>>> at at
>>> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
>>> at at
>>> com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:118)
>>> at at
>>> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
>>> at at
>>> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
>>> at at
>>> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
>>> at at
>>> com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
>>> at at
>>> com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)
>>> at at
>>> com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:642)
>>> at at
>>> com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:326)
>>> at at
>>> ch.qos.logback.core.joran.event.SaxEventRecorder.recordEvents(SaxEventRecorder.java:61)
>>> at at
>>> ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:134)
>>> at at
>>> ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:99)
>>> at at
>>> ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:49)
>>> at at
>>> ch.qos.logback.classic.util.ContextInitializer.configureByResource(ContextInitializer.java:77)
>>> at at
>>> ch.qos.logback.classic.util.ContextInitializer.autoConfig(ContextInitializer.java:152)
>>> at at
>>> org.slf4j.impl.StaticLoggerBinder.init(StaticLoggerBinder.java:85)
>>> at at
>>> org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:55)
>>> at at org.slf4j.LoggerFactory.bind(LoggerFactory.java:141)
>>> at at
>>> org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:120)
>>> at at
>>> org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:331)
>>> at at
>>> org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:283)
>>> at at
>>> org.apache.nifi.bootstrap.RunNiFi.<init>(RunNiFi.java:117)
>>> at at
>>> org.apache.nifi.bootstrap.RunNiFi.main(RunNiFi.java:199)
>>>
>>>
>>> On Mon, Jun 6, 2016 at 4:10 PM Andrew Psaltis <ps...@gmail.com>
>>> wrote:
>>>
>>>> Hi Stephane,
>>>> In the conf directory of your nifi install there is a logback.xml file
>>>> that needs to be edited. This configuration file controls the internal
>>>> logging done by nifi. For each of the appenders you will see an XML
>>>> elements that reference the "logs" directory. For example here is the
>>>> config for the app.log
>>>>
>>>> <appender name="APP_FILE"
>>>> class="ch.qos.logback.core.rolling.RollingFileAppender">
>>>> * <file>logs/nifi-app.log</file> <--- This should be changed*
>>>> <rollingPolicy
>>>> class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
>>>> <!--
>>>> For daily rollover, use 'app_%d.log'.
>>>> For hourly rollover, use 'app_%d{yyyy-MM-dd_HH}.log'.
>>>> To GZIP rolled files, replace '.log' with '.log.gz'.
>>>> To ZIP rolled files, replace '.log' with '.log.zip'.
>>>> -->
>>>> *
>>>> <fileNamePattern>./logs/nifi-app_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern>
>>>> <--This should be changed*
>>>> <timeBasedFileNamingAndTriggeringPolicy
>>>> class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
>>>> <maxFileSize>100MB</maxFileSize>
>>>> </timeBasedFileNamingAndTriggeringPolicy>
>>>> <!-- keep 30 log files worth of history -->
>>>> <maxHistory>30</maxHistory>
>>>> </rollingPolicy>
>>>> <encoder
>>>> class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
>>>> <pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
>>>> <immediateFlush>true</immediateFlush>
>>>> </encoder>
>>>> </appender>
>>>>
>>>>
>>>> You would want to change the references above in to your attached
>>>> volume. I have marked them with bold above. In the logback config file you
>>>> will see two other appenders as well, these same changes should be done
>>>> there as well.
>>>>
>>>> Thanks,
>>>> Andrew
>>>>
>>>>
>>>> On Mon, Jun 6, 2016 at 6:56 AM, Stéphane Maarek <
>>>> stephane.maarek@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I'm running NiFi on a t1.micro in AWS. I have attached a 100GB EBS
>>>>> volume at /mnt/xvdf and all the NiFi databases are there. I have a problem
>>>>> though. It seems that NiFi keeps an internal log and I can't change the
>>>>> position of it, so it's quickly clogging my t1.micro disk. See below. How
>>>>> can I improve / change that? And what's the "work" folder for?
>>>>>
>>>>> [ec2-user@ ~]$ du -s *
>>>>> 5271404 nifi
>>>>> [ec2-user@ ~]$ cd nifi
>>>>> [ec2-user@ nifi]$ du -s *
>>>>> 28 bin
>>>>> 756 conf
>>>>> 8412 docs
>>>>> 427468 lib
>>>>> 64 LICENSE
>>>>> 4331244 logs
>>>>> 44 NOTICE
>>>>> 8 README
>>>>> 144 state
>>>>> 503232 work
>>>>>
>>>>>
>>>>> Thanks!
>>>>> Stephane
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Thanks,
>>>> Andrew
>>>>
>>>> Subscribe to my book: Streaming Data <http://manning.com/psaltis>
>>>> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306>
>>>> twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata>
>>>>
>>>
>
>
> --
> Thanks,
> Andrew
>
> Subscribe to my book: Streaming Data <http://manning.com/psaltis>
> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306>
> twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata>
>
Re: How to control NiFi logging
Posted by Andrew Psaltis <ps...@gmail.com>.
You are correct, sorry about that Stephane!
On Mon, Jun 6, 2016 at 8:23 AM, Stéphane Maarek <st...@gmail.com>
wrote:
> Actually, it seems I only needed to change one line (versus two)
>
> <appender name="BOOTSTRAP_FILE"
> class="ch.qos.logback.core.rolling.RollingFileAppender">
> <file>/mnt/xvdf/logs/nifi-bootstrap.log</file>
> <rollingPolicy
> class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
> <!--
> For daily rollover, use 'user_%d.log'.
> For hourly rollover, use 'user_%d{yyyy-MM-dd_HH}.log'.
> To GZIP rolled files, replace '.log' with '.log.gz'.
> To ZIP rolled files, replace '.log' with '.log.zip'.
> -->
> <fileNamePattern>./logs/nifi-bootstrap_%d.log</fileNamePattern>
>
> On Mon, Jun 6, 2016 at 4:20 PM Stéphane Maarek <st...@gmail.com>
> wrote:
>
>> Hi Andrew,
>>
>> After changing the file I'm getting the following error. I changed the
>> lines you indicated.
>>
>> Failed to auto configure default logger context
>> Reported exception:
>> ch.qos.logback.core.joran.spi.JoranException: Problem parsing XML
>> document. See previously reported errors.
>> at
>> ch.qos.logback.core.joran.event.SaxEventRecorder.recordEvents(SaxEventRecorder.java:67)
>> at
>> ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:134)
>> at
>> ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:99)
>> at
>> ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:49)
>> at
>> ch.qos.logback.classic.util.ContextInitializer.configureByResource(ContextInitializer.java:77)
>> at
>> ch.qos.logback.classic.util.ContextInitializer.autoConfig(ContextInitializer.java:152)
>> at
>> org.slf4j.impl.StaticLoggerBinder.init(StaticLoggerBinder.java:85)
>> at
>> org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:55)
>> at org.slf4j.LoggerFactory.bind(LoggerFactory.java:141)
>> at
>> org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:120)
>> at
>> org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:331)
>> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:283)
>> at org.apache.nifi.bootstrap.RunNiFi.<init>(RunNiFi.java:117)
>> at org.apache.nifi.bootstrap.RunNiFi.main(RunNiFi.java:199)
>> Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1;
>> Premature end of file.
>> at
>> com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1239)
>> at
>> com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:642)
>> at
>> com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:326)
>> at
>> ch.qos.logback.core.joran.event.SaxEventRecorder.recordEvents(SaxEventRecorder.java:61)
>> ... 13 more
>> 06:16:11,055 |-INFO in ch.qos.logback.classic.LoggerContext[default] -
>> Could NOT find resource [logback.groovy]
>> 06:16:11,055 |-INFO in ch.qos.logback.classic.LoggerContext[default] -
>> Could NOT find resource [logback-test.xml]
>> 06:16:11,055 |-INFO in ch.qos.logback.classic.LoggerContext[default] -
>> Found resource [logback.xml] at [file:/home/ec2-user/nifi/conf/logback.xml]
>> 06:16:11,127 |-ERROR in
>> ch.qos.logback.core.joran.event.SaxEventRecorder@682ea4b1 - XML_PARSING
>> - Parsing fatal error on line 1 and column 1 org.xml.sax.SAXParseException;
>> lineNumber: 1; columnNumber: 1; Premature end of file.
>> at org.xml.sax.SAXParseException: Premature end of file.
>> at at
>> com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:198)
>> at at
>> com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
>> at at
>> com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:400)
>> at at
>> com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:327)
>> at at
>> com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1438)
>> at at
>> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:1019)
>> at at
>> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
>> at at
>> com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:118)
>> at at
>> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
>> at at
>> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
>> at at
>> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
>> at at
>> com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
>> at at
>> com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)
>> at at
>> com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:642)
>> at at
>> com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:326)
>> at at
>> ch.qos.logback.core.joran.event.SaxEventRecorder.recordEvents(SaxEventRecorder.java:61)
>> at at
>> ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:134)
>> at at
>> ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:99)
>> at at
>> ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:49)
>> at at
>> ch.qos.logback.classic.util.ContextInitializer.configureByResource(ContextInitializer.java:77)
>> at at
>> ch.qos.logback.classic.util.ContextInitializer.autoConfig(ContextInitializer.java:152)
>> at at
>> org.slf4j.impl.StaticLoggerBinder.init(StaticLoggerBinder.java:85)
>> at at
>> org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:55)
>> at at org.slf4j.LoggerFactory.bind(LoggerFactory.java:141)
>> at at
>> org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:120)
>> at at
>> org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:331)
>> at at
>> org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:283)
>> at at
>> org.apache.nifi.bootstrap.RunNiFi.<init>(RunNiFi.java:117)
>> at at
>> org.apache.nifi.bootstrap.RunNiFi.main(RunNiFi.java:199)
>>
>>
>> On Mon, Jun 6, 2016 at 4:10 PM Andrew Psaltis <ps...@gmail.com>
>> wrote:
>>
>>> Hi Stephane,
>>> In the conf directory of your nifi install there is a logback.xml file
>>> that needs to be edited. This configuration file controls the internal
>>> logging done by nifi. For each of the appenders you will see an XML
>>> elements that reference the "logs" directory. For example here is the
>>> config for the app.log
>>>
>>> <appender name="APP_FILE"
>>> class="ch.qos.logback.core.rolling.RollingFileAppender">
>>> * <file>logs/nifi-app.log</file> <--- This should be changed*
>>> <rollingPolicy
>>> class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
>>> <!--
>>> For daily rollover, use 'app_%d.log'.
>>> For hourly rollover, use 'app_%d{yyyy-MM-dd_HH}.log'.
>>> To GZIP rolled files, replace '.log' with '.log.gz'.
>>> To ZIP rolled files, replace '.log' with '.log.zip'.
>>> -->
>>> *
>>> <fileNamePattern>./logs/nifi-app_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern>
>>> <--This should be changed*
>>> <timeBasedFileNamingAndTriggeringPolicy
>>> class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
>>> <maxFileSize>100MB</maxFileSize>
>>> </timeBasedFileNamingAndTriggeringPolicy>
>>> <!-- keep 30 log files worth of history -->
>>> <maxHistory>30</maxHistory>
>>> </rollingPolicy>
>>> <encoder
>>> class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
>>> <pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
>>> <immediateFlush>true</immediateFlush>
>>> </encoder>
>>> </appender>
>>>
>>>
>>> You would want to change the references above in to your attached
>>> volume. I have marked them with bold above. In the logback config file you
>>> will see two other appenders as well, these same changes should be done
>>> there as well.
>>>
>>> Thanks,
>>> Andrew
>>>
>>>
>>> On Mon, Jun 6, 2016 at 6:56 AM, Stéphane Maarek <
>>> stephane.maarek@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I'm running NiFi on a t1.micro in AWS. I have attached a 100GB EBS
>>>> volume at /mnt/xvdf and all the NiFi databases are there. I have a problem
>>>> though. It seems that NiFi keeps an internal log and I can't change the
>>>> position of it, so it's quickly clogging my t1.micro disk. See below. How
>>>> can I improve / change that? And what's the "work" folder for?
>>>>
>>>> [ec2-user@ ~]$ du -s *
>>>> 5271404 nifi
>>>> [ec2-user@ ~]$ cd nifi
>>>> [ec2-user@ nifi]$ du -s *
>>>> 28 bin
>>>> 756 conf
>>>> 8412 docs
>>>> 427468 lib
>>>> 64 LICENSE
>>>> 4331244 logs
>>>> 44 NOTICE
>>>> 8 README
>>>> 144 state
>>>> 503232 work
>>>>
>>>>
>>>> Thanks!
>>>> Stephane
>>>>
>>>
>>>
>>>
>>> --
>>> Thanks,
>>> Andrew
>>>
>>> Subscribe to my book: Streaming Data <http://manning.com/psaltis>
>>> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306>
>>> twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata>
>>>
>>
--
Thanks,
Andrew
Subscribe to my book: Streaming Data <http://manning.com/psaltis>
<https://www.linkedin.com/pub/andrew-psaltis/1/17b/306>
twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata>
Re: How to control NiFi logging
Posted by Stéphane Maarek <st...@gmail.com>.
Actually, it seems I only needed to change one line (versus two)
<appender name="BOOTSTRAP_FILE"
class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>/mnt/xvdf/logs/nifi-bootstrap.log</file>
<rollingPolicy
class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<!--
For daily rollover, use 'user_%d.log'.
For hourly rollover, use 'user_%d{yyyy-MM-dd_HH}.log'.
To GZIP rolled files, replace '.log' with '.log.gz'.
To ZIP rolled files, replace '.log' with '.log.zip'.
-->
<fileNamePattern>./logs/nifi-bootstrap_%d.log</fileNamePattern>
On Mon, Jun 6, 2016 at 4:20 PM Stéphane Maarek <st...@gmail.com>
wrote:
> Hi Andrew,
>
> After changing the file I'm getting the following error. I changed the
> lines you indicated.
>
> Failed to auto configure default logger context
> Reported exception:
> ch.qos.logback.core.joran.spi.JoranException: Problem parsing XML
> document. See previously reported errors.
> at
> ch.qos.logback.core.joran.event.SaxEventRecorder.recordEvents(SaxEventRecorder.java:67)
> at
> ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:134)
> at
> ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:99)
> at
> ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:49)
> at
> ch.qos.logback.classic.util.ContextInitializer.configureByResource(ContextInitializer.java:77)
> at
> ch.qos.logback.classic.util.ContextInitializer.autoConfig(ContextInitializer.java:152)
> at
> org.slf4j.impl.StaticLoggerBinder.init(StaticLoggerBinder.java:85)
> at
> org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:55)
> at org.slf4j.LoggerFactory.bind(LoggerFactory.java:141)
> at
> org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:120)
> at
> org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:331)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:283)
> at org.apache.nifi.bootstrap.RunNiFi.<init>(RunNiFi.java:117)
> at org.apache.nifi.bootstrap.RunNiFi.main(RunNiFi.java:199)
> Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1;
> Premature end of file.
> at
> com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1239)
> at
> com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:642)
> at
> com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:326)
> at
> ch.qos.logback.core.joran.event.SaxEventRecorder.recordEvents(SaxEventRecorder.java:61)
> ... 13 more
> 06:16:11,055 |-INFO in ch.qos.logback.classic.LoggerContext[default] -
> Could NOT find resource [logback.groovy]
> 06:16:11,055 |-INFO in ch.qos.logback.classic.LoggerContext[default] -
> Could NOT find resource [logback-test.xml]
> 06:16:11,055 |-INFO in ch.qos.logback.classic.LoggerContext[default] -
> Found resource [logback.xml] at [file:/home/ec2-user/nifi/conf/logback.xml]
> 06:16:11,127 |-ERROR in
> ch.qos.logback.core.joran.event.SaxEventRecorder@682ea4b1 - XML_PARSING -
> Parsing fatal error on line 1 and column 1 org.xml.sax.SAXParseException;
> lineNumber: 1; columnNumber: 1; Premature end of file.
> at org.xml.sax.SAXParseException: Premature end of file.
> at at
> com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:198)
> at at
> com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
> at at
> com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:400)
> at at
> com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:327)
> at at
> com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1438)
> at at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:1019)
> at at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
> at at
> com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:118)
> at at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
> at at
> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
> at at
> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
> at at
> com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
> at at
> com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)
> at at
> com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:642)
> at at
> com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:326)
> at at
> ch.qos.logback.core.joran.event.SaxEventRecorder.recordEvents(SaxEventRecorder.java:61)
> at at
> ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:134)
> at at
> ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:99)
> at at
> ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:49)
> at at
> ch.qos.logback.classic.util.ContextInitializer.configureByResource(ContextInitializer.java:77)
> at at
> ch.qos.logback.classic.util.ContextInitializer.autoConfig(ContextInitializer.java:152)
> at at
> org.slf4j.impl.StaticLoggerBinder.init(StaticLoggerBinder.java:85)
> at at
> org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:55)
> at at org.slf4j.LoggerFactory.bind(LoggerFactory.java:141)
> at at
> org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:120)
> at at
> org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:331)
> at at
> org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:283)
> at at
> org.apache.nifi.bootstrap.RunNiFi.<init>(RunNiFi.java:117)
> at at org.apache.nifi.bootstrap.RunNiFi.main(RunNiFi.java:199)
>
>
> On Mon, Jun 6, 2016 at 4:10 PM Andrew Psaltis <ps...@gmail.com>
> wrote:
>
>> Hi Stephane,
>> In the conf directory of your nifi install there is a logback.xml file
>> that needs to be edited. This configuration file controls the internal
>> logging done by nifi. For each of the appenders you will see an XML
>> elements that reference the "logs" directory. For example here is the
>> config for the app.log
>>
>> <appender name="APP_FILE"
>> class="ch.qos.logback.core.rolling.RollingFileAppender">
>> * <file>logs/nifi-app.log</file> <--- This should be changed*
>> <rollingPolicy
>> class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
>> <!--
>> For daily rollover, use 'app_%d.log'.
>> For hourly rollover, use 'app_%d{yyyy-MM-dd_HH}.log'.
>> To GZIP rolled files, replace '.log' with '.log.gz'.
>> To ZIP rolled files, replace '.log' with '.log.zip'.
>> -->
>> *
>> <fileNamePattern>./logs/nifi-app_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern>
>> <--This should be changed*
>> <timeBasedFileNamingAndTriggeringPolicy
>> class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
>> <maxFileSize>100MB</maxFileSize>
>> </timeBasedFileNamingAndTriggeringPolicy>
>> <!-- keep 30 log files worth of history -->
>> <maxHistory>30</maxHistory>
>> </rollingPolicy>
>> <encoder
>> class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
>> <pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
>> <immediateFlush>true</immediateFlush>
>> </encoder>
>> </appender>
>>
>>
>> You would want to change the references above in to your attached volume.
>> I have marked them with bold above. In the logback config file you will see
>> two other appenders as well, these same changes should be done there as
>> well.
>>
>> Thanks,
>> Andrew
>>
>>
>> On Mon, Jun 6, 2016 at 6:56 AM, Stéphane Maarek <
>> stephane.maarek@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I'm running NiFi on a t1.micro in AWS. I have attached a 100GB EBS
>>> volume at /mnt/xvdf and all the NiFi databases are there. I have a problem
>>> though. It seems that NiFi keeps an internal log and I can't change the
>>> position of it, so it's quickly clogging my t1.micro disk. See below. How
>>> can I improve / change that? And what's the "work" folder for?
>>>
>>> [ec2-user@ ~]$ du -s *
>>> 5271404 nifi
>>> [ec2-user@ ~]$ cd nifi
>>> [ec2-user@ nifi]$ du -s *
>>> 28 bin
>>> 756 conf
>>> 8412 docs
>>> 427468 lib
>>> 64 LICENSE
>>> 4331244 logs
>>> 44 NOTICE
>>> 8 README
>>> 144 state
>>> 503232 work
>>>
>>>
>>> Thanks!
>>> Stephane
>>>
>>
>>
>>
>> --
>> Thanks,
>> Andrew
>>
>> Subscribe to my book: Streaming Data <http://manning.com/psaltis>
>> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306>
>> twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata>
>>
>
Re: How to control NiFi logging
Posted by Stéphane Maarek <st...@gmail.com>.
Hi Andrew,
After changing the file I'm getting the following error. I changed the
lines you indicated.
Failed to auto configure default logger context
Reported exception:
ch.qos.logback.core.joran.spi.JoranException: Problem parsing XML document.
See previously reported errors.
at
ch.qos.logback.core.joran.event.SaxEventRecorder.recordEvents(SaxEventRecorder.java:67)
at
ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:134)
at
ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:99)
at
ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:49)
at
ch.qos.logback.classic.util.ContextInitializer.configureByResource(ContextInitializer.java:77)
at
ch.qos.logback.classic.util.ContextInitializer.autoConfig(ContextInitializer.java:152)
at
org.slf4j.impl.StaticLoggerBinder.init(StaticLoggerBinder.java:85)
at
org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:55)
at org.slf4j.LoggerFactory.bind(LoggerFactory.java:141)
at
org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:120)
at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:331)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:283)
at org.apache.nifi.bootstrap.RunNiFi.<init>(RunNiFi.java:117)
at org.apache.nifi.bootstrap.RunNiFi.main(RunNiFi.java:199)
Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1;
Premature end of file.
at
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1239)
at
com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:642)
at
com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:326)
at
ch.qos.logback.core.joran.event.SaxEventRecorder.recordEvents(SaxEventRecorder.java:61)
... 13 more
06:16:11,055 |-INFO in ch.qos.logback.classic.LoggerContext[default] -
Could NOT find resource [logback.groovy]
06:16:11,055 |-INFO in ch.qos.logback.classic.LoggerContext[default] -
Could NOT find resource [logback-test.xml]
06:16:11,055 |-INFO in ch.qos.logback.classic.LoggerContext[default] -
Found resource [logback.xml] at [file:/home/ec2-user/nifi/conf/logback.xml]
06:16:11,127 |-ERROR in
ch.qos.logback.core.joran.event.SaxEventRecorder@682ea4b1 - XML_PARSING -
Parsing fatal error on line 1 and column 1 org.xml.sax.SAXParseException;
lineNumber: 1; columnNumber: 1; Premature end of file.
at org.xml.sax.SAXParseException: Premature end of file.
at at
com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:198)
at at
com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
at at
com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:400)
at at
com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:327)
at at
com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1438)
at at
com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:1019)
at at
com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
at at
com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:118)
at at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
at at
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
at at
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
at at
com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
at at
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)
at at
com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:642)
at at
com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:326)
at at
ch.qos.logback.core.joran.event.SaxEventRecorder.recordEvents(SaxEventRecorder.java:61)
at at
ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:134)
at at
ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:99)
at at
ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:49)
at at
ch.qos.logback.classic.util.ContextInitializer.configureByResource(ContextInitializer.java:77)
at at
ch.qos.logback.classic.util.ContextInitializer.autoConfig(ContextInitializer.java:152)
at at
org.slf4j.impl.StaticLoggerBinder.init(StaticLoggerBinder.java:85)
at at
org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:55)
at at org.slf4j.LoggerFactory.bind(LoggerFactory.java:141)
at at
org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:120)
at at
org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:331)
at at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:283)
at at
org.apache.nifi.bootstrap.RunNiFi.<init>(RunNiFi.java:117)
at at org.apache.nifi.bootstrap.RunNiFi.main(RunNiFi.java:199)
On Mon, Jun 6, 2016 at 4:10 PM Andrew Psaltis <ps...@gmail.com>
wrote:
> Hi Stephane,
> In the conf directory of your nifi install there is a logback.xml file
> that needs to be edited. This configuration file controls the internal
> logging done by nifi. For each of the appenders you will see an XML
> elements that reference the "logs" directory. For example here is the
> config for the app.log
>
> <appender name="APP_FILE"
> class="ch.qos.logback.core.rolling.RollingFileAppender">
> * <file>logs/nifi-app.log</file> <--- This should be changed*
> <rollingPolicy
> class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
> <!--
> For daily rollover, use 'app_%d.log'.
> For hourly rollover, use 'app_%d{yyyy-MM-dd_HH}.log'.
> To GZIP rolled files, replace '.log' with '.log.gz'.
> To ZIP rolled files, replace '.log' with '.log.zip'.
> -->
> *
> <fileNamePattern>./logs/nifi-app_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern>
> <--This should be changed*
> <timeBasedFileNamingAndTriggeringPolicy
> class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
> <maxFileSize>100MB</maxFileSize>
> </timeBasedFileNamingAndTriggeringPolicy>
> <!-- keep 30 log files worth of history -->
> <maxHistory>30</maxHistory>
> </rollingPolicy>
> <encoder
> class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
> <pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
> <immediateFlush>true</immediateFlush>
> </encoder>
> </appender>
>
>
> You would want to change the references above in to your attached volume.
> I have marked them with bold above. In the logback config file you will see
> two other appenders as well, these same changes should be done there as
> well.
>
> Thanks,
> Andrew
>
>
> On Mon, Jun 6, 2016 at 6:56 AM, Stéphane Maarek <stephane.maarek@gmail.com
> > wrote:
>
>> Hi,
>>
>> I'm running NiFi on a t1.micro in AWS. I have attached a 100GB EBS volume
>> at /mnt/xvdf and all the NiFi databases are there. I have a problem though.
>> It seems that NiFi keeps an internal log and I can't change the position of
>> it, so it's quickly clogging my t1.micro disk. See below. How can I improve
>> / change that? And what's the "work" folder for?
>>
>> [ec2-user@ ~]$ du -s *
>> 5271404 nifi
>> [ec2-user@ ~]$ cd nifi
>> [ec2-user@ nifi]$ du -s *
>> 28 bin
>> 756 conf
>> 8412 docs
>> 427468 lib
>> 64 LICENSE
>> 4331244 logs
>> 44 NOTICE
>> 8 README
>> 144 state
>> 503232 work
>>
>>
>> Thanks!
>> Stephane
>>
>
>
>
> --
> Thanks,
> Andrew
>
> Subscribe to my book: Streaming Data <http://manning.com/psaltis>
> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306>
> twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata>
>
Re: How to control NiFi logging
Posted by Andrew Psaltis <ps...@gmail.com>.
Hi Stephane,
In the conf directory of your nifi install there is a logback.xml file that
needs to be edited. This configuration file controls the internal logging
done by nifi. For each of the appenders you will see an XML elements that
reference the "logs" directory. For example here is the config for the
app.log
<appender name="APP_FILE"
class="ch.qos.logback.core.rolling.RollingFileAppender">
* <file>logs/nifi-app.log</file> <--- This should be changed*
<rollingPolicy
class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<!--
For daily rollover, use 'app_%d.log'.
For hourly rollover, use 'app_%d{yyyy-MM-dd_HH}.log'.
To GZIP rolled files, replace '.log' with '.log.gz'.
To ZIP rolled files, replace '.log' with '.log.zip'.
-->
*
<fileNamePattern>./logs/nifi-app_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern>
<--This should be changed*
<timeBasedFileNamingAndTriggeringPolicy
class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
<maxFileSize>100MB</maxFileSize>
</timeBasedFileNamingAndTriggeringPolicy>
<!-- keep 30 log files worth of history -->
<maxHistory>30</maxHistory>
</rollingPolicy>
<encoder
class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
<pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
<immediateFlush>true</immediateFlush>
</encoder>
</appender>
You would want to change the references above in to your attached volume. I
have marked them with bold above. In the logback config file you will see
two other appenders as well, these same changes should be done there as
well.
Thanks,
Andrew
On Mon, Jun 6, 2016 at 6:56 AM, Stéphane Maarek <st...@gmail.com>
wrote:
> Hi,
>
> I'm running NiFi on a t1.micro in AWS. I have attached a 100GB EBS volume
> at /mnt/xvdf and all the NiFi databases are there. I have a problem though.
> It seems that NiFi keeps an internal log and I can't change the position of
> it, so it's quickly clogging my t1.micro disk. See below. How can I improve
> / change that? And what's the "work" folder for?
>
> [ec2-user@ ~]$ du -s *
> 5271404 nifi
> [ec2-user@ ~]$ cd nifi
> [ec2-user@ nifi]$ du -s *
> 28 bin
> 756 conf
> 8412 docs
> 427468 lib
> 64 LICENSE
> 4331244 logs
> 44 NOTICE
> 8 README
> 144 state
> 503232 work
>
>
> Thanks!
> Stephane
>
--
Thanks,
Andrew
Subscribe to my book: Streaming Data <http://manning.com/psaltis>
<https://www.linkedin.com/pub/andrew-psaltis/1/17b/306>
twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata>