You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "lordgamez (via GitHub)" <gi...@apache.org> on 2024/01/10 11:44:41 UTC
[PR] MINIFICPP-2284 Update documentation to reflect user feedback [nifi-minifi-cpp]
lordgamez opened a new pull request, #1713:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1713
- Document all repository options
- Add documentation on logging configuration
- Additional clarification and notes
https://issues.apache.org/jira/browse/MINIFICPP-2284
------------------------
Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with MINIFICPP-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [ ] Has your PR been rebased against the latest commit within the target branch (typically main)?
- [ ] Is your initial contribution a single, squashed commit?
### For code changes:
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file?
- [ ] If applicable, have you updated the NOTICE file?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINIFICPP-2284 Update documentation to reflect user feedback [nifi-minifi-cpp]
Posted by "lordgamez (via GitHub)" <gi...@apache.org>.
lordgamez commented on code in PR #1713:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1713#discussion_r1447342775
##########
CONFIGURE.md:
##########
@@ -287,6 +341,60 @@ Another option to define a state storage is to use the following properties in t
nifi.state.storage.local.auto.persistence.interval=0 sec
+### Configuring Repositories
+
+Apache MiNiFi C++ uses three repositories similarily to Apache NiFi:
+- The Flow File Repository holds the metadata (like flow file attributes) of the flow Files.
+- The Content Repository holds the content of the flow files.
+- The Provenance Repository holds the history of the flow files.
+
+The underlying implementation to use for these repositories can be configured in the minifi.properties file.
+
+The Flow File Repository can be configured with the `nifi.flowfile.repository.class.name` property. If not specified, it uses the `FlowFileRepository` class by default, which stores the flow file metadata in a RocksDB database. Alternatively it can be configured to use a `VolatileFlowFileRepository` that keeps the state in memory and flushes upon restart, or the `NoOpRepository` for not keeping any state.
+
+ # in minifi.properties
+ nifi.flowfile.repository.class.name=VolatileFlowFileRepository
+
+The Content Repository can be configured with the `nifi.content.repository.class.name` property. If not specified, it uses the `DatabaseContentRepository` class by default, which persists the content in a RocksDB database. `DatabaseContentRepository` is also the default value specified in the minifi.properties file. Alternatively it can be configured to use a `VolatileContentRepository` that keeps the state in memory and flushes upon restart, or the `FileSystemRepository` to keep the state in regular files.
+
+**NOTE:** RocksDB database has a limit of 4GB for the size of a database object. Due to this if you are expecting to process larger flow files than 4GB you should use the `FileSystemRepository`. The downside of using `FileSystemRepository` is that it does not have the transactional guarantees of the RocksDB repository implementation.
+
+ # in minifi.properties
+ nifi.content.repository.class.name=FileSystemRepository
+
+The Provenance Repository can be configured with the `nifi.provenance.repository.class.name` property. If not specified, it uses the `ProvenanceRepository` class by default, which persists the provenance events in a RocksDB database. Alternatively it can be configured to use a `VolatileProvenanceRepository` that keeps the state in memory and flushes upon restart, or the `NoOpRepository` to not keep track of the provenance events. By default we do not keep track of the provenance data, so `NoOpRepository` is the value specified in the default minifi.properties file.
+
+ # in minifi.properties
+ nifi.flowfile.repository.class.name=NoOpRepository
+
+
+### Configuring Volatile Repositories
+As stated before each of the repositories can be configured to be volatile (state kept in memory and flushed upon restart) or persistent. Volatile repositories have some additional options, that can be specified in the following ways:
Review Comment:
In this case the repository write replaces the oldest entry in round robin manner, so the oldest entries will be lost. I added a note about this in 999a2039a9457abf91dae7af2f75a5910d7ce86f
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINIFICPP-2284 Update documentation to reflect user feedback [nifi-minifi-cpp]
Posted by "lordgamez (via GitHub)" <gi...@apache.org>.
lordgamez commented on code in PR #1713:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1713#discussion_r1448830392
##########
CONFIGURE.md:
##########
@@ -78,8 +132,8 @@ MiNiFi supports YAML and JSON configuration formats. The desired configuration f
### Scheduling strategies
Currently Apache NiFi MiNiFi C++ supports TIMER_DRIVEN, EVENT_DRIVEN, and CRON_DRIVEN. TIMER_DRIVEN uses periods to execute your processor(s) at given intervals.
-The EVENT_DRIVEN strategy awaits for data be available or some other notification mechanism to trigger execution. CRON_DRIVEN executes at the desired intervals
-based on the CRON periods. Apache NiFi MiNiFi C++ supports standard CRON expressions without intervals ( */5 * * * * ).
+The EVENT_DRIVEN strategy awaits for data be available or some other notification mechanism to trigger execution. EVENT_DRIVEN is the recommended strategy for non-source processors.
+CRON_DRIVEN executes at the desired intervals based on the CRON periods. Apache NiFi MiNiFi C++ supports standard CRON expressions without intervals ( */5 * * * * ).
Review Comment:
I'm not sure what was this referring to, but as it is not clear I removed this limitation in 4e4434732dac8f91122fc910ea067c7c752283c6
##########
CONFIGURE.md:
##########
@@ -287,6 +341,62 @@ Another option to define a state storage is to use the following properties in t
nifi.state.storage.local.auto.persistence.interval=0 sec
+### Configuring Repositories
+
+Apache MiNiFi C++ uses three repositories similarily to Apache NiFi:
Review Comment:
Updated in 4e4434732dac8f91122fc910ea067c7c752283c6
##########
CONFIGURE.md:
##########
@@ -287,6 +341,62 @@ Another option to define a state storage is to use the following properties in t
nifi.state.storage.local.auto.persistence.interval=0 sec
+### Configuring Repositories
+
+Apache MiNiFi C++ uses three repositories similarily to Apache NiFi:
+- The Flow File Repository holds the metadata (like flow file attributes) of the flow Files.
Review Comment:
Updated in 4e4434732dac8f91122fc910ea067c7c752283c6
##########
CONFIGURE.md:
##########
@@ -287,6 +341,62 @@ Another option to define a state storage is to use the following properties in t
nifi.state.storage.local.auto.persistence.interval=0 sec
+### Configuring Repositories
+
+Apache MiNiFi C++ uses three repositories similarily to Apache NiFi:
+- The Flow File Repository holds the metadata (like flow file attributes) of the flow Files.
+- The Content Repository holds the content of the flow files.
+- The Provenance Repository holds the history of the flow files.
+
+The underlying implementation to use for these repositories can be configured in the minifi.properties file.
+
+The Flow File Repository can be configured with the `nifi.flowfile.repository.class.name` property. If not specified, it uses the `FlowFileRepository` class by default, which stores the flow file metadata in a RocksDB database. Alternatively it can be configured to use a `VolatileFlowFileRepository` that keeps the state in memory and flushes upon restart, or the `NoOpRepository` for not keeping any state.
Review Comment:
Updated in 4e4434732dac8f91122fc910ea067c7c752283c6
##########
CONFIGURE.md:
##########
@@ -287,6 +341,62 @@ Another option to define a state storage is to use the following properties in t
nifi.state.storage.local.auto.persistence.interval=0 sec
+### Configuring Repositories
+
+Apache MiNiFi C++ uses three repositories similarily to Apache NiFi:
+- The Flow File Repository holds the metadata (like flow file attributes) of the flow Files.
+- The Content Repository holds the content of the flow files.
+- The Provenance Repository holds the history of the flow files.
+
+The underlying implementation to use for these repositories can be configured in the minifi.properties file.
+
+The Flow File Repository can be configured with the `nifi.flowfile.repository.class.name` property. If not specified, it uses the `FlowFileRepository` class by default, which stores the flow file metadata in a RocksDB database. Alternatively it can be configured to use a `VolatileFlowFileRepository` that keeps the state in memory and flushes upon restart, or the `NoOpRepository` for not keeping any state.
+
+ # in minifi.properties
+ nifi.flowfile.repository.class.name=VolatileFlowFileRepository
+
+The Content Repository can be configured with the `nifi.content.repository.class.name` property. If not specified, it uses the `DatabaseContentRepository` class by default, which persists the content in a RocksDB database. `DatabaseContentRepository` is also the default value specified in the minifi.properties file. Alternatively it can be configured to use a `VolatileContentRepository` that keeps the state in memory and flushes upon restart, or the `FileSystemRepository` to keep the state in regular files.
+
+**NOTE:** RocksDB database has a limit of 4GB for the size of a database object. Due to this if you are expecting to process larger flow files than 4GB you should use the `FileSystemRepository`. The downside of using `FileSystemRepository` is that it does not have the transactional guarantees of the RocksDB repository implementation.
Review Comment:
Updated in 4e4434732dac8f91122fc910ea067c7c752283c6
##########
CONFIGURE.md:
##########
@@ -287,6 +341,62 @@ Another option to define a state storage is to use the following properties in t
nifi.state.storage.local.auto.persistence.interval=0 sec
+### Configuring Repositories
+
+Apache MiNiFi C++ uses three repositories similarily to Apache NiFi:
+- The Flow File Repository holds the metadata (like flow file attributes) of the flow Files.
+- The Content Repository holds the content of the flow files.
+- The Provenance Repository holds the history of the flow files.
+
+The underlying implementation to use for these repositories can be configured in the minifi.properties file.
+
+The Flow File Repository can be configured with the `nifi.flowfile.repository.class.name` property. If not specified, it uses the `FlowFileRepository` class by default, which stores the flow file metadata in a RocksDB database. Alternatively it can be configured to use a `VolatileFlowFileRepository` that keeps the state in memory and flushes upon restart, or the `NoOpRepository` for not keeping any state.
+
+ # in minifi.properties
+ nifi.flowfile.repository.class.name=VolatileFlowFileRepository
+
+The Content Repository can be configured with the `nifi.content.repository.class.name` property. If not specified, it uses the `DatabaseContentRepository` class by default, which persists the content in a RocksDB database. `DatabaseContentRepository` is also the default value specified in the minifi.properties file. Alternatively it can be configured to use a `VolatileContentRepository` that keeps the state in memory and flushes upon restart, or the `FileSystemRepository` to keep the state in regular files.
+
+**NOTE:** RocksDB database has a limit of 4GB for the size of a database object. Due to this if you are expecting to process larger flow files than 4GB you should use the `FileSystemRepository`. The downside of using `FileSystemRepository` is that it does not have the transactional guarantees of the RocksDB repository implementation.
+
+ # in minifi.properties
+ nifi.content.repository.class.name=FileSystemRepository
+
+The Provenance Repository can be configured with the `nifi.provenance.repository.class.name` property. If not specified, it uses the `ProvenanceRepository` class by default, which persists the provenance events in a RocksDB database. Alternatively it can be configured to use a `VolatileProvenanceRepository` that keeps the state in memory and flushes upon restart, or the `NoOpRepository` to not keep track of the provenance events. By default we do not keep track of the provenance data, so `NoOpRepository` is the value specified in the default minifi.properties file.
+
+ # in minifi.properties
+ nifi.flowfile.repository.class.name=NoOpRepository
+
+
+### Configuring Volatile Repositories
+As stated before each of the repositories can be configured to be volatile (state kept in memory and flushed upon restart) or persistent. Volatile repositories have some additional options, that can be specified in the following ways:
+
+ # in minifi.properties
+ # For Volatile Repositories:
+ nifi.flowfile.repository.class.name=VolatileFlowFileRepository
+ nifi.provenance.repository.class.name=VolatileProvenanceRepository
+ nifi.content.repository.class.name=VolatileContentRepository
+
+ # configuration options
+ # maximum number of entries to keep in memory
+ nifi.volatile.repository.options.flowfile.max.count=10000
+ # maximum number of bytes to keep in memory, also limited by option above
+ nifi.volatile.repository.options.flowfile.max.bytes=1M
+
+ # maximum number of entries to keep in memory
+ nifi.volatile.repository.options.provenance.max.count=10000
+ # maximum number of bytes to keep in memory, also limited by option above
+ nifi.volatile.repository.options.provenance.max.bytes=1M
+
+ # maximum number of entries to keep in memory
+ nifi.volatile.repository.options.content.max.count=100000
+ # maximum number of bytes to keep in memory, also limited by option above
+ nifi.volatile.repository.options.content.max.bytes=1M
+ # limits locking for the content repository
+ nifi.volatile.repository.options.content.minimal.locking=true
Review Comment:
I updated the values and I also found a bug in the code, where the class name of the volatile provenance repository was incorrect, so the example was not working in the production code. Fixed the issues in 4e4434732dac8f91122fc910ea067c7c752283c6
##########
CONFIGURE.md:
##########
@@ -560,3 +633,92 @@ To notify the agent which extensions it should load see [Loading extensions](Ext
### JNI Functionality
Please see the [JNI Configuration Guide](JNI.md).
+
+## Log configuration
+By default the application logs for Apache MiNiFi C++ can be found in the ${MINIFI_HOME}/logs/minifi-app.log file with default INFO level logging. The logger can be reconfigured in the ${MINIFI_HOME}/conf/minifi-log.properties file to use different output streams, log level, and output format.
+
+### Log appenders
+In the configuration file it is possible to define different output streams by defining what type of appenders (log sinks) should be used for logging. The following appenders are supported:
+- rollingappender - logs written to this log sink are written to a file with a rolling policy
+- stdout - logs written to this log sink are written to the standard output
+- stderr - logs written to this log sink are written to the standard error
+- syslog - logs written to this log sink are written to the syslog
+- alert - logs written to this log sink are critical logs forwarded through HTTP as alerts
+- nullappender - logs written to this log sink are discarded
+
+Defining a new appender can be done in the format `appender.<appender name>=<appender type>` followed by defining its properties. For example here is how to define a rolling file appender:
+
+ # in minifi-log.properties
+ appender.rolling=rollingappender
+ appender.rolling.directory=${MINIFI_HOME}/logs
+ appender.rolling.file_name=minifi-app.log
+ appender.rolling.max_files=3
+ appender.rolling.max_file_size=5 MB
+
+Here's an example of an alert appender with its available properties:
+
+ # in minifi-log.properties
+ appender.alert1=alert
+ appender.alert1.url=<URL>
+ appender.alert1.filter=<regex pattern to match logs against>
+ appender.alert1.rate.limit=10 min
+ appender.alert1.flush.period=5 s
+ appender.alert1.batch.size=100 KB
+ appender.alert1.buffer.limit=1 MB
+ appender.alert1.level=TRACE
+ appender.alert1.ssl.context.service=<Name of the SSLContextService>
+
+### Log levels
+After the appenders are defined you can set the log level and logging target for each of them. Appenders can be set to log everything on a specific log level, but you can also define some appenders to only write logs from specific namespaces or classes with different log levels. The log level can be set to one of the following values: OFF, TRACE, DEBUG, INFO, WARN, ERROR. The log levels can be set in the following way:
Review Comment:
Good point, added in 4e4434732dac8f91122fc910ea067c7c752283c6
##########
CONFIGURE.md:
##########
@@ -560,3 +633,92 @@ To notify the agent which extensions it should load see [Loading extensions](Ext
### JNI Functionality
Please see the [JNI Configuration Guide](JNI.md).
+
+## Log configuration
+By default the application logs for Apache MiNiFi C++ can be found in the ${MINIFI_HOME}/logs/minifi-app.log file with default INFO level logging. The logger can be reconfigured in the ${MINIFI_HOME}/conf/minifi-log.properties file to use different output streams, log level, and output format.
+
+### Log appenders
+In the configuration file it is possible to define different output streams by defining what type of appenders (log sinks) should be used for logging. The following appenders are supported:
+- rollingappender - logs written to this log sink are written to a file with a rolling policy
+- stdout - logs written to this log sink are written to the standard output
+- stderr - logs written to this log sink are written to the standard error
+- syslog - logs written to this log sink are written to the syslog
+- alert - logs written to this log sink are critical logs forwarded through HTTP as alerts
+- nullappender - logs written to this log sink are discarded
+
+Defining a new appender can be done in the format `appender.<appender name>=<appender type>` followed by defining its properties. For example here is how to define a rolling file appender:
+
+ # in minifi-log.properties
+ appender.rolling=rollingappender
+ appender.rolling.directory=${MINIFI_HOME}/logs
+ appender.rolling.file_name=minifi-app.log
+ appender.rolling.max_files=3
+ appender.rolling.max_file_size=5 MB
+
+Here's an example of an alert appender with its available properties:
+
+ # in minifi-log.properties
+ appender.alert1=alert
+ appender.alert1.url=<URL>
+ appender.alert1.filter=<regex pattern to match logs against>
+ appender.alert1.rate.limit=10 min
+ appender.alert1.flush.period=5 s
+ appender.alert1.batch.size=100 KB
+ appender.alert1.buffer.limit=1 MB
+ appender.alert1.level=TRACE
+ appender.alert1.ssl.context.service=<Name of the SSLContextService>
+
+### Log levels
+After the appenders are defined you can set the log level and logging target for each of them. Appenders can be set to log everything on a specific log level, but you can also define some appenders to only write logs from specific namespaces or classes with different log levels. The log level can be set to one of the following values: OFF, TRACE, DEBUG, INFO, WARN, ERROR. The log levels can be set in the following way:
+
+ # in minifi-log.properties
+ # Set the rolling appender to log everything with INFO level
+ logger.root=INFO,rolling
+ # But only write logs from the org::apache::nifi::minifi::FlowController class with ERROR level
+ logger.org::apache::nifi::minifi::core::logging::FlowController=ERROR,rolling
+
+ # Write all error logs in the minifi namespace to the alert1 appender
+ logger.org::apache::nifi::minifi=ERROR,alert1
+
+ # Logs from the FlowController class should be written to the stderr and rolling appenders with DEBUG level
+ logger.org::apache::nifi::minifi::FlowController=DEBUG,stderr,rolling
+
+ # Log warnings and errors from the LoggerConfiguration class to all appenders
+ logger.org::apache::nifi::minifi::core::logging::LoggerConfiguration=WARN
+
+### Log pattern
+The log pattern can be changed if needed using the `spdlog.pattern` property. The default log pattern is the following, but there are additional examples in the default minifi-log.properties file:
+
+ # in minifi-log.properties
+ spdlog.pattern=[%Y-%m-%d %H:%M:%S.%e] [%n] [%l] %v
Review Comment:
Added link in 4e4434732dac8f91122fc910ea067c7c752283c6
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINIFICPP-2284 Update documentation to reflect user feedback [nifi-minifi-cpp]
Posted by "fgerlits (via GitHub)" <gi...@apache.org>.
fgerlits commented on code in PR #1713:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1713#discussion_r1448847931
##########
CONFIGURE.md:
##########
@@ -78,8 +132,8 @@ MiNiFi supports YAML and JSON configuration formats. The desired configuration f
### Scheduling strategies
Currently Apache NiFi MiNiFi C++ supports TIMER_DRIVEN, EVENT_DRIVEN, and CRON_DRIVEN. TIMER_DRIVEN uses periods to execute your processor(s) at given intervals.
-The EVENT_DRIVEN strategy awaits for data be available or some other notification mechanism to trigger execution. CRON_DRIVEN executes at the desired intervals
-based on the CRON periods. Apache NiFi MiNiFi C++ supports standard CRON expressions without intervals ( */5 * * * * ).
+The EVENT_DRIVEN strategy awaits for data be available or some other notification mechanism to trigger execution. EVENT_DRIVEN is the recommended strategy for non-source processors.
+CRON_DRIVEN executes at the desired intervals based on the CRON periods. Apache NiFi MiNiFi C++ supports standard CRON expressions without intervals ( */5 * * * * ).
Review Comment:
I think the rest of the sentence was useful; I would only remove the "without intervals" part.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINIFICPP-2284 Update documentation to reflect user feedback [nifi-minifi-cpp]
Posted by "szaszm (via GitHub)" <gi...@apache.org>.
szaszm closed pull request #1713: MINIFICPP-2284 Update documentation to reflect user feedback
URL: https://github.com/apache/nifi-minifi-cpp/pull/1713
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINIFICPP-2284 Update documentation to reflect user feedback [nifi-minifi-cpp]
Posted by "szaszm (via GitHub)" <gi...@apache.org>.
szaszm commented on code in PR #1713:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1713#discussion_r1447296805
##########
CONFIGURE.md:
##########
@@ -560,3 +631,92 @@ To notify the agent which extensions it should load see [Loading extensions](Ext
### JNI Functionality
Please see the [JNI Configuration Guide](JNI.md).
+
+## Log configuration
+By default the application logs for Apache MiNiFi C++ can be found in the ${MINIFI_HOME}/logs/minifi-app.log file with default INFO level logging. The logger can be reconfigured in the ${MINIFI_HOME}/conf/minifi-log.properties file to use different output streams, log level, and output format.
+
+### Log appenders
+In the configuration file it is possible to define different output streams by defining what type of appenders (log sinks) should be used for logging. The following appenders are supported:
+- rollingappender - logs written to this log sink are written to a file with a rolling policy
+- stdout - logs written to this log sink are written to the standard output
+- stderr - logs written to this log sink are written to the standard error
+- syslog - logs written to this log sink are written to the syslog
+- alert - logs written to this log sink are critical logs forwarded through HTTP as alerts
+- nullappender - logs written to this log sink are discarded
+
+Defining a new appender can be done in the format `appender.<appender name>=<appender type>` followed by defining its properties. For example here is how to define a rolling file appender:
+
+ # in minifi-log.properties
+ appender.rolling=rollingappender
+ appender.rolling.directory=${MINIFI_HOME}/logs
+ appender.rolling.file_name=minifi-app.log
+ appender.rolling.max_files=3
+ appender.rolling.max_file_size=5 MB
+
+Here's an example of an alert appender with its available properties:
+
+ # in minifi-log.properties
+ appender.alert1=alert
+ appender.alert1.url=<URL>
+ appender.alert1.filter=<regex pattern to match logs against>
+ appender.alert1.rate.limit=10 min
+ appender.alert1.flush.period=5 s
+ appender.alert1.batch.size=100 KB
+ appender.alert1.buffer.limit=1 MB
+ appender.alert1.level=TRACE
+ appender.alert1.ssl.context.service=<Name of the SSLContextService>
+
+### Log levels
+After the appenders are defined you can set the log level and logging target for each of them. Appenders can be set to log everything on a specific log level, but you can also define some appenders to only write logs from specific namespaces or classes with different log levels. The log level can be set to one of the following values: OFF, TRACE, DEBUG, INFO, WARN, ERROR. The log levels can be set in the following way:
+
+ # in minifi-log.properties
+ # Set the rolling appender to log everything with INFO level
+ logger.root=INFO,rolling
+ # But only write logs from the org::apache::nifi::minifi::FlowController class with ERROR level
+ logger.org::apache::nifi::minifi::core::logging::LoggerConfiguration=ERROR,rolling
Review Comment:
That's a different class
##########
CONFIGURE.md:
##########
@@ -287,6 +341,60 @@ Another option to define a state storage is to use the following properties in t
nifi.state.storage.local.auto.persistence.interval=0 sec
+### Configuring Repositories
+
+Apache MiNiFi C++ uses three repositories similarily to Apache NiFi:
+- The Flow File Repository holds the metadata (like flow file attributes) of the flow Files.
+- The Content Repository holds the content of the flow files.
+- The Provenance Repository holds the history of the flow files.
+
+The underlying implementation to use for these repositories can be configured in the minifi.properties file.
+
+The Flow File Repository can be configured with the `nifi.flowfile.repository.class.name` property. If not specified, it uses the `FlowFileRepository` class by default, which stores the flow file metadata in a RocksDB database. Alternatively it can be configured to use a `VolatileFlowFileRepository` that keeps the state in memory and flushes upon restart, or the `NoOpRepository` for not keeping any state.
+
+ # in minifi.properties
+ nifi.flowfile.repository.class.name=VolatileFlowFileRepository
+
+The Content Repository can be configured with the `nifi.content.repository.class.name` property. If not specified, it uses the `DatabaseContentRepository` class by default, which persists the content in a RocksDB database. `DatabaseContentRepository` is also the default value specified in the minifi.properties file. Alternatively it can be configured to use a `VolatileContentRepository` that keeps the state in memory and flushes upon restart, or the `FileSystemRepository` to keep the state in regular files.
+
+**NOTE:** RocksDB database has a limit of 4GB for the size of a database object. Due to this if you are expecting to process larger flow files than 4GB you should use the `FileSystemRepository`. The downside of using `FileSystemRepository` is that it does not have the transactional guarantees of the RocksDB repository implementation.
+
+ # in minifi.properties
+ nifi.content.repository.class.name=FileSystemRepository
+
+The Provenance Repository can be configured with the `nifi.provenance.repository.class.name` property. If not specified, it uses the `ProvenanceRepository` class by default, which persists the provenance events in a RocksDB database. Alternatively it can be configured to use a `VolatileProvenanceRepository` that keeps the state in memory and flushes upon restart, or the `NoOpRepository` to not keep track of the provenance events. By default we do not keep track of the provenance data, so `NoOpRepository` is the value specified in the default minifi.properties file.
+
+ # in minifi.properties
+ nifi.flowfile.repository.class.name=NoOpRepository
+
+
+### Configuring Volatile Repositories
+As stated before each of the repositories can be configured to be volatile (state kept in memory and flushed upon restart) or persistent. Volatile repositories have some additional options, that can be specified in the following ways:
Review Comment:
If I remember correctly, volatile repos can behave in unexpected ways, when reaching their limits, probably breaking the configuration. Maybe @adamdebreceni remembers how exactly they break, from #1435 ?
If so, we should probably document this behavior.
##########
README.md:
##########
@@ -154,7 +154,7 @@ and rebuild.
* perl -- Required for OpenSSL configuration
* NASM -- Required for OpenSSL only on Windows
-**NOTE** On Windows if Strawberry Perl is used the `${StrawberryPerlRoot}\c\bin` directory should not be part of the %PATH% variable or make sure to find git's patch executable before that as Strawberry Perl's patch.exe will be found as the patch executable in the configure phase instead of the git patch executable. Also on Windows CMake's CPack is used for MSI generation, building WIX files and calling WIX toolset tools to create an MSI. If Chocolatey package manager is used its CPack can conflict with CMake, so make sure that CMake's CPack is found in the %PATH% before that.
+**NOTE:** On Windows if Strawberry Perl is used the `${StrawberryPerlRoot}\c\bin` directory should not be part of the PATH environment variable. Alternatively make sure that git's patch executable path comes before Strawberry Perl's patch.exe path in the PATH environment variable, so that the git patch executable is found in the in the CMake configuration phase. Also on Windows CMake's CPack is used for MSI generation, building WIX files and calling WIX toolset tools to create an MSI. If Chocolatey package manager is used its CPack can conflict with CMake, so make sure that CMake's CPack is found in the %PATH% before that.
Review Comment:
I found it not to be enough to move the Strawberry Perl c/bin in the path, I had to actually rename or remove the patch.exe on my VM to not cause any issues.
I think we should recommend the scoop package of perl: it's strawberry perl internally, but doesn't pollute the path with its toolchain, and doesn't cause these issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINIFICPP-2284 Update documentation to reflect user feedback [nifi-minifi-cpp]
Posted by "lordgamez (via GitHub)" <gi...@apache.org>.
lordgamez commented on code in PR #1713:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1713#discussion_r1448882409
##########
CONFIGURE.md:
##########
@@ -78,8 +132,8 @@ MiNiFi supports YAML and JSON configuration formats. The desired configuration f
### Scheduling strategies
Currently Apache NiFi MiNiFi C++ supports TIMER_DRIVEN, EVENT_DRIVEN, and CRON_DRIVEN. TIMER_DRIVEN uses periods to execute your processor(s) at given intervals.
-The EVENT_DRIVEN strategy awaits for data be available or some other notification mechanism to trigger execution. CRON_DRIVEN executes at the desired intervals
-based on the CRON periods. Apache NiFi MiNiFi C++ supports standard CRON expressions without intervals ( */5 * * * * ).
+The EVENT_DRIVEN strategy awaits for data be available or some other notification mechanism to trigger execution. EVENT_DRIVEN is the recommended strategy for non-source processors.
+CRON_DRIVEN executes at the desired intervals based on the CRON periods. Apache NiFi MiNiFi C++ supports standard CRON expressions without intervals ( */5 * * * * ).
Review Comment:
Readded in 8333c11812fda11ca06c6fe99e902b1fd58e50a7
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINIFICPP-2284 Update documentation to reflect user feedback [nifi-minifi-cpp]
Posted by "lordgamez (via GitHub)" <gi...@apache.org>.
lordgamez commented on code in PR #1713:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1713#discussion_r1447343003
##########
CONFIGURE.md:
##########
@@ -560,3 +631,92 @@ To notify the agent which extensions it should load see [Loading extensions](Ext
### JNI Functionality
Please see the [JNI Configuration Guide](JNI.md).
+
+## Log configuration
+By default the application logs for Apache MiNiFi C++ can be found in the ${MINIFI_HOME}/logs/minifi-app.log file with default INFO level logging. The logger can be reconfigured in the ${MINIFI_HOME}/conf/minifi-log.properties file to use different output streams, log level, and output format.
+
+### Log appenders
+In the configuration file it is possible to define different output streams by defining what type of appenders (log sinks) should be used for logging. The following appenders are supported:
+- rollingappender - logs written to this log sink are written to a file with a rolling policy
+- stdout - logs written to this log sink are written to the standard output
+- stderr - logs written to this log sink are written to the standard error
+- syslog - logs written to this log sink are written to the syslog
+- alert - logs written to this log sink are critical logs forwarded through HTTP as alerts
+- nullappender - logs written to this log sink are discarded
+
+Defining a new appender can be done in the format `appender.<appender name>=<appender type>` followed by defining its properties. For example here is how to define a rolling file appender:
+
+ # in minifi-log.properties
+ appender.rolling=rollingappender
+ appender.rolling.directory=${MINIFI_HOME}/logs
+ appender.rolling.file_name=minifi-app.log
+ appender.rolling.max_files=3
+ appender.rolling.max_file_size=5 MB
+
+Here's an example of an alert appender with its available properties:
+
+ # in minifi-log.properties
+ appender.alert1=alert
+ appender.alert1.url=<URL>
+ appender.alert1.filter=<regex pattern to match logs against>
+ appender.alert1.rate.limit=10 min
+ appender.alert1.flush.period=5 s
+ appender.alert1.batch.size=100 KB
+ appender.alert1.buffer.limit=1 MB
+ appender.alert1.level=TRACE
+ appender.alert1.ssl.context.service=<Name of the SSLContextService>
+
+### Log levels
+After the appenders are defined you can set the log level and logging target for each of them. Appenders can be set to log everything on a specific log level, but you can also define some appenders to only write logs from specific namespaces or classes with different log levels. The log level can be set to one of the following values: OFF, TRACE, DEBUG, INFO, WARN, ERROR. The log levels can be set in the following way:
+
+ # in minifi-log.properties
+ # Set the rolling appender to log everything with INFO level
+ logger.root=INFO,rolling
+ # But only write logs from the org::apache::nifi::minifi::FlowController class with ERROR level
+ logger.org::apache::nifi::minifi::core::logging::LoggerConfiguration=ERROR,rolling
Review Comment:
Good catch, updated in 999a2039a9457abf91dae7af2f75a5910d7ce86f
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINIFICPP-2284 Update documentation to reflect user feedback [nifi-minifi-cpp]
Posted by "lordgamez (via GitHub)" <gi...@apache.org>.
lordgamez commented on code in PR #1713:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1713#discussion_r1447344042
##########
README.md:
##########
@@ -154,7 +154,7 @@ and rebuild.
* perl -- Required for OpenSSL configuration
* NASM -- Required for OpenSSL only on Windows
-**NOTE** On Windows if Strawberry Perl is used the `${StrawberryPerlRoot}\c\bin` directory should not be part of the %PATH% variable or make sure to find git's patch executable before that as Strawberry Perl's patch.exe will be found as the patch executable in the configure phase instead of the git patch executable. Also on Windows CMake's CPack is used for MSI generation, building WIX files and calling WIX toolset tools to create an MSI. If Chocolatey package manager is used its CPack can conflict with CMake, so make sure that CMake's CPack is found in the %PATH% before that.
+**NOTE:** On Windows if Strawberry Perl is used the `${StrawberryPerlRoot}\c\bin` directory should not be part of the PATH environment variable. Alternatively make sure that git's patch executable path comes before Strawberry Perl's patch.exe path in the PATH environment variable, so that the git patch executable is found in the in the CMake configuration phase. Also on Windows CMake's CPack is used for MSI generation, building WIX files and calling WIX toolset tools to create an MSI. If Chocolatey package manager is used its CPack can conflict with CMake, so make sure that CMake's CPack is found in the %PATH% before that.
Review Comment:
Updated the doc with the scoop alternative and removed the info about moving the path in 999a2039a9457abf91dae7af2f75a5910d7ce86f
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINIFICPP-2284 Update documentation to reflect user feedback [nifi-minifi-cpp]
Posted by "szaszm (via GitHub)" <gi...@apache.org>.
szaszm commented on code in PR #1713:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1713#discussion_r1447508845
##########
README.md:
##########
@@ -154,7 +154,7 @@ and rebuild.
* perl -- Required for OpenSSL configuration
* NASM -- Required for OpenSSL only on Windows
-**NOTE** On Windows if Strawberry Perl is used the `${StrawberryPerlRoot}\c\bin` directory should not be part of the %PATH% variable or make sure to find git's patch executable before that as Strawberry Perl's patch.exe will be found as the patch executable in the configure phase instead of the git patch executable. Also on Windows CMake's CPack is used for MSI generation, building WIX files and calling WIX toolset tools to create an MSI. If Chocolatey package manager is used its CPack can conflict with CMake, so make sure that CMake's CPack is found in the %PATH% before that.
+**NOTE:** On Windows if Strawberry Perl is used the `${StrawberryPerlRoot}\c\bin` directory should not be part of the PATH environment variable. Alternatively make sure that git's patch executable path comes before Strawberry Perl's patch.exe path in the PATH environment variable, so that the git patch executable is found in the in the CMake configuration phase. Also on Windows CMake's CPack is used for MSI generation, building WIX files and calling WIX toolset tools to create an MSI. If Chocolatey package manager is used its CPack can conflict with CMake, so make sure that CMake's CPack is found in the %PATH% before that.
Review Comment:
Confirmed that moving is not enough, removing the path entry either on the CLI or GUI solves the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINIFICPP-2284 Update documentation to reflect user feedback [nifi-minifi-cpp]
Posted by "fgerlits (via GitHub)" <gi...@apache.org>.
fgerlits commented on code in PR #1713:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1713#discussion_r1448676610
##########
CONFIGURE.md:
##########
@@ -287,6 +341,62 @@ Another option to define a state storage is to use the following properties in t
nifi.state.storage.local.auto.persistence.interval=0 sec
+### Configuring Repositories
+
+Apache MiNiFi C++ uses three repositories similarily to Apache NiFi:
Review Comment:
typo:
```suggestion
Apache MiNiFi C++ uses three repositories similarly to Apache NiFi:
```
##########
CONFIGURE.md:
##########
@@ -287,6 +341,62 @@ Another option to define a state storage is to use the following properties in t
nifi.state.storage.local.auto.persistence.interval=0 sec
+### Configuring Repositories
+
+Apache MiNiFi C++ uses three repositories similarily to Apache NiFi:
+- The Flow File Repository holds the metadata (like flow file attributes) of the flow Files.
+- The Content Repository holds the content of the flow files.
+- The Provenance Repository holds the history of the flow files.
+
+The underlying implementation to use for these repositories can be configured in the minifi.properties file.
+
+The Flow File Repository can be configured with the `nifi.flowfile.repository.class.name` property. If not specified, it uses the `FlowFileRepository` class by default, which stores the flow file metadata in a RocksDB database. Alternatively it can be configured to use a `VolatileFlowFileRepository` that keeps the state in memory and flushes upon restart, or the `NoOpRepository` for not keeping any state.
Review Comment:
I think "so the state gets lost" would be clearer than "and flushes" ("flushes" suggests writing it out somewhere)
##########
CONFIGURE.md:
##########
@@ -78,8 +132,8 @@ MiNiFi supports YAML and JSON configuration formats. The desired configuration f
### Scheduling strategies
Currently Apache NiFi MiNiFi C++ supports TIMER_DRIVEN, EVENT_DRIVEN, and CRON_DRIVEN. TIMER_DRIVEN uses periods to execute your processor(s) at given intervals.
-The EVENT_DRIVEN strategy awaits for data be available or some other notification mechanism to trigger execution. CRON_DRIVEN executes at the desired intervals
-based on the CRON periods. Apache NiFi MiNiFi C++ supports standard CRON expressions without intervals ( */5 * * * * ).
+The EVENT_DRIVEN strategy awaits for data be available or some other notification mechanism to trigger execution. EVENT_DRIVEN is the recommended strategy for non-source processors.
+CRON_DRIVEN executes at the desired intervals based on the CRON periods. Apache NiFi MiNiFi C++ supports standard CRON expressions without intervals ( */5 * * * * ).
Review Comment:
What does "without intervals" refer to? I would guess it means "no ranges", but [the Cron unit tests](https://github.com/apache/nifi-minifi-cpp/blob/main/libminifi/test/unit/CronTests.cpp) show support for `"0 0-5 14 * * ?"`, `"0 15 10 ? * MON-FRI"` and `"0 15 10 ? * 6L 2002-2005"`. Maybe this limitation is out of date?
##########
CONFIGURE.md:
##########
@@ -287,6 +341,62 @@ Another option to define a state storage is to use the following properties in t
nifi.state.storage.local.auto.persistence.interval=0 sec
+### Configuring Repositories
+
+Apache MiNiFi C++ uses three repositories similarily to Apache NiFi:
+- The Flow File Repository holds the metadata (like flow file attributes) of the flow Files.
+- The Content Repository holds the content of the flow files.
+- The Provenance Repository holds the history of the flow files.
+
+The underlying implementation to use for these repositories can be configured in the minifi.properties file.
+
+The Flow File Repository can be configured with the `nifi.flowfile.repository.class.name` property. If not specified, it uses the `FlowFileRepository` class by default, which stores the flow file metadata in a RocksDB database. Alternatively it can be configured to use a `VolatileFlowFileRepository` that keeps the state in memory and flushes upon restart, or the `NoOpRepository` for not keeping any state.
+
+ # in minifi.properties
+ nifi.flowfile.repository.class.name=VolatileFlowFileRepository
+
+The Content Repository can be configured with the `nifi.content.repository.class.name` property. If not specified, it uses the `DatabaseContentRepository` class by default, which persists the content in a RocksDB database. `DatabaseContentRepository` is also the default value specified in the minifi.properties file. Alternatively it can be configured to use a `VolatileContentRepository` that keeps the state in memory and flushes upon restart, or the `FileSystemRepository` to keep the state in regular files.
+
+**NOTE:** RocksDB database has a limit of 4GB for the size of a database object. Due to this if you are expecting to process larger flow files than 4GB you should use the `FileSystemRepository`. The downside of using `FileSystemRepository` is that it does not have the transactional guarantees of the RocksDB repository implementation.
Review Comment:
minor, but I think "expect" would sound better:
```suggestion
**NOTE:** RocksDB database has a limit of 4GB for the size of a database object. Due to this if you expect to process larger flow files than 4GB you should use the `FileSystemRepository`. The downside of using `FileSystemRepository` is that it does not have the transactional guarantees of the RocksDB repository implementation.
```
##########
CONFIGURE.md:
##########
@@ -287,6 +341,62 @@ Another option to define a state storage is to use the following properties in t
nifi.state.storage.local.auto.persistence.interval=0 sec
+### Configuring Repositories
+
+Apache MiNiFi C++ uses three repositories similarily to Apache NiFi:
+- The Flow File Repository holds the metadata (like flow file attributes) of the flow Files.
Review Comment:
the capitalization looks weird:
```suggestion
- The Flow File Repository holds the metadata (like flow file attributes) of the flow files.
```
##########
CONFIGURE.md:
##########
@@ -287,6 +341,62 @@ Another option to define a state storage is to use the following properties in t
nifi.state.storage.local.auto.persistence.interval=0 sec
+### Configuring Repositories
+
+Apache MiNiFi C++ uses three repositories similarily to Apache NiFi:
+- The Flow File Repository holds the metadata (like flow file attributes) of the flow Files.
+- The Content Repository holds the content of the flow files.
+- The Provenance Repository holds the history of the flow files.
+
+The underlying implementation to use for these repositories can be configured in the minifi.properties file.
+
+The Flow File Repository can be configured with the `nifi.flowfile.repository.class.name` property. If not specified, it uses the `FlowFileRepository` class by default, which stores the flow file metadata in a RocksDB database. Alternatively it can be configured to use a `VolatileFlowFileRepository` that keeps the state in memory and flushes upon restart, or the `NoOpRepository` for not keeping any state.
+
+ # in minifi.properties
+ nifi.flowfile.repository.class.name=VolatileFlowFileRepository
+
+The Content Repository can be configured with the `nifi.content.repository.class.name` property. If not specified, it uses the `DatabaseContentRepository` class by default, which persists the content in a RocksDB database. `DatabaseContentRepository` is also the default value specified in the minifi.properties file. Alternatively it can be configured to use a `VolatileContentRepository` that keeps the state in memory and flushes upon restart, or the `FileSystemRepository` to keep the state in regular files.
+
+**NOTE:** RocksDB database has a limit of 4GB for the size of a database object. Due to this if you are expecting to process larger flow files than 4GB you should use the `FileSystemRepository`. The downside of using `FileSystemRepository` is that it does not have the transactional guarantees of the RocksDB repository implementation.
+
+ # in minifi.properties
+ nifi.content.repository.class.name=FileSystemRepository
+
+The Provenance Repository can be configured with the `nifi.provenance.repository.class.name` property. If not specified, it uses the `ProvenanceRepository` class by default, which persists the provenance events in a RocksDB database. Alternatively it can be configured to use a `VolatileProvenanceRepository` that keeps the state in memory and flushes upon restart, or the `NoOpRepository` to not keep track of the provenance events. By default we do not keep track of the provenance data, so `NoOpRepository` is the value specified in the default minifi.properties file.
+
+ # in minifi.properties
+ nifi.flowfile.repository.class.name=NoOpRepository
+
+
+### Configuring Volatile Repositories
+As stated before each of the repositories can be configured to be volatile (state kept in memory and flushed upon restart) or persistent. Volatile repositories have some additional options, that can be specified in the following ways:
+
+ # in minifi.properties
+ # For Volatile Repositories:
+ nifi.flowfile.repository.class.name=VolatileFlowFileRepository
+ nifi.provenance.repository.class.name=VolatileProvenanceRepository
+ nifi.content.repository.class.name=VolatileContentRepository
+
+ # configuration options
+ # maximum number of entries to keep in memory
+ nifi.volatile.repository.options.flowfile.max.count=10000
+ # maximum number of bytes to keep in memory, also limited by option above
+ nifi.volatile.repository.options.flowfile.max.bytes=1M
+
+ # maximum number of entries to keep in memory
+ nifi.volatile.repository.options.provenance.max.count=10000
+ # maximum number of bytes to keep in memory, also limited by option above
+ nifi.volatile.repository.options.provenance.max.bytes=1M
+
+ # maximum number of entries to keep in memory
+ nifi.volatile.repository.options.content.max.count=100000
+ # maximum number of bytes to keep in memory, also limited by option above
+ nifi.volatile.repository.options.content.max.bytes=1M
+ # limits locking for the content repository
+ nifi.volatile.repository.options.content.minimal.locking=true
Review Comment:
I suspect the values in this sample section used to be the default values, but they aren't any longer. Can you change them to the current default values, please?
##########
CONFIGURE.md:
##########
@@ -560,3 +633,92 @@ To notify the agent which extensions it should load see [Loading extensions](Ext
### JNI Functionality
Please see the [JNI Configuration Guide](JNI.md).
+
+## Log configuration
+By default the application logs for Apache MiNiFi C++ can be found in the ${MINIFI_HOME}/logs/minifi-app.log file with default INFO level logging. The logger can be reconfigured in the ${MINIFI_HOME}/conf/minifi-log.properties file to use different output streams, log level, and output format.
+
+### Log appenders
+In the configuration file it is possible to define different output streams by defining what type of appenders (log sinks) should be used for logging. The following appenders are supported:
+- rollingappender - logs written to this log sink are written to a file with a rolling policy
+- stdout - logs written to this log sink are written to the standard output
+- stderr - logs written to this log sink are written to the standard error
+- syslog - logs written to this log sink are written to the syslog
+- alert - logs written to this log sink are critical logs forwarded through HTTP as alerts
+- nullappender - logs written to this log sink are discarded
+
+Defining a new appender can be done in the format `appender.<appender name>=<appender type>` followed by defining its properties. For example here is how to define a rolling file appender:
+
+ # in minifi-log.properties
+ appender.rolling=rollingappender
+ appender.rolling.directory=${MINIFI_HOME}/logs
+ appender.rolling.file_name=minifi-app.log
+ appender.rolling.max_files=3
+ appender.rolling.max_file_size=5 MB
+
+Here's an example of an alert appender with its available properties:
+
+ # in minifi-log.properties
+ appender.alert1=alert
+ appender.alert1.url=<URL>
+ appender.alert1.filter=<regex pattern to match logs against>
+ appender.alert1.rate.limit=10 min
+ appender.alert1.flush.period=5 s
+ appender.alert1.batch.size=100 KB
+ appender.alert1.buffer.limit=1 MB
+ appender.alert1.level=TRACE
+ appender.alert1.ssl.context.service=<Name of the SSLContextService>
+
+### Log levels
+After the appenders are defined you can set the log level and logging target for each of them. Appenders can be set to log everything on a specific log level, but you can also define some appenders to only write logs from specific namespaces or classes with different log levels. The log level can be set to one of the following values: OFF, TRACE, DEBUG, INFO, WARN, ERROR. The log levels can be set in the following way:
Review Comment:
we have CRITICAL now, too
##########
CONFIGURE.md:
##########
@@ -560,3 +633,92 @@ To notify the agent which extensions it should load see [Loading extensions](Ext
### JNI Functionality
Please see the [JNI Configuration Guide](JNI.md).
+
+## Log configuration
+By default the application logs for Apache MiNiFi C++ can be found in the ${MINIFI_HOME}/logs/minifi-app.log file with default INFO level logging. The logger can be reconfigured in the ${MINIFI_HOME}/conf/minifi-log.properties file to use different output streams, log level, and output format.
+
+### Log appenders
+In the configuration file it is possible to define different output streams by defining what type of appenders (log sinks) should be used for logging. The following appenders are supported:
+- rollingappender - logs written to this log sink are written to a file with a rolling policy
+- stdout - logs written to this log sink are written to the standard output
+- stderr - logs written to this log sink are written to the standard error
+- syslog - logs written to this log sink are written to the syslog
+- alert - logs written to this log sink are critical logs forwarded through HTTP as alerts
+- nullappender - logs written to this log sink are discarded
+
+Defining a new appender can be done in the format `appender.<appender name>=<appender type>` followed by defining its properties. For example here is how to define a rolling file appender:
+
+ # in minifi-log.properties
+ appender.rolling=rollingappender
+ appender.rolling.directory=${MINIFI_HOME}/logs
+ appender.rolling.file_name=minifi-app.log
+ appender.rolling.max_files=3
+ appender.rolling.max_file_size=5 MB
+
+Here's an example of an alert appender with its available properties:
+
+ # in minifi-log.properties
+ appender.alert1=alert
+ appender.alert1.url=<URL>
+ appender.alert1.filter=<regex pattern to match logs against>
+ appender.alert1.rate.limit=10 min
+ appender.alert1.flush.period=5 s
+ appender.alert1.batch.size=100 KB
+ appender.alert1.buffer.limit=1 MB
+ appender.alert1.level=TRACE
+ appender.alert1.ssl.context.service=<Name of the SSLContextService>
+
+### Log levels
+After the appenders are defined you can set the log level and logging target for each of them. Appenders can be set to log everything on a specific log level, but you can also define some appenders to only write logs from specific namespaces or classes with different log levels. The log level can be set to one of the following values: OFF, TRACE, DEBUG, INFO, WARN, ERROR. The log levels can be set in the following way:
+
+ # in minifi-log.properties
+ # Set the rolling appender to log everything with INFO level
+ logger.root=INFO,rolling
+ # But only write logs from the org::apache::nifi::minifi::FlowController class with ERROR level
+ logger.org::apache::nifi::minifi::core::logging::FlowController=ERROR,rolling
+
+ # Write all error logs in the minifi namespace to the alert1 appender
+ logger.org::apache::nifi::minifi=ERROR,alert1
+
+ # Logs from the FlowController class should be written to the stderr and rolling appenders with DEBUG level
+ logger.org::apache::nifi::minifi::FlowController=DEBUG,stderr,rolling
+
+ # Log warnings and errors from the LoggerConfiguration class to all appenders
+ logger.org::apache::nifi::minifi::core::logging::LoggerConfiguration=WARN
+
+### Log pattern
+The log pattern can be changed if needed using the `spdlog.pattern` property. The default log pattern is the following, but there are additional examples in the default minifi-log.properties file:
+
+ # in minifi-log.properties
+ spdlog.pattern=[%Y-%m-%d %H:%M:%S.%e] [%n] [%l] %v
Review Comment:
we could link to https://github.com/gabime/spdlog/wiki/3.-Custom-formatting#customizing-format-using-set_pattern, as the meaning of some of the pattern placeholders is not obvious
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org