You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by GitBox <gi...@apache.org> on 2022/12/23 11:54:21 UTC
[GitHub] [nifi-minifi-cpp] fgerlits opened a new pull request, #1483: MINIFICPP-2019 Fix auto-generation of PROCESSORS.md
fgerlits opened a new pull request, #1483:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1483
https://issues.apache.org/jira/browse/MINIFICPP-2019
The `PROCESSORS.md` file included in this PR is almost auto-generated (by merging the output of `minifi docs ...` on Linux and on Windows), except for a few sections mentioned in https://issues.apache.org/jira/browse/MINIFICPP-2021.
`CollectKubernetesPodMetrics`, `ExecuteJavaProcessor` and `SourceInitiatedSubscriptionListener` were missing from `PROCESSORS.md`; I have added the auto-generated docs of these.
Also, the ordering of the processors was inconsistent; now they are ordered in case-insensitive lexicographic order.
---
Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [x] Does your PR title start with MINIFICPP-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target branch (typically main)?
- [ ] Is your initial contribution a single, squashed commit?
### For code changes:
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file?
- [ ] If applicable, have you updated the NOTICE file?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a diff in pull request #1483: MINIFICPP-2019 Fix auto-generation of PROCESSORS.md
Posted by "lordgamez (via GitHub)" <gi...@apache.org>.
lordgamez commented on code in PR #1483:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1483#discussion_r1098616074
##########
extensions/splunk/PutSplunkHTTP.h:
##########
@@ -40,7 +40,23 @@ class PutSplunkHTTP final : public SplunkHECProcessor {
PutSplunkHTTP& operator=(PutSplunkHTTP&&) = delete;
~PutSplunkHTTP() override = default;
- EXTENSIONAPI static constexpr const char* Description = "Sends the flow file contents to the specified Splunk HTTP Event Collector over HTTP or HTTPS. Supports HEC Index Acknowledgement.";
+ EXTENSIONAPI static constexpr const char* Description =
+ "Sends the flow file contents to the specified Splunk HTTP Event Collector (see https://docs.splunk.com/Documentation/SplunkCloud/latest/Data/UsetheHTTPEventCollector) over HTTP or HTTPS.\n"
+ "\n"
+ "The \"Source\", \"Source Type\", \"Host\" and \"Index\" properties are optional and will be set by Splunk if unspecified. If set,\n"
Review Comment:
Are the `\n` endline characters necessary to wrap the descriptions? Isn't the wrapping done automatically in the markdown files?
##########
extensions/aws/processors/PutS3Object.h:
##########
@@ -44,7 +44,10 @@ class PutS3Object : public S3Processor {
static const std::set<std::string> STORAGE_CLASSES;
static const std::set<std::string> SERVER_SIDE_ENCRYPTIONS;
- EXTENSIONAPI static constexpr const char* Description = "This Processor puts FlowFiles to an Amazon S3 Bucket.";
+ EXTENSIONAPI static constexpr const char* Description = "Puts FlowFiles to an Amazon S3 Bucket. The upload uses either the PutS3Object method. "
Review Comment:
```suggestion
EXTENSIONAPI static constexpr const char* Description = "Puts FlowFiles to an Amazon S3 Bucket. The upload uses the PutS3Object method. "
```
##########
minifi_main/TableFormatter.h:
##########
@@ -0,0 +1,47 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "core/PropertyValue.h"
+
+namespace org::apache::nifi::minifi::docs {
+
+class Table {
+ public:
+ explicit Table(std::vector<std::string> header) : header_{std::move(header)} {}
+ void addRow(std::vector<std::string> row);
+ [[nodiscard]] std::string toString() const;
+
+ private:
+ [[nodiscard]] std::vector<size_t> find_widths() const;
Review Comment:
This should be camelCase
##########
minifi_main/TableFormatter.h:
##########
@@ -0,0 +1,47 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "core/PropertyValue.h"
+
+namespace org::apache::nifi::minifi::docs {
+
+class Table {
+ public:
+ explicit Table(std::vector<std::string> header) : header_{std::move(header)} {}
+ void addRow(std::vector<std::string> row);
+ [[nodiscard]] std::string toString() const;
+
+ private:
+ [[nodiscard]] std::vector<size_t> find_widths() const;
+
+ std::vector<std::string> header_;
+ std::vector<std::vector<std::string>> rows_;
+};
+
+std::string formatName(const std::string& name, bool is_required);
+std::string formatAllowableValues(const std::vector<org::apache::nifi::minifi::core::PropertyValue>& values);
+std::string formatDescription(std::string description, bool supports_expression_language = false);
+std::string formatSeparator(const std::vector<size_t>& widths);
+std::string formatRow(const std::vector<std::string>& items, const std::vector<size_t>& widths);
Review Comment:
This might benefit from a few unit tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a diff in pull request #1483: MINIFICPP-2019 Fix auto-generation of PROCESSORS.md
Posted by "fgerlits (via GitHub)" <gi...@apache.org>.
fgerlits commented on code in PR #1483:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1483#discussion_r1135686012
##########
PROCESSORS.md:
##########
@@ -502,16 +574,15 @@ Deletes an object from a Google Cloud Bucket.
In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
-| Name | Default Value | Allowable Values | Description |
-|--------------------------------------|---------------|-----------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------|
-| **Bucket** | ${gcs.bucket} | | Bucket of the object.<br>**Supports Expression Language: true** |
-| **Key** | ${filename} | | Name of the object.<br>**Supports Expression Language: true** |
-| **Number of retries** | 6 | integers | How many retry attempts should be made before routing to the failure relationship. |
-| **GCP Credentials Provider Service** | | [GCPCredentialsControllerService](CONTROLLERS.md#GCPCredentialsControllerService) | The Controller Service used to obtain Google Cloud Platform credentials. |
-| Server Side Encryption Key | | | An AES256 Encryption Key (encoded in base64) for server-side encryption of the object.<br>**Supports Expression Language: true** |
-| Object Generation | | | The generation of the Object to download. If left empty, then it will download the latest generation.<br>**Supports Expression Language: true** |
-| Endpoint Override URL | | | Overrides the default Google Cloud Storage endpoints |
-
+| Name | Default Value | Allowable Values | Description |
+|--------------------------------------|---------------|------------------|--------------------------------------------------------------------------------------------------------------------------------------------------|
+| **GCP Credentials Provider Service** | | | The Controller Service used to obtain Google Cloud Platform credentials. |
Review Comment:
I don't want to include a markup link, but I have added some more info in b2d6f83319296b9d1849cd6d7c835050e088c017.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a diff in pull request #1483: MINIFICPP-2019 Fix auto-generation of PROCESSORS.md
Posted by "fgerlits (via GitHub)" <gi...@apache.org>.
fgerlits commented on code in PR #1483:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1483#discussion_r1135687938
##########
PROCESSORS.md:
##########
@@ -589,91 +688,97 @@ In the list below, the names of required properties appear in bold. Any other pr
### Description
-Executes a script given the flow file and a process session. The script is responsible for handling the incoming flow file (transfer to SUCCESS or remove, e.g.) as well as any flow files created by the script. If the handling is incomplete or incorrect, the session will be rolled back. Scripts must define an onTrigger function which accepts NiFi Context and Property objects. For efficiency, scripts are executed once when the processor is run, then the onTrigger method is called for each incoming flowfile. This enables scripts to keep state if they wish, although there will be a script context per concurrent task of the processor. In order to, e.g., compute an arithmetic sum based on incoming flow file information, set the concurrent tasks to 1. The python script files are expected to contain `describe(procesor)` and `onTrigger(context, session)`.
+Executes a script given the flow file and a process session. The script is responsible for handling the incoming flow file (transfer to SUCCESS or remove, e.g.) as well as any flow files created by the script. If the handling is incomplete or incorrect, the session will be rolled back.Scripts must define an onTrigger function which accepts NiFi Context and Property objects. For efficiency, scripts are executed once when the processor is run, then the onTrigger method is called for each incoming flowfile. This enables scripts to keep state if they wish, although there will be a script context per concurrent task of the processor. In order to, e.g., compute an arithmetic sum based on incoming flow file information, set the concurrent tasks to 1.
### Properties
In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
| Name | Default Value | Allowable Values | Description |
|-----------------------------|---------------|------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| Script File | | | Path to script file to execute. Only one of Script File or Script Body may be used |
+| Script Body | | | Script to execute. Only one of Script File or Script Body may be used |
| Module Directory | | | Comma-separated list of paths to files and/or directories which contain modules required by the script |
| **Reload on Script Change** | true | | If true and Script File property is used, then script file will be reloaded if it has changed, otherwise the first loaded version will be used at all times. |
-| Script Body | | | Script to execute |
-| Script File | | | Path to script file to execute. Only one of Script File or Script Body may be used |
+
### Relationships
-| Name | Description |
-|---------|------------------|
-| failure | Script failures |
-| success | Script successes |
+| Name | Description |
+|---------|-----------------|
+| success | Script succeeds |
+| failure | Script fails |
-## ExecuteSQL
+## ExecuteScript
### Description
-Execute provided SQL query. Query result rows will be outputted as new flow files with attribute keys equal to result column names and values equal to result values. There will be one output FlowFile per result row. This processor can be scheduled to run using the standard timer-based scheduling methods, or it can be triggered by an incoming FlowFile. If it is triggered by an incoming FlowFile, then attributes of that FlowFile will be available when evaluating the query.
+Executes a script given the flow file and a process session. The script is responsible for handling the incoming flow file (transfer to SUCCESS or remove, e.g.) as well as any flow files created by the script. If the handling is incomplete or incorrect, the session will be rolled back.Scripts must define an onTrigger function which accepts NiFi Context and Property objects. For efficiency, scripts are executed once when the processor is run, then the onTrigger method is called for each incoming flowfile. This enables scripts to keep state if they wish, although there will be a script context per concurrent task of the processor. In order to, e.g., compute an arithmetic sum based on incoming flow file information, set the concurrent tasks to 1.
+
### Properties
In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
-| Name | Default Value | Allowable Values | Description |
-|----------------------------|---------------|----------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| **DB Controller Service** | | | Database Controller Service. |
-| **Max Rows Per Flow File** | 0 | | The maximum number of result rows that will be included in a single FlowFile. This will allow you to break up very large result sets into multiple FlowFiles. If the value specified is zero, then all rows are returned in a single FlowFile. |
-| **Output Format** | JSON-Pretty | JSON<br/>JSON-Pretty | Set the output format type. |
-| SQL select query | | | The SQL select query to execute. The query can be empty, a constant value, or built from attributes using Expression Language. If this property is specified, it will be used regardless of the content of incoming flowfiles. If this property is empty, the content of the incoming flow file is expected to contain a valid SQL select query, to be issued by the processor to the database. |
+| Name | Default Value | Allowable Values | Description |
+|-------------------|---------------|------------------|--------------------------------------------------------------------------------------------------------|
+| **Script Engine** | python | lua<br/>python | The engine to execute scripts (python, lua) |
+| Script File | | | Path to script file to execute. Only one of Script File or Script Body may be used |
+| Script Body | | | Body of script to execute. Only one of Script File or Script Body may be used |
+| Module Directory | | | Comma-separated list of paths to files and/or directories which contain modules required by the script |
+
### Relationships
-| Name | Description |
-|---------|----------------------------------------------------------|
-| success | Successfully created FlowFile from SQL query result set. |
-| failure | Flow files containing malformed sql statements |
+| Name | Description |
+|---------|------------------|
+| success | Script successes |
+| failure | Script failures |
-## ExecuteScript
+## ExecuteSQL
### Description
-Executes a script given the flow file and a process session. The script is responsible for handling the incoming flow file (transfer to SUCCESS or remove, e.g.) as well as any flow files created by the script. If the handling is incomplete or incorrect, the session will be rolled back. Scripts must define an onTrigger function which accepts NiFi Context and Property objects. For efficiency, scripts are executed once when the processor is run, then the onTrigger method is called for each incoming flowfile. This enables scripts to keep state if they wish, although there will be a script context per concurrent task of the processor. In order to, e.g., compute an arithmetic sum based on incoming flow file information, set the concurrent tasks to 1.
+ExecuteSQL to execute SELECT statement via ODBC.
Review Comment:
fixed in f43e7ef373d2f4579b76e199b92d2a339df5cfd4
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a diff in pull request #1483: MINIFICPP-2019 Fix auto-generation of PROCESSORS.md
Posted by "fgerlits (via GitHub)" <gi...@apache.org>.
fgerlits commented on code in PR #1483:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1483#discussion_r1101486033
##########
extensions/aws/processors/PutS3Object.h:
##########
@@ -44,7 +44,10 @@ class PutS3Object : public S3Processor {
static const std::set<std::string> STORAGE_CLASSES;
static const std::set<std::string> SERVER_SIDE_ENCRYPTIONS;
- EXTENSIONAPI static constexpr const char* Description = "This Processor puts FlowFiles to an Amazon S3 Bucket.";
+ EXTENSIONAPI static constexpr const char* Description = "Puts FlowFiles to an Amazon S3 Bucket. The upload uses either the PutS3Object method. "
Review Comment:
fixed in 5331a2377aa7ca6b3b259ebabe6e88314480578d
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a diff in pull request #1483: MINIFICPP-2019 Fix auto-generation of PROCESSORS.md
Posted by "szaszm (via GitHub)" <gi...@apache.org>.
szaszm commented on code in PR #1483:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1483#discussion_r1133780010
##########
PROCESSORS.md:
##########
@@ -249,165 +305,176 @@ In the list below, the names of required properties appear in bold. Any other pr
### Description
Compresses or decompresses the contents of FlowFiles using a user-specified compression algorithm and updates the mime.type attribute as appropriate
+
### Properties
In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
-| Name | Default Value | Allowable Values | Description |
-|--------------------|-------------------------|------------------|--------------------------------------------------------------------------------|
-| Compression Format | use mime.type attribute | | The compression format to use. |
-| Compression Level | 1 | | The compression level to use; this is valid only when using GZIP compression. |
-| Mode | compress | | Indicates whether the processor should compress content or decompress content. |
-| Update Filename | false | | Determines if filename extension need to be updated |
+| Name | Default Value | Allowable Values | Description |
+|--------------------|-------------------------|------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| Mode | compress | compress<br/>decompress | Indicates whether the processor should compress content or decompress content. |
+| Compression Level | 1 | | The compression level to use; this is valid only when using GZIP compression. |
+| Compression Format | use mime.type attribute | bzip2<br/>gzip<br/>lzma<br/>use mime.type attribute<br/>xz-lzma2 | The compression format to use. |
+| Update Filename | false | | Determines if filename extension need to be updated |
+| Encapsulate in TAR | true | | If true, on compression the FlowFile is added to a TAR archive and then compressed, and on decompression a compressed, TAR-encapsulated FlowFile is expected.<br/>If false, on compression the content of the FlowFile simply gets compressed, and on decompression a simple compressed content is expected.<br/>true is the behaviour compatible with older MiNiFi C++ versions, false is the behaviour compatible with NiFi. |
+| Batch Size | 1 | | Maximum number of FlowFiles processed in a single session |
+
### Relationships
| Name | Description |
|---------|---------------------------------------------------------------------------------------------------------------|
-| failure | FlowFiles will be transferred to the failure relationship if they fail to compress/decompress |
| success | FlowFiles will be transferred to the success relationship after successfully being compressed or decompressed |
+| failure | FlowFiles will be transferred to the failure relationship if they fail to compress/decompress |
## ConsumeJournald
+
### Description
-Consume systemd-journald journal messages. Available on Linux only.
+
+Consume systemd-journald journal messages. Creates one flow file per message. Fields are mapped to attributes. Realtime timestamp is mapped to the 'timestamp' attribute. Available on Linux only.
### Properties
-All properties are required with a default value, making them effectively optional. None of the properties support the NiFi Expression Language.
-| Name | Default Value | Allowable Values | Description |
-|----------------------|---------------|------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------|
-| Batch Size | 1000 | Positive numbers | The maximum number of entries processed in a single execution. |
-| Payload Format | Syslog | Raw<br>Syslog | Configures flow file content formatting.<br>Raw: only the message.<br>Syslog: similar to syslog or journalctl output. |
-| Include Timestamp | true | true<br>false | Include message timestamp in the 'timestamp' attribute. |
-| Journal Type | System | User<br>System<br>Both | Type of journal to consume. |
-| Process Old Messages | false | true<br>false | Process events created before the first usage (schedule) of the processor instance. |
-| Timestamp Format | %x %X %Z | [date format](https://howardhinnant.github.io/date/date.html#to_stream_formatting) | Format string to use when creating the timestamp attribute or writing messages in the syslog format. ISO/ISO 8601/ISO8601 are equivalent to "%FT%T%Ez". |
+In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
+
+| Name | Default Value | Allowable Values | Description |
+|--------------------------|---------------|--------------------------|-----------------------------------------------------------------------------------------------------------------|
+| **Batch Size** | 1000 | | The maximum number of entries processed in a single execution. |
+| **Payload Format** | Syslog | Raw<br/>Syslog | Configures flow file content formatting. Raw: only the message. Syslog: similar to syslog or journalctl output. |
+| **Include Timestamp** | true | | Include message timestamp in the 'timestamp' attribute. |
+| **Journal Type** | System | Both<br/>System<br/>User | Type of journal to consume. |
+| **Process Old Messages** | false | | Process events created before the first usage (schedule) of the processor instance. |
+| **Timestamp Format** | %x %X %Z | | Format string to use when creating the timestamp attribute or writing messages in the syslog format. |
Review Comment:
Can we re-add some of the lost info? I think the format string specs and the ISO mapping are important to be documented.
##########
PROCESSORS.md:
##########
@@ -589,91 +688,97 @@ In the list below, the names of required properties appear in bold. Any other pr
### Description
-Executes a script given the flow file and a process session. The script is responsible for handling the incoming flow file (transfer to SUCCESS or remove, e.g.) as well as any flow files created by the script. If the handling is incomplete or incorrect, the session will be rolled back. Scripts must define an onTrigger function which accepts NiFi Context and Property objects. For efficiency, scripts are executed once when the processor is run, then the onTrigger method is called for each incoming flowfile. This enables scripts to keep state if they wish, although there will be a script context per concurrent task of the processor. In order to, e.g., compute an arithmetic sum based on incoming flow file information, set the concurrent tasks to 1. The python script files are expected to contain `describe(procesor)` and `onTrigger(context, session)`.
+Executes a script given the flow file and a process session. The script is responsible for handling the incoming flow file (transfer to SUCCESS or remove, e.g.) as well as any flow files created by the script. If the handling is incomplete or incorrect, the session will be rolled back.Scripts must define an onTrigger function which accepts NiFi Context and Property objects. For efficiency, scripts are executed once when the processor is run, then the onTrigger method is called for each incoming flowfile. This enables scripts to keep state if they wish, although there will be a script context per concurrent task of the processor. In order to, e.g., compute an arithmetic sum based on incoming flow file information, set the concurrent tasks to 1.
### Properties
In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
| Name | Default Value | Allowable Values | Description |
|-----------------------------|---------------|------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| Script File | | | Path to script file to execute. Only one of Script File or Script Body may be used |
+| Script Body | | | Script to execute. Only one of Script File or Script Body may be used |
| Module Directory | | | Comma-separated list of paths to files and/or directories which contain modules required by the script |
| **Reload on Script Change** | true | | If true and Script File property is used, then script file will be reloaded if it has changed, otherwise the first loaded version will be used at all times. |
-| Script Body | | | Script to execute |
-| Script File | | | Path to script file to execute. Only one of Script File or Script Body may be used |
+
### Relationships
-| Name | Description |
-|---------|------------------|
-| failure | Script failures |
-| success | Script successes |
+| Name | Description |
+|---------|-----------------|
+| success | Script succeeds |
+| failure | Script fails |
-## ExecuteSQL
+## ExecuteScript
### Description
-Execute provided SQL query. Query result rows will be outputted as new flow files with attribute keys equal to result column names and values equal to result values. There will be one output FlowFile per result row. This processor can be scheduled to run using the standard timer-based scheduling methods, or it can be triggered by an incoming FlowFile. If it is triggered by an incoming FlowFile, then attributes of that FlowFile will be available when evaluating the query.
+Executes a script given the flow file and a process session. The script is responsible for handling the incoming flow file (transfer to SUCCESS or remove, e.g.) as well as any flow files created by the script. If the handling is incomplete or incorrect, the session will be rolled back.Scripts must define an onTrigger function which accepts NiFi Context and Property objects. For efficiency, scripts are executed once when the processor is run, then the onTrigger method is called for each incoming flowfile. This enables scripts to keep state if they wish, although there will be a script context per concurrent task of the processor. In order to, e.g., compute an arithmetic sum based on incoming flow file information, set the concurrent tasks to 1.
+
### Properties
In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
-| Name | Default Value | Allowable Values | Description |
-|----------------------------|---------------|----------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| **DB Controller Service** | | | Database Controller Service. |
-| **Max Rows Per Flow File** | 0 | | The maximum number of result rows that will be included in a single FlowFile. This will allow you to break up very large result sets into multiple FlowFiles. If the value specified is zero, then all rows are returned in a single FlowFile. |
-| **Output Format** | JSON-Pretty | JSON<br/>JSON-Pretty | Set the output format type. |
-| SQL select query | | | The SQL select query to execute. The query can be empty, a constant value, or built from attributes using Expression Language. If this property is specified, it will be used regardless of the content of incoming flowfiles. If this property is empty, the content of the incoming flow file is expected to contain a valid SQL select query, to be issued by the processor to the database. |
+| Name | Default Value | Allowable Values | Description |
+|-------------------|---------------|------------------|--------------------------------------------------------------------------------------------------------|
+| **Script Engine** | python | lua<br/>python | The engine to execute scripts (python, lua) |
+| Script File | | | Path to script file to execute. Only one of Script File or Script Body may be used |
+| Script Body | | | Body of script to execute. Only one of Script File or Script Body may be used |
+| Module Directory | | | Comma-separated list of paths to files and/or directories which contain modules required by the script |
+
### Relationships
-| Name | Description |
-|---------|----------------------------------------------------------|
-| success | Successfully created FlowFile from SQL query result set. |
-| failure | Flow files containing malformed sql statements |
+| Name | Description |
+|---------|------------------|
+| success | Script successes |
+| failure | Script failures |
-## ExecuteScript
+## ExecuteSQL
### Description
-Executes a script given the flow file and a process session. The script is responsible for handling the incoming flow file (transfer to SUCCESS or remove, e.g.) as well as any flow files created by the script. If the handling is incomplete or incorrect, the session will be rolled back. Scripts must define an onTrigger function which accepts NiFi Context and Property objects. For efficiency, scripts are executed once when the processor is run, then the onTrigger method is called for each incoming flowfile. This enables scripts to keep state if they wish, although there will be a script context per concurrent task of the processor. In order to, e.g., compute an arithmetic sum based on incoming flow file information, set the concurrent tasks to 1.
+ExecuteSQL to execute SELECT statement via ODBC.
Review Comment:
The old description was much more detailed here. Can we revert?
##########
PROCESSORS.md:
##########
@@ -249,165 +305,176 @@ In the list below, the names of required properties appear in bold. Any other pr
### Description
Compresses or decompresses the contents of FlowFiles using a user-specified compression algorithm and updates the mime.type attribute as appropriate
+
### Properties
In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
-| Name | Default Value | Allowable Values | Description |
-|--------------------|-------------------------|------------------|--------------------------------------------------------------------------------|
-| Compression Format | use mime.type attribute | | The compression format to use. |
-| Compression Level | 1 | | The compression level to use; this is valid only when using GZIP compression. |
-| Mode | compress | | Indicates whether the processor should compress content or decompress content. |
-| Update Filename | false | | Determines if filename extension need to be updated |
+| Name | Default Value | Allowable Values | Description |
+|--------------------|-------------------------|------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| Mode | compress | compress<br/>decompress | Indicates whether the processor should compress content or decompress content. |
+| Compression Level | 1 | | The compression level to use; this is valid only when using GZIP compression. |
+| Compression Format | use mime.type attribute | bzip2<br/>gzip<br/>lzma<br/>use mime.type attribute<br/>xz-lzma2 | The compression format to use. |
+| Update Filename | false | | Determines if filename extension need to be updated |
+| Encapsulate in TAR | true | | If true, on compression the FlowFile is added to a TAR archive and then compressed, and on decompression a compressed, TAR-encapsulated FlowFile is expected.<br/>If false, on compression the content of the FlowFile simply gets compressed, and on decompression a simple compressed content is expected.<br/>true is the behaviour compatible with older MiNiFi C++ versions, false is the behaviour compatible with NiFi. |
+| Batch Size | 1 | | Maximum number of FlowFiles processed in a single session |
+
### Relationships
| Name | Description |
|---------|---------------------------------------------------------------------------------------------------------------|
-| failure | FlowFiles will be transferred to the failure relationship if they fail to compress/decompress |
| success | FlowFiles will be transferred to the success relationship after successfully being compressed or decompressed |
+| failure | FlowFiles will be transferred to the failure relationship if they fail to compress/decompress |
## ConsumeJournald
+
### Description
-Consume systemd-journald journal messages. Available on Linux only.
+
+Consume systemd-journald journal messages. Creates one flow file per message. Fields are mapped to attributes. Realtime timestamp is mapped to the 'timestamp' attribute. Available on Linux only.
### Properties
-All properties are required with a default value, making them effectively optional. None of the properties support the NiFi Expression Language.
-| Name | Default Value | Allowable Values | Description |
-|----------------------|---------------|------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------|
-| Batch Size | 1000 | Positive numbers | The maximum number of entries processed in a single execution. |
-| Payload Format | Syslog | Raw<br>Syslog | Configures flow file content formatting.<br>Raw: only the message.<br>Syslog: similar to syslog or journalctl output. |
-| Include Timestamp | true | true<br>false | Include message timestamp in the 'timestamp' attribute. |
-| Journal Type | System | User<br>System<br>Both | Type of journal to consume. |
-| Process Old Messages | false | true<br>false | Process events created before the first usage (schedule) of the processor instance. |
-| Timestamp Format | %x %X %Z | [date format](https://howardhinnant.github.io/date/date.html#to_stream_formatting) | Format string to use when creating the timestamp attribute or writing messages in the syslog format. ISO/ISO 8601/ISO8601 are equivalent to "%FT%T%Ez". |
+In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
+
+| Name | Default Value | Allowable Values | Description |
+|--------------------------|---------------|--------------------------|-----------------------------------------------------------------------------------------------------------------|
+| **Batch Size** | 1000 | | The maximum number of entries processed in a single execution. |
+| **Payload Format** | Syslog | Raw<br/>Syslog | Configures flow file content formatting. Raw: only the message. Syslog: similar to syslog or journalctl output. |
+| **Include Timestamp** | true | | Include message timestamp in the 'timestamp' attribute. |
+| **Journal Type** | System | Both<br/>System<br/>User | Type of journal to consume. |
+| **Process Old Messages** | false | | Process events created before the first usage (schedule) of the processor instance. |
+| **Timestamp Format** | %x %X %Z | | Format string to use when creating the timestamp attribute or writing messages in the syslog format. |
### Relationships
-| Name | Description |
-|---------|--------------------------------|
-| success | Journal messages as flow files |
+| Name | Description |
+|---------|-----------------------------------------|
+| success | Successfully consumed journal messages. |
## ConsumeKafka
### Description
Consumes messages from Apache Kafka and transform them into MiNiFi FlowFiles. The application should make sure that the processor is triggered at regular intervals, even if no messages are expected, to serve any queued callbacks waiting to be called. Rebalancing can also only happen on trigger.
+
### Properties
In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
-| Name | Default Value | Allowable Values | Description |
-|------------------------------|----------------|--------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| Duplicate Header Handling | Keep Latest | Comma-separated Merge<br>Keep First<br>Keep Latest<br> | For headers to be added as attributes, this option specifies how to handle cases where multiple headers are present with the same key. For example in case of receiving these two headers: "Accept: text/html" and "Accept: application/xml" and we want to attach the value of "Accept" as a FlowFile attribute:<br/> - "Keep First" attaches: "Accept -> text/html"<br/> - "Keep Latest" attaches: "Accept -> application/xml"<br/> - "Comma-separated Merge" attaches: "Accept -> text/html, application/xml" |
-| **Group ID** | | | A Group ID is used to identify consumers that are within the same consumer group. Corresponds to Kafka's 'group.id' property.<br/>**Supports Expression Language: true** |
-| Headers To Add As Attributes | | | A comma separated list to match against all message headers. Any message header whose name matches an item from the list will be added to the FlowFile as an Attribute. If not specified, no Header values will be added as FlowFile attributes. The behaviour on when multiple headers of the same name are present is set using the DuplicateHeaderHandling attribute. |
-| **Honor Transactions** | true | | Specifies whether or not MiNiFi should honor transactional guarantees when communicating with Kafka. If false, the Processor will use an "isolation level" of read_uncomitted. This means that messages will be received as soon as they are written to Kafka but will be pulled, even if the producer cancels the transactions. If this value is true, MiNiFi will not receive any messages for which the producer's transaction was canceled, but this can result in some latency since the consumer must wait for the producer to finish its entire transaction instead of pulling as the messages become available. |
-| **Kafka Brokers** | localhost:9092 | | A comma-separated list of known Kafka Brokers in the format <host>:<port>.<br/>**Supports Expression Language: true** |
-| Kerberos Keytab Path | | | The path to the location on the local filesystem where the kerberos keytab is located. Read permission on the file is required. |
-| Kerberos Principal | | | Keberos Principal |
-| Kerberos Service Name | | | Kerberos Service Name |
-| **Key Attribute Encoding** | UTF-8 | Hex<br>UTF-8<br> | FlowFiles that are emitted have an attribute named 'kafka.key'. This property dictates how the value of the attribute should be encoded. |
-| Max Poll Records | 10000 | | Specifies the maximum number of records Kafka should return when polling each time the processor is triggered. |
-| **Max Poll Time** | 4 seconds | | Specifies the maximum amount of time the consumer can use for polling data from the brokers. Polling is a blocking operation, so the upper limit of this value is specified in 4 seconds. |
-| Message Demarcator | | | Since KafkaConsumer receives messages in batches, you have an option to output FlowFiles which contains all Kafka messages in a single batch for a given topic and partition and this property allows you to provide a string (interpreted as UTF-8) to use for demarcating apart multiple Kafka messages. This is an optional property and if not provided each Kafka message received will result in a single FlowFile which time it is triggered. <br/>**Supports Expression Language: true** |
-| Message Header Encoding | UTF-8 | Hex<br>UTF-8<br> | Any message header that is found on a Kafka message will be added to the outbound FlowFile as an attribute. This property indicates the Character Encoding to use for deserializing the headers. |
-| **Offset Reset** | latest | earliest<br>latest<br>none<br> | Allows you to manage the condition when there is no initial offset in Kafka or if the current offset does not exist any more on the server (e.g. because that data has been deleted). Corresponds to Kafka's 'auto.offset.reset' property. |
-| Password | | | The password for the given username when the SASL Mechanism is sasl_plaintext |
-| SASL Mechanism | GSSAPI | GSSAPI<br/>PLAIN | The SASL mechanism to use for authentication. Corresponds to Kafka's 'sasl.mechanism' property. |
-| **Security Protocol** | plaintext | plaintext<br/>ssl<br/>sasl_plaintext<br/>sasl_ssl | Protocol used to communicate with brokers. Corresponds to Kafka's 'security.protocol' property. |
-| Session Timeout | 60 seconds | | Client group session and failure detection timeout. The consumer sends periodic heartbeats to indicate its liveness to the broker. If no hearts are received by the broker for a group member within the session timeout, the broker will remove the consumer from the group and trigger a rebalance. The allowed range is configured with the broker configuration properties group.min.session.timeout.ms and group.max.session.timeout.ms. |
-| SSL Context Service | | | SSL Context Service Name |
-| **Topic Name Format** | Names | Names<br>Patterns<br> | Specifies whether the Topic(s) provided are a comma separated list of names or a single regular expression. Using regular expressions does not automatically discover Kafka topics created after the processor started. |
-| **Topic Names** | | | The name of the Kafka Topic(s) to pull from. Multiple topic names are supported as a comma separated list.<br/>**Supports Expression Language: true** |
-| Username | | | The username when the SASL Mechanism is sasl_plaintext |
-### Properties
+| Name | Default Value | Allowable Values | Description |
+|------------------------------|----------------|------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| SSL Context Service | | | SSL Context Service Name |
+| **Security Protocol** | plaintext | plaintext<br/>sasl_plaintext<br/>sasl_ssl<br/>ssl | Protocol used to communicate with brokers. Corresponds to Kafka's 'security.protocol' property. |
+| Kerberos Service Name | | | Kerberos Service Name |
+| Kerberos Principal | | | Keberos Principal |
+| Kerberos Keytab Path | | | The path to the location on the local filesystem where the kerberos keytab is located. Read permission on the file is required. |
+| **SASL Mechanism** | GSSAPI | GSSAPI<br/>PLAIN | The SASL mechanism to use for authentication. Corresponds to Kafka's 'sasl.mechanism' property. |
+| Username | | | The username when the SASL Mechanism is sasl_plaintext |
+| Password | | | The password for the given username when the SASL Mechanism is sasl_plaintext |
+| **Kafka Brokers** | localhost:9092 | | A comma-separated list of known Kafka Brokers in the format <host>:<port>.<br/>**Supports Expression Language: true** |
+| **Topic Names** | | | The name of the Kafka Topic(s) to pull from. Multiple topic names are supported as a comma separated list.<br/>**Supports Expression Language: true** |
+| **Topic Name Format** | Names | Names<br/>Patterns | Specifies whether the Topic(s) provided are a comma separated list of names or a single regular expression. Using regular expressions does not automatically discover Kafka topics created after the processor started. |
+| **Honor Transactions** | true | | Specifies whether or not MiNiFi should honor transactional guarantees when communicating with Kafka. If false, the Processor will use an "isolation level" of read_uncomitted. This means that messages will be received as soon as they are written to Kafka but will be pulled, even if the producer cancels the transactions. If this value is true, MiNiFi will not receive any messages for which the producer's transaction was canceled, but this can result in some latency since the consumer must wait for the producer to finish its entire transaction instead of pulling as the messages become available. |
+| **Group ID** | | | A Group ID is used to identify consumers that are within the same consumer group. Corresponds to Kafka's 'group.id' property.<br/>**Supports Expression Language: true** |
+| **Offset Reset** | latest | earliest<br/>latest<br/>none | Allows you to manage the condition when there is no initial offset in Kafka or if the current offset does not exist any more on the server (e.g. because that data has been deleted). Corresponds to Kafka's 'auto.offset.reset' property. |
+| **Key Attribute Encoding** | UTF-8 | Hex<br/>UTF-8 | FlowFiles that are emitted have an attribute named 'kafka.key'. This property dictates how the value of the attribute should be encoded. |
+| Message Demarcator | | | Since KafkaConsumer receives messages in batches, you have an option to output FlowFiles which contains all Kafka messages in a single batch for a given topic and partition and this property allows you to provide a string (interpreted as UTF-8) to use for demarcating apart multiple Kafka messages. This is an optional property and if not provided each Kafka message received will result in a single FlowFile which time it is triggered. <br/>**Supports Expression Language: true** |
+| Message Header Encoding | UTF-8 | Hex<br/>UTF-8 | Any message header that is found on a Kafka message will be added to the outbound FlowFile as an attribute. This property indicates the Character Encoding to use for deserializing the headers. |
+| Headers To Add As Attributes | | | A comma separated list to match against all message headers. Any message header whose name matches an item from the list will be added to the FlowFile as an Attribute. If not specified, no Header values will be added as FlowFile attributes. The behaviour on when multiple headers of the same name are present is set using the DuplicateHeaderHandling attribute. |
Review Comment:
```suggestion
| Headers To Add As Attributes | | | A comma separated list to match against all message headers. Any message header whose name matches an item from the list will be added to the FlowFile as an Attribute. If not specified, no Header values will be added as FlowFile attributes. The behaviour on when multiple headers of the same name are present is set using the Duplicate Header Handling property |
```
##########
PROCESSORS.md:
##########
@@ -749,16 +885,15 @@ Fetches a file from a Google Cloud Bucket. Designed to be used in tandem with Li
In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
-| Name | Default Value | Allowable Values | Description |
-|--------------------------------------|---------------|-----------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------|
-| **Bucket** | ${gcs.bucket} | | Bucket of the object.<br>**Supports Expression Language: true** |
-| **Key** | ${filename} | | Name of the object.<br>**Supports Expression Language: true** |
-| **Number of retries** | 6 | integers | How many retry attempts should be made before routing to the failure relationship. |
-| **GCP Credentials Provider Service** | | [GCPCredentialsControllerService](CONTROLLERS.md#GCPCredentialsControllerService) | The Controller Service used to obtain Google Cloud Platform credentials. |
-| Server Side Encryption Key | | | An AES256 Encryption Key (encoded in base64) for server-side encryption of the object.<br>**Supports Expression Language: true** |
-| Object Generation | | | The generation of the Object to download. If left empty, then it will download the latest generation.<br>**Supports Expression Language: true** |
-| Endpoint Override URL | | | Overrides the default Google Cloud Storage endpoints |
-
+| Name | Default Value | Allowable Values | Description |
+|--------------------------------------|---------------|------------------|--------------------------------------------------------------------------------------------------------------------------------------------------|
+| **GCP Credentials Provider Service** | | | The Controller Service used to obtain Google Cloud Platform credentials. |
+| **Number of retries** | 6 | | How many retry attempts should be made before routing to the failure relationship. |
+| Endpoint Override URL | | | Overrides the default Google Cloud Storage endpoints<br/>**Supports Expression Language: true** |
+| Bucket | ${gcs.bucket} | | Bucket of the object.<br/>**Supports Expression Language: true** |
+| Key | ${filename} | | Name of the object.<br/>**Supports Expression Language: true** |
+| Server Side Encryption Key | | | The AES256 Encryption Key (encoded in base64) for server-side decryption of the object.<br/>**Supports Expression Language: true** |
Review Comment:
There is a difference in the description here: encryption (old) vs decryption (new).
##########
PROCESSORS.md:
##########
@@ -502,16 +574,15 @@ Deletes an object from a Google Cloud Bucket.
In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
-| Name | Default Value | Allowable Values | Description |
-|--------------------------------------|---------------|-----------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------|
-| **Bucket** | ${gcs.bucket} | | Bucket of the object.<br>**Supports Expression Language: true** |
-| **Key** | ${filename} | | Name of the object.<br>**Supports Expression Language: true** |
-| **Number of retries** | 6 | integers | How many retry attempts should be made before routing to the failure relationship. |
-| **GCP Credentials Provider Service** | | [GCPCredentialsControllerService](CONTROLLERS.md#GCPCredentialsControllerService) | The Controller Service used to obtain Google Cloud Platform credentials. |
-| Server Side Encryption Key | | | An AES256 Encryption Key (encoded in base64) for server-side encryption of the object.<br>**Supports Expression Language: true** |
-| Object Generation | | | The generation of the Object to download. If left empty, then it will download the latest generation.<br>**Supports Expression Language: true** |
-| Endpoint Override URL | | | Overrides the default Google Cloud Storage endpoints |
-
+| Name | Default Value | Allowable Values | Description |
+|--------------------------------------|---------------|------------------|--------------------------------------------------------------------------------------------------------------------------------------------------|
+| **GCP Credentials Provider Service** | | | The Controller Service used to obtain Google Cloud Platform credentials. |
Review Comment:
Is there any way to keep the association between this property and the GCPCredentialsControllerService?
##########
PROCESSORS.md:
##########
@@ -589,91 +688,97 @@ In the list below, the names of required properties appear in bold. Any other pr
### Description
-Executes a script given the flow file and a process session. The script is responsible for handling the incoming flow file (transfer to SUCCESS or remove, e.g.) as well as any flow files created by the script. If the handling is incomplete or incorrect, the session will be rolled back. Scripts must define an onTrigger function which accepts NiFi Context and Property objects. For efficiency, scripts are executed once when the processor is run, then the onTrigger method is called for each incoming flowfile. This enables scripts to keep state if they wish, although there will be a script context per concurrent task of the processor. In order to, e.g., compute an arithmetic sum based on incoming flow file information, set the concurrent tasks to 1. The python script files are expected to contain `describe(procesor)` and `onTrigger(context, session)`.
+Executes a script given the flow file and a process session. The script is responsible for handling the incoming flow file (transfer to SUCCESS or remove, e.g.) as well as any flow files created by the script. If the handling is incomplete or incorrect, the session will be rolled back.Scripts must define an onTrigger function which accepts NiFi Context and Property objects. For efficiency, scripts are executed once when the processor is run, then the onTrigger method is called for each incoming flowfile. This enables scripts to keep state if they wish, although there will be a script context per concurrent task of the processor. In order to, e.g., compute an arithmetic sum based on incoming flow file information, set the concurrent tasks to 1.
Review Comment:
We had an additional sentence here, that I'd rather not delete.
##########
PROCESSORS.md:
##########
@@ -249,165 +305,176 @@ In the list below, the names of required properties appear in bold. Any other pr
### Description
Compresses or decompresses the contents of FlowFiles using a user-specified compression algorithm and updates the mime.type attribute as appropriate
+
### Properties
In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
-| Name | Default Value | Allowable Values | Description |
-|--------------------|-------------------------|------------------|--------------------------------------------------------------------------------|
-| Compression Format | use mime.type attribute | | The compression format to use. |
-| Compression Level | 1 | | The compression level to use; this is valid only when using GZIP compression. |
-| Mode | compress | | Indicates whether the processor should compress content or decompress content. |
-| Update Filename | false | | Determines if filename extension need to be updated |
+| Name | Default Value | Allowable Values | Description |
+|--------------------|-------------------------|------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| Mode | compress | compress<br/>decompress | Indicates whether the processor should compress content or decompress content. |
+| Compression Level | 1 | | The compression level to use; this is valid only when using GZIP compression. |
+| Compression Format | use mime.type attribute | bzip2<br/>gzip<br/>lzma<br/>use mime.type attribute<br/>xz-lzma2 | The compression format to use. |
+| Update Filename | false | | Determines if filename extension need to be updated |
+| Encapsulate in TAR | true | | If true, on compression the FlowFile is added to a TAR archive and then compressed, and on decompression a compressed, TAR-encapsulated FlowFile is expected.<br/>If false, on compression the content of the FlowFile simply gets compressed, and on decompression a simple compressed content is expected.<br/>true is the behaviour compatible with older MiNiFi C++ versions, false is the behaviour compatible with NiFi. |
+| Batch Size | 1 | | Maximum number of FlowFiles processed in a single session |
+
### Relationships
| Name | Description |
|---------|---------------------------------------------------------------------------------------------------------------|
-| failure | FlowFiles will be transferred to the failure relationship if they fail to compress/decompress |
| success | FlowFiles will be transferred to the success relationship after successfully being compressed or decompressed |
+| failure | FlowFiles will be transferred to the failure relationship if they fail to compress/decompress |
## ConsumeJournald
+
### Description
-Consume systemd-journald journal messages. Available on Linux only.
+
+Consume systemd-journald journal messages. Creates one flow file per message. Fields are mapped to attributes. Realtime timestamp is mapped to the 'timestamp' attribute. Available on Linux only.
### Properties
-All properties are required with a default value, making them effectively optional. None of the properties support the NiFi Expression Language.
-| Name | Default Value | Allowable Values | Description |
-|----------------------|---------------|------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------|
-| Batch Size | 1000 | Positive numbers | The maximum number of entries processed in a single execution. |
-| Payload Format | Syslog | Raw<br>Syslog | Configures flow file content formatting.<br>Raw: only the message.<br>Syslog: similar to syslog or journalctl output. |
-| Include Timestamp | true | true<br>false | Include message timestamp in the 'timestamp' attribute. |
-| Journal Type | System | User<br>System<br>Both | Type of journal to consume. |
-| Process Old Messages | false | true<br>false | Process events created before the first usage (schedule) of the processor instance. |
-| Timestamp Format | %x %X %Z | [date format](https://howardhinnant.github.io/date/date.html#to_stream_formatting) | Format string to use when creating the timestamp attribute or writing messages in the syslog format. ISO/ISO 8601/ISO8601 are equivalent to "%FT%T%Ez". |
+In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
+
+| Name | Default Value | Allowable Values | Description |
+|--------------------------|---------------|--------------------------|-----------------------------------------------------------------------------------------------------------------|
+| **Batch Size** | 1000 | | The maximum number of entries processed in a single execution. |
+| **Payload Format** | Syslog | Raw<br/>Syslog | Configures flow file content formatting. Raw: only the message. Syslog: similar to syslog or journalctl output. |
+| **Include Timestamp** | true | | Include message timestamp in the 'timestamp' attribute. |
+| **Journal Type** | System | Both<br/>System<br/>User | Type of journal to consume. |
+| **Process Old Messages** | false | | Process events created before the first usage (schedule) of the processor instance. |
+| **Timestamp Format** | %x %X %Z | | Format string to use when creating the timestamp attribute or writing messages in the syslog format. |
### Relationships
-| Name | Description |
-|---------|--------------------------------|
-| success | Journal messages as flow files |
+| Name | Description |
+|---------|-----------------------------------------|
+| success | Successfully consumed journal messages. |
## ConsumeKafka
### Description
Consumes messages from Apache Kafka and transform them into MiNiFi FlowFiles. The application should make sure that the processor is triggered at regular intervals, even if no messages are expected, to serve any queued callbacks waiting to be called. Rebalancing can also only happen on trigger.
+
### Properties
In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
-| Name | Default Value | Allowable Values | Description |
-|------------------------------|----------------|--------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| Duplicate Header Handling | Keep Latest | Comma-separated Merge<br>Keep First<br>Keep Latest<br> | For headers to be added as attributes, this option specifies how to handle cases where multiple headers are present with the same key. For example in case of receiving these two headers: "Accept: text/html" and "Accept: application/xml" and we want to attach the value of "Accept" as a FlowFile attribute:<br/> - "Keep First" attaches: "Accept -> text/html"<br/> - "Keep Latest" attaches: "Accept -> application/xml"<br/> - "Comma-separated Merge" attaches: "Accept -> text/html, application/xml" |
-| **Group ID** | | | A Group ID is used to identify consumers that are within the same consumer group. Corresponds to Kafka's 'group.id' property.<br/>**Supports Expression Language: true** |
-| Headers To Add As Attributes | | | A comma separated list to match against all message headers. Any message header whose name matches an item from the list will be added to the FlowFile as an Attribute. If not specified, no Header values will be added as FlowFile attributes. The behaviour on when multiple headers of the same name are present is set using the DuplicateHeaderHandling attribute. |
-| **Honor Transactions** | true | | Specifies whether or not MiNiFi should honor transactional guarantees when communicating with Kafka. If false, the Processor will use an "isolation level" of read_uncomitted. This means that messages will be received as soon as they are written to Kafka but will be pulled, even if the producer cancels the transactions. If this value is true, MiNiFi will not receive any messages for which the producer's transaction was canceled, but this can result in some latency since the consumer must wait for the producer to finish its entire transaction instead of pulling as the messages become available. |
-| **Kafka Brokers** | localhost:9092 | | A comma-separated list of known Kafka Brokers in the format <host>:<port>.<br/>**Supports Expression Language: true** |
-| Kerberos Keytab Path | | | The path to the location on the local filesystem where the kerberos keytab is located. Read permission on the file is required. |
-| Kerberos Principal | | | Keberos Principal |
-| Kerberos Service Name | | | Kerberos Service Name |
-| **Key Attribute Encoding** | UTF-8 | Hex<br>UTF-8<br> | FlowFiles that are emitted have an attribute named 'kafka.key'. This property dictates how the value of the attribute should be encoded. |
-| Max Poll Records | 10000 | | Specifies the maximum number of records Kafka should return when polling each time the processor is triggered. |
-| **Max Poll Time** | 4 seconds | | Specifies the maximum amount of time the consumer can use for polling data from the brokers. Polling is a blocking operation, so the upper limit of this value is specified in 4 seconds. |
-| Message Demarcator | | | Since KafkaConsumer receives messages in batches, you have an option to output FlowFiles which contains all Kafka messages in a single batch for a given topic and partition and this property allows you to provide a string (interpreted as UTF-8) to use for demarcating apart multiple Kafka messages. This is an optional property and if not provided each Kafka message received will result in a single FlowFile which time it is triggered. <br/>**Supports Expression Language: true** |
-| Message Header Encoding | UTF-8 | Hex<br>UTF-8<br> | Any message header that is found on a Kafka message will be added to the outbound FlowFile as an attribute. This property indicates the Character Encoding to use for deserializing the headers. |
-| **Offset Reset** | latest | earliest<br>latest<br>none<br> | Allows you to manage the condition when there is no initial offset in Kafka or if the current offset does not exist any more on the server (e.g. because that data has been deleted). Corresponds to Kafka's 'auto.offset.reset' property. |
-| Password | | | The password for the given username when the SASL Mechanism is sasl_plaintext |
-| SASL Mechanism | GSSAPI | GSSAPI<br/>PLAIN | The SASL mechanism to use for authentication. Corresponds to Kafka's 'sasl.mechanism' property. |
-| **Security Protocol** | plaintext | plaintext<br/>ssl<br/>sasl_plaintext<br/>sasl_ssl | Protocol used to communicate with brokers. Corresponds to Kafka's 'security.protocol' property. |
-| Session Timeout | 60 seconds | | Client group session and failure detection timeout. The consumer sends periodic heartbeats to indicate its liveness to the broker. If no hearts are received by the broker for a group member within the session timeout, the broker will remove the consumer from the group and trigger a rebalance. The allowed range is configured with the broker configuration properties group.min.session.timeout.ms and group.max.session.timeout.ms. |
-| SSL Context Service | | | SSL Context Service Name |
-| **Topic Name Format** | Names | Names<br>Patterns<br> | Specifies whether the Topic(s) provided are a comma separated list of names or a single regular expression. Using regular expressions does not automatically discover Kafka topics created after the processor started. |
-| **Topic Names** | | | The name of the Kafka Topic(s) to pull from. Multiple topic names are supported as a comma separated list.<br/>**Supports Expression Language: true** |
-| Username | | | The username when the SASL Mechanism is sasl_plaintext |
-### Properties
+| Name | Default Value | Allowable Values | Description |
+|------------------------------|----------------|------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| SSL Context Service | | | SSL Context Service Name |
+| **Security Protocol** | plaintext | plaintext<br/>sasl_plaintext<br/>sasl_ssl<br/>ssl | Protocol used to communicate with brokers. Corresponds to Kafka's 'security.protocol' property. |
+| Kerberos Service Name | | | Kerberos Service Name |
+| Kerberos Principal | | | Keberos Principal |
+| Kerberos Keytab Path | | | The path to the location on the local filesystem where the kerberos keytab is located. Read permission on the file is required. |
+| **SASL Mechanism** | GSSAPI | GSSAPI<br/>PLAIN | The SASL mechanism to use for authentication. Corresponds to Kafka's 'sasl.mechanism' property. |
+| Username | | | The username when the SASL Mechanism is sasl_plaintext |
+| Password | | | The password for the given username when the SASL Mechanism is sasl_plaintext |
+| **Kafka Brokers** | localhost:9092 | | A comma-separated list of known Kafka Brokers in the format <host>:<port>.<br/>**Supports Expression Language: true** |
+| **Topic Names** | | | The name of the Kafka Topic(s) to pull from. Multiple topic names are supported as a comma separated list.<br/>**Supports Expression Language: true** |
+| **Topic Name Format** | Names | Names<br/>Patterns | Specifies whether the Topic(s) provided are a comma separated list of names or a single regular expression. Using regular expressions does not automatically discover Kafka topics created after the processor started. |
+| **Honor Transactions** | true | | Specifies whether or not MiNiFi should honor transactional guarantees when communicating with Kafka. If false, the Processor will use an "isolation level" of read_uncomitted. This means that messages will be received as soon as they are written to Kafka but will be pulled, even if the producer cancels the transactions. If this value is true, MiNiFi will not receive any messages for which the producer's transaction was canceled, but this can result in some latency since the consumer must wait for the producer to finish its entire transaction instead of pulling as the messages become available. |
+| **Group ID** | | | A Group ID is used to identify consumers that are within the same consumer group. Corresponds to Kafka's 'group.id' property.<br/>**Supports Expression Language: true** |
+| **Offset Reset** | latest | earliest<br/>latest<br/>none | Allows you to manage the condition when there is no initial offset in Kafka or if the current offset does not exist any more on the server (e.g. because that data has been deleted). Corresponds to Kafka's 'auto.offset.reset' property. |
+| **Key Attribute Encoding** | UTF-8 | Hex<br/>UTF-8 | FlowFiles that are emitted have an attribute named 'kafka.key'. This property dictates how the value of the attribute should be encoded. |
+| Message Demarcator | | | Since KafkaConsumer receives messages in batches, you have an option to output FlowFiles which contains all Kafka messages in a single batch for a given topic and partition and this property allows you to provide a string (interpreted as UTF-8) to use for demarcating apart multiple Kafka messages. This is an optional property and if not provided each Kafka message received will result in a single FlowFile which time it is triggered. <br/>**Supports Expression Language: true** |
+| Message Header Encoding | UTF-8 | Hex<br/>UTF-8 | Any message header that is found on a Kafka message will be added to the outbound FlowFile as an attribute. This property indicates the Character Encoding to use for deserializing the headers. |
+| Headers To Add As Attributes | | | A comma separated list to match against all message headers. Any message header whose name matches an item from the list will be added to the FlowFile as an Attribute. If not specified, no Header values will be added as FlowFile attributes. The behaviour on when multiple headers of the same name are present is set using the DuplicateHeaderHandling attribute. |
+| Duplicate Header Handling | Keep Latest | Comma-separated Merge<br/>Keep First<br/>Keep Latest | For headers to be added as attributes, this option specifies how to handle cases where multiple headers are present with the same key. For example in case of receiving these two headers: "Accept: text/html" and "Accept: application/xml" and we want to attach the value of "Accept" as a FlowFile attribute:<br/> - "Keep First" attaches: "Accept -> text/html"<br/> - "Keep Latest" attaches: "Accept -> application/xml"<br/> - "Comma-separated Merge" attaches: "Accept -> text/html, application/xml"<br/> |
+| Max Poll Records | 10000 | | Specifies the maximum number of records Kafka should return when polling each time the processor is triggered. |
+| **Max Poll Time** | 4 seconds | | Specifies the maximum amount of time the consumer can use for polling data from the brokers. Polling is a blocking operation, so the upper limit of this value is specified in 4 seconds. |
+| Session Timeout | 60 seconds | | Client group session and failure detection timeout. The consumer sends periodic heartbeats to indicate its liveness to the broker. If no hearts are received by the broker for a group member within the session timeout, the broker will remove the consumer from the group and trigger a rebalance. The allowed range is configured with the broker configuration properties group.min.session.timeout.ms and group.max.session.timeout.ms. |
+
+### Relationships
| Name | Description |
|---------|---------------------------------------------------------------------------------------------------------------------------------|
| success | Incoming Kafka messages as flowfiles. Depending on the demarcation strategy, this can be one or multiple flowfiles per message. |
+
## ConsumeMQTT
### Description
This Processor gets the contents of a FlowFile from a MQTT broker for a specified topic. The the payload of the MQTT message becomes content of a FlowFile
+
### Properties
In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
-| Name | Default Value | Allowable Values | Description |
-|-----------------------------|---------------|-----------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------|
-| **Broker URI** | | | The URI to use to connect to the MQTT broker |
-| Client ID | | | MQTT client ID to use. WARNING: Must not be empty when using MQTT 3.1.0! |
-| MQTT Version | 3.x AUTO | 3.x AUTO, 3.1.0, 3.1.1, 5.0 | The MQTT specification version when connecting to the broker. |
-| **Topic** | | | The topic to subscribe to. |
-| Clean Session | true | | Whether to start afresh rather than remembering previous subscriptions. If true, then make broker forget subscriptions after disconnected. MQTT 3.x only. |
-| Clean Start | true | | Whether to start afresh rather than remembering previous subscriptions. MQTT 5.x only. |
-| Session Expiry Interval | 0 s | | Time to delete session on broker after client is disconnected. MQTT 5.x only. |
-| Queue Max Message | 1000 | | Maximum number of messages allowed on the received MQTT queue |
-| Attribute From Content Type | | | Name of FlowFile attribute to be filled from content type of received message. MQTT 5.x only. |
-| Topic Alias Maximum | 0 | | Maximum number of topic aliases to use. If set to 0, then topic aliases cannot be used. MQTT 5.x only. |
-| Receive Maximum | 65535 | | Maximum number of unacknowledged messages allowed. MQTT 5.x only. |
-| Quality of Service | 0 | 0, 1 and 2 | The Quality of Service (QoS) to receive the message with. |
-| Connection Timeout | 10 sec | | Maximum time interval the client will wait for the network connection to the MQTT broker |
-| Keep Alive Interval | 60 sec | | Defines the maximum time interval between messages being received from the broker |
-| Last Will Topic | | | The topic to send the client's Last Will to. If the Last Will topic is not set then a Last Will will not be sent |
-| Last Will Message | | | The message to send as the client's Last Will. If the Last Will Message is empty, Last Will will be deleted from the broker |
-| Last Will QoS | 0 | 0, 1 and 2 | The Quality of Service (QoS) to send the last will with. |
-| Last Will Retain | false | | Whether to retain the client's Last Will |
-| Last Will Content Type | | | Content type of the client's Last Will. MQTT 5.x only. |
-| Username | | | Username to use when connecting to the broker |
-| Password | | | Password to use when connecting to the broker |
-| Security Protocol | | | Protocol used to communicate with brokers |
-| Security CA | | | File or directory path to CA certificate(s) for verifying the broker's key |
-| Security Cert | | | Path to client's public key (PEM) used for authentication |
-| Security Private Key | | | Path to client's private key (PEM) used for authentication |
-| Security Pass Phrase | | | Private key passphrase |
-
+| Name | Default Value | Allowable Values | Description |
+|-----------------------------|---------------|--------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------|
+| **Broker URI** | | | The URI to use to connect to the MQTT broker |
+| Client ID | | | MQTT client ID to use. WARNING: Must not be empty when using MQTT 3.1.0! |
+| MQTT Version | 3.x AUTO | 3.1.0<br/>3.1.1<br/>3.x AUTO<br/>5.0 | The MQTT specification version when connecting to the broker. |
+| **Topic** | | | The topic to subscribe to. |
+| Clean Session | true | | Whether to start afresh rather than remembering previous subscriptions. If true, then make broker forget subscriptions after disconnected. MQTT 3.x only. |
+| Clean Start | true | | Whether to start afresh rather than remembering previous subscriptions. MQTT 5.x only. |
+| Session Expiry Interval | 0 s | | Time to delete session on broker after client is disconnected. MQTT 5.x only. |
+| Queue Max Message | 1000 | | Maximum number of messages allowed on the received MQTT queue |
+| Attribute From Content Type | | | Name of FlowFile attribute to be filled from content type of received message. MQTT 5.x only. |
+| Topic Alias Maximum | 0 | | Maximum number of topic aliases to use. If set to 0, then topic aliases cannot be used. MQTT 5.x only. |
+| Receive Maximum | 65535 | | Maximum number of unacknowledged messages allowed. MQTT 5.x only. |
+| Quality of Service | 0 | 0<br/>1<br/>2 | The Quality of Service (QoS) of messages. |
+| Connection Timeout | 10 sec | | Maximum time interval the client will wait for the network connection to the MQTT broker |
+| Keep Alive Interval | 60 sec | | Defines the maximum time interval between messages sent or received |
+| Last Will Topic | | | The topic to send the client's Last Will to. If the Last Will topic is not set then a Last Will will not be sent |
+| Last Will Message | | | The message to send as the client's Last Will. If the Last Will Message is empty, Last Will will be deleted from the broker |
+| Last Will QoS | 0 | 0<br/>1<br/>2 | The Quality of Service (QoS) to send the last will with. |
+| Last Will Retain | false | | Whether to retain the client's Last Will |
+| Last Will Content Type | | | Content type of the client's Last Will. MQTT 5.x only. |
+| Username | | | Username to use when connecting to the broker |
+| Password | | | Password to use when connecting to the broker |
+| Security Protocol | | | Protocol used to communicate with brokers |
+| Security CA | | | File or directory path to CA certificate(s) for verifying the broker's key |
+| Security Cert | | | Path to client's public key (PEM) used for authentication |
+| Security Private Key | | | Path to client's private key (PEM) used for authentication |
+| Security Pass Phrase | | | Private key passphrase |
### Relationships
| Name | Description |
|---------|----------------------------------------------------------------------------------------------|
| success | FlowFiles that are sent successfully to the destination are transferred to this relationship |
-
### Output Attributes
| Attribute | Description |
|---------------|---------------------------|
| _mqtt.broker_ | URI of the sending broker |
| _mqtt.topic_ | Topic of the message |
+
## ConsumeWindowsEventLog
### Description
Windows Event Log Subscribe Callback to receive FlowFiles from Events on Windows.
+
### Properties
In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
-| Name | Default Value | Allowable Values | Description |
-|------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| **Channel** | System | | The Windows Event Log Channel to listen to.<br/>**Supports Expression Language: true** |
-| **Query** | * | | XPath Query to filter events. (See https://msdn.microsoft.com/en-us/library/windows/desktop/dd996910(v=vs.85).aspx for examples.)<br/>**Supports Expression Language: true** |
-| **Max Buffer Size** | 1 MB | | The individual Event Log XMLs are rendered to a buffer. This specifies the maximum size in bytes that the buffer will be allowed to grow to. (Limiting the maximum size of an individual Event XML.) |
-| **Inactive Duration To Reconnect** | 10 min | | If no new event logs are processed for the specified time period, this processor will try reconnecting to recover from a state where any further messages cannot be consumed. Such situation can happen if Windows Event Log service is restarted, or ERROR_EVT_QUERY_RESULT_STALE (15011) is returned. Setting no duration, e.g. '0 ms' disables auto-reconnection. |
-| Identifier Match Regex | .*Sid | | Regular Expression to match Subject Identifier Fields. These will be placed into the attributes of the FlowFile |
-| Apply Identifier Function | true | true<br>false | If true it will resolve SIDs matched in the 'Identifier Match Regex' to the DOMAIN\USERNAME associated with that ID |
-| Resolve Metadata in Attributes | true | true<br>false | If true, any metadata that is resolved ( such as IDs or keyword metadata ) will be placed into attributes, otherwise it will be replaced in the XML or text output |
-| Event Header Delimiter | | | If set, the chosen delimiter will be used in the Event output header. Otherwise, a colon followed by spaces will be used. |
-| Event Header | LOG_NAME=Log Name, SOURCE = Source, TIME_CREATED = Date,EVENT_RECORDID=Record ID,EVENTID = Event ID,TASK_CATEGORY = Task Category,LEVEL = Level,KEYWORDS = Keywords,USER = User,COMPUTER = Computer, EVENT_TYPE = EventType | | Comma seperated list of key/value pairs with the following keys LOG_NAME, SOURCE, TIME_CREATED,EVENT_RECORDID,EVENTID,TASK_CATEGORY,LEVEL,KEYWORDS,USER,COMPUTER, and EVENT_TYPE. Eliminating fields will remove them from the header. |
-| **Output Format** | Both | XML<br>Plaintext<br>JSON<br>Both | Set the output format type. In case 'Both' is selected the processor generates two flow files for every event captured in format XML and Plaintext |
-| **JSON Format** | Simple | Simple<br>Raw<br>Flattened | Set the json format type. Only applicable if Output Format is set to 'JSON' |
-| Batch Commit Size | 1000 | | Maximum number of Events to consume and create to Flow Files from before committing. |
-| **Process Old Events** | false | true<br>false | This property defines if old events (which are created before first time server is started) should be processed. |
-| State Directory | CWELState | | DEPRECATED. Only use it for state migration from the state file, supplying the legacy state directory. |
-| Cache SID Lookups | true | true<br>false | Determines whether SID to name lookups are cached in memory |
+| Name | Default Value | Allowable Values | Description |
+|------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| **Channel** | System | | The Windows Event Log Channel to listen to.<br/>**Supports Expression Language: true** |
+| **Query** | * | | XPath Query to filter events. (See https://msdn.microsoft.com/en-us/library/windows/desktop/dd996910(v=vs.85).aspx for examples.)<br/>**Supports Expression Language: true** |
+| **Max Buffer Size** | 1 MB | | The individual Event Log XMLs are rendered to a buffer. This specifies the maximum size in bytes that the buffer will be allowed to grow to. (Limiting the maximum size of an individual Event XML.) |
+| **Inactive Duration To Reconnect** | 10 min | | If no new event logs are processed for the specified time period, this processor will try reconnecting to recover from a state where any further messages cannot be consumed. Such situation can happen if Windows Event Log service is restarted, or ERROR_EVT_QUERY_RESULT_STALE (15011) is returned. Setting no duration, e.g. '0 ms' disables auto-reconnection. |
+| Identifier Match Regex | .*Sid | | Regular Expression to match Subject Identifier Fields. These will be placed into the attributes of the FlowFile |
+| Apply Identifier Function | true | | If true it will resolve SIDs matched in the 'Identifier Match Regex' to the DOMAIN\USERNAME associated with that ID |
+| Resolve Metadata in Attributes | true | | If true, any metadata that is resolved ( such as IDs or keyword metadata ) will be placed into attributes, otherwise it will be replaced in the XML or text output |
+| Event Header Delimiter | | | If set, the chosen delimiter will be used in the Event output header. Otherwise, a colon followed by spaces will be used. |
+| Event Header | LOG_NAME=Log Name, SOURCE = Source, TIME_CREATED = Date,EVENT_RECORDID=Record ID,EVENTID = Event ID,TASK_CATEGORY = Task Category,LEVEL = Level,KEYWORDS = Keywords,USER = User,COMPUTER = Computer, EVENT_TYPE = EventType | | Comma seperated list of key/value pairs with the following keys LOG_NAME, SOURCE, TIME_CREATED,EVENT_RECORDID,EVENTID,TASK_CATEGORY,LEVEL,KEYWORDS,USER,COMPUTER, and EVENT_TYPE. Eliminating fields will remove them from the header. |
+| **Output Format** | Both | Both<br/>JSON<br/>Plaintext<br/>XML | Set the output format type. In case 'Both' is selected the processor generates two flow files for every event captured in format XML and Plaintext |
+| **JSON Format** | Simple | Flattened<br/>Raw<br/>Simple | Set the json format type. Only applicable if Output Format is set to 'JSON' |
Review Comment:
I'd rather reorder the options to the old order.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a diff in pull request #1483: MINIFICPP-2019 Fix auto-generation of PROCESSORS.md
Posted by "fgerlits (via GitHub)" <gi...@apache.org>.
fgerlits commented on code in PR #1483:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1483#discussion_r1101493234
##########
minifi_main/TableFormatter.h:
##########
@@ -0,0 +1,47 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "core/PropertyValue.h"
+
+namespace org::apache::nifi::minifi::docs {
+
+class Table {
+ public:
+ explicit Table(std::vector<std::string> header) : header_{std::move(header)} {}
+ void addRow(std::vector<std::string> row);
+ [[nodiscard]] std::string toString() const;
+
+ private:
+ [[nodiscard]] std::vector<size_t> find_widths() const;
+
+ std::vector<std::string> header_;
+ std::vector<std::vector<std::string>> rows_;
+};
+
+std::string formatName(const std::string& name, bool is_required);
+std::string formatAllowableValues(const std::vector<org::apache::nifi::minifi::core::PropertyValue>& values);
+std::string formatDescription(std::string description, bool supports_expression_language = false);
+std::string formatSeparator(const std::vector<size_t>& widths);
+std::string formatRow(const std::vector<std::string>& items, const std::vector<size_t>& widths);
Review Comment:
fixed in 4a69170c4b1a1a7bb67d9348075d5828c88ac9af
I have only tested the `Table` class and not the free helper functions, because I am going to remove or hide those in the follow-up PR #1492.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a diff in pull request #1483: MINIFICPP-2019 Fix auto-generation of PROCESSORS.md
Posted by "fgerlits (via GitHub)" <gi...@apache.org>.
fgerlits commented on code in PR #1483:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1483#discussion_r1135684000
##########
PROCESSORS.md:
##########
@@ -249,165 +305,176 @@ In the list below, the names of required properties appear in bold. Any other pr
### Description
Compresses or decompresses the contents of FlowFiles using a user-specified compression algorithm and updates the mime.type attribute as appropriate
+
### Properties
In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
-| Name | Default Value | Allowable Values | Description |
-|--------------------|-------------------------|------------------|--------------------------------------------------------------------------------|
-| Compression Format | use mime.type attribute | | The compression format to use. |
-| Compression Level | 1 | | The compression level to use; this is valid only when using GZIP compression. |
-| Mode | compress | | Indicates whether the processor should compress content or decompress content. |
-| Update Filename | false | | Determines if filename extension need to be updated |
+| Name | Default Value | Allowable Values | Description |
+|--------------------|-------------------------|------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| Mode | compress | compress<br/>decompress | Indicates whether the processor should compress content or decompress content. |
+| Compression Level | 1 | | The compression level to use; this is valid only when using GZIP compression. |
+| Compression Format | use mime.type attribute | bzip2<br/>gzip<br/>lzma<br/>use mime.type attribute<br/>xz-lzma2 | The compression format to use. |
+| Update Filename | false | | Determines if filename extension need to be updated |
+| Encapsulate in TAR | true | | If true, on compression the FlowFile is added to a TAR archive and then compressed, and on decompression a compressed, TAR-encapsulated FlowFile is expected.<br/>If false, on compression the content of the FlowFile simply gets compressed, and on decompression a simple compressed content is expected.<br/>true is the behaviour compatible with older MiNiFi C++ versions, false is the behaviour compatible with NiFi. |
+| Batch Size | 1 | | Maximum number of FlowFiles processed in a single session |
+
### Relationships
| Name | Description |
|---------|---------------------------------------------------------------------------------------------------------------|
-| failure | FlowFiles will be transferred to the failure relationship if they fail to compress/decompress |
| success | FlowFiles will be transferred to the success relationship after successfully being compressed or decompressed |
+| failure | FlowFiles will be transferred to the failure relationship if they fail to compress/decompress |
## ConsumeJournald
+
### Description
-Consume systemd-journald journal messages. Available on Linux only.
+
+Consume systemd-journald journal messages. Creates one flow file per message. Fields are mapped to attributes. Realtime timestamp is mapped to the 'timestamp' attribute. Available on Linux only.
### Properties
-All properties are required with a default value, making them effectively optional. None of the properties support the NiFi Expression Language.
-| Name | Default Value | Allowable Values | Description |
-|----------------------|---------------|------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------|
-| Batch Size | 1000 | Positive numbers | The maximum number of entries processed in a single execution. |
-| Payload Format | Syslog | Raw<br>Syslog | Configures flow file content formatting.<br>Raw: only the message.<br>Syslog: similar to syslog or journalctl output. |
-| Include Timestamp | true | true<br>false | Include message timestamp in the 'timestamp' attribute. |
-| Journal Type | System | User<br>System<br>Both | Type of journal to consume. |
-| Process Old Messages | false | true<br>false | Process events created before the first usage (schedule) of the processor instance. |
-| Timestamp Format | %x %X %Z | [date format](https://howardhinnant.github.io/date/date.html#to_stream_formatting) | Format string to use when creating the timestamp attribute or writing messages in the syslog format. ISO/ISO 8601/ISO8601 are equivalent to "%FT%T%Ez". |
+In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
+
+| Name | Default Value | Allowable Values | Description |
+|--------------------------|---------------|--------------------------|-----------------------------------------------------------------------------------------------------------------|
+| **Batch Size** | 1000 | | The maximum number of entries processed in a single execution. |
+| **Payload Format** | Syslog | Raw<br/>Syslog | Configures flow file content formatting. Raw: only the message. Syslog: similar to syslog or journalctl output. |
+| **Include Timestamp** | true | | Include message timestamp in the 'timestamp' attribute. |
+| **Journal Type** | System | Both<br/>System<br/>User | Type of journal to consume. |
+| **Process Old Messages** | false | | Process events created before the first usage (schedule) of the processor instance. |
+| **Timestamp Format** | %x %X %Z | | Format string to use when creating the timestamp attribute or writing messages in the syslog format. |
### Relationships
-| Name | Description |
-|---------|--------------------------------|
-| success | Journal messages as flow files |
+| Name | Description |
+|---------|-----------------------------------------|
+| success | Successfully consumed journal messages. |
## ConsumeKafka
### Description
Consumes messages from Apache Kafka and transform them into MiNiFi FlowFiles. The application should make sure that the processor is triggered at regular intervals, even if no messages are expected, to serve any queued callbacks waiting to be called. Rebalancing can also only happen on trigger.
+
### Properties
In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
-| Name | Default Value | Allowable Values | Description |
-|------------------------------|----------------|--------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| Duplicate Header Handling | Keep Latest | Comma-separated Merge<br>Keep First<br>Keep Latest<br> | For headers to be added as attributes, this option specifies how to handle cases where multiple headers are present with the same key. For example in case of receiving these two headers: "Accept: text/html" and "Accept: application/xml" and we want to attach the value of "Accept" as a FlowFile attribute:<br/> - "Keep First" attaches: "Accept -> text/html"<br/> - "Keep Latest" attaches: "Accept -> application/xml"<br/> - "Comma-separated Merge" attaches: "Accept -> text/html, application/xml" |
-| **Group ID** | | | A Group ID is used to identify consumers that are within the same consumer group. Corresponds to Kafka's 'group.id' property.<br/>**Supports Expression Language: true** |
-| Headers To Add As Attributes | | | A comma separated list to match against all message headers. Any message header whose name matches an item from the list will be added to the FlowFile as an Attribute. If not specified, no Header values will be added as FlowFile attributes. The behaviour on when multiple headers of the same name are present is set using the DuplicateHeaderHandling attribute. |
-| **Honor Transactions** | true | | Specifies whether or not MiNiFi should honor transactional guarantees when communicating with Kafka. If false, the Processor will use an "isolation level" of read_uncomitted. This means that messages will be received as soon as they are written to Kafka but will be pulled, even if the producer cancels the transactions. If this value is true, MiNiFi will not receive any messages for which the producer's transaction was canceled, but this can result in some latency since the consumer must wait for the producer to finish its entire transaction instead of pulling as the messages become available. |
-| **Kafka Brokers** | localhost:9092 | | A comma-separated list of known Kafka Brokers in the format <host>:<port>.<br/>**Supports Expression Language: true** |
-| Kerberos Keytab Path | | | The path to the location on the local filesystem where the kerberos keytab is located. Read permission on the file is required. |
-| Kerberos Principal | | | Keberos Principal |
-| Kerberos Service Name | | | Kerberos Service Name |
-| **Key Attribute Encoding** | UTF-8 | Hex<br>UTF-8<br> | FlowFiles that are emitted have an attribute named 'kafka.key'. This property dictates how the value of the attribute should be encoded. |
-| Max Poll Records | 10000 | | Specifies the maximum number of records Kafka should return when polling each time the processor is triggered. |
-| **Max Poll Time** | 4 seconds | | Specifies the maximum amount of time the consumer can use for polling data from the brokers. Polling is a blocking operation, so the upper limit of this value is specified in 4 seconds. |
-| Message Demarcator | | | Since KafkaConsumer receives messages in batches, you have an option to output FlowFiles which contains all Kafka messages in a single batch for a given topic and partition and this property allows you to provide a string (interpreted as UTF-8) to use for demarcating apart multiple Kafka messages. This is an optional property and if not provided each Kafka message received will result in a single FlowFile which time it is triggered. <br/>**Supports Expression Language: true** |
-| Message Header Encoding | UTF-8 | Hex<br>UTF-8<br> | Any message header that is found on a Kafka message will be added to the outbound FlowFile as an attribute. This property indicates the Character Encoding to use for deserializing the headers. |
-| **Offset Reset** | latest | earliest<br>latest<br>none<br> | Allows you to manage the condition when there is no initial offset in Kafka or if the current offset does not exist any more on the server (e.g. because that data has been deleted). Corresponds to Kafka's 'auto.offset.reset' property. |
-| Password | | | The password for the given username when the SASL Mechanism is sasl_plaintext |
-| SASL Mechanism | GSSAPI | GSSAPI<br/>PLAIN | The SASL mechanism to use for authentication. Corresponds to Kafka's 'sasl.mechanism' property. |
-| **Security Protocol** | plaintext | plaintext<br/>ssl<br/>sasl_plaintext<br/>sasl_ssl | Protocol used to communicate with brokers. Corresponds to Kafka's 'security.protocol' property. |
-| Session Timeout | 60 seconds | | Client group session and failure detection timeout. The consumer sends periodic heartbeats to indicate its liveness to the broker. If no hearts are received by the broker for a group member within the session timeout, the broker will remove the consumer from the group and trigger a rebalance. The allowed range is configured with the broker configuration properties group.min.session.timeout.ms and group.max.session.timeout.ms. |
-| SSL Context Service | | | SSL Context Service Name |
-| **Topic Name Format** | Names | Names<br>Patterns<br> | Specifies whether the Topic(s) provided are a comma separated list of names or a single regular expression. Using regular expressions does not automatically discover Kafka topics created after the processor started. |
-| **Topic Names** | | | The name of the Kafka Topic(s) to pull from. Multiple topic names are supported as a comma separated list.<br/>**Supports Expression Language: true** |
-| Username | | | The username when the SASL Mechanism is sasl_plaintext |
-### Properties
+| Name | Default Value | Allowable Values | Description |
+|------------------------------|----------------|------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| SSL Context Service | | | SSL Context Service Name |
+| **Security Protocol** | plaintext | plaintext<br/>sasl_plaintext<br/>sasl_ssl<br/>ssl | Protocol used to communicate with brokers. Corresponds to Kafka's 'security.protocol' property. |
+| Kerberos Service Name | | | Kerberos Service Name |
+| Kerberos Principal | | | Keberos Principal |
+| Kerberos Keytab Path | | | The path to the location on the local filesystem where the kerberos keytab is located. Read permission on the file is required. |
+| **SASL Mechanism** | GSSAPI | GSSAPI<br/>PLAIN | The SASL mechanism to use for authentication. Corresponds to Kafka's 'sasl.mechanism' property. |
+| Username | | | The username when the SASL Mechanism is sasl_plaintext |
+| Password | | | The password for the given username when the SASL Mechanism is sasl_plaintext |
+| **Kafka Brokers** | localhost:9092 | | A comma-separated list of known Kafka Brokers in the format <host>:<port>.<br/>**Supports Expression Language: true** |
+| **Topic Names** | | | The name of the Kafka Topic(s) to pull from. Multiple topic names are supported as a comma separated list.<br/>**Supports Expression Language: true** |
+| **Topic Name Format** | Names | Names<br/>Patterns | Specifies whether the Topic(s) provided are a comma separated list of names or a single regular expression. Using regular expressions does not automatically discover Kafka topics created after the processor started. |
+| **Honor Transactions** | true | | Specifies whether or not MiNiFi should honor transactional guarantees when communicating with Kafka. If false, the Processor will use an "isolation level" of read_uncomitted. This means that messages will be received as soon as they are written to Kafka but will be pulled, even if the producer cancels the transactions. If this value is true, MiNiFi will not receive any messages for which the producer's transaction was canceled, but this can result in some latency since the consumer must wait for the producer to finish its entire transaction instead of pulling as the messages become available. |
+| **Group ID** | | | A Group ID is used to identify consumers that are within the same consumer group. Corresponds to Kafka's 'group.id' property.<br/>**Supports Expression Language: true** |
+| **Offset Reset** | latest | earliest<br/>latest<br/>none | Allows you to manage the condition when there is no initial offset in Kafka or if the current offset does not exist any more on the server (e.g. because that data has been deleted). Corresponds to Kafka's 'auto.offset.reset' property. |
+| **Key Attribute Encoding** | UTF-8 | Hex<br/>UTF-8 | FlowFiles that are emitted have an attribute named 'kafka.key'. This property dictates how the value of the attribute should be encoded. |
+| Message Demarcator | | | Since KafkaConsumer receives messages in batches, you have an option to output FlowFiles which contains all Kafka messages in a single batch for a given topic and partition and this property allows you to provide a string (interpreted as UTF-8) to use for demarcating apart multiple Kafka messages. This is an optional property and if not provided each Kafka message received will result in a single FlowFile which time it is triggered. <br/>**Supports Expression Language: true** |
+| Message Header Encoding | UTF-8 | Hex<br/>UTF-8 | Any message header that is found on a Kafka message will be added to the outbound FlowFile as an attribute. This property indicates the Character Encoding to use for deserializing the headers. |
+| Headers To Add As Attributes | | | A comma separated list to match against all message headers. Any message header whose name matches an item from the list will be added to the FlowFile as an Attribute. If not specified, no Header values will be added as FlowFile attributes. The behaviour on when multiple headers of the same name are present is set using the DuplicateHeaderHandling attribute. |
+| Duplicate Header Handling | Keep Latest | Comma-separated Merge<br/>Keep First<br/>Keep Latest | For headers to be added as attributes, this option specifies how to handle cases where multiple headers are present with the same key. For example in case of receiving these two headers: "Accept: text/html" and "Accept: application/xml" and we want to attach the value of "Accept" as a FlowFile attribute:<br/> - "Keep First" attaches: "Accept -> text/html"<br/> - "Keep Latest" attaches: "Accept -> application/xml"<br/> - "Comma-separated Merge" attaches: "Accept -> text/html, application/xml"<br/> |
+| Max Poll Records | 10000 | | Specifies the maximum number of records Kafka should return when polling each time the processor is triggered. |
+| **Max Poll Time** | 4 seconds | | Specifies the maximum amount of time the consumer can use for polling data from the brokers. Polling is a blocking operation, so the upper limit of this value is specified in 4 seconds. |
+| Session Timeout | 60 seconds | | Client group session and failure detection timeout. The consumer sends periodic heartbeats to indicate its liveness to the broker. If no hearts are received by the broker for a group member within the session timeout, the broker will remove the consumer from the group and trigger a rebalance. The allowed range is configured with the broker configuration properties group.min.session.timeout.ms and group.max.session.timeout.ms. |
+
+### Relationships
| Name | Description |
|---------|---------------------------------------------------------------------------------------------------------------------------------|
| success | Incoming Kafka messages as flowfiles. Depending on the demarcation strategy, this can be one or multiple flowfiles per message. |
+
## ConsumeMQTT
### Description
This Processor gets the contents of a FlowFile from a MQTT broker for a specified topic. The the payload of the MQTT message becomes content of a FlowFile
+
### Properties
In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
-| Name | Default Value | Allowable Values | Description |
-|-----------------------------|---------------|-----------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------|
-| **Broker URI** | | | The URI to use to connect to the MQTT broker |
-| Client ID | | | MQTT client ID to use. WARNING: Must not be empty when using MQTT 3.1.0! |
-| MQTT Version | 3.x AUTO | 3.x AUTO, 3.1.0, 3.1.1, 5.0 | The MQTT specification version when connecting to the broker. |
-| **Topic** | | | The topic to subscribe to. |
-| Clean Session | true | | Whether to start afresh rather than remembering previous subscriptions. If true, then make broker forget subscriptions after disconnected. MQTT 3.x only. |
-| Clean Start | true | | Whether to start afresh rather than remembering previous subscriptions. MQTT 5.x only. |
-| Session Expiry Interval | 0 s | | Time to delete session on broker after client is disconnected. MQTT 5.x only. |
-| Queue Max Message | 1000 | | Maximum number of messages allowed on the received MQTT queue |
-| Attribute From Content Type | | | Name of FlowFile attribute to be filled from content type of received message. MQTT 5.x only. |
-| Topic Alias Maximum | 0 | | Maximum number of topic aliases to use. If set to 0, then topic aliases cannot be used. MQTT 5.x only. |
-| Receive Maximum | 65535 | | Maximum number of unacknowledged messages allowed. MQTT 5.x only. |
-| Quality of Service | 0 | 0, 1 and 2 | The Quality of Service (QoS) to receive the message with. |
-| Connection Timeout | 10 sec | | Maximum time interval the client will wait for the network connection to the MQTT broker |
-| Keep Alive Interval | 60 sec | | Defines the maximum time interval between messages being received from the broker |
-| Last Will Topic | | | The topic to send the client's Last Will to. If the Last Will topic is not set then a Last Will will not be sent |
-| Last Will Message | | | The message to send as the client's Last Will. If the Last Will Message is empty, Last Will will be deleted from the broker |
-| Last Will QoS | 0 | 0, 1 and 2 | The Quality of Service (QoS) to send the last will with. |
-| Last Will Retain | false | | Whether to retain the client's Last Will |
-| Last Will Content Type | | | Content type of the client's Last Will. MQTT 5.x only. |
-| Username | | | Username to use when connecting to the broker |
-| Password | | | Password to use when connecting to the broker |
-| Security Protocol | | | Protocol used to communicate with brokers |
-| Security CA | | | File or directory path to CA certificate(s) for verifying the broker's key |
-| Security Cert | | | Path to client's public key (PEM) used for authentication |
-| Security Private Key | | | Path to client's private key (PEM) used for authentication |
-| Security Pass Phrase | | | Private key passphrase |
-
+| Name | Default Value | Allowable Values | Description |
+|-----------------------------|---------------|--------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------|
+| **Broker URI** | | | The URI to use to connect to the MQTT broker |
+| Client ID | | | MQTT client ID to use. WARNING: Must not be empty when using MQTT 3.1.0! |
+| MQTT Version | 3.x AUTO | 3.1.0<br/>3.1.1<br/>3.x AUTO<br/>5.0 | The MQTT specification version when connecting to the broker. |
+| **Topic** | | | The topic to subscribe to. |
+| Clean Session | true | | Whether to start afresh rather than remembering previous subscriptions. If true, then make broker forget subscriptions after disconnected. MQTT 3.x only. |
+| Clean Start | true | | Whether to start afresh rather than remembering previous subscriptions. MQTT 5.x only. |
+| Session Expiry Interval | 0 s | | Time to delete session on broker after client is disconnected. MQTT 5.x only. |
+| Queue Max Message | 1000 | | Maximum number of messages allowed on the received MQTT queue |
+| Attribute From Content Type | | | Name of FlowFile attribute to be filled from content type of received message. MQTT 5.x only. |
+| Topic Alias Maximum | 0 | | Maximum number of topic aliases to use. If set to 0, then topic aliases cannot be used. MQTT 5.x only. |
+| Receive Maximum | 65535 | | Maximum number of unacknowledged messages allowed. MQTT 5.x only. |
+| Quality of Service | 0 | 0<br/>1<br/>2 | The Quality of Service (QoS) of messages. |
+| Connection Timeout | 10 sec | | Maximum time interval the client will wait for the network connection to the MQTT broker |
+| Keep Alive Interval | 60 sec | | Defines the maximum time interval between messages sent or received |
+| Last Will Topic | | | The topic to send the client's Last Will to. If the Last Will topic is not set then a Last Will will not be sent |
+| Last Will Message | | | The message to send as the client's Last Will. If the Last Will Message is empty, Last Will will be deleted from the broker |
+| Last Will QoS | 0 | 0<br/>1<br/>2 | The Quality of Service (QoS) to send the last will with. |
+| Last Will Retain | false | | Whether to retain the client's Last Will |
+| Last Will Content Type | | | Content type of the client's Last Will. MQTT 5.x only. |
+| Username | | | Username to use when connecting to the broker |
+| Password | | | Password to use when connecting to the broker |
+| Security Protocol | | | Protocol used to communicate with brokers |
+| Security CA | | | File or directory path to CA certificate(s) for verifying the broker's key |
+| Security Cert | | | Path to client's public key (PEM) used for authentication |
+| Security Private Key | | | Path to client's private key (PEM) used for authentication |
+| Security Pass Phrase | | | Private key passphrase |
### Relationships
| Name | Description |
|---------|----------------------------------------------------------------------------------------------|
| success | FlowFiles that are sent successfully to the destination are transferred to this relationship |
-
### Output Attributes
| Attribute | Description |
|---------------|---------------------------|
| _mqtt.broker_ | URI of the sending broker |
| _mqtt.topic_ | Topic of the message |
+
## ConsumeWindowsEventLog
### Description
Windows Event Log Subscribe Callback to receive FlowFiles from Events on Windows.
+
### Properties
In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
-| Name | Default Value | Allowable Values | Description |
-|------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| **Channel** | System | | The Windows Event Log Channel to listen to.<br/>**Supports Expression Language: true** |
-| **Query** | * | | XPath Query to filter events. (See https://msdn.microsoft.com/en-us/library/windows/desktop/dd996910(v=vs.85).aspx for examples.)<br/>**Supports Expression Language: true** |
-| **Max Buffer Size** | 1 MB | | The individual Event Log XMLs are rendered to a buffer. This specifies the maximum size in bytes that the buffer will be allowed to grow to. (Limiting the maximum size of an individual Event XML.) |
-| **Inactive Duration To Reconnect** | 10 min | | If no new event logs are processed for the specified time period, this processor will try reconnecting to recover from a state where any further messages cannot be consumed. Such situation can happen if Windows Event Log service is restarted, or ERROR_EVT_QUERY_RESULT_STALE (15011) is returned. Setting no duration, e.g. '0 ms' disables auto-reconnection. |
-| Identifier Match Regex | .*Sid | | Regular Expression to match Subject Identifier Fields. These will be placed into the attributes of the FlowFile |
-| Apply Identifier Function | true | true<br>false | If true it will resolve SIDs matched in the 'Identifier Match Regex' to the DOMAIN\USERNAME associated with that ID |
-| Resolve Metadata in Attributes | true | true<br>false | If true, any metadata that is resolved ( such as IDs or keyword metadata ) will be placed into attributes, otherwise it will be replaced in the XML or text output |
-| Event Header Delimiter | | | If set, the chosen delimiter will be used in the Event output header. Otherwise, a colon followed by spaces will be used. |
-| Event Header | LOG_NAME=Log Name, SOURCE = Source, TIME_CREATED = Date,EVENT_RECORDID=Record ID,EVENTID = Event ID,TASK_CATEGORY = Task Category,LEVEL = Level,KEYWORDS = Keywords,USER = User,COMPUTER = Computer, EVENT_TYPE = EventType | | Comma seperated list of key/value pairs with the following keys LOG_NAME, SOURCE, TIME_CREATED,EVENT_RECORDID,EVENTID,TASK_CATEGORY,LEVEL,KEYWORDS,USER,COMPUTER, and EVENT_TYPE. Eliminating fields will remove them from the header. |
-| **Output Format** | Both | XML<br>Plaintext<br>JSON<br>Both | Set the output format type. In case 'Both' is selected the processor generates two flow files for every event captured in format XML and Plaintext |
-| **JSON Format** | Simple | Simple<br>Raw<br>Flattened | Set the json format type. Only applicable if Output Format is set to 'JSON' |
-| Batch Commit Size | 1000 | | Maximum number of Events to consume and create to Flow Files from before committing. |
-| **Process Old Events** | false | true<br>false | This property defines if old events (which are created before first time server is started) should be processed. |
-| State Directory | CWELState | | DEPRECATED. Only use it for state migration from the state file, supplying the legacy state directory. |
-| Cache SID Lookups | true | true<br>false | Determines whether SID to name lookups are cached in memory |
+| Name | Default Value | Allowable Values | Description |
+|------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| **Channel** | System | | The Windows Event Log Channel to listen to.<br/>**Supports Expression Language: true** |
+| **Query** | * | | XPath Query to filter events. (See https://msdn.microsoft.com/en-us/library/windows/desktop/dd996910(v=vs.85).aspx for examples.)<br/>**Supports Expression Language: true** |
+| **Max Buffer Size** | 1 MB | | The individual Event Log XMLs are rendered to a buffer. This specifies the maximum size in bytes that the buffer will be allowed to grow to. (Limiting the maximum size of an individual Event XML.) |
+| **Inactive Duration To Reconnect** | 10 min | | If no new event logs are processed for the specified time period, this processor will try reconnecting to recover from a state where any further messages cannot be consumed. Such situation can happen if Windows Event Log service is restarted, or ERROR_EVT_QUERY_RESULT_STALE (15011) is returned. Setting no duration, e.g. '0 ms' disables auto-reconnection. |
+| Identifier Match Regex | .*Sid | | Regular Expression to match Subject Identifier Fields. These will be placed into the attributes of the FlowFile |
+| Apply Identifier Function | true | | If true it will resolve SIDs matched in the 'Identifier Match Regex' to the DOMAIN\USERNAME associated with that ID |
+| Resolve Metadata in Attributes | true | | If true, any metadata that is resolved ( such as IDs or keyword metadata ) will be placed into attributes, otherwise it will be replaced in the XML or text output |
+| Event Header Delimiter | | | If set, the chosen delimiter will be used in the Event output header. Otherwise, a colon followed by spaces will be used. |
+| Event Header | LOG_NAME=Log Name, SOURCE = Source, TIME_CREATED = Date,EVENT_RECORDID=Record ID,EVENTID = Event ID,TASK_CATEGORY = Task Category,LEVEL = Level,KEYWORDS = Keywords,USER = User,COMPUTER = Computer, EVENT_TYPE = EventType | | Comma seperated list of key/value pairs with the following keys LOG_NAME, SOURCE, TIME_CREATED,EVENT_RECORDID,EVENTID,TASK_CATEGORY,LEVEL,KEYWORDS,USER,COMPUTER, and EVENT_TYPE. Eliminating fields will remove them from the header. |
+| **Output Format** | Both | Both<br/>JSON<br/>Plaintext<br/>XML | Set the output format type. In case 'Both' is selected the processor generates two flow files for every event captured in format XML and Plaintext |
+| **JSON Format** | Simple | Flattened<br/>Raw<br/>Simple | Set the json format type. Only applicable if Output Format is set to 'JSON' |
Review Comment:
I've reordered them in 065d45dd97a9124d4390b9c8bbecdf5219cb16d5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a diff in pull request #1483: MINIFICPP-2019 Fix auto-generation of PROCESSORS.md
Posted by "fgerlits (via GitHub)" <gi...@apache.org>.
fgerlits commented on code in PR #1483:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1483#discussion_r1101490310
##########
extensions/splunk/PutSplunkHTTP.h:
##########
@@ -40,7 +40,23 @@ class PutSplunkHTTP final : public SplunkHECProcessor {
PutSplunkHTTP& operator=(PutSplunkHTTP&&) = delete;
~PutSplunkHTTP() override = default;
- EXTENSIONAPI static constexpr const char* Description = "Sends the flow file contents to the specified Splunk HTTP Event Collector over HTTP or HTTPS. Supports HEC Index Acknowledgement.";
+ EXTENSIONAPI static constexpr const char* Description =
+ "Sends the flow file contents to the specified Splunk HTTP Event Collector (see https://docs.splunk.com/Documentation/SplunkCloud/latest/Data/UsetheHTTPEventCollector) over HTTP or HTTPS.\n"
+ "\n"
+ "The \"Source\", \"Source Type\", \"Host\" and \"Index\" properties are optional and will be set by Splunk if unspecified. If set,\n"
Review Comment:
I think it's useful if `PROCESSORS.md` can be read in both a markdown reader (eg. the GitHub UI) and as a plain text file (eg. in `less` or `vim`). Newlines are useful in the second case, and harmless in the first case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a diff in pull request #1483: MINIFICPP-2019 Fix auto-generation of PROCESSORS.md
Posted by "fgerlits (via GitHub)" <gi...@apache.org>.
fgerlits commented on code in PR #1483:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1483#discussion_r1135686813
##########
PROCESSORS.md:
##########
@@ -589,91 +688,97 @@ In the list below, the names of required properties appear in bold. Any other pr
### Description
-Executes a script given the flow file and a process session. The script is responsible for handling the incoming flow file (transfer to SUCCESS or remove, e.g.) as well as any flow files created by the script. If the handling is incomplete or incorrect, the session will be rolled back. Scripts must define an onTrigger function which accepts NiFi Context and Property objects. For efficiency, scripts are executed once when the processor is run, then the onTrigger method is called for each incoming flowfile. This enables scripts to keep state if they wish, although there will be a script context per concurrent task of the processor. In order to, e.g., compute an arithmetic sum based on incoming flow file information, set the concurrent tasks to 1. The python script files are expected to contain `describe(procesor)` and `onTrigger(context, session)`.
+Executes a script given the flow file and a process session. The script is responsible for handling the incoming flow file (transfer to SUCCESS or remove, e.g.) as well as any flow files created by the script. If the handling is incomplete or incorrect, the session will be rolled back.Scripts must define an onTrigger function which accepts NiFi Context and Property objects. For efficiency, scripts are executed once when the processor is run, then the onTrigger method is called for each incoming flowfile. This enables scripts to keep state if they wish, although there will be a script context per concurrent task of the processor. In order to, e.g., compute an arithmetic sum based on incoming flow file information, set the concurrent tasks to 1.
Review Comment:
fixed in dbd1c1a31b42d364b9e46f8f8b961b747b02f6c1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a diff in pull request #1483: MINIFICPP-2019 Fix auto-generation of PROCESSORS.md
Posted by "fgerlits (via GitHub)" <gi...@apache.org>.
fgerlits commented on code in PR #1483:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1483#discussion_r1135691120
##########
PROCESSORS.md:
##########
@@ -749,16 +885,15 @@ Fetches a file from a Google Cloud Bucket. Designed to be used in tandem with Li
In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
-| Name | Default Value | Allowable Values | Description |
-|--------------------------------------|---------------|-----------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------|
-| **Bucket** | ${gcs.bucket} | | Bucket of the object.<br>**Supports Expression Language: true** |
-| **Key** | ${filename} | | Name of the object.<br>**Supports Expression Language: true** |
-| **Number of retries** | 6 | integers | How many retry attempts should be made before routing to the failure relationship. |
-| **GCP Credentials Provider Service** | | [GCPCredentialsControllerService](CONTROLLERS.md#GCPCredentialsControllerService) | The Controller Service used to obtain Google Cloud Platform credentials. |
-| Server Side Encryption Key | | | An AES256 Encryption Key (encoded in base64) for server-side encryption of the object.<br>**Supports Expression Language: true** |
-| Object Generation | | | The generation of the Object to download. If left empty, then it will download the latest generation.<br>**Supports Expression Language: true** |
-| Endpoint Override URL | | | Overrides the default Google Cloud Storage endpoints |
-
+| Name | Default Value | Allowable Values | Description |
+|--------------------------------------|---------------|------------------|--------------------------------------------------------------------------------------------------------------------------------------------------|
+| **GCP Credentials Provider Service** | | | The Controller Service used to obtain Google Cloud Platform credentials. |
+| **Number of retries** | 6 | | How many retry attempts should be made before routing to the failure relationship. |
+| Endpoint Override URL | | | Overrides the default Google Cloud Storage endpoints<br/>**Supports Expression Language: true** |
+| Bucket | ${gcs.bucket} | | Bucket of the object.<br/>**Supports Expression Language: true** |
+| Key | ${filename} | | Name of the object.<br/>**Supports Expression Language: true** |
+| Server Side Encryption Key | | | The AES256 Encryption Key (encoded in base64) for server-side decryption of the object.<br/>**Supports Expression Language: true** |
Review Comment:
Since this is in `FetchGCSObject`, I think "decryption" is correct, and "encryption" was a typo.
@martinzink please correct me if I'm wrong.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez closed pull request #1483: MINIFICPP-2019 Fix auto-generation of PROCESSORS.md
Posted by "lordgamez (via GitHub)" <gi...@apache.org>.
lordgamez closed pull request #1483: MINIFICPP-2019 Fix auto-generation of PROCESSORS.md
URL: https://github.com/apache/nifi-minifi-cpp/pull/1483
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a diff in pull request #1483: MINIFICPP-2019 Fix auto-generation of PROCESSORS.md
Posted by "lordgamez (via GitHub)" <gi...@apache.org>.
lordgamez commented on code in PR #1483:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1483#discussion_r1102514118
##########
extensions/splunk/PutSplunkHTTP.h:
##########
@@ -40,7 +40,23 @@ class PutSplunkHTTP final : public SplunkHECProcessor {
PutSplunkHTTP& operator=(PutSplunkHTTP&&) = delete;
~PutSplunkHTTP() override = default;
- EXTENSIONAPI static constexpr const char* Description = "Sends the flow file contents to the specified Splunk HTTP Event Collector over HTTP or HTTPS. Supports HEC Index Acknowledgement.";
+ EXTENSIONAPI static constexpr const char* Description =
+ "Sends the flow file contents to the specified Splunk HTTP Event Collector (see https://docs.splunk.com/Documentation/SplunkCloud/latest/Data/UsetheHTTPEventCollector) over HTTP or HTTPS.\n"
+ "\n"
+ "The \"Source\", \"Source Type\", \"Host\" and \"Index\" properties are optional and will be set by Splunk if unspecified. If set,\n"
Review Comment:
Sounds reasonable, thanks for elaborating!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a diff in pull request #1483: MINIFICPP-2019 Fix auto-generation of PROCESSORS.md
Posted by "fgerlits (via GitHub)" <gi...@apache.org>.
fgerlits commented on code in PR #1483:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1483#discussion_r1101491092
##########
minifi_main/TableFormatter.h:
##########
@@ -0,0 +1,47 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "core/PropertyValue.h"
+
+namespace org::apache::nifi::minifi::docs {
+
+class Table {
+ public:
+ explicit Table(std::vector<std::string> header) : header_{std::move(header)} {}
+ void addRow(std::vector<std::string> row);
+ [[nodiscard]] std::string toString() const;
+
+ private:
+ [[nodiscard]] std::vector<size_t> find_widths() const;
Review Comment:
fixed in b617491a4791459a244daf8f8b2dcae30e29c475
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a diff in pull request #1483: MINIFICPP-2019 Fix auto-generation of PROCESSORS.md
Posted by "fgerlits (via GitHub)" <gi...@apache.org>.
fgerlits commented on code in PR #1483:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1483#discussion_r1135682353
##########
PROCESSORS.md:
##########
@@ -249,165 +305,176 @@ In the list below, the names of required properties appear in bold. Any other pr
### Description
Compresses or decompresses the contents of FlowFiles using a user-specified compression algorithm and updates the mime.type attribute as appropriate
+
### Properties
In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
-| Name | Default Value | Allowable Values | Description |
-|--------------------|-------------------------|------------------|--------------------------------------------------------------------------------|
-| Compression Format | use mime.type attribute | | The compression format to use. |
-| Compression Level | 1 | | The compression level to use; this is valid only when using GZIP compression. |
-| Mode | compress | | Indicates whether the processor should compress content or decompress content. |
-| Update Filename | false | | Determines if filename extension need to be updated |
+| Name | Default Value | Allowable Values | Description |
+|--------------------|-------------------------|------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| Mode | compress | compress<br/>decompress | Indicates whether the processor should compress content or decompress content. |
+| Compression Level | 1 | | The compression level to use; this is valid only when using GZIP compression. |
+| Compression Format | use mime.type attribute | bzip2<br/>gzip<br/>lzma<br/>use mime.type attribute<br/>xz-lzma2 | The compression format to use. |
+| Update Filename | false | | Determines if filename extension need to be updated |
+| Encapsulate in TAR | true | | If true, on compression the FlowFile is added to a TAR archive and then compressed, and on decompression a compressed, TAR-encapsulated FlowFile is expected.<br/>If false, on compression the content of the FlowFile simply gets compressed, and on decompression a simple compressed content is expected.<br/>true is the behaviour compatible with older MiNiFi C++ versions, false is the behaviour compatible with NiFi. |
+| Batch Size | 1 | | Maximum number of FlowFiles processed in a single session |
+
### Relationships
| Name | Description |
|---------|---------------------------------------------------------------------------------------------------------------|
-| failure | FlowFiles will be transferred to the failure relationship if they fail to compress/decompress |
| success | FlowFiles will be transferred to the success relationship after successfully being compressed or decompressed |
+| failure | FlowFiles will be transferred to the failure relationship if they fail to compress/decompress |
## ConsumeJournald
+
### Description
-Consume systemd-journald journal messages. Available on Linux only.
+
+Consume systemd-journald journal messages. Creates one flow file per message. Fields are mapped to attributes. Realtime timestamp is mapped to the 'timestamp' attribute. Available on Linux only.
### Properties
-All properties are required with a default value, making them effectively optional. None of the properties support the NiFi Expression Language.
-| Name | Default Value | Allowable Values | Description |
-|----------------------|---------------|------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------|
-| Batch Size | 1000 | Positive numbers | The maximum number of entries processed in a single execution. |
-| Payload Format | Syslog | Raw<br>Syslog | Configures flow file content formatting.<br>Raw: only the message.<br>Syslog: similar to syslog or journalctl output. |
-| Include Timestamp | true | true<br>false | Include message timestamp in the 'timestamp' attribute. |
-| Journal Type | System | User<br>System<br>Both | Type of journal to consume. |
-| Process Old Messages | false | true<br>false | Process events created before the first usage (schedule) of the processor instance. |
-| Timestamp Format | %x %X %Z | [date format](https://howardhinnant.github.io/date/date.html#to_stream_formatting) | Format string to use when creating the timestamp attribute or writing messages in the syslog format. ISO/ISO 8601/ISO8601 are equivalent to "%FT%T%Ez". |
+In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
+
+| Name | Default Value | Allowable Values | Description |
+|--------------------------|---------------|--------------------------|-----------------------------------------------------------------------------------------------------------------|
+| **Batch Size** | 1000 | | The maximum number of entries processed in a single execution. |
+| **Payload Format** | Syslog | Raw<br/>Syslog | Configures flow file content formatting. Raw: only the message. Syslog: similar to syslog or journalctl output. |
+| **Include Timestamp** | true | | Include message timestamp in the 'timestamp' attribute. |
+| **Journal Type** | System | Both<br/>System<br/>User | Type of journal to consume. |
+| **Process Old Messages** | false | | Process events created before the first usage (schedule) of the processor instance. |
+| **Timestamp Format** | %x %X %Z | | Format string to use when creating the timestamp attribute or writing messages in the syslog format. |
### Relationships
-| Name | Description |
-|---------|--------------------------------|
-| success | Journal messages as flow files |
+| Name | Description |
+|---------|-----------------------------------------|
+| success | Successfully consumed journal messages. |
## ConsumeKafka
### Description
Consumes messages from Apache Kafka and transform them into MiNiFi FlowFiles. The application should make sure that the processor is triggered at regular intervals, even if no messages are expected, to serve any queued callbacks waiting to be called. Rebalancing can also only happen on trigger.
+
### Properties
In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
-| Name | Default Value | Allowable Values | Description |
-|------------------------------|----------------|--------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| Duplicate Header Handling | Keep Latest | Comma-separated Merge<br>Keep First<br>Keep Latest<br> | For headers to be added as attributes, this option specifies how to handle cases where multiple headers are present with the same key. For example in case of receiving these two headers: "Accept: text/html" and "Accept: application/xml" and we want to attach the value of "Accept" as a FlowFile attribute:<br/> - "Keep First" attaches: "Accept -> text/html"<br/> - "Keep Latest" attaches: "Accept -> application/xml"<br/> - "Comma-separated Merge" attaches: "Accept -> text/html, application/xml" |
-| **Group ID** | | | A Group ID is used to identify consumers that are within the same consumer group. Corresponds to Kafka's 'group.id' property.<br/>**Supports Expression Language: true** |
-| Headers To Add As Attributes | | | A comma separated list to match against all message headers. Any message header whose name matches an item from the list will be added to the FlowFile as an Attribute. If not specified, no Header values will be added as FlowFile attributes. The behaviour on when multiple headers of the same name are present is set using the DuplicateHeaderHandling attribute. |
-| **Honor Transactions** | true | | Specifies whether or not MiNiFi should honor transactional guarantees when communicating with Kafka. If false, the Processor will use an "isolation level" of read_uncomitted. This means that messages will be received as soon as they are written to Kafka but will be pulled, even if the producer cancels the transactions. If this value is true, MiNiFi will not receive any messages for which the producer's transaction was canceled, but this can result in some latency since the consumer must wait for the producer to finish its entire transaction instead of pulling as the messages become available. |
-| **Kafka Brokers** | localhost:9092 | | A comma-separated list of known Kafka Brokers in the format <host>:<port>.<br/>**Supports Expression Language: true** |
-| Kerberos Keytab Path | | | The path to the location on the local filesystem where the kerberos keytab is located. Read permission on the file is required. |
-| Kerberos Principal | | | Keberos Principal |
-| Kerberos Service Name | | | Kerberos Service Name |
-| **Key Attribute Encoding** | UTF-8 | Hex<br>UTF-8<br> | FlowFiles that are emitted have an attribute named 'kafka.key'. This property dictates how the value of the attribute should be encoded. |
-| Max Poll Records | 10000 | | Specifies the maximum number of records Kafka should return when polling each time the processor is triggered. |
-| **Max Poll Time** | 4 seconds | | Specifies the maximum amount of time the consumer can use for polling data from the brokers. Polling is a blocking operation, so the upper limit of this value is specified in 4 seconds. |
-| Message Demarcator | | | Since KafkaConsumer receives messages in batches, you have an option to output FlowFiles which contains all Kafka messages in a single batch for a given topic and partition and this property allows you to provide a string (interpreted as UTF-8) to use for demarcating apart multiple Kafka messages. This is an optional property and if not provided each Kafka message received will result in a single FlowFile which time it is triggered. <br/>**Supports Expression Language: true** |
-| Message Header Encoding | UTF-8 | Hex<br>UTF-8<br> | Any message header that is found on a Kafka message will be added to the outbound FlowFile as an attribute. This property indicates the Character Encoding to use for deserializing the headers. |
-| **Offset Reset** | latest | earliest<br>latest<br>none<br> | Allows you to manage the condition when there is no initial offset in Kafka or if the current offset does not exist any more on the server (e.g. because that data has been deleted). Corresponds to Kafka's 'auto.offset.reset' property. |
-| Password | | | The password for the given username when the SASL Mechanism is sasl_plaintext |
-| SASL Mechanism | GSSAPI | GSSAPI<br/>PLAIN | The SASL mechanism to use for authentication. Corresponds to Kafka's 'sasl.mechanism' property. |
-| **Security Protocol** | plaintext | plaintext<br/>ssl<br/>sasl_plaintext<br/>sasl_ssl | Protocol used to communicate with brokers. Corresponds to Kafka's 'security.protocol' property. |
-| Session Timeout | 60 seconds | | Client group session and failure detection timeout. The consumer sends periodic heartbeats to indicate its liveness to the broker. If no hearts are received by the broker for a group member within the session timeout, the broker will remove the consumer from the group and trigger a rebalance. The allowed range is configured with the broker configuration properties group.min.session.timeout.ms and group.max.session.timeout.ms. |
-| SSL Context Service | | | SSL Context Service Name |
-| **Topic Name Format** | Names | Names<br>Patterns<br> | Specifies whether the Topic(s) provided are a comma separated list of names or a single regular expression. Using regular expressions does not automatically discover Kafka topics created after the processor started. |
-| **Topic Names** | | | The name of the Kafka Topic(s) to pull from. Multiple topic names are supported as a comma separated list.<br/>**Supports Expression Language: true** |
-| Username | | | The username when the SASL Mechanism is sasl_plaintext |
-### Properties
+| Name | Default Value | Allowable Values | Description |
+|------------------------------|----------------|------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| SSL Context Service | | | SSL Context Service Name |
+| **Security Protocol** | plaintext | plaintext<br/>sasl_plaintext<br/>sasl_ssl<br/>ssl | Protocol used to communicate with brokers. Corresponds to Kafka's 'security.protocol' property. |
+| Kerberos Service Name | | | Kerberos Service Name |
+| Kerberos Principal | | | Keberos Principal |
+| Kerberos Keytab Path | | | The path to the location on the local filesystem where the kerberos keytab is located. Read permission on the file is required. |
+| **SASL Mechanism** | GSSAPI | GSSAPI<br/>PLAIN | The SASL mechanism to use for authentication. Corresponds to Kafka's 'sasl.mechanism' property. |
+| Username | | | The username when the SASL Mechanism is sasl_plaintext |
+| Password | | | The password for the given username when the SASL Mechanism is sasl_plaintext |
+| **Kafka Brokers** | localhost:9092 | | A comma-separated list of known Kafka Brokers in the format <host>:<port>.<br/>**Supports Expression Language: true** |
+| **Topic Names** | | | The name of the Kafka Topic(s) to pull from. Multiple topic names are supported as a comma separated list.<br/>**Supports Expression Language: true** |
+| **Topic Name Format** | Names | Names<br/>Patterns | Specifies whether the Topic(s) provided are a comma separated list of names or a single regular expression. Using regular expressions does not automatically discover Kafka topics created after the processor started. |
+| **Honor Transactions** | true | | Specifies whether or not MiNiFi should honor transactional guarantees when communicating with Kafka. If false, the Processor will use an "isolation level" of read_uncomitted. This means that messages will be received as soon as they are written to Kafka but will be pulled, even if the producer cancels the transactions. If this value is true, MiNiFi will not receive any messages for which the producer's transaction was canceled, but this can result in some latency since the consumer must wait for the producer to finish its entire transaction instead of pulling as the messages become available. |
+| **Group ID** | | | A Group ID is used to identify consumers that are within the same consumer group. Corresponds to Kafka's 'group.id' property.<br/>**Supports Expression Language: true** |
+| **Offset Reset** | latest | earliest<br/>latest<br/>none | Allows you to manage the condition when there is no initial offset in Kafka or if the current offset does not exist any more on the server (e.g. because that data has been deleted). Corresponds to Kafka's 'auto.offset.reset' property. |
+| **Key Attribute Encoding** | UTF-8 | Hex<br/>UTF-8 | FlowFiles that are emitted have an attribute named 'kafka.key'. This property dictates how the value of the attribute should be encoded. |
+| Message Demarcator | | | Since KafkaConsumer receives messages in batches, you have an option to output FlowFiles which contains all Kafka messages in a single batch for a given topic and partition and this property allows you to provide a string (interpreted as UTF-8) to use for demarcating apart multiple Kafka messages. This is an optional property and if not provided each Kafka message received will result in a single FlowFile which time it is triggered. <br/>**Supports Expression Language: true** |
+| Message Header Encoding | UTF-8 | Hex<br/>UTF-8 | Any message header that is found on a Kafka message will be added to the outbound FlowFile as an attribute. This property indicates the Character Encoding to use for deserializing the headers. |
+| Headers To Add As Attributes | | | A comma separated list to match against all message headers. Any message header whose name matches an item from the list will be added to the FlowFile as an Attribute. If not specified, no Header values will be added as FlowFile attributes. The behaviour on when multiple headers of the same name are present is set using the DuplicateHeaderHandling attribute. |
Review Comment:
fixed in f48606ccce5a4ce7bcd3ddb73ed5eb013d03b5c8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a diff in pull request #1483: MINIFICPP-2019 Fix auto-generation of PROCESSORS.md
Posted by "fgerlits (via GitHub)" <gi...@apache.org>.
fgerlits commented on code in PR #1483:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1483#discussion_r1135681565
##########
PROCESSORS.md:
##########
@@ -249,165 +305,176 @@ In the list below, the names of required properties appear in bold. Any other pr
### Description
Compresses or decompresses the contents of FlowFiles using a user-specified compression algorithm and updates the mime.type attribute as appropriate
+
### Properties
In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
-| Name | Default Value | Allowable Values | Description |
-|--------------------|-------------------------|------------------|--------------------------------------------------------------------------------|
-| Compression Format | use mime.type attribute | | The compression format to use. |
-| Compression Level | 1 | | The compression level to use; this is valid only when using GZIP compression. |
-| Mode | compress | | Indicates whether the processor should compress content or decompress content. |
-| Update Filename | false | | Determines if filename extension need to be updated |
+| Name | Default Value | Allowable Values | Description |
+|--------------------|-------------------------|------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| Mode | compress | compress<br/>decompress | Indicates whether the processor should compress content or decompress content. |
+| Compression Level | 1 | | The compression level to use; this is valid only when using GZIP compression. |
+| Compression Format | use mime.type attribute | bzip2<br/>gzip<br/>lzma<br/>use mime.type attribute<br/>xz-lzma2 | The compression format to use. |
+| Update Filename | false | | Determines if filename extension need to be updated |
+| Encapsulate in TAR | true | | If true, on compression the FlowFile is added to a TAR archive and then compressed, and on decompression a compressed, TAR-encapsulated FlowFile is expected.<br/>If false, on compression the content of the FlowFile simply gets compressed, and on decompression a simple compressed content is expected.<br/>true is the behaviour compatible with older MiNiFi C++ versions, false is the behaviour compatible with NiFi. |
+| Batch Size | 1 | | Maximum number of FlowFiles processed in a single session |
+
### Relationships
| Name | Description |
|---------|---------------------------------------------------------------------------------------------------------------|
-| failure | FlowFiles will be transferred to the failure relationship if they fail to compress/decompress |
| success | FlowFiles will be transferred to the success relationship after successfully being compressed or decompressed |
+| failure | FlowFiles will be transferred to the failure relationship if they fail to compress/decompress |
## ConsumeJournald
+
### Description
-Consume systemd-journald journal messages. Available on Linux only.
+
+Consume systemd-journald journal messages. Creates one flow file per message. Fields are mapped to attributes. Realtime timestamp is mapped to the 'timestamp' attribute. Available on Linux only.
### Properties
-All properties are required with a default value, making them effectively optional. None of the properties support the NiFi Expression Language.
-| Name | Default Value | Allowable Values | Description |
-|----------------------|---------------|------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------|
-| Batch Size | 1000 | Positive numbers | The maximum number of entries processed in a single execution. |
-| Payload Format | Syslog | Raw<br>Syslog | Configures flow file content formatting.<br>Raw: only the message.<br>Syslog: similar to syslog or journalctl output. |
-| Include Timestamp | true | true<br>false | Include message timestamp in the 'timestamp' attribute. |
-| Journal Type | System | User<br>System<br>Both | Type of journal to consume. |
-| Process Old Messages | false | true<br>false | Process events created before the first usage (schedule) of the processor instance. |
-| Timestamp Format | %x %X %Z | [date format](https://howardhinnant.github.io/date/date.html#to_stream_formatting) | Format string to use when creating the timestamp attribute or writing messages in the syslog format. ISO/ISO 8601/ISO8601 are equivalent to "%FT%T%Ez". |
+In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
+
+| Name | Default Value | Allowable Values | Description |
+|--------------------------|---------------|--------------------------|-----------------------------------------------------------------------------------------------------------------|
+| **Batch Size** | 1000 | | The maximum number of entries processed in a single execution. |
+| **Payload Format** | Syslog | Raw<br/>Syslog | Configures flow file content formatting. Raw: only the message. Syslog: similar to syslog or journalctl output. |
+| **Include Timestamp** | true | | Include message timestamp in the 'timestamp' attribute. |
+| **Journal Type** | System | Both<br/>System<br/>User | Type of journal to consume. |
+| **Process Old Messages** | false | | Process events created before the first usage (schedule) of the processor instance. |
+| **Timestamp Format** | %x %X %Z | | Format string to use when creating the timestamp attribute or writing messages in the syslog format. |
Review Comment:
fixed in 0864076cedae2ffd5df44bf6a430c93c35d3a24f
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org