You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@nifi.apache.org by sz...@apache.org on 2024/04/15 17:05:14 UTC
(nifi-minifi-cpp) 01/05: MINIFICPP-2309 Add JSON flow config examples
This is an automated email from the ASF dual-hosted git repository.
szaszm pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit 45971b514528337ce9f6ed8928442bef0cef85ab
Author: Gabor Gyimesi <ga...@gmail.com>
AuthorDate: Mon Apr 15 18:24:45 2024 +0200
MINIFICPP-2309 Add JSON flow config examples
- Also add a few new use case examples
Closes #1740
Signed-off-by: Marton Szasz <sz...@apache.org>
---
examples/README.md | 110 ++--
examples/azure_storage_config.json | 52 ++
examples/azure_storage_config.nifi.schema.json | 254 ++++++++
examples/consumejournald_config.json | 49 ++
examples/consumejournald_config.nifi.schema.json | 213 +++++++
examples/consumekafka_config.json | 59 ++
examples/cwel_config.json | 54 ++
examples/getfile_putfile_config.json | 54 ++
examples/grafana_loki_config.json | 52 ++
...tailfile_config.yml => grafana_loki_config.yml} | 36 +-
examples/http_post_config.json | 142 +++++
examples/http_post_config.nifi.schema.json | 647 +++++++++++++++++++++
examples/http_post_config.yml | 4 +-
examples/kubernetes_tailfile_config.json | 72 +++
examples/lists3_fetchs3_config.json | 85 +++
examples/lists3_fetchs3_config.nifi.schema.json | 492 ++++++++++++++++
.../merge_compress_and_upload_to_gcs_config.json | 157 +++++
.../merge_compress_and_upload_to_gcs_config.yml | 116 ++++
examples/mqtt_config.json | 52 ++
examples/mqtt_config.yml | 32 +
examples/pdh_config.json | 53 ++
examples/process_data_with_scripts.json | 105 ++++
examples/publishkafka_config.json | 55 ++
examples/publishkafka_ssl_config.json | 60 ++
examples/publishkafka_ssl_config.yml | 3 +-
examples/puts3_config.json | 51 ++
examples/querydbtable_config.json | 61 ++
examples/querydbtable_config.nifi.schema.json | 245 ++++++++
examples/querydbtable_config.yml | 1 -
examples/site_to_site_config.json | 49 ++
examples/site_to_site_config.nifi.schema.json | 195 +++++++
examples/splittext_puttcp_config.json | 84 +++
examples/splittext_puttcp_config.yml | 53 ++
examples/tailfile_config.json | 50 ++
examples/tailfile_config.yml | 3 +-
35 files changed, 3741 insertions(+), 59 deletions(-)
diff --git a/examples/README.md b/examples/README.md
index 788bf47e8..07d7a2454 100644
--- a/examples/README.md
+++ b/examples/README.md
@@ -14,7 +14,13 @@
-->
# Apache NiFi - MiNiFi - C++ Examples
-The following examples show simple flow configurations for several common use cases. Use the path of these yaml files as the flow configuration set in the `nifi.flow.configuration.file` property of `conf/minifi.properties` file or replace the default `conf/config.yml` file to try them out.
+The following examples show simple flow configurations for several common use cases. Apache MiNiFi C++ supports flow configurations using yaml and json formats. Use the path of these json and yaml files as the flow configuration set in the `nifi.flow.configuration.file` property of `conf/minifi.properties` file or replace the default `conf/config.yml` file to try them out.
+
+For the json flow configuration format there are two supported schemas:
+ - The default schema mimics the yaml configuration format using json syntax and json naming conventions. See the examples without the `nifi.schema.json` suffix.
+ - The NiFi schema mimics NiFi's json flow configuration format, having some additional json properties added to the default schema from NiFi's json flow configuration format. See the examples with the `nifi.schema.json` suffix.
+
+The json schema can be generated using the MiNiFi C++ binary with the `--schema` option. The generated schema can be used to validate the json flow configuration files.
- [Filesystem Operations](#filesystem-operations)
- [Getting File Data and Putting It in an Output Directory](#getting-file-data-and-putting-it-in-an-output-directory)
@@ -37,61 +43,67 @@ The following examples show simple flow configurations for several common use ca
- [Upload Blob to Azure Storage](#upload-blob-to-azure-storage)
- [Put Object in AWS S3 Bucket](#put-object-in-aws-s3-bucket)
- [List and Fetch Content from AWS S3 Bucket](#list-and-fetch-content-from-aws-s3-bucket)
+ - [Merge, compress and upload files to Google Cloud Storage](#merge-compress-and-upload-files-to-google-cloud-storage)
- [SQL Operations](#sql-operations)
- [Query Database Table](#query-database-table)
-- [ExecuteScript](#ExecuteScript)
- - [Reverse flowfile's content with lua](#reverse-content-with-lua)
- - [Reverse flowfile's content with python](#reverse-content-with-python)
+- [ExecuteScript](#executescript)
+ - [Reverse Content with Scripts](#reverse-content-with-scripts)
+- [Grafana Loki](#grafana-loki)
+ - [Send logs to Grafana Loki](#send-logs-to-grafana-loki)
+- [MQTT Operations](#mqtt-operations)
+ - [Publish Message to MQTT Broker](#publish-message-to-mqtt-broker)
+- [Network Operations](#network-operations)
+ - [Send Data to Remote Host through TCP](#send-data-to-remote-host-through-tcp)
## Filesystem Operations
### Getting File Data and Putting It in an Output Directory
-Using the [getfile_putfile_config.yml](getfile_putfile_config.yml) flow configuration MiNiFi gets all files of minimum 1MB size from the `/tmp/getfile_dir` directory and puts them in the `/tmp/out_dir` output directory.
+Using the [getfile_putfile_config.yml](getfile_putfile_config.yml)/[getfile_putfile_config.json](getfile_putfile_config.json) flow configuration MiNiFi gets all files of minimum 1MB size from the `/tmp/getfile_dir` directory and puts them in the `/tmp/out_dir` output directory.
-The flow: GetFile ➔ success ➔ PutFile
+The flow: [GetFile](../PROCESSORS.md#getfile) ➔ [PutFile](../PROCESSORS.md#putfile)
### Tailing a Single File
-Using the [tailfile_config.yml](tailfile_config.yml) flow configuration MiNiFi tails a single file `/tmp/test_file.log` and creates flowfiles from every single line, then logs the flowfile attributes.
+Using the [tailfile_config.yml](tailfile_config.yml)/[tailfile_config.json](tailfile_config.json) flow configuration MiNiFi tails a single file `/tmp/test_file.log` and creates flowfiles from every single line, then logs the flowfile attributes.
-The flow: TailFile ➔ success ➔ LogAttribute
+The flow: [TailFile](../PROCESSORS.md#tailfile) ➔ [LogAttribute](../PROCESSORS.md#logattribute)
## Windows Specific Processors
### Consuming Windows Event Logs
-Using the [cwel_config.yml](cwel_config.yml) flow configuration MiNiFi queries all Windows system events and puts them to the `C:\temp\` directory in flattened JSON format.
+Using the [cwel_config.yml](cwel_config.yml)/[cwel_config.json](cwel_config.json) flow configuration MiNiFi queries all Windows system events and puts them to the `C:\temp\` directory in flattened JSON format.
-The flow: ConsumeWindowsEventLog ➔ PutFile
+The flow: [ConsumeWindowsEventLog](../PROCESSORS.md#consumewindowseventlog) ➔ [PutFile](../PROCESSORS.md#putfile)
### Reading System Performance Data
-Using the [pdh_config.yml](pdh_config.yml) flow configuration MiNiFi reads CPU and Disk performance data through Windows' Performance Data Helper (PDH) component and puts the data to the `C:\temp\` directory in a compact JSON format.
+Using the [pdh_config.yml](pdh_config.yml)/[pdh_config.json](pdh_config.json) flow configuration MiNiFi reads CPU and Disk performance data through Windows' Performance Data Helper (PDH) component and puts the data to the `C:\temp\` directory in a compact JSON format.
-The flow: PerformanceDataMonitor ➔ PutFile
+The flow: [PerformanceDataMonitor](../PROCESSORS.md#performancedatamonitor) ➔ [PutFile](../PROCESSORS.md#putfile)
## Linux Specific Processors
### Consume Systemd-Journald System Journal Messages
-Using the [consumejournald_config.yml](consumejournald_config.yml) flow configuration MiNiFi reads systemd-journald journal messages and logs them on `info` level.
+Using the [consumejournald_config.yml](consumejournald_config.yml)/[consumejournald_config.json](consumejournald_config.json)/[consumejournald_config.nifi.schema.json](consumejournald_config.nifi.schema.json) flow configuration MiNiFi reads systemd-journald journal messages and logs them on `info` level.
-The flow: ConsumeJournald ➔ LogAttribute
+The flow: [ConsumeJournald](../PROCESSORS.md#consumejournald) ➔ [LogAttribute](../PROCESSORS.md#logattribute)
## HTTP Operations
### HTTP POST Invocation
-Using the [http_post_config.yml](http_post_config.yml) flow configuration MiNiFi transfers flowfile data received from the GetFile processor by invoking an HTTP endpoint with POST method.
+Using the [http_post_config.yml](http_post_config.yml)/[http_post_config.json](http_post_config.json)/[http_post_config.nifi.schema.json](http_post_config.nifi.schema.json) flow configuration MiNiFi transfers flowfile data received from the GetFile processor by invoking an HTTP endpoint with POST method.
-The flow: GetFile ➔ success ➔ InvokeHTTP
+The flow: [GetFile](../PROCESSORS.md#getfile) ➔ [InvokeHTTP](../PROCESSORS.md#invokehttp)
## Site to Site Operations
### Transfer Data to Remote Nifi Instance
-Using the [site_to_site_config.yml](site_to_site_config.yml) flow configuration MiNiFi transfers data received from the GetFile processor to a remote NiFi instance located at `http://nifi:8080/nifi`.
+Using the [site_to_site_config.yml](site_to_site_config.yml)/[site_to_site_config.json](site_to_site_config.json)/[site_to_site_config.nifi.schema.json](site_to_site_config.nifi.schema.json) flow configuration MiNiFi transfers data received from the GetFile processor to a remote NiFi instance located at `http://nifi:8080/nifi`.
### [Site-2-Site Bi-directional Configuration](BidirectionalSiteToSite/README.md)
@@ -99,49 +111,55 @@ Using the [site_to_site_config.yml](site_to_site_config.yml) flow configuration
### Publish Message to Kafka Broker
-Using the [publishkafka_config.yml](publishkafka_config.yml) flow configuration MiNiFi publishes data received from the GetFile processor to a configured Kafka broker's `test` topic.
+Using the [publishkafka_config.yml](publishkafka_config.yml)/[publishkafka_config.json](publishkafka_config.json) flow configuration MiNiFi publishes data received from the GetFile processor to a configured Kafka broker's `test` topic.
-The flow: GetFile ➔ success ➔ PublishKafka
+The flow: [GetFile](../PROCESSORS.md#getfile) ➔ [PublishKafka](../PROCESSORS.md#publishkafka)
### Publish Message to Kafka Broker Through SSL
-Using the [publishkafka_ssl_config.yml](publishkafka_ssl_config.yml) flow configuration MiNiFi publishes data received from the GetFile processor to a configured Kafka broker's `test` topic through SSL connection.
+Using the [publishkafka_ssl_config.yml](publishkafka_ssl_config.yml)/[publishkafka_ssl_config.json](publishkafka_ssl_config.json) flow configuration MiNiFi publishes data received from the GetFile processor to a configured Kafka broker's `test` topic through SSL connection.
-The flow: GetFile ➔ success ➔ PublishKafka
+The flow: [GetFile](../PROCESSORS.md#getfile) ➔ [PublishKafka](../PROCESSORS.md#publishkafka)
### Consume Messages from Kafka
-Using the [consumekafka_config.yml](consumekafka_config.yml) flow configuration MiNiFi consumes messages from the configured Kafka broker's `ConsumeKafkaTest` topic from the earliest available message. The messages are forwarded to the `PutFile` processor and put in the `/tmp/output` directory.
+Using the [consumekafka_config.yml](consumekafka_config.yml)/[consumekafka_config.json](consumekafka_config.json) flow configuration MiNiFi consumes messages from the configured Kafka broker's `ConsumeKafkaTest` topic from the earliest available message. The messages are forwarded to the `PutFile` processor and put in the `/tmp/output` directory.
-The flow: ConsumeKafka ➔ success ➔ PutFile
+The flow: [ConsumeKafka](../PROCESSORS.md#consumekafka) ➔ [PutFile](../PROCESSORS.md#putfile)
## Public Cloud Operations
### Upload Blob to Azure Storage
-Using the [azure_storage_config.yml](azure_storage_config.yml) flow configuration MiNiFi uploads data received from the GetFile processor to Azure's blob storage container `test-container`.
+Using the [azure_storage_config.yml](azure_storage_config.yml)/[azure_storage_config.json](azure_storage_config.json)/[azure_storage_config.nifi.schema.json](azure_storage_config.nifi.schema.json) flow configuration MiNiFi uploads data received from the GetFile processor to Azure's blob storage container `test-container`.
-The flow: GetFile ➔ success ➔ PutAzureBlobStorage
+The flow: [GetFile](../PROCESSORS.md#getfile) ➔ [PutAzureBlobStorage](../PROCESSORS.md#putAzureblobstorage)
### Put Object in AWS S3 Bucket
-Using the [puts3_config.yml](puts3_config.yml) flow configuration MiNiFi uploads data received from the GetFile processor to AWS S3 bucket `test_bucket`.
+Using the [puts3_config.yml](puts3_config.yml)/[puts3_config.json](puts3_config.json) flow configuration MiNiFi uploads data received from the GetFile processor to AWS S3 bucket `test_bucket`.
-The flow: GetFile ➔ success ➔ PutS3Object
+The flow: [GetFile](../PROCESSORS.md#getfile) ➔ [PutS3Object](../PROCESSORS.md#puts3object)
### List and Fetch Content from AWS S3 Bucket
-Using the [lists3_fetchs3_config.yml](lists3_fetchs3_config.yml) flow configuration MiNiFi lists S3 bucket `test_bucket` and fetches its contents in flowfiles then logs the attributes. The flow uses `AWSCredentialsService` controller service to provide credentials for all S3 processors. It has `Use Default Credentials` property set which retrieves credentials from AWS default credentials provider chain (environment variables, configuration file, instance profile).
+Using the [lists3_fetchs3_config.yml](lists3_fetchs3_config.yml)/[lists3_fetchs3_config.json](lists3_fetchs3_config.json)/[lists3_fetchs3_config.nifi.schema.json](lists3_fetchs3_config.nifi.schema.json) flow configuration MiNiFi lists S3 bucket `test_bucket` and fetches its contents in flowfiles then logs the attributes. The flow uses `AWSCredentialsService` controller service to provide credentials for all S3 processors. It has `Use Default Credentials` property set which retrieves cred [...]
+
+The flow: [ListS3](../PROCESSORS.md#lists3) ➔ [FetchS3Object](../PROCESSORS.md#fetchs3object) ➔ [LogAttribute](../PROCESSORS.md#logattribute)
+
+### Merge, compress and upload files to Google Cloud Storage
+
+Using the [merge_compress_and_upload_to_gcs_config.yml](merge_compress_and_upload_to_gcs_config.yml)/[merge_compress_and_upload_to_gcs_config.json](merge_compress_and_upload_to_gcs_config.json) flow configuration MiNiFi tails a file, creates a flow file with an added `google_cloud_storage` attrbute from every new line, then merges every 10 lines, compresses the merged content in gzip format, finally uploads it to Google Cloud Storage. The flow uses `GCSStorageService` controller service [...]
-The flow: ListS3 ➔ FetchS3Object ➔ LogAttribute
+The flow: [TailFile](../PROCESSORS.md#tailfile) ➔ [UpdateAttribute](../PROCESSORS.md#updateattribute) ➔ [MergeContent](../PROCESSORS.md#mergecontent) ➔ [CompressContent](../PROCESSORS.md#compresscontent) ➔ [PutGCSObject](../PROCESSORS.md#putgcsobject)
## SQL Operations
### Query Database Table
-Using the [querydbtable_config.yml](querydbtable_config.yml) flow configuration MiNiFi queries the `id` and `name` columns of the `users` table with a `where` clause and the results are put in the `/tmp/output` directory. The database connection data is set in the `ODBCService` controller service.
+Using the [querydbtable_config.yml](querydbtable_config.yml)/[querydbtable_config.json](querydbtable_config.json)/[querydbtable_config.nifi.schema.json](querydbtable_config.nifi.schema.json) flow configuration MiNiFi queries the `id` and `name` columns of the `users` table with a `where` clause and the results are put in the `/tmp/output` directory. The database connection data is set in the `ODBCService` controller service.
-The flow: QueryDatabaseTable ➔ PutFile
+The flow: [QueryDatabaseTable](../PROCESSORS.md#querydatabasetable) ➔ [PutFile](../PROCESSORS.md#putfile)
## ExecuteScript
@@ -149,10 +167,34 @@ ExecuteScript supports [Lua](https://www.lua.org/) and [Python](https://www.pyth
### Reverse Content with Scripts
-Using the [process_data_with_scripts.yml](process_data_with_scripts.yml) flow configuration MiNiFi generates a flowfile
-then reverses its content with [reverse_flow_file_content.py](scripts/python/reverse_flow_file_content.py) or [reverse_flow_file_content.lua](scripts/lua/reverse_flow_file_content.lua)
+Using the [process_data_with_scripts.yml](process_data_with_scripts.yml)/[process_data_with_scripts.json](process_data_with_scripts.json) flow configuration MiNiFi generates a flowfile
+then reverses its content with [reverse_flow_file_content.py](scripts/python/reverse_flow_file_content.py) or [reverse_flow_file_content.lua](scripts/lua/reverse_flow_file_content.lua)
then writes the result to _./reversed_flow_files/_
-The flow: [GenerateFlowFile](../PROCESSORS.md#generateflowfile) ➔ [ExecuteScript](../PROCESSORS.md#executescript) ➔ [PutFile](../PROCESSORS.md#putfile)
+The flow: [GenerateFlowFile](../PROCESSORS.md#generateflowfile) ➔ [ExecuteScript](../PROCESSORS.md#executescript) ➔ [PutFile](../PROCESSORS.md#putfile)
Additional script examples can be found [here](scripts/README.md).
+
+## Grafana Loki
+
+### Send logs to Grafana Loki
+
+Using the [grafana_loki_config.yml](grafana_loki_config.yml)/[grafana_loki_config.json](grafana_loki_config.json) flow configuration MiNiFi tails a log file and sends the log lines to the Grafana Loki server configured on the localhost on the port 3100, with the job=minifi and id=logs labels using Grafana Loki's REST API.
+
+The flow: [TailFile](../PROCESSORS.md#tailfile) ➔ [PushGrafanaLokiREST](../PROCESSORS.md#pushgrafanalokirest)
+
+## MQTT Operations
+
+### Publish Message to MQTT Broker
+
+Using the [mqtt_config.yml](mqtt_config.yml)/[mqtt_config.json](mqtt_config.json) flow configuration MiNiFi publishes the data from the files located under the `/tmp/input` directory to the `testtopic` topic of the MQTT broker configured on localhost on the port 1883.
+
+The flow: [GetFile](../PROCESSORS.md#getfile) ➔ [PublishMQTT](../PROCESSORS.md#publishmqtt)
+
+## Network Operations
+
+### Send Data to Remote Host through TCP
+
+Using the [splittext_puttcp_config.yml](splittext_puttcp_config.yml)/[splittext_puttcp_config.json](splittext_puttcp_config.json) flow configuration MiNiFi splits the content of the files located under the `/tmp/input` directory into separate lines, while skipping the first 3 header lines, and sends them to the remote address 192.168.1.5 on the port 8081 through TCP.
+
+The flow: [GetFile](../PROCESSORS.md#getfile) ➔ [SplitText](../PROCESSORS.md#splittext) ➔ [PutTCP](../PROCESSORS.md#puttcp)
diff --git a/examples/azure_storage_config.json b/examples/azure_storage_config.json
new file mode 100644
index 000000000..6e895df03
--- /dev/null
+++ b/examples/azure_storage_config.json
@@ -0,0 +1,52 @@
+{
+ "rootGroup": {
+ "name": "MiNiFi Flow",
+ "processors": [
+ {
+ "name": "Get files from /tmp/input",
+ "identifier": "d5fef466-fef0-4d90-814e-9c19a67e454a",
+ "type": "org.apache.nifi.processors.standard.GetFile",
+ "schedulingStrategy": "TIMER_DRIVEN",
+ "schedulingPeriod": "2 sec",
+ "properties": {
+ "Input Directory": "/tmp/input"
+ },
+ "autoTerminatedRelationships": []
+ },
+ {
+ "name": "Upload Blob to test-container in Azure Storage",
+ "identifier": "e7b3f9c6-d1ee-4d9b-b676-48c872cc7ae3",
+ "type": "org.apache.nifi.processors.standard.PutAzureBlobStorage",
+ "schedulingStrategy": "EVENT_DRIVEN",
+ "properties": {
+ "Blob": "test-blob",
+ "Storage Account Name": "test-account",
+ "Storage Account Key": "test-key",
+ "Container Name": "test-container",
+ "Create Container": "true"
+ },
+ "autoTerminatedRelationships": [
+ "success",
+ "failure"
+ ]
+ }
+ ],
+ "connections": [
+ {
+ "identifier": "781aa97f-534f-416a-985c-dc046e2a527a",
+ "name": "GetFile/success/PutAzureBlobStorage",
+ "source": {
+ "id": "d5fef466-fef0-4d90-814e-9c19a67e454a"
+ },
+ "destination": {
+ "id": "e7b3f9c6-d1ee-4d9b-b676-48c872cc7ae3"
+ },
+ "selectedRelationships": [
+ "success"
+ ]
+ }
+ ],
+ "remoteProcessGroups": [],
+ "controllerServices": []
+ }
+}
diff --git a/examples/azure_storage_config.nifi.schema.json b/examples/azure_storage_config.nifi.schema.json
new file mode 100644
index 000000000..3f7caf19f
--- /dev/null
+++ b/examples/azure_storage_config.nifi.schema.json
@@ -0,0 +1,254 @@
+{
+ "encodingVersion": {
+ "majorVersion": 2,
+ "minorVersion": 0
+ },
+ "maxTimerDrivenThreadCount": 1,
+ "maxEventDrivenThreadCount": 1,
+ "parameterContexts": [],
+ "rootGroup": {
+ "identifier": "1c2f34d2-ee57-448a-b150-8140f0f7643c",
+ "instanceIdentifier": "9af45830-320e-437e-8204-dcd3c2933aa6",
+ "name": "root",
+ "position": {
+ "x": 0.0,
+ "y": 0.0
+ },
+ "processGroups": [],
+ "remoteProcessGroups": [],
+ "processors": [
+ {
+ "identifier": "af053760-caa2-4624-b70e-0ef8a802864f",
+ "instanceIdentifier": "35196325-c33f-486e-ba36-646d66ee3369",
+ "name": "Upload Blob to test-container in Azure Storage",
+ "comments": "",
+ "position": {
+ "x": 1101.0,
+ "y": 257.0
+ },
+ "type": "org.apache.nifi.minifi.azure.processors.PutAzureBlobStorage",
+ "bundle": {
+ "group": "org.apache.nifi.minifi",
+ "artifact": "minifi-azure",
+ "version": "0.15.0"
+ },
+ "properties": {
+ "Blob": "test-blob",
+ "Storage Account Name": "test-account",
+ "Container Name": "test-container",
+ "Use Managed Identity Credentials": "false",
+ "Storage Account Key": "test-key",
+ "Create Container": "false"
+ },
+ "propertyDescriptors": {
+ "Common Storage Account Endpoint Suffix": {
+ "name": "Common Storage Account Endpoint Suffix",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Blob": {
+ "name": "Blob",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Storage Account Name": {
+ "name": "Storage Account Name",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Azure Storage Credentials Service": {
+ "name": "Azure Storage Credentials Service",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Container Name": {
+ "name": "Container Name",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Use Managed Identity Credentials": {
+ "name": "Use Managed Identity Credentials",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Storage Account Key": {
+ "name": "Storage Account Key",
+ "identifiesControllerService": false,
+ "sensitive": true
+ },
+ "SAS Token": {
+ "name": "SAS Token",
+ "identifiesControllerService": false,
+ "sensitive": true
+ },
+ "Create Container": {
+ "name": "Create Container",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Connection String": {
+ "name": "Connection String",
+ "identifiesControllerService": false,
+ "sensitive": false
+ }
+ },
+ "style": {},
+ "schedulingStrategy": "EVENT_DRIVEN",
+ "executionNode": "ALL",
+ "penaltyDuration": "30000 ms",
+ "yieldDuration": "1000 ms",
+ "bulletinLevel": "WARN",
+ "runDurationMillis": 0,
+ "concurrentlySchedulableTaskCount": 1,
+ "autoTerminatedRelationships": [
+ "success",
+ "failure"
+ ],
+ "componentType": "PROCESSOR",
+ "groupIdentifier": "1c2f34d2-ee57-448a-b150-8140f0f7643c"
+ },
+ {
+ "identifier": "e7947180-caf7-4ddf-8b02-9e03d8f77d9d",
+ "instanceIdentifier": "2d995b54-4b70-456e-9f76-49265a1fc489",
+ "name": "Get files from /tmp/input",
+ "comments": "",
+ "position": {
+ "x": 361.0,
+ "y": 248.0
+ },
+ "type": "org.apache.nifi.minifi.processors.GetFile",
+ "bundle": {
+ "group": "org.apache.nifi.minifi",
+ "artifact": "minifi-standard-processors",
+ "version": "0.15.0"
+ },
+ "properties": {
+ "Keep Source File": "false",
+ "Minimum File Age": "0 sec",
+ "Polling Interval": "0 sec",
+ "Input Directory": "/tmp/input",
+ "Maximum File Age": "0 sec",
+ "Batch Size": "10",
+ "Maximum File Size": "0 B",
+ "Minimum File Size": "0 B",
+ "Ignore Hidden Files": "true",
+ "Recurse Subdirectories": "true",
+ "File Filter": ".*"
+ },
+ "propertyDescriptors": {
+ "Keep Source File": {
+ "name": "Keep Source File",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Minimum File Age": {
+ "name": "Minimum File Age",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Polling Interval": {
+ "name": "Polling Interval",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Input Directory": {
+ "name": "Input Directory",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Maximum File Age": {
+ "name": "Maximum File Age",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Batch Size": {
+ "name": "Batch Size",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Maximum File Size": {
+ "name": "Maximum File Size",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Minimum File Size": {
+ "name": "Minimum File Size",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Ignore Hidden Files": {
+ "name": "Ignore Hidden Files",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Recurse Subdirectories": {
+ "name": "Recurse Subdirectories",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "File Filter": {
+ "name": "File Filter",
+ "identifiesControllerService": false,
+ "sensitive": false
+ }
+ },
+ "style": {},
+ "schedulingPeriod": "2000 ms",
+ "schedulingStrategy": "TIMER_DRIVEN",
+ "executionNode": "ALL",
+ "penaltyDuration": "30000 ms",
+ "yieldDuration": "1000 ms",
+ "bulletinLevel": "WARN",
+ "runDurationMillis": 0,
+ "concurrentlySchedulableTaskCount": 1,
+ "autoTerminatedRelationships": [],
+ "componentType": "PROCESSOR",
+ "groupIdentifier": "1c2f34d2-ee57-448a-b150-8140f0f7643c"
+ }
+ ],
+ "inputPorts": [],
+ "outputPorts": [],
+ "connections": [
+ {
+ "identifier": "a380ee83-29fc-4a00-b2c6-2cf71e1d8b0e",
+ "instanceIdentifier": "84861a85-32d9-49b2-874e-99011cb28257",
+ "name": "Get files from /tmp/input/success/Upload Blob to test-container in Azure Storage",
+ "position": {
+ "x": 0.0,
+ "y": 0.0
+ },
+ "source": {
+ "id": "e7947180-caf7-4ddf-8b02-9e03d8f77d9d",
+ "type": "PROCESSOR",
+ "groupId": "1c2f34d2-ee57-448a-b150-8140f0f7643c",
+ "name": "Get files from /tmp/input",
+ "instanceIdentifier": "2d995b54-4b70-456e-9f76-49265a1fc489"
+ },
+ "destination": {
+ "id": "af053760-caa2-4624-b70e-0ef8a802864f",
+ "type": "PROCESSOR",
+ "groupId": "1c2f34d2-ee57-448a-b150-8140f0f7643c",
+ "name": "Upload Blob to test-container in Azure Storage",
+ "instanceIdentifier": "35196325-c33f-486e-ba36-646d66ee3369"
+ },
+ "labelIndex": 1,
+ "zIndex": 0,
+ "selectedRelationships": [
+ "success"
+ ],
+ "backPressureObjectThreshold": 2000,
+ "backPressureDataSizeThreshold": "100 MB",
+ "flowFileExpiration": "0 seconds",
+ "prioritizers": [],
+ "bends": [],
+ "componentType": "CONNECTION",
+ "groupIdentifier": "1c2f34d2-ee57-448a-b150-8140f0f7643c"
+ }
+ ],
+ "labels": [],
+ "funnels": [],
+ "controllerServices": [],
+ "variables": {},
+ "componentType": "PROCESS_GROUP"
+ }
+}
diff --git a/examples/consumejournald_config.json b/examples/consumejournald_config.json
new file mode 100644
index 000000000..0dad3d088
--- /dev/null
+++ b/examples/consumejournald_config.json
@@ -0,0 +1,49 @@
+{
+ "rootGroup": {
+ "name": "MiNiFi Flow",
+ "processors": [
+ {
+ "name": "Consume journal messages",
+ "identifier": "66fbb65d-8b11-43a9-aaad-746a07d2fe35",
+ "type": "org.apache.nifi.minifi.extensions.systemd.ConsumeJournald",
+ "schedulingStrategy": "TIMER_DRIVEN",
+ "schedulingPeriod": "5 sec",
+ "properties": {},
+ "autoTerminatedRelationships": []
+ },
+ {
+ "name": "Log messages to info level",
+ "identifier": "f7281113-9576-4680-9a87-a8e961101b5c",
+ "type": "org.apache.nifi.minifi.processors.LogAttribute",
+ "schedulingStrategy": "EVENT_DRIVEN",
+ "properties": {
+ "FlowFiles To Log": "0",
+ "Log Level": "info",
+ "Log Payload": true,
+ "Log Prefix": "=====",
+ "Maxumim Payload Line Length": "0"
+ },
+ "autoTerminatedRelationships": [
+ "success"
+ ]
+ }
+ ],
+ "connections": [
+ {
+ "name": "ConsumeJournald/success/LogAttribute",
+ "identifier": "f57713d9-fa0b-4416-948d-4b316b1a1c0d",
+ "source": {
+ "id": "66fbb65d-8b11-43a9-aaad-746a07d2fe35"
+ },
+ "destination": {
+ "id": "f7281113-9576-4680-9a87-a8e961101b5c"
+ },
+ "selectedRelationships": [
+ "success"
+ ]
+ }
+ ],
+ "remoteProcessGroups": [],
+ "controllerServices": []
+ }
+}
diff --git a/examples/consumejournald_config.nifi.schema.json b/examples/consumejournald_config.nifi.schema.json
new file mode 100644
index 000000000..59c84a31c
--- /dev/null
+++ b/examples/consumejournald_config.nifi.schema.json
@@ -0,0 +1,213 @@
+{
+ "encodingVersion": {
+ "majorVersion": 2,
+ "minorVersion": 0
+ },
+ "maxTimerDrivenThreadCount": 1,
+ "maxEventDrivenThreadCount": 1,
+ "parameterContexts": [],
+ "rootGroup": {
+ "identifier": "1c2f34d2-ee57-448a-b150-8140f0f7643c",
+ "instanceIdentifier": "9af45830-320e-437e-8204-dcd3c2933aa6",
+ "name": "root",
+ "position": {
+ "x": 0.0,
+ "y": 0.0
+ },
+ "processGroups": [],
+ "remoteProcessGroups": [],
+ "processors": [
+ {
+ "identifier": "4b4c6065-da7e-4316-82fd-440273932a20",
+ "instanceIdentifier": "10f04bb1-ee11-4fad-8533-09cc975363bf",
+ "name": "Consume journal messages",
+ "comments": "",
+ "position": {
+ "x": 95.0,
+ "y": 127.0
+ },
+ "type": "org.apache.nifi.minifi.extensions.systemd.ConsumeJournald",
+ "bundle": {
+ "group": "org.apache.nifi.minifi",
+ "artifact": "minifi-systemd",
+ "version": "0.15.0"
+ },
+ "properties": {
+ "Timestamp Format": "%x %X %Z",
+ "Journal Type": "System",
+ "Batch Size": "1000",
+ "Payload Format": "Syslog",
+ "Process Old Messages": "false",
+ "Include Timestamp": "true"
+ },
+ "propertyDescriptors": {
+ "Timestamp Format": {
+ "name": "Timestamp Format",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Journal Type": {
+ "name": "Journal Type",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Batch Size": {
+ "name": "Batch Size",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Payload Format": {
+ "name": "Payload Format",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Process Old Messages": {
+ "name": "Process Old Messages",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Include Timestamp": {
+ "name": "Include Timestamp",
+ "identifiesControllerService": false,
+ "sensitive": false
+ }
+ },
+ "style": {},
+ "schedulingPeriod": "5 sec",
+ "schedulingStrategy": "TIMER_DRIVEN",
+ "executionNode": "ALL",
+ "penaltyDuration": "30000 ms",
+ "yieldDuration": "1000 ms",
+ "bulletinLevel": "WARN",
+ "runDurationMillis": 0,
+ "concurrentlySchedulableTaskCount": 1,
+ "autoTerminatedRelationships": [],
+ "componentType": "PROCESSOR",
+ "groupIdentifier": "1c2f34d2-ee57-448a-b150-8140f0f7643c"
+ },
+ {
+ "identifier": "bff103a6-6ee1-44c3-b084-3e410b428e17",
+ "instanceIdentifier": "c0707e12-622e-4c1d-b0fa-e200926995d7",
+ "name": "Log messages to info level",
+ "comments": "",
+ "position": {
+ "x": 390.0,
+ "y": 131.0
+ },
+ "type": "org.apache.nifi.minifi.processors.LogAttribute",
+ "bundle": {
+ "group": "org.apache.nifi.minifi",
+ "artifact": "minifi-standard-processors",
+ "version": "0.15.0"
+ },
+ "properties": {
+ "FlowFiles To Log": "0",
+ "Log Level": "info",
+ "Log Payload": "true",
+ "Log Prefix": "=====",
+ "Hexencode Payload": "false",
+ "Maximum Payload Line Length": "0"
+ },
+ "propertyDescriptors": {
+ "FlowFiles To Log": {
+ "name": "FlowFiles To Log",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Log Level": {
+ "name": "Log Level",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Attributes to Ignore": {
+ "name": "Attributes to Ignore",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Attributes to Log": {
+ "name": "Attributes to Log",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Log Payload": {
+ "name": "Log Payload",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Log Prefix": {
+ "name": "Log Prefix",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Hexencode Payload": {
+ "name": "Hexencode Payload",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Maximum Payload Line Length": {
+ "name": "Maximum Payload Line Length",
+ "identifiesControllerService": false,
+ "sensitive": false
+ }
+ },
+ "style": {},
+ "schedulingStrategy": "EVENT_DRIVEN",
+ "executionNode": "ALL",
+ "penaltyDuration": "30000 ms",
+ "yieldDuration": "1000 ms",
+ "bulletinLevel": "WARN",
+ "runDurationMillis": 0,
+ "concurrentlySchedulableTaskCount": 1,
+ "autoTerminatedRelationships": [
+ "success"
+ ],
+ "componentType": "PROCESSOR",
+ "groupIdentifier": "1c2f34d2-ee57-448a-b150-8140f0f7643c"
+ }
+ ],
+ "inputPorts": [],
+ "outputPorts": [],
+ "connections": [
+ {
+ "identifier": "0ed54cca-7d1e-4299-9ddf-05b7f6d48828",
+ "instanceIdentifier": "fe35f524-e4f1-4e2d-a79f-40cf536eeca4",
+ "name": "Consume journal messages/success/Log messages to info level",
+ "position": {
+ "x": 0.0,
+ "y": 0.0
+ },
+ "source": {
+ "id": "4b4c6065-da7e-4316-82fd-440273932a20",
+ "type": "PROCESSOR",
+ "groupId": "1c2f34d2-ee57-448a-b150-8140f0f7643c",
+ "name": "Consume journal messages",
+ "instanceIdentifier": "10f04bb1-ee11-4fad-8533-09cc975363bf"
+ },
+ "destination": {
+ "id": "bff103a6-6ee1-44c3-b084-3e410b428e17",
+ "type": "PROCESSOR",
+ "groupId": "1c2f34d2-ee57-448a-b150-8140f0f7643c",
+ "name": "Log messages to info level",
+ "instanceIdentifier": "c0707e12-622e-4c1d-b0fa-e200926995d7"
+ },
+ "labelIndex": 1,
+ "zIndex": 0,
+ "selectedRelationships": [
+ "success"
+ ],
+ "backPressureObjectThreshold": 2000,
+ "backPressureDataSizeThreshold": "100 MB",
+ "flowFileExpiration": "0 seconds",
+ "prioritizers": [],
+ "bends": [],
+ "componentType": "CONNECTION",
+ "groupIdentifier": "1c2f34d2-ee57-448a-b150-8140f0f7643c"
+ }
+ ],
+ "labels": [],
+ "funnels": [],
+ "controllerServices": [],
+ "variables": {},
+ "componentType": "PROCESS_GROUP"
+ }
+}
diff --git a/examples/consumekafka_config.json b/examples/consumekafka_config.json
new file mode 100644
index 000000000..9436d3207
--- /dev/null
+++ b/examples/consumekafka_config.json
@@ -0,0 +1,59 @@
+{
+ "rootGroup": {
+ "name": "MiNiFi Flow",
+ "processors": [
+ {
+ "name": "Consume Kafka messages from ConsumeKafkaTest topic",
+ "identifier": "37d6e8e5-6284-423e-8620-c5168a71add7",
+ "type": "org.apache.nifi.processors.standard.ConsumeKafka",
+ "schedulingStrategy": "TIMER_DRIVEN",
+ "schedulingPeriod": "1 sec",
+ "properties": {
+ "Group ID": "test_group",
+ "Honor Transactions": "true",
+ "Kafka Brokers": "kafka-broker:9092",
+ "Key Attribute Encoding": "UTF-8",
+ "Max Poll Time": "4 sec",
+ "Message Header Encoding": "UTF-8",
+ "Offset Reset": "earliest",
+ "Session Timeout": "60 sec",
+ "Topic Name Format": "Names",
+ "Topic Names": "ConsumeKafkaTest"
+ },
+ "autoTerminatedRelationships": [
+ "failure"
+ ]
+ },
+ {
+ "name": "Put files to /tmp/output",
+ "identifier": "7f9c6f0b-e936-499e-b659-acfaf6872772",
+ "type": "org.apache.nifi.processors.standard.PutFile",
+ "schedulingStrategy": "EVENT_DRIVEN",
+ "properties": {
+ "Directory": "/tmp/output"
+ },
+ "autoTerminatedRelationships": [
+ "success",
+ "failure"
+ ]
+ }
+ ],
+ "connections": [
+ {
+ "name": "ConsumeKafka/success/PutFile",
+ "identifier": "c9ac662f-015a-4b09-9c05-14599e30f196",
+ "source": {
+ "id": "37d6e8e5-6284-423e-8620-c5168a71add7"
+ },
+ "destination": {
+ "id": "7f9c6f0b-e936-499e-b659-acfaf6872772"
+ },
+ "selectedRelationships": [
+ "success"
+ ]
+ }
+ ],
+ "remoteProcessGroups": [],
+ "controllerServices": []
+ }
+}
diff --git a/examples/cwel_config.json b/examples/cwel_config.json
new file mode 100644
index 000000000..38516e894
--- /dev/null
+++ b/examples/cwel_config.json
@@ -0,0 +1,54 @@
+{
+ "rootGroup": {
+ "name": "MiNiFi Flow",
+ "processors": [
+ {
+ "name": "Consume Windows Events in JSON format",
+ "identifier": "2f2a3b47-f5ba-49f6-82b5-bc1c86b96e27",
+ "type": "org.apache.nifi.minifi.processors.ConsumeWindowsEventLog",
+ "schedulingStrategy": "TIMER_DRIVEN",
+ "schedulingPeriod": "1000 ms",
+ "properties": {
+ "Channel": "System",
+ "Query": "*",
+ "Output Format": "JSON",
+ "JSON Format": "Flattened",
+ "Batch Commit Size": "10"
+ },
+ "autoTerminatedRelationships": []
+ },
+ {
+ "name": "Put files to C:\\temp\\",
+ "identifier": "e143601d-de4f-44ba-a6ec-d1f97d77ec94",
+ "type": "org.apache.nifi.minifi.processors.PutFile",
+ "schedulingStrategy": "EVENT_DRIVEN",
+ "properties": {
+ "Conflict Resolution Strategy": "fail",
+ "Create Missing Directories": "true",
+ "Directory": "C:\\temp\\"
+ },
+ "autoTerminatedRelationships": [
+ "failure",
+ "success"
+ ]
+ }
+ ],
+ "connections": [
+ {
+ "name": "ConsumeWindowsEventLog/success/PutFile",
+ "identifier": "098a56ba-f4bf-4323-a3f3-6f8a5e3586bf",
+ "source": {
+ "id": "2f2a3b47-f5ba-49f6-82b5-bc1c86b96e27"
+ },
+ "destination": {
+ "id": "e143601d-de4f-44ba-a6ec-d1f97d77ec94"
+ },
+ "selectedRelationships": [
+ "success"
+ ]
+ }
+ ],
+ "controllerServices": [],
+ "remoteProcessGroups": []
+ }
+}
diff --git a/examples/getfile_putfile_config.json b/examples/getfile_putfile_config.json
new file mode 100644
index 000000000..57ae076f4
--- /dev/null
+++ b/examples/getfile_putfile_config.json
@@ -0,0 +1,54 @@
+{
+ "rootGroup": {
+ "name": "MiNiFi Flow",
+ "processors": [
+ {
+ "name": "Get files from /tmp/getfile_dir",
+ "identifier": "2f2a3b47-f5ba-49f6-82b5-bc1c86b96e27",
+ "type": "org.apache.nifi.processors.standard.GetFile",
+ "schedulingStrategy": "TIMER_DRIVEN",
+ "schedulingPeriod": "1 sec",
+ "properties": {
+ "Input Directory": "/tmp/input",
+ "Keep Source File": "true",
+ "Recurse Subdirectories": "true",
+ "Ignore Hidden Files": "false",
+ "Minimum File Size": "1 MB"
+ },
+ "autoTerminatedRelationships": []
+ },
+ {
+ "name": "Put files to /tmp/out_dir",
+ "identifier": "e143601d-de4f-44ba-a6ec-d1f97d77ec94",
+ "type": "org.apache.nifi.processors.standard.PutFile",
+ "schedulingStrategy": "EVENT_DRIVEN",
+ "properties": {
+ "Conflict Resolution Strategy": "fail",
+ "Create Missing Directories": "true",
+ "Directory": "/tmp/out_dir"
+ },
+ "autoTerminatedRelationships": [
+ "success",
+ "failure"
+ ]
+ }
+ ],
+ "connections": [
+ {
+ "identifier": "098a56ba-f4bf-4323-a3f3-6f8a5e3586bf",
+ "name": "GetFile/success/PutFile",
+ "source": {
+ "id": "2f2a3b47-f5ba-49f6-82b5-bc1c86b96e27"
+ },
+ "destination": {
+ "id": "e143601d-de4f-44ba-a6ec-d1f97d77ec94"
+ },
+ "selectedRelationships": [
+ "success"
+ ]
+ }
+ ],
+ "remoteProcessGroups": [],
+ "controllerServices": []
+ }
+}
diff --git a/examples/grafana_loki_config.json b/examples/grafana_loki_config.json
new file mode 100644
index 000000000..672767133
--- /dev/null
+++ b/examples/grafana_loki_config.json
@@ -0,0 +1,52 @@
+{
+ "rootGroup": {
+ "name": "MiNiFi Flow",
+ "processors": [
+ {
+ "name": "Tail /tmp/input/test_file.log",
+ "identifier": "962c76c4-013e-40c0-ac21-9fd6db4593a3",
+ "type": "org.apache.nifi.processors.standard.TailFile",
+ "schedulingStrategy": "TIMER_DRIVEN",
+ "schedulingPeriod": "1 sec",
+ "properties": {
+ "File to Tail": "/tmp/input/test_file.log"
+ },
+ "autoTerminatedRelationships": []
+ },
+ {
+ "name": "PushGrafanaLokiREST",
+ "identifier": "6a9f6c0e-f70c-4778-be8b-ca2494eb0042",
+ "type": "org.apache.nifi.processors.standard.PushGrafanaLokiREST",
+ "schedulingStrategy": "EVENT_DRIVEN",
+ "properties": {
+ "Url": "http://127.0.0.1:3100/",
+ "Stream Labels": "job=minifi,id=logs",
+ "Log Line Metadata Attributes": "filename",
+ "Max Batch Size": "50"
+ },
+ "autoTerminatedRelationships": [
+ "success",
+ "failure"
+ ]
+ }
+ ],
+ "funnels": [],
+ "connections": [
+ {
+ "identifier": "32f1aa1b-05c8-4fc1-941f-1e3dc70cb1b2",
+ "name": "TailFile/success/PushGrafanaLokiREST",
+ "source": {
+ "id": "962c76c4-013e-40c0-ac21-9fd6db4593a3"
+ },
+ "destination": {
+ "id": "6a9f6c0e-f70c-4778-be8b-ca2494eb0042"
+ },
+ "selectedRelationships": [
+ "success"
+ ]
+ }
+ ],
+ "remoteProcessGroups": [],
+ "controllerServices": []
+ }
+}
diff --git a/examples/tailfile_config.yml b/examples/grafana_loki_config.yml
similarity index 64%
copy from examples/tailfile_config.yml
copy to examples/grafana_loki_config.yml
index ada27ffb8..d7dbe7984 100644
--- a/examples/tailfile_config.yml
+++ b/examples/grafana_loki_config.yml
@@ -17,31 +17,31 @@ MiNiFi Config Version: 3
Flow Controller:
name: MiNiFi Flow
Processors:
-- name: Log tailfile attributes
- id: 41141af1-3c43-4fa4-8236-2b207b0fe9d8
- class: org.apache.nifi.minifi.processors.LogAttribute
+- name: Tail /tmp/input/test_file.log
+ id: 962c76c4-013e-40c0-ac21-9fd6db4593a3
+ class: org.apache.nifi.minifi.processors.TailFile
scheduling strategy: TIMER_DRIVEN
scheduling period: 1000 ms
+ Properties:
+ File to Tail: /tmp/input/test_file.log
+- name: PushGrafanaLokiREST
+ id: 6a9f6c0e-f70c-4778-be8b-ca2494eb0042
+ class: org.apache.nifi.minifi.processors.PushGrafanaLokiREST
+ scheduling strategy: EVENT_DRIVEN
auto-terminated relationships list:
+ - failure
- success
Properties:
- FlowFiles To Log: '0'
-- name: Tail /tmp/test_file.log
- id: 0ef22940-ac0a-4813-9ad9-fe41c2b51fff
- class: org.apache.nifi.minifi.processors.TailFile
- scheduling strategy: TIMER_DRIVEN
- scheduling period: 1000 ms
- Properties:
- File to Tail: /tmp/test_file.log
- Initial Start Position: Beginning of File
- Input Delimiter: \n
- tail-mode: Single file
+ Url: http://127.0.0.1:3100/
+ Stream Labels: job=minifi,id=logs
+ Log Line Metadata Attributes: filename
+ Max Batch Size: '50'
Connections:
-- name: TailFile/success/LogAttribute
- id: eda7f758-c548-46d6-aba0-faa6e9937d8c
- source id: 0ef22940-ac0a-4813-9ad9-fe41c2b51fff
+- name: TailFile/success/PushGrafanaLokiREST
+ id: 32f1aa1b-05c8-4fc1-941f-1e3dc70cb1b2
+ source id: 962c76c4-013e-40c0-ac21-9fd6db4593a3
source relationship names:
- success
- destination id: 41141af1-3c43-4fa4-8236-2b207b0fe9d8
+ destination id: 6a9f6c0e-f70c-4778-be8b-ca2494eb0042
Controller Services: []
Remote Process Groups: []
diff --git a/examples/http_post_config.json b/examples/http_post_config.json
new file mode 100644
index 000000000..879953963
--- /dev/null
+++ b/examples/http_post_config.json
@@ -0,0 +1,142 @@
+{
+ "rootGroup": {
+ "name": "MiNiFi Flow",
+ "processors": [
+ {
+ "name": "Get files from /tmp/input",
+ "identifier": "962790e7-ea35-4096-9362-96f527288669",
+ "type": "org.apache.nifi.processors.standard.GetFile",
+ "schedulingStrategy": "TIMER_DRIVEN",
+ "schedulingPeriod": "2 sec",
+ "properties": {
+ "Input Directory": "/tmp/input",
+ "Keep Source File": "true"
+ },
+ "autoTerminatedRelationships": []
+ },
+ {
+ "name": "Invoke POST request on remote URL",
+ "identifier": "33b373b6-6f19-4194-b45a-1ef73c357a8e",
+ "type": "org.apache.nifi.processors.standard.InvokeHTTP",
+ "schedulingStrategy": "EVENT_DRIVEN",
+ "properties": {
+ "HTTP Method": "POST",
+ "Remote URL": "http://1.2.3.4:8080/contentListener"
+ },
+ "autoTerminatedRelationships": [
+ "success",
+ "response"
+ ]
+ },
+ {
+ "name": "Retry on HTTP 5xx status codes",
+ "identifier": "3b1c6971-541b-4530-9f72-175757500699",
+ "type": "org.apache.nifi.processors.standard.RetryFlowFile",
+ "schedulingStrategy": "EVENT_DRIVEN",
+ "properties": {
+ "Maximum Retries": "5"
+ },
+ "autoTerminatedRelationships": [
+ "failure"
+ ]
+ },
+ {
+ "name": "Log failed flow files",
+ "identifier": "1f669bb0-6d50-41e6-9230-90c1cc3e21f9",
+ "type": "org.apache.nifi.minifi.processors.LogAttribute",
+ "schedulingStrategy": "EVENT_DRIVEN",
+ "properties": {
+ "FlowFiles To Log": "0",
+ "Log Level": "warn",
+ "Log Prefix": "=== Failed HTTP request ===",
+ "Maximum Payload Line Length": "0"
+ },
+ "autoTerminatedRelationships": [
+ "success"
+ ]
+ }
+ ],
+ "connections": [
+ {
+ "name": "GetFile/success/InvokeHTTP",
+ "identifier": "168cc31e-be1d-4ed3-bebc-8bf652dde104",
+ "source": {
+ "id": "962790e7-ea35-4096-9362-96f527288669"
+ },
+ "destination": {
+ "id": "33b373b6-6f19-4194-b45a-1ef73c357a8e"
+ },
+ "selectedRelationships": [
+ "success"
+ ]
+ },
+ {
+ "name": "InvokeHTTP/failure/LogAttribute",
+ "identifier": "0d55009a-adcc-4722-b889-570b6d26cfee",
+ "source": {
+ "id": "33b373b6-6f19-4194-b45a-1ef73c357a8e"
+ },
+ "destination": {
+ "id": "1f669bb0-6d50-41e6-9230-90c1cc3e21f9"
+ },
+ "selectedRelationships": [
+ "failure"
+ ]
+ },
+ {
+ "name": "InvokeHTTP/retry/RetryFlowFile",
+ "identifier": "c8a96f1d-48cd-4ee3-8a2f-081c3e3c7bcd",
+ "source": {
+ "id": "33b373b6-6f19-4194-b45a-1ef73c357a8e"
+ },
+ "destination": {
+ "id": "3b1c6971-541b-4530-9f72-175757500699"
+ },
+ "selectedRelationships": [
+ "retry"
+ ]
+ },
+ {
+ "name": "RetryFlowFile/retry/InvokeHTTP",
+ "identifier": "cadf51f0-3468-423a-9d12-5c40270aa7ca",
+ "source": {
+ "id": "3b1c6971-541b-4530-9f72-175757500699"
+ },
+ "destination": {
+ "id": "33b373b6-6f19-4194-b45a-1ef73c357a8e"
+ },
+ "selectedRelationships": [
+ "retry"
+ ]
+ },
+ {
+ "name": "RetryFlowFile/retries_exceeded/LogAttribute",
+ "identifier": "160fc0f4-6efe-4274-8bc7-2a416570bdeb",
+ "source": {
+ "id": "3b1c6971-541b-4530-9f72-175757500699"
+ },
+ "destination": {
+ "id": "1f669bb0-6d50-41e6-9230-90c1cc3e21f9"
+ },
+ "selectedRelationships": [
+ "retries_exceeded"
+ ]
+ },
+ {
+ "name": "InvokeHTTP/no retry/LogAttribute",
+ "identifier": "8f6008ba-e1c7-433b-88f1-1e461bb82e29",
+ "source": {
+ "id": "33b373b6-6f19-4194-b45a-1ef73c357a8e"
+ },
+ "destination": {
+ "id": "1f669bb0-6d50-41e6-9230-90c1cc3e21f9"
+ },
+ "selectedRelationships": [
+ "no retry"
+ ]
+ }
+ ],
+ "controllerServices": [],
+ "remoteProcessGroups": []
+ }
+}
diff --git a/examples/http_post_config.nifi.schema.json b/examples/http_post_config.nifi.schema.json
new file mode 100644
index 000000000..c334bc58d
--- /dev/null
+++ b/examples/http_post_config.nifi.schema.json
@@ -0,0 +1,647 @@
+{
+ "encodingVersion": {
+ "majorVersion": 2,
+ "minorVersion": 0
+ },
+ "maxTimerDrivenThreadCount": 1,
+ "maxEventDrivenThreadCount": 1,
+ "parameterContexts": [],
+ "rootGroup": {
+ "identifier": "1c2f34d2-ee57-448a-b150-8140f0f7643c",
+ "instanceIdentifier": "9af45830-320e-437e-8204-dcd3c2933aa6",
+ "name": "root",
+ "position": {
+ "x": 0.0,
+ "y": 0.0
+ },
+ "processGroups": [],
+ "remoteProcessGroups": [],
+ "processors": [
+ {
+ "identifier": "375ce4dc-8005-4a39-92e8-5402c72f541c",
+ "instanceIdentifier": "44898895-5d15-4965-8677-aba1c11f2876",
+ "name": "Log failed flow files",
+ "comments": "",
+ "position": {
+ "x": 957.0,
+ "y": 293.0
+ },
+ "type": "org.apache.nifi.minifi.processors.LogAttribute",
+ "bundle": {
+ "group": "org.apache.nifi.minifi",
+ "artifact": "minifi-standard-processors",
+ "version": "0.15.0"
+ },
+ "properties": {
+ "FlowFiles To Log": "0",
+ "Log Level": "warn",
+ "Log Payload": "false",
+ "Log Prefix": "=== Failed HTTP request ===",
+ "Hexencode Payload": "false",
+ "Maximum Payload Line Length": "0"
+ },
+ "propertyDescriptors": {
+ "FlowFiles To Log": {
+ "name": "FlowFiles To Log",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Log Level": {
+ "name": "Log Level",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Attributes to Ignore": {
+ "name": "Attributes to Ignore",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Attributes to Log": {
+ "name": "Attributes to Log",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Log Payload": {
+ "name": "Log Payload",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Log Prefix": {
+ "name": "Log Prefix",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Hexencode Payload": {
+ "name": "Hexencode Payload",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Maximum Payload Line Length": {
+ "name": "Maximum Payload Line Length",
+ "identifiesControllerService": false,
+ "sensitive": false
+ }
+ },
+ "style": {},
+ "schedulingStrategy": "EVENT_DRIVEN",
+ "executionNode": "ALL",
+ "penaltyDuration": "30000 ms",
+ "yieldDuration": "1000 ms",
+ "bulletinLevel": "WARN",
+ "runDurationMillis": 0,
+ "concurrentlySchedulableTaskCount": 1,
+ "autoTerminatedRelationships": [
+ "success"
+ ],
+ "componentType": "PROCESSOR",
+ "groupIdentifier": "1c2f34d2-ee57-448a-b150-8140f0f7643c"
+ },
+ {
+ "identifier": "7f5c10d0-e7d8-42c4-aacd-96d9d1353da4",
+ "instanceIdentifier": "fc1ea604-88e5-4d5a-bef3-95b071442558",
+ "name": "Invoke POST request on remote URL",
+ "comments": "",
+ "position": {
+ "x": 354.0,
+ "y": 100.0
+ },
+ "type": "org.apache.nifi.minifi.processors.InvokeHTTP",
+ "bundle": {
+ "group": "org.apache.nifi.minifi",
+ "artifact": "minifi-http-curl",
+ "version": "0.15.0"
+ },
+ "properties": {
+ "Invalid HTTP Header Field Handling Strategy": "transform",
+ "Read Timeout": "15 s",
+ "Send Message Body": "true",
+ "Connection Timeout": "5 s",
+ "send-message-body": "true",
+ "Content-type": "application/octet-stream",
+ "Always Output Response": "false",
+ "HTTP Method": "POST",
+ "Include Date Header": "true",
+ "Use Chunked Encoding": "false",
+ "Disable Peer Verification": "false",
+ "Penalize on \"No Retry\"": "false",
+ "Follow Redirects": "true",
+ "Remote URL": "http://1.2.3.4:8080/contentListener"
+ },
+ "propertyDescriptors": {
+ "Proxy Host": {
+ "name": "Proxy Host",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Upload Speed Limit": {
+ "name": "Upload Speed Limit",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Attributes to Send": {
+ "name": "Attributes to Send",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Invalid HTTP Header Field Handling Strategy": {
+ "name": "Invalid HTTP Header Field Handling Strategy",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Download Speed Limit": {
+ "name": "Download Speed Limit",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Read Timeout": {
+ "name": "Read Timeout",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "invokehttp-proxy-password": {
+ "name": "invokehttp-proxy-password",
+ "displayName": "Proxy Password",
+ "identifiesControllerService": false,
+ "sensitive": true
+ },
+ "Send Message Body": {
+ "name": "Send Message Body",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Proxy Port": {
+ "name": "Proxy Port",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "invokehttp-proxy-username": {
+ "name": "invokehttp-proxy-username",
+ "displayName": "Proxy Username",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Put Response Body in Attribute": {
+ "name": "Put Response Body in Attribute",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Connection Timeout": {
+ "name": "Connection Timeout",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "send-message-body": {
+ "name": "send-message-body",
+ "displayName": "Send Body",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Content-type": {
+ "name": "Content-type",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "SSL Context Service": {
+ "name": "SSL Context Service",
+ "identifiesControllerService": true,
+ "sensitive": false
+ },
+ "Always Output Response": {
+ "name": "Always Output Response",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "HTTP Method": {
+ "name": "HTTP Method",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Include Date Header": {
+ "name": "Include Date Header",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Use Chunked Encoding": {
+ "name": "Use Chunked Encoding",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Disable Peer Verification": {
+ "name": "Disable Peer Verification",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Penalize on \"No Retry\"": {
+ "name": "Penalize on \"No Retry\"",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Follow Redirects": {
+ "name": "Follow Redirects",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Remote URL": {
+ "name": "Remote URL",
+ "identifiesControllerService": false,
+ "sensitive": false
+ }
+ },
+ "style": {},
+ "schedulingStrategy": "EVENT_DRIVEN",
+ "executionNode": "ALL",
+ "penaltyDuration": "30000 ms",
+ "yieldDuration": "1000 ms",
+ "bulletinLevel": "WARN",
+ "runDurationMillis": 0,
+ "concurrentlySchedulableTaskCount": 1,
+ "autoTerminatedRelationships": [
+ "success",
+ "response"
+ ],
+ "componentType": "PROCESSOR",
+ "groupIdentifier": "1c2f34d2-ee57-448a-b150-8140f0f7643c"
+ },
+ {
+ "identifier": "bbf3ddd1-0761-411e-930b-fec2c282d10e",
+ "instanceIdentifier": "28de5978-6696-491a-906b-eb7a43145393",
+ "name": "Retry on HTTP 5xx status codes",
+ "comments": "",
+ "position": {
+ "x": 366.0,
+ "y": 356.0
+ },
+ "type": "org.apache.nifi.minifi.processors.RetryFlowFile",
+ "bundle": {
+ "group": "org.apache.nifi.minifi",
+ "artifact": "minifi-standard-processors",
+ "version": "0.15.0"
+ },
+ "properties": {
+ "Reuse Mode": "Fail on Reuse",
+ "Retry Attribute": "flowfile.retries",
+ "Penalize Retries": "true",
+ "Maximum Retries": "5",
+ "Fail on Non-numerical Overwrite": "false"
+ },
+ "propertyDescriptors": {
+ "Reuse Mode": {
+ "name": "Reuse Mode",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Retry Attribute": {
+ "name": "Retry Attribute",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Penalize Retries": {
+ "name": "Penalize Retries",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Maximum Retries": {
+ "name": "Maximum Retries",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Fail on Non-numerical Overwrite": {
+ "name": "Fail on Non-numerical Overwrite",
+ "identifiesControllerService": false,
+ "sensitive": false
+ }
+ },
+ "style": {},
+ "schedulingStrategy": "EVENT_DRIVEN",
+ "executionNode": "ALL",
+ "penaltyDuration": "30000 ms",
+ "yieldDuration": "1000 ms",
+ "bulletinLevel": "WARN",
+ "runDurationMillis": 0,
+ "concurrentlySchedulableTaskCount": 1,
+ "autoTerminatedRelationships": [
+ "failure"
+ ],
+ "componentType": "PROCESSOR",
+ "groupIdentifier": "1c2f34d2-ee57-448a-b150-8140f0f7643c"
+ },
+ {
+ "identifier": "570e4ec6-105a-4da6-aba8-0d8ca069c12a",
+ "instanceIdentifier": "25d38cad-02d6-4f2e-b3a4-17786df17a07",
+ "name": "Get files from /tmp/input",
+ "comments": "",
+ "position": {
+ "x": 71.0,
+ "y": 90.0
+ },
+ "type": "org.apache.nifi.minifi.processors.GetFile",
+ "bundle": {
+ "group": "org.apache.nifi.minifi",
+ "artifact": "minifi-standard-processors",
+ "version": "0.15.0"
+ },
+ "properties": {
+ "Keep Source File": "true",
+ "Minimum File Age": "0 sec",
+ "Polling Interval": "0 sec",
+ "Input Directory": "/tmp/input",
+ "Maximum File Age": "0 sec",
+ "Batch Size": "10",
+ "Maximum File Size": "0 B",
+ "Minimum File Size": "0 B",
+ "Ignore Hidden Files": "true",
+ "Recurse Subdirectories": "true",
+ "File Filter": ".*"
+ },
+ "propertyDescriptors": {
+ "Keep Source File": {
+ "name": "Keep Source File",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Minimum File Age": {
+ "name": "Minimum File Age",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Polling Interval": {
+ "name": "Polling Interval",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Input Directory": {
+ "name": "Input Directory",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Maximum File Age": {
+ "name": "Maximum File Age",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Batch Size": {
+ "name": "Batch Size",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Maximum File Size": {
+ "name": "Maximum File Size",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Minimum File Size": {
+ "name": "Minimum File Size",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Ignore Hidden Files": {
+ "name": "Ignore Hidden Files",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Recurse Subdirectories": {
+ "name": "Recurse Subdirectories",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "File Filter": {
+ "name": "File Filter",
+ "identifiesControllerService": false,
+ "sensitive": false
+ }
+ },
+ "style": {},
+ "schedulingPeriod": "2 sec",
+ "schedulingStrategy": "TIMER_DRIVEN",
+ "executionNode": "ALL",
+ "penaltyDuration": "30000 ms",
+ "yieldDuration": "1000 ms",
+ "bulletinLevel": "WARN",
+ "runDurationMillis": 0,
+ "concurrentlySchedulableTaskCount": 1,
+ "autoTerminatedRelationships": [],
+ "componentType": "PROCESSOR",
+ "groupIdentifier": "1c2f34d2-ee57-448a-b150-8140f0f7643c"
+ }
+ ],
+ "inputPorts": [],
+ "outputPorts": [],
+ "connections": [
+ {
+ "identifier": "79cb7a8f-6e9d-40b5-9897-35667f9a85db",
+ "instanceIdentifier": "a18e5acf-b232-4f7d-814a-dc97dcab5368",
+ "name": "Invoke POST request on remote URL/no retry/Log failed flow files",
+ "position": {
+ "x": 0.0,
+ "y": 0.0
+ },
+ "source": {
+ "id": "7f5c10d0-e7d8-42c4-aacd-96d9d1353da4",
+ "type": "PROCESSOR",
+ "groupId": "1c2f34d2-ee57-448a-b150-8140f0f7643c",
+ "name": "Invoke POST request on remote URL",
+ "instanceIdentifier": "fc1ea604-88e5-4d5a-bef3-95b071442558"
+ },
+ "destination": {
+ "id": "375ce4dc-8005-4a39-92e8-5402c72f541c",
+ "type": "PROCESSOR",
+ "groupId": "1c2f34d2-ee57-448a-b150-8140f0f7643c",
+ "name": "Log failed flow files",
+ "instanceIdentifier": "44898895-5d15-4965-8677-aba1c11f2876"
+ },
+ "labelIndex": 1,
+ "zIndex": 0,
+ "selectedRelationships": [
+ "no retry"
+ ],
+ "backPressureObjectThreshold": 2000,
+ "backPressureDataSizeThreshold": "100 MB",
+ "flowFileExpiration": "0 seconds",
+ "prioritizers": [],
+ "bends": [],
+ "componentType": "CONNECTION",
+ "groupIdentifier": "1c2f34d2-ee57-448a-b150-8140f0f7643c"
+ },
+ {
+ "identifier": "edf56257-a055-48e6-a7e4-87b53a950a87",
+ "instanceIdentifier": "f3216baf-e7ee-4c3c-83b4-f33482d40dca",
+ "name": "Get files from /tmp/input/success/Invoke POST request on remote URL",
+ "position": {
+ "x": 0.0,
+ "y": 0.0
+ },
+ "source": {
+ "id": "570e4ec6-105a-4da6-aba8-0d8ca069c12a",
+ "type": "PROCESSOR",
+ "groupId": "1c2f34d2-ee57-448a-b150-8140f0f7643c",
+ "name": "Get files from /tmp/input",
+ "instanceIdentifier": "25d38cad-02d6-4f2e-b3a4-17786df17a07"
+ },
+ "destination": {
+ "id": "7f5c10d0-e7d8-42c4-aacd-96d9d1353da4",
+ "type": "PROCESSOR",
+ "groupId": "1c2f34d2-ee57-448a-b150-8140f0f7643c",
+ "name": "Invoke POST request on remote URL",
+ "instanceIdentifier": "fc1ea604-88e5-4d5a-bef3-95b071442558"
+ },
+ "labelIndex": 1,
+ "zIndex": 0,
+ "selectedRelationships": [
+ "success"
+ ],
+ "backPressureObjectThreshold": 2000,
+ "backPressureDataSizeThreshold": "100 MB",
+ "flowFileExpiration": "0 seconds",
+ "prioritizers": [],
+ "bends": [],
+ "componentType": "CONNECTION",
+ "groupIdentifier": "1c2f34d2-ee57-448a-b150-8140f0f7643c"
+ },
+ {
+ "identifier": "e1766b79-913e-4964-a748-38da1eaf9c68",
+ "instanceIdentifier": "26dafca1-adaa-4ca2-90d5-14d4fad0b32b",
+ "name": "Retry on HTTP 5xx status codes/retries_exceeded/Log failed flow files",
+ "position": {
+ "x": 0.0,
+ "y": 0.0
+ },
+ "source": {
+ "id": "bbf3ddd1-0761-411e-930b-fec2c282d10e",
+ "type": "PROCESSOR",
+ "groupId": "1c2f34d2-ee57-448a-b150-8140f0f7643c",
+ "name": "Retry on HTTP 5xx status codes",
+ "instanceIdentifier": "28de5978-6696-491a-906b-eb7a43145393"
+ },
+ "destination": {
+ "id": "375ce4dc-8005-4a39-92e8-5402c72f541c",
+ "type": "PROCESSOR",
+ "groupId": "1c2f34d2-ee57-448a-b150-8140f0f7643c",
+ "name": "Log failed flow files",
+ "instanceIdentifier": "44898895-5d15-4965-8677-aba1c11f2876"
+ },
+ "labelIndex": 1,
+ "zIndex": 0,
+ "selectedRelationships": [
+ "retries_exceeded"
+ ],
+ "backPressureObjectThreshold": 2000,
+ "backPressureDataSizeThreshold": "100 MB",
+ "flowFileExpiration": "0 seconds",
+ "prioritizers": [],
+ "bends": [],
+ "componentType": "CONNECTION",
+ "groupIdentifier": "1c2f34d2-ee57-448a-b150-8140f0f7643c"
+ },
+ {
+ "identifier": "43bed49d-842a-4144-abdf-9768d784973d",
+ "instanceIdentifier": "c4722313-9e83-468a-ac0a-edb3cac8ffeb",
+ "name": "Invoke POST request on remote URL/failure/Log failed flow files",
+ "position": {
+ "x": 0.0,
+ "y": 0.0
+ },
+ "source": {
+ "id": "7f5c10d0-e7d8-42c4-aacd-96d9d1353da4",
+ "type": "PROCESSOR",
+ "groupId": "1c2f34d2-ee57-448a-b150-8140f0f7643c",
+ "name": "Invoke POST request on remote URL",
+ "instanceIdentifier": "fc1ea604-88e5-4d5a-bef3-95b071442558"
+ },
+ "destination": {
+ "id": "375ce4dc-8005-4a39-92e8-5402c72f541c",
+ "type": "PROCESSOR",
+ "groupId": "1c2f34d2-ee57-448a-b150-8140f0f7643c",
+ "name": "Log failed flow files",
+ "instanceIdentifier": "44898895-5d15-4965-8677-aba1c11f2876"
+ },
+ "labelIndex": 1,
+ "zIndex": 0,
+ "selectedRelationships": [
+ "failure"
+ ],
+ "backPressureObjectThreshold": 2000,
+ "backPressureDataSizeThreshold": "100 MB",
+ "flowFileExpiration": "0 seconds",
+ "prioritizers": [],
+ "bends": [],
+ "componentType": "CONNECTION",
+ "groupIdentifier": "1c2f34d2-ee57-448a-b150-8140f0f7643c"
+ },
+ {
+ "identifier": "0f1d984a-114f-4ce0-af3b-79a372ac2a7e",
+ "instanceIdentifier": "5b127baa-1371-45ed-a4cc-04ed870d4f32",
+ "name": "Invoke POST request on remote URL/retry/Retry on HTTP 5xx status codes",
+ "position": {
+ "x": 0.0,
+ "y": 0.0
+ },
+ "source": {
+ "id": "7f5c10d0-e7d8-42c4-aacd-96d9d1353da4",
+ "type": "PROCESSOR",
+ "groupId": "1c2f34d2-ee57-448a-b150-8140f0f7643c",
+ "name": "Invoke POST request on remote URL",
+ "instanceIdentifier": "fc1ea604-88e5-4d5a-bef3-95b071442558"
+ },
+ "destination": {
+ "id": "bbf3ddd1-0761-411e-930b-fec2c282d10e",
+ "type": "PROCESSOR",
+ "groupId": "1c2f34d2-ee57-448a-b150-8140f0f7643c",
+ "name": "Retry on HTTP 5xx status codes",
+ "instanceIdentifier": "28de5978-6696-491a-906b-eb7a43145393"
+ },
+ "labelIndex": 1,
+ "zIndex": 0,
+ "selectedRelationships": [
+ "retry"
+ ],
+ "backPressureObjectThreshold": 2000,
+ "backPressureDataSizeThreshold": "100 MB",
+ "flowFileExpiration": "0 seconds",
+ "prioritizers": [],
+ "bends": [],
+ "componentType": "CONNECTION",
+ "groupIdentifier": "1c2f34d2-ee57-448a-b150-8140f0f7643c"
+ },
+ {
+ "identifier": "a8353d2f-777c-4dda-bd32-32f30db25f09",
+ "instanceIdentifier": "791904a8-f4e0-46f2-9909-87428c31ab63",
+ "name": "Retry on HTTP 5xx status codes/retry/Invoke POST request on remote URL",
+ "position": {
+ "x": 0.0,
+ "y": 0.0
+ },
+ "source": {
+ "id": "bbf3ddd1-0761-411e-930b-fec2c282d10e",
+ "type": "PROCESSOR",
+ "groupId": "1c2f34d2-ee57-448a-b150-8140f0f7643c",
+ "name": "Retry on HTTP 5xx status codes",
+ "instanceIdentifier": "28de5978-6696-491a-906b-eb7a43145393"
+ },
+ "destination": {
+ "id": "7f5c10d0-e7d8-42c4-aacd-96d9d1353da4",
+ "type": "PROCESSOR",
+ "groupId": "1c2f34d2-ee57-448a-b150-8140f0f7643c",
+ "name": "Invoke POST request on remote URL",
+ "instanceIdentifier": "fc1ea604-88e5-4d5a-bef3-95b071442558"
+ },
+ "labelIndex": 1,
+ "zIndex": 0,
+ "selectedRelationships": [
+ "retry"
+ ],
+ "backPressureObjectThreshold": 2000,
+ "backPressureDataSizeThreshold": "100 MB",
+ "flowFileExpiration": "0 seconds",
+ "prioritizers": [],
+ "bends": [],
+ "componentType": "CONNECTION",
+ "groupIdentifier": "1c2f34d2-ee57-448a-b150-8140f0f7643c"
+ }
+ ],
+ "labels": [],
+ "funnels": [],
+ "controllerServices": [],
+ "variables": {},
+ "componentType": "PROCESS_GROUP"
+ }
+}
diff --git a/examples/http_post_config.yml b/examples/http_post_config.yml
index 9b7f5d85a..efe6adcb4 100644
--- a/examples/http_post_config.yml
+++ b/examples/http_post_config.yml
@@ -59,11 +59,11 @@ Connections:
source id: 962790e7-ea35-4096-9362-96f527288669
source relationship name: success
destination id: 33b373b6-6f19-4194-b45a-1ef73c357a8e
-- name: InvokeHTTP/failure/InvokeHTTP
+- name: InvokeHTTP/failure/LogAttribute
id: 0d55009a-adcc-4722-b889-570b6d26cfee
source id: 33b373b6-6f19-4194-b45a-1ef73c357a8e
source relationship name: failure
- destination id: 33b373b6-6f19-4194-b45a-1ef73c357a8e
+ destination id: 1f669bb0-6d50-41e6-9230-90c1cc3e21f9
- name: InvokeHTTP/retry/RetryFlowFile
id: c8a96f1d-48cd-4ee3-8a2f-081c3e3c7bcd
source id: 33b373b6-6f19-4194-b45a-1ef73c357a8e
diff --git a/examples/kubernetes_tailfile_config.json b/examples/kubernetes_tailfile_config.json
new file mode 100644
index 000000000..7d1664b7d
--- /dev/null
+++ b/examples/kubernetes_tailfile_config.json
@@ -0,0 +1,72 @@
+{
+ "rootGroup": {
+ "name": "Read Kubernetes pod logs and publish them to Kafka",
+ "processors": [
+ {
+ "name": "Tail Kubernetes log files",
+ "identifier": "891efd7f-2814-4068-9efd-5f258f4990d5",
+ "type": "org.apache.nifi.minifi.processors.TailFile",
+ "schedulingStrategy": "TIMER_DRIVEN",
+ "schedulingPeriod": "1 sec",
+ "properties": {
+ "tail-mode": "Multiple file",
+ "tail-base-directory": "/var/log/pods/${namespace}_${pod}_${uid}/${container}",
+ "File to Tail": ".*\\.log",
+ "Attribute Provider Service": "KubernetesControllerService",
+ "Lookup frequency": "10 min",
+ "Recursive lookup": "false",
+ "Initial Start Position": "Beginning of File",
+ "Input Delimiter": "\\n",
+ "Rolling Filename Pattern": "${filename}.log.*"
+ },
+ "autoTerminatedRelationships": []
+ },
+ {
+ "name": "Publish messages to Kafka topic test",
+ "identifier": "fb880b73-bff7-4775-a854-e048ae09e07e",
+ "type": "org.apache.nifi.processors.standard.PublishKafka",
+ "schedulingStrategy": "EVENT_DRIVEN",
+ "properties": {
+ "Batch Size": "10",
+ "Client Name": "test-client",
+ "Compress Codec": "none",
+ "Delivery Guarantee": "1",
+ "Known Brokers": "kafka-broker:9092",
+ "Message Timeout": "12 sec",
+ "Request Timeout": "10 sec",
+ "Topic Name": "test"
+ },
+ "autoTerminatedRelationships": [
+ "success",
+ "failure"
+ ]
+ }
+ ],
+ "connections": [
+ {
+ "name": "TailFile/success/PublishKafka",
+ "identifier": "d6675f90-62a6-4f98-b67f-00efeab78e5e",
+ "source": {
+ "id": "891efd7f-2814-4068-9efd-5f258f4990d5"
+ },
+ "destination": {
+ "id": "fb880b73-bff7-4775-a854-e048ae09e07e"
+ },
+ "selectedRelationships": [
+ "success"
+ ]
+ }
+ ],
+ "remoteProcessGroups": [],
+ "controllerServices": [
+ {
+ "name": "KubernetesControllerService",
+ "identifier": "568559dc-3c81-4b01-bc08-760fadb953b0",
+ "type": "org.apache.nifi.minifi.controllers.KubernetesControllerService",
+ "properties": {
+ "Namespace Filter": "default"
+ }
+ }
+ ]
+ }
+}
diff --git a/examples/lists3_fetchs3_config.json b/examples/lists3_fetchs3_config.json
new file mode 100644
index 000000000..eaa13a511
--- /dev/null
+++ b/examples/lists3_fetchs3_config.json
@@ -0,0 +1,85 @@
+{
+ "rootGroup": {
+ "name": "MiNiFi Flow",
+ "processors": [
+ {
+ "name": "Fetch S3 object from test_bucket",
+ "identifier": "6ccd8be9-4aa3-4b86-ae10-0a68668f1c25",
+ "type": "org.apache.nifi.minifi.aws.processors.FetchS3Object",
+ "schedulingStrategy": "EVENT_DRIVEN",
+ "properties": {
+ "AWS Credentials Provider service": "AWSCredentialsService",
+ "Bucket": "test_bucket",
+ "Communications Timeout": "30 sec"
+ },
+ "autoTerminatedRelationships": [
+ "failure"
+ ]
+ },
+ {
+ "name": "List test_bucket S3 bucket",
+ "identifier": "6f2c20f5-e990-4d9f-bfd6-f7471e3df469",
+ "type": "org.apache.nifi.minifi.aws.processors.ListS3",
+ "schedulingStrategy": "TIMER_DRIVEN",
+ "schedulingPeriod": "10000 ms",
+ "properties": {
+ "AWS Credentials Provider service": "AWSCredentialsService",
+ "Bucket": "test_bucket",
+ "Communications Timeout": "30 sec"
+ },
+ "autoTerminatedRelationships": []
+ },
+ {
+ "name": "Log S3 object attributes",
+ "identifier": "73fd7f54-7639-4f27-b20b-889708f493cb",
+ "type": "org.apache.nifi.minifi.processors.LogAttribute",
+ "schedulingStrategy": "EVENT_DRIVEN",
+ "properties": {
+ "FlowFiles To Log": "0"
+ },
+ "autoTerminatedRelationships": [
+ "success"
+ ]
+ }
+ ],
+ "connections": [
+ {
+ "name": "FetchS3Object/success/LogAttribute",
+ "identifier": "2c13ddd6-543e-4e37-a295-e4cb8fe85fa5",
+ "source": {
+ "id": "6ccd8be9-4aa3-4b86-ae10-0a68668f1c25"
+ },
+ "destination": {
+ "id": "73fd7f54-7639-4f27-b20b-889708f493cb"
+ },
+ "selectedRelationships": [
+ "success"
+ ]
+ },
+ {
+ "name": "ListS3/success/FetchS3Object",
+ "identifier": "7895c12e-9657-4baa-a8ec-6b8cb11f32cd",
+ "source": {
+ "id": "6f2c20f5-e990-4d9f-bfd6-f7471e3df469"
+ },
+ "destination": {
+ "id": "6ccd8be9-4aa3-4b86-ae10-0a68668f1c25"
+ },
+ "selectedRelationships": [
+ "success"
+ ]
+ }
+ ],
+ "remoteProcessGroups": [],
+ "controllerServices": [
+ {
+ "name": "AWSCredentialsService",
+ "identifier": "2094d776-2006-4d02-9bb9-28eac9d0fc95",
+ "type": "org.apache.nifi.minifi.aws.controllers.AWSCredentialsService",
+ "properties": {
+ "Use Default Credentials": "true"
+ }
+ }
+ ]
+ }
+}
diff --git a/examples/lists3_fetchs3_config.nifi.schema.json b/examples/lists3_fetchs3_config.nifi.schema.json
new file mode 100644
index 000000000..93e74a534
--- /dev/null
+++ b/examples/lists3_fetchs3_config.nifi.schema.json
@@ -0,0 +1,492 @@
+{
+ "encodingVersion": {
+ "majorVersion": 2,
+ "minorVersion": 0
+ },
+ "maxTimerDrivenThreadCount": 1,
+ "maxEventDrivenThreadCount": 1,
+ "parameterContexts": [],
+ "rootGroup": {
+ "identifier": "1c2f34d2-ee57-448a-b150-8140f0f7643c",
+ "instanceIdentifier": "9af45830-320e-437e-8204-dcd3c2933aa6",
+ "name": "root",
+ "position": {
+ "x": 0.0,
+ "y": 0.0
+ },
+ "processGroups": [],
+ "remoteProcessGroups": [],
+ "processors": [
+ {
+ "identifier": "1f27e7fd-4463-40a2-acfd-fd3d5cafae50",
+ "instanceIdentifier": "5e273549-c27b-48fa-b3a3-c1ff837749ea",
+ "name": "Log S3 object attributes",
+ "comments": "",
+ "position": {
+ "x": 447.0,
+ "y": 100.0
+ },
+ "type": "org.apache.nifi.minifi.processors.LogAttribute",
+ "bundle": {
+ "group": "org.apache.nifi.minifi",
+ "artifact": "minifi-standard-processors",
+ "version": "0.15.0"
+ },
+ "properties": {
+ "FlowFiles To Log": "0",
+ "Log Payload": "false",
+ "Hexencode Payload": "false",
+ "Maximum Payload Line Length": "0"
+ },
+ "propertyDescriptors": {
+ "FlowFiles To Log": {
+ "name": "FlowFiles To Log",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Log Level": {
+ "name": "Log Level",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Attributes to Ignore": {
+ "name": "Attributes to Ignore",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Attributes to Log": {
+ "name": "Attributes to Log",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Log Payload": {
+ "name": "Log Payload",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Log Prefix": {
+ "name": "Log Prefix",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Hexencode Payload": {
+ "name": "Hexencode Payload",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Maximum Payload Line Length": {
+ "name": "Maximum Payload Line Length",
+ "identifiesControllerService": false,
+ "sensitive": false
+ }
+ },
+ "style": {},
+ "schedulingStrategy": "EVENT_DRIVEN",
+ "executionNode": "ALL",
+ "penaltyDuration": "30000 ms",
+ "yieldDuration": "1000 ms",
+ "bulletinLevel": "WARN",
+ "runDurationMillis": 0,
+ "concurrentlySchedulableTaskCount": 1,
+ "autoTerminatedRelationships": [
+ "success"
+ ],
+ "componentType": "PROCESSOR",
+ "groupIdentifier": "1c2f34d2-ee57-448a-b150-8140f0f7643c"
+ },
+ {
+ "identifier": "1b8dc9d5-9f61-4739-80b6-638fbb4ff8f1",
+ "instanceIdentifier": "8396d665-31b7-43b7-84f0-41cfc884e029",
+ "name": "Fetch S3 object from test_bucket",
+ "comments": "",
+ "position": {
+ "x": 247.0,
+ "y": 103.0
+ },
+ "type": "org.apache.nifi.minifi.aws.processors.FetchS3Object",
+ "bundle": {
+ "group": "org.apache.nifi.minifi",
+ "artifact": "minifi-aws",
+ "version": "0.15.0"
+ },
+ "properties": {
+ "Bucket": "test_bucket",
+ "Use Default Credentials": "false",
+ "Communications Timeout": "30 sec",
+ "Region": "us-west-2",
+ "Requester Pays": "false",
+ "AWS Credentials Provider service": "AWSCredentialsService"
+ },
+ "propertyDescriptors": {
+ "Proxy Host": {
+ "name": "Proxy Host",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Proxy Username": {
+ "name": "Proxy Username",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Access Key": {
+ "name": "Access Key",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Proxy Password": {
+ "name": "Proxy Password",
+ "identifiesControllerService": false,
+ "sensitive": true
+ },
+ "Endpoint Override URL": {
+ "name": "Endpoint Override URL",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Proxy Port": {
+ "name": "Proxy Port",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Object Key": {
+ "name": "Object Key",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "AWS Credentials Provider service": {
+ "name": "AWS Credentials Provider service",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Bucket": {
+ "name": "Bucket",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Use Default Credentials": {
+ "name": "Use Default Credentials",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Version": {
+ "name": "Version",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Communications Timeout": {
+ "name": "Communications Timeout",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Secret Key": {
+ "name": "Secret Key",
+ "identifiesControllerService": false,
+ "sensitive": true
+ },
+ "Credentials File": {
+ "name": "Credentials File",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Region": {
+ "name": "Region",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Requester Pays": {
+ "name": "Requester Pays",
+ "identifiesControllerService": false,
+ "sensitive": false
+ }
+ },
+ "style": {},
+ "schedulingStrategy": "EVENT_DRIVEN",
+ "executionNode": "ALL",
+ "penaltyDuration": "30000 ms",
+ "yieldDuration": "1000 ms",
+ "bulletinLevel": "WARN",
+ "runDurationMillis": 0,
+ "concurrentlySchedulableTaskCount": 1,
+ "autoTerminatedRelationships": [
+ "failure"
+ ],
+ "componentType": "PROCESSOR",
+ "groupIdentifier": "1c2f34d2-ee57-448a-b150-8140f0f7643c"
+ },
+ {
+ "identifier": "d472aea5-3f4e-45bc-accb-d6ad4f783622",
+ "instanceIdentifier": "713e1f3f-33c4-44fb-8055-a82d760755a9",
+ "name": "List test_bucket S3 bucket",
+ "comments": "",
+ "position": {
+ "x": 36.0,
+ "y": 96.0
+ },
+ "type": "org.apache.nifi.minifi.aws.processors.ListS3",
+ "bundle": {
+ "group": "org.apache.nifi.minifi",
+ "artifact": "minifi-aws",
+ "version": "0.15.0"
+ },
+ "properties": {
+ "Minimum Object Age": "0 sec",
+ "Bucket": "test_bucket",
+ "Use Default Credentials": "false",
+ "Communications Timeout": "30 sec",
+ "Region": "us-west-2",
+ "Use Versions": "false",
+ "Write User Metadata": "false",
+ "Requester Pays": "false",
+ "AWS Credentials Provider service": "AWSCredentialsService",
+ "Write Object Tags": "false"
+ },
+ "propertyDescriptors": {
+ "Proxy Host": {
+ "name": "Proxy Host",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Proxy Username": {
+ "name": "Proxy Username",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Minimum Object Age": {
+ "name": "Minimum Object Age",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Access Key": {
+ "name": "Access Key",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Delimiter": {
+ "name": "Delimiter",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Proxy Password": {
+ "name": "Proxy Password",
+ "identifiesControllerService": false,
+ "sensitive": true
+ },
+ "Endpoint Override URL": {
+ "name": "Endpoint Override URL",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Prefix": {
+ "name": "Prefix",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Proxy Port": {
+ "name": "Proxy Port",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "AWS Credentials Provider service": {
+ "name": "AWS Credentials Provider service",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Write Object Tags": {
+ "name": "Write Object Tags",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Bucket": {
+ "name": "Bucket",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Use Default Credentials": {
+ "name": "Use Default Credentials",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Communications Timeout": {
+ "name": "Communications Timeout",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Secret Key": {
+ "name": "Secret Key",
+ "identifiesControllerService": false,
+ "sensitive": true
+ },
+ "Credentials File": {
+ "name": "Credentials File",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Region": {
+ "name": "Region",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Use Versions": {
+ "name": "Use Versions",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Write User Metadata": {
+ "name": "Write User Metadata",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Requester Pays": {
+ "name": "Requester Pays",
+ "identifiesControllerService": false,
+ "sensitive": false
+ }
+ },
+ "style": {},
+ "schedulingPeriod": "10000 ms",
+ "schedulingStrategy": "TIMER_DRIVEN",
+ "executionNode": "ALL",
+ "penaltyDuration": "30000 ms",
+ "yieldDuration": "1000 ms",
+ "bulletinLevel": "WARN",
+ "runDurationMillis": 0,
+ "concurrentlySchedulableTaskCount": 1,
+ "autoTerminatedRelationships": [],
+ "componentType": "PROCESSOR",
+ "groupIdentifier": "1c2f34d2-ee57-448a-b150-8140f0f7643c"
+ }
+ ],
+ "inputPorts": [],
+ "outputPorts": [],
+ "connections": [
+ {
+ "identifier": "1140c720-9c7c-4ff9-9bb4-1b08bcfca185",
+ "instanceIdentifier": "1e01b749-1e33-400c-b997-f89f29cde46f",
+ "name": "List test_bucket S3 bucket/success/Fetch S3 object from test_bucket",
+ "position": {
+ "x": 0.0,
+ "y": 0.0
+ },
+ "source": {
+ "id": "d472aea5-3f4e-45bc-accb-d6ad4f783622",
+ "type": "PROCESSOR",
+ "groupId": "1c2f34d2-ee57-448a-b150-8140f0f7643c",
+ "name": "List test_bucket S3 bucket",
+ "instanceIdentifier": "713e1f3f-33c4-44fb-8055-a82d760755a9"
+ },
+ "destination": {
+ "id": "1b8dc9d5-9f61-4739-80b6-638fbb4ff8f1",
+ "type": "PROCESSOR",
+ "groupId": "1c2f34d2-ee57-448a-b150-8140f0f7643c",
+ "name": "Fetch S3 object from test_bucket",
+ "instanceIdentifier": "8396d665-31b7-43b7-84f0-41cfc884e029"
+ },
+ "labelIndex": 1,
+ "zIndex": 0,
+ "selectedRelationships": [
+ "success"
+ ],
+ "backPressureObjectThreshold": 2000,
+ "backPressureDataSizeThreshold": "100 MB",
+ "flowFileExpiration": "0 seconds",
+ "prioritizers": [],
+ "bends": [],
+ "componentType": "CONNECTION",
+ "groupIdentifier": "1c2f34d2-ee57-448a-b150-8140f0f7643c"
+ },
+ {
+ "identifier": "dc1b813f-ac8d-4081-9fa0-7dc58ab8a05c",
+ "instanceIdentifier": "501be62a-5793-4c51-8fa8-6cbcad14961d",
+ "name": "Fetch S3 object from test_bucket/success/Log S3 object attributes",
+ "position": {
+ "x": 0.0,
+ "y": 0.0
+ },
+ "source": {
+ "id": "1b8dc9d5-9f61-4739-80b6-638fbb4ff8f1",
+ "type": "PROCESSOR",
+ "groupId": "1c2f34d2-ee57-448a-b150-8140f0f7643c",
+ "name": "Fetch S3 object from test_bucket",
+ "instanceIdentifier": "8396d665-31b7-43b7-84f0-41cfc884e029"
+ },
+ "destination": {
+ "id": "1f27e7fd-4463-40a2-acfd-fd3d5cafae50",
+ "type": "PROCESSOR",
+ "groupId": "1c2f34d2-ee57-448a-b150-8140f0f7643c",
+ "name": "Log S3 object attributes",
+ "instanceIdentifier": "5e273549-c27b-48fa-b3a3-c1ff837749ea"
+ },
+ "labelIndex": 1,
+ "zIndex": 0,
+ "selectedRelationships": [
+ "success"
+ ],
+ "backPressureObjectThreshold": 2000,
+ "backPressureDataSizeThreshold": "100 MB",
+ "flowFileExpiration": "0 seconds",
+ "prioritizers": [],
+ "bends": [],
+ "componentType": "CONNECTION",
+ "groupIdentifier": "1c2f34d2-ee57-448a-b150-8140f0f7643c"
+ }
+ ],
+ "labels": [],
+ "funnels": [],
+ "controllerServices": [
+ {
+ "identifier": "b4a19d3f-2db7-493e-9f13-6130ef4ccae0",
+ "instanceIdentifier": "77c68d5a-52da-4b5b-8a04-e915e6fe91ca",
+ "name": "AWSCredentialsService",
+ "position": {
+ "x": 0.0,
+ "y": 0.0
+ },
+ "type": "org.apache.nifi.minifi.aws.controllers.AWSCredentialsService",
+ "bundle": {
+ "group": "org.apache.nifi.minifi",
+ "artifact": "minifi-aws",
+ "version": "0.15.0"
+ },
+ "properties": {
+ "Use Default Credentials": "true"
+ },
+ "propertyDescriptors": {
+ "Access Key": {
+ "name": "Access Key",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Use Default Credentials": {
+ "name": "Use Default Credentials",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Secret Key": {
+ "name": "Secret Key",
+ "identifiesControllerService": false,
+ "sensitive": true
+ },
+ "Credentials File": {
+ "name": "Credentials File",
+ "identifiesControllerService": false,
+ "sensitive": false
+ }
+ },
+ "controllerServiceApis": [
+ {
+ "type": "org.apache.nifi.minifi.aws.controllers.AWSCredentialsService",
+ "bundle": {
+ "group": "org.apache.nifi.minifi",
+ "artifact": "minifi-aws",
+ "version": "0.15.0"
+ }
+ }
+ ],
+ "componentType": "CONTROLLER_SERVICE",
+ "groupIdentifier": "1c2f34d2-ee57-448a-b150-8140f0f7643c"
+ }
+ ],
+ "variables": {},
+ "componentType": "PROCESS_GROUP"
+ }
+}
diff --git a/examples/merge_compress_and_upload_to_gcs_config.json b/examples/merge_compress_and_upload_to_gcs_config.json
new file mode 100644
index 000000000..f79ffaba5
--- /dev/null
+++ b/examples/merge_compress_and_upload_to_gcs_config.json
@@ -0,0 +1,157 @@
+{
+ "rootGroup": {
+ "name": "MiNiFi Flow",
+ "processors": [
+ {
+ "name": "Compress to tar.gz format",
+ "identifier": "333d994b-0637-4c46-b515-89464523ccb8",
+ "type": "org.apache.nifi.minifi.processors.CompressContent",
+ "maxConcurrentTasks": 1,
+ "schedulingStrategy": "EVENT_DRIVEN",
+ "autoTerminatedRelationships": [
+ "failure"
+ ],
+ "properties": {
+ "Batch Size": 1,
+ "Compression Format": "gzip",
+ "Compression Level": 6,
+ "Encapsulate in TAR": true,
+ "Mode": "compress",
+ "Update Filename": true
+ }
+ },
+ {
+ "name": "Tail /tmp/test_file.log",
+ "identifier": "47bd6218-14ab-4180-95ff-3db72e82d97c",
+ "type": "org.apache.nifi.minifi.processors.TailFile",
+ "schedulingStrategy": "TIMER_DRIVEN",
+ "schedulingPeriod": "1000 ms",
+ "properties": {
+ "File to Tail": "/tmp/test_file.log",
+ "Initial Start Position": "Beginning of File",
+ "Input Delimiter": "\\n",
+ "tail-mode": "Single file"
+ }
+ },
+ {
+ "name": "Merge every 10 lines",
+ "identifier": "d527e0ff-0504-4954-b42f-15771a6e7366",
+ "type": "org.apache.nifi.minifi.processors.MergeContent",
+ "maxConcurrentTasks": 1,
+ "schedulingStrategy": "EVENT_DRIVEN",
+ "autoTerminatedRelationships": [
+ "original",
+ "failure"
+ ],
+ "properties": {
+ "Attribute Strategy": "Keep Only Common Attributes",
+ "Batch Size": 1,
+ "Correlation Attribute Name": "test",
+ "Delimiter Strategy": "Filename",
+ "Keep Path": false,
+ "Maximum Number of Entries": 10,
+ "Maximum number of Bins": 100,
+ "Merge Format": "Binary Concatenation",
+ "Merge Strategy": "Bin-Packing Algorithm",
+ "Minimum Group Size": 0,
+ "Minimum Number of Entries": 10
+ }
+ },
+ {
+ "name": "Upload to Google Cloud Storage",
+ "identifier": "58da4d62-1f29-4fc6-92b8-c223f081d701",
+ "type": "org.apache.nifi.minifi.aws.processors.PutGCSObject",
+ "maxConcurrentTasks": 1,
+ "schedulingStrategy": "EVENT_DRIVEN",
+ "autoTerminatedRelationships": [
+ "success",
+ "failure"
+ ],
+ "properties": {
+ "Bucket": "test-bucket",
+ "Number of retries": 2,
+ "GCP Credentials Provider Service": "GCPCredentialsControllerService"
+ }
+ },
+ {
+ "name": "Add upload target attribute",
+ "identifier": "78000004-98b5-44ec-a25c-1f0e0134ffe8",
+ "type": "org.apache.nifi.minifi.processors.UpdateAttribute",
+ "maxConcurrentTasks": 1,
+ "schedulingStrategy": "EVENT_DRIVEN",
+ "autoTerminatedRelationships": [
+ "failure"
+ ],
+ "properties": {
+ "upload_target": "google_cloud_storage"
+ }
+ }
+ ],
+ "funnels": [],
+ "connections": [
+ {
+ "identifier": "b9558376-04e8-4e42-a4cd-41e31754e585",
+ "name": "CompressContent/success/PutGCSObject",
+ "source": {
+ "id": "333d994b-0637-4c46-b515-89464523ccb8"
+ },
+ "destination": {
+ "id": "58da4d62-1f29-4fc6-92b8-c223f081d701"
+ },
+ "selectedRelationships": [
+ "success"
+ ]
+ },
+ {
+ "identifier": "7ebc1173-5d15-4894-836e-e2597496dfe9",
+ "name": "TailFile/success/UpdateAttribute",
+ "source": {
+ "id": "47bd6218-14ab-4180-95ff-3db72e82d97c"
+ },
+ "destination": {
+ "id": "78000004-98b5-44ec-a25c-1f0e0134ffe8"
+ },
+ "selectedRelationships": [
+ "success"
+ ]
+ },
+ {
+ "identifier": "5c6d3cec-17c7-479f-919a-99ac8762de5e",
+ "name": "MergeContent/merged/CompressContent",
+ "source": {
+ "id": "d527e0ff-0504-4954-b42f-15771a6e7366"
+ },
+ "destination": {
+ "id": "333d994b-0637-4c46-b515-89464523ccb8"
+ },
+ "selectedRelationships": [
+ "merged"
+ ]
+ },
+ {
+ "identifier": "8a247c4d-0c9f-495f-8a00-5c8a21f56a9e",
+ "name": "UpdateAttribute/success/MergeContent",
+ "source": {
+ "id": "78000004-98b5-44ec-a25c-1f0e0134ffe8"
+ },
+ "destination": {
+ "id": "d527e0ff-0504-4954-b42f-15771a6e7366"
+ },
+ "selectedRelationships": [
+ "success"
+ ]
+ }
+ ],
+ "remoteProcessGroups": [],
+ "controllerServices": [
+ {
+ "name": "GCPCredentialsControllerService",
+ "identifier": "c1c49f60-c054-4df8-bce9-8f0660dcc55e",
+ "type": "org.apache.nifi.minifi.controllers.GCPCredentialsControllerService",
+ "properties": {
+ "Credentials Location": "Use Anonymous credentials"
+ }
+ }
+ ]
+ }
+}
diff --git a/examples/merge_compress_and_upload_to_gcs_config.yml b/examples/merge_compress_and_upload_to_gcs_config.yml
new file mode 100644
index 000000000..066a02e5f
--- /dev/null
+++ b/examples/merge_compress_and_upload_to_gcs_config.yml
@@ -0,0 +1,116 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+MiNiFi Config Version: 3
+Flow Controller:
+ name: MiNiFi Flow
+Processors:
+ - id: 333d994b-0637-4c46-b515-89464523ccb8
+ name: Compress to tar.gz format
+ class: org.apache.nifi.minifi.processors.CompressContent
+ max concurrent tasks: 1
+ scheduling strategy: EVENT_DRIVEN
+ auto-terminated relationships list:
+ - failure
+ Properties:
+ Batch Size: 1
+ Compression Format: gzip
+ Compression Level: 6
+ Encapsulate in TAR: true
+ Mode: compress
+ Update Filename: true
+ - name: Tail /tmp/test_file.log
+ id: 47bd6218-14ab-4180-95ff-3db72e82d97c
+ class: org.apache.nifi.minifi.processors.TailFile
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 1000 ms
+ Properties:
+ File to Tail: /tmp/test_file.log
+ Initial Start Position: Beginning of File
+ Input Delimiter: \n
+ tail-mode: Single file
+ - id: d527e0ff-0504-4954-b42f-15771a6e7366
+ name: Merge every 10 lines
+ class: org.apache.nifi.minifi.processors.MergeContent
+ max concurrent tasks: 1
+ scheduling strategy: EVENT_DRIVEN
+ auto-terminated relationships list:
+ - original
+ - failure
+ Properties:
+ Attribute Strategy: Keep Only Common Attributes
+ Batch Size: 1
+ Correlation Attribute Name: test
+ Delimiter Strategy: Filename
+ Keep Path: false
+ Maximum Number of Entries: 10
+ Maximum number of Bins: 100
+ Merge Format: Binary Concatenation
+ Merge Strategy: Bin-Packing Algorithm
+ Minimum Group Size: 0
+ Minimum Number of Entries: 10
+ - id: 58da4d62-1f29-4fc6-92b8-c223f081d701
+ name: Upload to Google Cloud Storage
+ class: org.apache.nifi.minifi.aws.processors.PutGCSObject
+ max concurrent tasks: 1
+ scheduling strategy: EVENT_DRIVEN
+ auto-terminated relationships list:
+ - success
+ - failure
+ Properties:
+ Bucket: test-bucket
+ Number of retries: 2
+ GCP Credentials Provider Service: GCPCredentialsControllerService
+ - id: 78000004-98b5-44ec-a25c-1f0e0134ffe8
+ name: Add upload target attribute
+ class: org.apache.nifi.minifi.processors.UpdateAttribute
+ max concurrent tasks: 1
+ scheduling strategy: EVENT_DRIVEN
+ auto-terminated relationships list:
+ - failure
+ Properties:
+ upload_target: google_cloud_storage
+Controller Services:
+ - name: GCPCredentialsControllerService
+ id: c1c49f60-c054-4df8-bce9-8f0660dcc55e
+ class: org.apache.nifi.minifi.controllers.GCPCredentialsControllerService
+ Properties:
+ Credentials Location: Use Anonymous credentials
+Connections:
+ - id: b9558376-04e8-4e42-a4cd-41e31754e585
+ name: CompressContent/success/PutGCSObject
+ source id: 333d994b-0637-4c46-b515-89464523ccb8
+ source relationship names:
+ - success
+ destination id: 58da4d62-1f29-4fc6-92b8-c223f081d701
+ - id: 7ebc1173-5d15-4894-836e-e2597496dfe9
+ name: TailFile/success/UpdateAttribute
+ source id: 47bd6218-14ab-4180-95ff-3db72e82d97c
+ source relationship names:
+ - success
+ destination id: 78000004-98b5-44ec-a25c-1f0e0134ffe8
+ - id: 5c6d3cec-17c7-479f-919a-99ac8762de5e
+ name: MergeContent/merged/CompressContent
+ source id: d527e0ff-0504-4954-b42f-15771a6e7366
+ source relationship names:
+ - merged
+ destination id: 333d994b-0637-4c46-b515-89464523ccb8
+ - id: 8a247c4d-0c9f-495f-8a00-5c8a21f56a9e
+ name: UpdateAttribute/success/MergeContent
+ source id: 78000004-98b5-44ec-a25c-1f0e0134ffe8
+ source relationship names:
+ - success
+ destination id: d527e0ff-0504-4954-b42f-15771a6e7366
+Remote Process Groups: []
diff --git a/examples/mqtt_config.json b/examples/mqtt_config.json
new file mode 100644
index 000000000..834fdf205
--- /dev/null
+++ b/examples/mqtt_config.json
@@ -0,0 +1,52 @@
+{
+ "rootGroup": {
+ "name": "MiNiFi Flow",
+ "processors": [
+ {
+ "name": "Get files from /tmp/input",
+ "identifier": "1dbeda1d-1c89-4f33-a8fa-4c3b76abaed1",
+ "type": "org.apache.nifi.processors.standard.GetFile",
+ "schedulingStrategy": "TIMER_DRIVEN",
+ "schedulingPeriod": "2 sec",
+ "properties": {
+ "Input Directory": "/tmp/input"
+ },
+ "autoTerminatedRelationships": []
+ },
+ {
+ "name": "Publish data to MQTT broker",
+ "identifier": "6ecff0a9-9d1b-45c6-99b7-3e4db40e0cd1",
+ "type": "org.apache.nifi.processors.standard.PublishMQTT",
+ "schedulingStrategy": "EVENT_DRIVEN",
+ "properties": {
+ "Broker URI": "127.0.0.1:1883",
+ "Topic": "testtopic",
+ "Client ID": "publisher-client",
+ "MQTT Version": "3.x AUTO"
+ },
+ "autoTerminatedRelationships": [
+ "failure",
+ "success"
+ ]
+ }
+ ],
+ "funnels": [],
+ "connections": [
+ {
+ "identifier": "3783d336-1d81-452b-9d34-8be77bc8fce5",
+ "name": "GetFile/success/PublishMQTT",
+ "source": {
+ "id": "1dbeda1d-1c89-4f33-a8fa-4c3b76abaed1"
+ },
+ "destination": {
+ "id": "6ecff0a9-9d1b-45c6-99b7-3e4db40e0cd1"
+ },
+ "selectedRelationships": [
+ "success"
+ ]
+ }
+ ],
+ "remoteProcessGroups": [],
+ "controllerServices": []
+ }
+}
diff --git a/examples/mqtt_config.yml b/examples/mqtt_config.yml
new file mode 100644
index 000000000..0ccc873ba
--- /dev/null
+++ b/examples/mqtt_config.yml
@@ -0,0 +1,32 @@
+MiNiFi Config Version: 3
+Flow Controller:
+ name: MiNiFi Flow
+Processors:
+- name: Get files from /tmp/input
+ id: 1dbeda1d-1c89-4f33-a8fa-4c3b76abaed1
+ class: org.apache.nifi.processors.standard.GetFile
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 2000 ms
+ Properties:
+ Input Directory: /tmp/input
+- name: Publish data to MQTT broker
+ id: 6ecff0a9-9d1b-45c6-99b7-3e4db40e0cd1
+ class: org.apache.nifi.processors.standard.PublishMQTT
+ scheduling strategy: EVENT_DRIVEN
+ auto-terminated relationships list:
+ - failure
+ - success
+ Properties:
+ Broker URI: 127.0.0.1:1883
+ Topic: testtopic
+ Client ID: publisher-client
+ MQTT Version: '3.x AUTO'
+Connections:
+- name: GetFile/success/PublishMQTT
+ id: 3783d336-1d81-452b-9d34-8be77bc8fce5
+ source id: 1dbeda1d-1c89-4f33-a8fa-4c3b76abaed1
+ source relationship names:
+ - success
+ destination id: 6ecff0a9-9d1b-45c6-99b7-3e4db40e0cd1
+Controller Services: []
+Remote Process Groups: []
diff --git a/examples/pdh_config.json b/examples/pdh_config.json
new file mode 100644
index 000000000..2d199c38c
--- /dev/null
+++ b/examples/pdh_config.json
@@ -0,0 +1,53 @@
+{
+ "rootGroup": {
+ "name": "MiNiFi Flow",
+ "processors": [
+ {
+ "name": "Read disk and CPU performance data",
+ "identifier": "2f2a3b47-f5ba-49f6-82b5-bc1c86b96e27",
+ "type": "org.apache.nifi.minifi.processors.PerformanceDataMonitor",
+ "schedulingStrategy": "TIMER_DRIVEN",
+ "schedulingPeriod": "10000 ms",
+ "properties": {
+ "Predefined Groups": "Disk,CPU",
+ "Custom PDH Counters": "\\System\\Processes,\\Process(*)\\ID Process,\\Process(*)\\Private Bytes",
+ "Output Format": "JSON",
+ "Output Compactness": "Compact"
+ },
+ "autoTerminatedRelationships": []
+ },
+ {
+ "name": "Put files to C:\\temp\\",
+ "identifier": "e143601d-de4f-44ba-a6ec-d1f97d77ec94",
+ "type": "org.apache.nifi.minifi.processors.PutFile",
+ "schedulingStrategy": "EVENT_DRIVEN",
+ "properties": {
+ "Conflict Resolution Strategy": "fail",
+ "Create Missing Directories": "true",
+ "Directory": "C:\\temp\\"
+ },
+ "autoTerminatedRelationships": [
+ "failure",
+ "success"
+ ]
+ }
+ ],
+ "connections": [
+ {
+ "name": "PerformanceDataMonitor/success/PutFile",
+ "identifier": "098a56ba-f4bf-4323-a3f3-6f8a5e3586bf",
+ "source": {
+ "id": "2f2a3b47-f5ba-49f6-82b5-bc1c86b96e27"
+ },
+ "destination": {
+ "id": "e143601d-de4f-44ba-a6ec-d1f97d77ec94"
+ },
+ "selectedRelationships": [
+ "success"
+ ]
+ }
+ ],
+ "remoteProcessGroups": [],
+ "controllerServices": []
+ }
+}
diff --git a/examples/process_data_with_scripts.json b/examples/process_data_with_scripts.json
new file mode 100644
index 000000000..12d2b981e
--- /dev/null
+++ b/examples/process_data_with_scripts.json
@@ -0,0 +1,105 @@
+{
+ "rootGroup": {
+ "name": "MiNiFi Flow",
+ "processors": [
+ {
+ "name": "ExecuteScript",
+ "identifier": "9e5dc411-cc01-4359-adea-ff70f07b3402",
+ "type": "org.apache.nifi.minifi.processors.ExecuteScript",
+ "schedulingStrategy": "TIMER_DRIVEN",
+ "schedulingPeriod": "1000 ms",
+ "penaltyDuration": "30000 ms",
+ "yieldDuration": "1000 ms",
+ "properties": {
+ "Module Directory": "",
+ "Script Body": "",
+ "Script Engine": "python",
+ "Script File": "../../examples/scripts/python/reverse_flow_file_content.py"
+ },
+ "autoTerminatedRelationships": [
+ "failure"
+ ]
+ },
+ {
+ "name": "GenerateFlowFile",
+ "identifier": "b3972d8e-f0de-420a-a259-4d46017699e7",
+ "type": "org.apache.nifi.minifi.processors.GenerateFlowFile",
+ "schedulingStrategy": "TIMER_DRIVEN",
+ "schedulingPeriod": "1000 ms",
+ "penaltyDuration": "30000 ms",
+ "yieldDuration": "1000 ms",
+ "properties": {
+ "Batch Size": "1",
+ "Custom Text": "here is a generated uuid: ${UUID()}",
+ "Data Format": "Text",
+ "File Size": "1 kB",
+ "Unique FlowFiles": "false"
+ },
+ "autoTerminatedRelationships": []
+ },
+ {
+ "name": "PutFile",
+ "identifier": "ea186cb6-48e6-40e6-98f6-b78568d9f91d",
+ "type": "org.apache.nifi.minifi.processors.PutFile",
+ "schedulingStrategy": "TIMER_DRIVEN",
+ "schedulingPeriod": "1000 ms",
+ "penaltyDuration": "30000 ms",
+ "yieldDuration": "1000 ms",
+ "properties": {
+ "Conflict Resolution Strategy": "fail",
+ "Create Missing Directories": "true",
+ "Directory": "./reversed_flow_files/",
+ "Directory Permissions": "",
+ "Maximum File Count": "-1",
+ "Permissions": ""
+ },
+ "autoTerminatedRelationships": [
+ "failure",
+ "success"
+ ]
+ }
+ ],
+ "connections": [
+ {
+ "identifier": "938cc13f-ae5a-43fc-ab10-e71cecef652e",
+ "name": "ExecuteScript/success/PutFile",
+ "source": {
+ "id": "9e5dc411-cc01-4359-adea-ff70f07b3402"
+ },
+ "destination": {
+ "id": "ea186cb6-48e6-40e6-98f6-b78568d9f91d"
+ },
+ "selectedRelationships": [
+ "success"
+ ],
+ "maxWorkQueueSize": 10000,
+ "maxWorkQueueDataSize": "10 MB",
+ "flowFileExpiration": "0 seconds",
+ "queuePrioritizerClass": ""
+ },
+ {
+ "identifier": "7612fcc2-c80e-4967-aeec-2b1076376f49",
+ "name": "GenerateFlowFile/success/ExecuteScript",
+ "source": {
+ "id": "b3972d8e-f0de-420a-a259-4d46017699e7"
+ },
+ "destination": {
+ "id": "9e5dc411-cc01-4359-adea-ff70f07b3402"
+ },
+ "selectedRelationships": [
+ "success"
+ ],
+ "maxWorkQueueSize": 10000,
+ "maxWorkQueueDataSize": "10 MB",
+ "flowFileExpiration": "0 seconds",
+ "queuePrioritizerClass": ""
+ }
+ ],
+ "remoteProcessGroups": [],
+ "controllerServices": [],
+ "processGroups": [],
+ "inputPorts": [],
+ "outputPorts": [],
+ "funnels": []
+ }
+}
diff --git a/examples/publishkafka_config.json b/examples/publishkafka_config.json
new file mode 100644
index 000000000..27cee97fb
--- /dev/null
+++ b/examples/publishkafka_config.json
@@ -0,0 +1,55 @@
+{
+ "rootGroup": {
+ "name": "MiNiFi Flow",
+ "processors": [
+ {
+ "name": "Get files from /tmp/input",
+ "identifier": "7fd166aa-0662-4c42-affa-88f6fb39807f",
+ "type": "org.apache.nifi.processors.standard.GetFile",
+ "schedulingStrategy": "TIMER_DRIVEN",
+ "schedulingPeriod": "2 sec",
+ "properties": {
+ "Input Directory": "/tmp/input"
+ },
+ "autoTerminatedRelationships": []
+ },
+ {
+ "name": "Publish messages to Kafka topic test",
+ "identifier": "8a534b4a-2b4a-4e1e-ab07-8a09fa08f848",
+ "type": "org.apache.nifi.processors.standard.PublishKafka",
+ "schedulingStrategy": "EVENT_DRIVEN",
+ "properties": {
+ "Batch Size": "10",
+ "Client Name": "test-client",
+ "Compress Codec": "none",
+ "Delivery Guarantee": "1",
+ "Known Brokers": "kafka-broker:9092",
+ "Message Timeout": "12 sec",
+ "Request Timeout": "10 sec",
+ "Topic Name": "test"
+ },
+ "autoTerminatedRelationships": [
+ "success",
+ "failure"
+ ]
+ }
+ ],
+ "connections": [
+ {
+ "identifier": "1edd529e-eee9-4b05-9e35-f1607bb0243b",
+ "name": "GetFile/success/PublishKafka",
+ "source": {
+ "id": "7fd166aa-0662-4c42-affa-88f6fb39807f"
+ },
+ "destination": {
+ "id": "8a534b4a-2b4a-4e1e-ab07-8a09fa08f848"
+ },
+ "selectedRelationships": [
+ "success"
+ ]
+ }
+ ],
+ "remoteProcessGroups": [],
+ "controllerServices": []
+ }
+}
diff --git a/examples/publishkafka_ssl_config.json b/examples/publishkafka_ssl_config.json
new file mode 100644
index 000000000..4eea265e3
--- /dev/null
+++ b/examples/publishkafka_ssl_config.json
@@ -0,0 +1,60 @@
+{
+ "rootGroup": {
+ "name": "MiNiFi Flow",
+ "processors": [
+ {
+ "name": "Get files from /tmp/input",
+ "identifier": "a78f9d7d-cc68-4f30-aac1-ee716d42bf9d",
+ "type": "org.apache.nifi.processors.standard.GetFile",
+ "schedulingStrategy": "TIMER_DRIVEN",
+ "schedulingPeriod": "2 sec",
+ "properties": {
+ "Input Directory": "/tmp/input"
+ },
+ "autoTerminatedRelationships": []
+ },
+ {
+ "name": "Publish messages to Kafka topic test",
+ "identifier": "8c0791c7-920c-4541-bcde-f5d559f42ed2",
+ "type": "org.apache.nifi.processors.standard.PublishKafka",
+ "schedulingStrategy": "EVENT_DRIVEN",
+ "properties": {
+ "Batch Size": "10",
+ "Client Name": "test_client",
+ "Compress Codec": "none",
+ "Delivery Guarantee": "1",
+ "Known Brokers": "kafka-broker:9093",
+ "Message Timeout": "12 sec",
+ "Request Timeout": "10 sec",
+ "Security CA": "/tmp/resources/certs/ca-cert",
+ "Security Cert": "/tmp/resources/certs/client_test_client_client.pem",
+ "Security Pass Phrase": "abcdefgh",
+ "Security Private Key": "/tmp/resources/certs/client_test_client_client.key",
+ "Security Protocol": "ssl",
+ "Topic Name": "test"
+ },
+ "autoTerminatedRelationships": [
+ "success",
+ "failure"
+ ]
+ }
+ ],
+ "connections": [
+ {
+ "identifier": "ab45e106-3d97-4a59-8021-e50aa68f1a32",
+ "name": "GetFile/success/PublishKafka",
+ "source": {
+ "id": "a78f9d7d-cc68-4f30-aac1-ee716d42bf9d"
+ },
+ "destination": {
+ "id": "8c0791c7-920c-4541-bcde-f5d559f42ed2"
+ },
+ "selectedRelationships": [
+ "success"
+ ]
+ }
+ ],
+ "remoteProcessGroups": [],
+ "controllerServices": []
+ }
+}
diff --git a/examples/publishkafka_ssl_config.yml b/examples/publishkafka_ssl_config.yml
index 2f903624e..6800fc000 100644
--- a/examples/publishkafka_ssl_config.yml
+++ b/examples/publishkafka_ssl_config.yml
@@ -38,9 +38,8 @@ Processors:
Delivery Guarantee: 1
Known Brokers: kafka-broker:9093
Message Timeout: 12 sec
- Message Timeout Phrase: 12 sec
Request Timeout: 10 sec
- Security CA Key: /tmp/resources/certs/ca-cert
+ Security CA: /tmp/resources/certs/ca-cert
Security Cert: /tmp/resources/certs/client_test_client_client.pem
Security Pass Phrase: abcdefgh
Security Private Key: /tmp/resources/certs/client_test_client_client.key
diff --git a/examples/puts3_config.json b/examples/puts3_config.json
new file mode 100644
index 000000000..6df399659
--- /dev/null
+++ b/examples/puts3_config.json
@@ -0,0 +1,51 @@
+{
+ "rootGroup": {
+ "name": "MiNiFi Flow",
+ "processors": [
+ {
+ "name": "Get files from /tmp/input",
+ "identifier": "bfd376a8-c845-453b-9f74-4cf6f3a95e74",
+ "type": "org.apache.nifi.processors.standard.GetFile",
+ "schedulingStrategy": "TIMER_DRIVEN",
+ "schedulingPeriod": "2 sec",
+ "properties": {
+ "Input Directory": "/tmp/input"
+ },
+ "autoTerminatedRelationships": []
+ },
+ {
+ "name": "Upload to S3 test_bucket",
+ "identifier": "8c9e2512-e814-4b73-ac5c-f33753332128",
+ "type": "org.apache.nifi.processors.standard.PutS3Object",
+ "schedulingStrategy": "TIMER_DRIVEN",
+ "schedulingPeriod": "1 sec",
+ "properties": {
+ "Access Key": "test_access_key",
+ "Bucket": "test_bucket",
+ "Secret Key": "test_secret"
+ },
+ "autoTerminatedRelationships": [
+ "success",
+ "failure"
+ ]
+ }
+ ],
+ "connections": [
+ {
+ "identifier": "ad443bb0-0254-47e9-b113-7122244a46b3",
+ "name": "GetFile/success/PutS3Object",
+ "source": {
+ "id": "bfd376a8-c845-453b-9f74-4cf6f3a95e74"
+ },
+ "destination": {
+ "id": "8c9e2512-e814-4b73-ac5c-f33753332128"
+ },
+ "selectedRelationships": [
+ "success"
+ ]
+ }
+ ],
+ "remoteProcessGroups": [],
+ "controllerServices": []
+ }
+}
diff --git a/examples/querydbtable_config.json b/examples/querydbtable_config.json
new file mode 100644
index 000000000..92075476b
--- /dev/null
+++ b/examples/querydbtable_config.json
@@ -0,0 +1,61 @@
+{
+ "rootGroup": {
+ "name": "MiNiFi Flow",
+ "processors": [
+ {
+ "name": "Query users table",
+ "identifier": "0bd6ebcd-7fef-47e3-8e36-4413d39d01d5",
+ "type": "org.apache.nifi.processors.standard.QueryDatabaseTable",
+ "schedulingStrategy": "TIMER_DRIVEN",
+ "schedulingPeriod": "1 sec",
+ "properties": {
+ "Columns to Return": "id,name",
+ "DB Controller Service": "ODBCService",
+ "Output Format": "JSON",
+ "Table Name": "users",
+ "Where Clause": "id = 1"
+ },
+ "autoTerminatedRelationships": []
+ },
+ {
+ "name": "Put files to /tmp/output",
+ "identifier": "32700228-e078-42a6-89d7-03e4e478c9da",
+ "type": "org.apache.nifi.processors.standard.PutFile",
+ "schedulingStrategy": "EVENT_DRIVEN",
+ "properties": {
+ "Directory": "/tmp/output"
+ },
+ "autoTerminatedRelationships": [
+ "success",
+ "failure"
+ ]
+ }
+ ],
+ "connections": [
+ {
+ "identifier": "2308c4f7-4710-4bd1-a949-141265771b2f",
+ "name": "QueryDatabaseTable/success/PutFile",
+ "source": {
+ "id": "0bd6ebcd-7fef-47e3-8e36-4413d39d01d5"
+ },
+ "destination": {
+ "id": "32700228-e078-42a6-89d7-03e4e478c9da"
+ },
+ "selectedRelationships": [
+ "success"
+ ]
+ }
+ ],
+ "remoteProcessGroups": [],
+ "controllerServices": [
+ {
+ "name": "ODBCService",
+ "identifier": "64fafacd-74ad-4c3b-b1cc-a816ef128e09",
+ "type": "ODBCService",
+ "properties": {
+ "Connection String": "Driver={PostgreSQL ANSI};Server=postgresql-server;Port=5432;Database=postgres;Uid=postgres;Pwd=password;"
+ }
+ }
+ ]
+ }
+}
diff --git a/examples/querydbtable_config.nifi.schema.json b/examples/querydbtable_config.nifi.schema.json
new file mode 100644
index 000000000..c01d1e756
--- /dev/null
+++ b/examples/querydbtable_config.nifi.schema.json
@@ -0,0 +1,245 @@
+{
+ "encodingVersion": {
+ "majorVersion": 2,
+ "minorVersion": 0
+ },
+ "maxTimerDrivenThreadCount": 1,
+ "maxEventDrivenThreadCount": 1,
+ "parameterContexts": [],
+ "rootGroup": {
+ "identifier": "1c2f34d2-ee57-448a-b150-8140f0f7643c",
+ "instanceIdentifier": "9af45830-320e-437e-8204-dcd3c2933aa6",
+ "name": "root",
+ "position": {
+ "x": 0.0,
+ "y": 0.0
+ },
+ "processGroups": [],
+ "remoteProcessGroups": [],
+ "processors": [
+ {
+ "identifier": "0f469727-71d2-420f-aca5-d51599cb7507",
+ "instanceIdentifier": "9e22e432-9a97-4cd9-a7a4-d1ffeca5fbfe",
+ "name": "Query users table",
+ "comments": "",
+ "position": {
+ "x": 90.22606801867788,
+ "y": 3.5556140585132995
+ },
+ "type": "org.apache.nifi.minifi.processors.QueryDatabaseTable",
+ "bundle": {
+ "group": "org.apache.nifi.minifi",
+ "artifact": "minifi-sql",
+ "version": "0.15.0"
+ },
+ "properties": {
+ "Table Name": "users",
+ "Where Clause": "id = 1",
+ "DB Controller Service": "ODBCService",
+ "Columns to Return": "id,name",
+ "Output Format": "JSON",
+ "Max Rows Per Flow File": "0"
+ },
+ "propertyDescriptors": {
+ "Table Name": {
+ "name": "Table Name",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Where Clause": {
+ "name": "Where Clause",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "DB Controller Service": {
+ "name": "DB Controller Service",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Columns to Return": {
+ "name": "Columns to Return",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Maximum-value Columns": {
+ "name": "Maximum-value Columns",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Output Format": {
+ "name": "Output Format",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Max Rows Per Flow File": {
+ "name": "Max Rows Per Flow File",
+ "identifiesControllerService": false,
+ "sensitive": false
+ }
+ },
+ "style": {},
+ "schedulingPeriod": "1 sec",
+ "schedulingStrategy": "TIMER_DRIVEN",
+ "executionNode": "ALL",
+ "penaltyDuration": "30000 ms",
+ "yieldDuration": "1000 ms",
+ "bulletinLevel": "WARN",
+ "runDurationMillis": 0,
+ "concurrentlySchedulableTaskCount": 1,
+ "autoTerminatedRelationships": [],
+ "componentType": "PROCESSOR",
+ "groupIdentifier": "1c2f34d2-ee57-448a-b150-8140f0f7643c"
+ },
+ {
+ "identifier": "ca0439aa-2454-40f8-90fb-72b792d71bc1",
+ "instanceIdentifier": "bb8a7729-d7d4-4b82-9cd0-4e5c28a5d52e",
+ "name": "Put files to /tmp/output",
+ "comments": "",
+ "position": {
+ "x": 382.2260680186779,
+ "y": 12.5556140585133
+ },
+ "type": "org.apache.nifi.minifi.processors.PutFile",
+ "bundle": {
+ "group": "org.apache.nifi.minifi",
+ "artifact": "minifi-standard-processors",
+ "version": "0.15.0"
+ },
+ "properties": {
+ "Create Missing Directories": "true",
+ "Maximum File Count": "-1",
+ "Directory": "/tmp/output",
+ "Conflict Resolution Strategy": "fail"
+ },
+ "propertyDescriptors": {
+ "Permissions": {
+ "name": "Permissions",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Create Missing Directories": {
+ "name": "Create Missing Directories",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Maximum File Count": {
+ "name": "Maximum File Count",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Directory Permissions": {
+ "name": "Directory Permissions",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Directory": {
+ "name": "Directory",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Conflict Resolution Strategy": {
+ "name": "Conflict Resolution Strategy",
+ "identifiesControllerService": false,
+ "sensitive": false
+ }
+ },
+ "style": {},
+ "schedulingStrategy": "EVENT_DRIVEN",
+ "executionNode": "ALL",
+ "penaltyDuration": "30000 ms",
+ "yieldDuration": "1000 ms",
+ "bulletinLevel": "WARN",
+ "runDurationMillis": 0,
+ "concurrentlySchedulableTaskCount": 1,
+ "autoTerminatedRelationships": [
+ "success",
+ "failure"
+ ],
+ "componentType": "PROCESSOR",
+ "groupIdentifier": "1c2f34d2-ee57-448a-b150-8140f0f7643c"
+ }
+ ],
+ "inputPorts": [],
+ "outputPorts": [],
+ "connections": [
+ {
+ "identifier": "a0bf433a-5ea8-4f5b-87f2-5d060fb368ca",
+ "instanceIdentifier": "ed677727-9dc1-4f9a-82bc-06f2e99b9908",
+ "name": "Query users table/success/Put files to /tmp/output",
+ "position": {
+ "x": 0.0,
+ "y": 0.0
+ },
+ "source": {
+ "id": "0f469727-71d2-420f-aca5-d51599cb7507",
+ "type": "PROCESSOR",
+ "groupId": "1c2f34d2-ee57-448a-b150-8140f0f7643c",
+ "name": "Query users table",
+ "instanceIdentifier": "9e22e432-9a97-4cd9-a7a4-d1ffeca5fbfe"
+ },
+ "destination": {
+ "id": "ca0439aa-2454-40f8-90fb-72b792d71bc1",
+ "type": "PROCESSOR",
+ "groupId": "1c2f34d2-ee57-448a-b150-8140f0f7643c",
+ "name": "Put files to /tmp/output",
+ "instanceIdentifier": "bb8a7729-d7d4-4b82-9cd0-4e5c28a5d52e"
+ },
+ "labelIndex": 1,
+ "zIndex": 0,
+ "selectedRelationships": [
+ "success"
+ ],
+ "backPressureObjectThreshold": 2000,
+ "backPressureDataSizeThreshold": "100 MB",
+ "flowFileExpiration": "0 seconds",
+ "prioritizers": [],
+ "bends": [],
+ "componentType": "CONNECTION",
+ "groupIdentifier": "1c2f34d2-ee57-448a-b150-8140f0f7643c"
+ }
+ ],
+ "labels": [],
+ "funnels": [],
+ "controllerServices": [
+ {
+ "identifier": "da6e66d2-b185-4ca2-9691-69afb82dfc0b",
+ "instanceIdentifier": "370368b3-75d8-4901-b478-bd039ca7f58a",
+ "name": "ODBCService",
+ "position": {
+ "x": 0.0,
+ "y": 0.0
+ },
+ "type": "org.apache.nifi.minifi.sql.controllers.ODBCService",
+ "bundle": {
+ "group": "org.apache.nifi.minifi",
+ "artifact": "minifi-sql",
+ "version": "0.15.0"
+ },
+ "properties": {
+ "Connection String": "Driver={PostgreSQL ANSI};Server=postgresql-server;Port=5432;Database=postgres;Uid=postgres;Pwd=password;"
+ },
+ "propertyDescriptors": {
+ "Connection String": {
+ "name": "Connection String",
+ "identifiesControllerService": false,
+ "sensitive": true
+ }
+ },
+ "controllerServiceApis": [
+ {
+ "type": "org.apache.nifi.minifi.sql.controllers.ODBCService",
+ "bundle": {
+ "group": "org.apache.nifi.minifi",
+ "artifact": "minifi-sql",
+ "version": "0.15.0"
+ }
+ }
+ ],
+ "componentType": "CONTROLLER_SERVICE",
+ "groupIdentifier": "1c2f34d2-ee57-448a-b150-8140f0f7643c"
+ }
+ ],
+ "variables": {},
+ "componentType": "PROCESS_GROUP"
+ }
+}
diff --git a/examples/querydbtable_config.yml b/examples/querydbtable_config.yml
index 1dd6161e3..175b7e363 100644
--- a/examples/querydbtable_config.yml
+++ b/examples/querydbtable_config.yml
@@ -32,7 +32,6 @@ Processors:
id: 32700228-e078-42a6-89d7-03e4e478c9da
class: org.apache.nifi.processors.standard.PutFile
scheduling strategy: EVENT_DRIVEN
- scheduling period: 1 sec
auto-terminated relationships list:
- success
- failure
diff --git a/examples/site_to_site_config.json b/examples/site_to_site_config.json
new file mode 100644
index 000000000..3742f19d3
--- /dev/null
+++ b/examples/site_to_site_config.json
@@ -0,0 +1,49 @@
+{
+ "rootGroup": {
+ "name": "MiNiFi Flow",
+ "processors": [
+ {
+ "name": "Get files from /tmp/input",
+ "identifier": "45c2a56e-7C29-42f3-8C1D-89c0c84a6ee7",
+ "type": "org.apache.nifi.processors.standard.GetFile",
+ "schedulingStrategy": "TIMER_DRIVEN",
+ "schedulingPeriod": "2 sec",
+ "properties": {
+ "Input Directory": "/tmp/input"
+ },
+ "autoTerminatedRelationships": []
+ }
+ ],
+ "connections": [
+ {
+ "identifier": "3952ac3c-c0a8-46fe-936a-e9faa7123275",
+ "name": "GetFile/success/RPG",
+ "source": {
+ "id": "45c2a56e-7C29-42f3-8C1D-89c0c84a6ee7"
+ },
+ "destination": {
+ "id": "4222cd3a-746e-3497-abd0-b7695faa750c"
+ },
+ "selectedRelationships": [
+ "success"
+ ]
+ }
+ ],
+ "remoteProcessGroups": [
+ {
+ "name": "RemoteProcessGroup",
+ "identifier": "38878f84-23cd-4aaa-9c45-45151bb2d681",
+ "targetUri": "http://127.0.0.1:8080/nifi",
+ "communicationsTimeout": "30 sec",
+ "yieldDuration": "3 sec",
+ "inputPorts": [
+ {
+ "identifier": "4222cd3a-746e-3497-abd0-b7695faa750c",
+ "name": "to_nifi"
+ }
+ ]
+ }
+ ],
+ "controllerServices": []
+ }
+}
diff --git a/examples/site_to_site_config.nifi.schema.json b/examples/site_to_site_config.nifi.schema.json
new file mode 100644
index 000000000..b4a94c495
--- /dev/null
+++ b/examples/site_to_site_config.nifi.schema.json
@@ -0,0 +1,195 @@
+{
+ "encodingVersion": {
+ "majorVersion": 2,
+ "minorVersion": 0
+ },
+ "maxTimerDrivenThreadCount": 1,
+ "maxEventDrivenThreadCount": 1,
+ "parameterContexts": [],
+ "rootGroup": {
+ "identifier": "1c2f34d2-ee57-448a-b150-8140f0f7643c",
+ "instanceIdentifier": "9af45830-320e-437e-8204-dcd3c2933aa6",
+ "name": "root",
+ "position": {
+ "x": 0.0,
+ "y": 0.0
+ },
+ "processGroups": [],
+ "remoteProcessGroups": [
+ {
+ "identifier": "7b8d0cc8-b565-4b98-b5e5-da30e15e8cb1",
+ "instanceIdentifier": "bcfea0da-bc85-4cf7-86ec-da8745567213",
+ "name": "http://localhost:8080/nifi",
+ "position": {
+ "x": 368.2260680186778,
+ "y": 52.5556140585133
+ },
+ "targetUri": "http://localhost:8080/nifi",
+ "targetUris": "http://localhost:8080/nifi",
+ "communicationsTimeout": "30 secs",
+ "yieldDuration": "3 sec",
+ "transportProtocol": "RAW",
+ "inputPorts": [
+ {
+ "identifier": "4222cd3a-746e-3497-abd0-b7695faa750c",
+ "instanceIdentifier": "4222cd3a-746e-3497-abd0-b7695faa750c",
+ "remoteGroupId": "7b8d0cc8-b565-4b98-b5e5-da30e15e8cb1",
+ "concurrentlySchedulableTaskCount": 1,
+ "useCompression": true,
+ "componentType": "REMOTE_INPUT_PORT",
+ "targetId": "4222cd3a-746e-3497-abd0-b7695faa750c",
+ "scheduledState": "RUNNING",
+ "groupIdentifier": "7b8d0cc8-b565-4b98-b5e5-da30e15e8cb1"
+ }
+ ],
+ "outputPorts": [],
+ "componentType": "REMOTE_PROCESS_GROUP",
+ "groupIdentifier": "1c2f34d2-ee57-448a-b150-8140f0f7643c"
+ }
+ ],
+ "processors": [
+ {
+ "identifier": "e974fc57-9bb5-42e1-a807-1da267e6b281",
+ "instanceIdentifier": "9def2e01-4843-486d-9e87-a60af90b874c",
+ "name": "Get files from /tmp/input",
+ "comments": "",
+ "position": {
+ "x": -0.7739701282948772,
+ "y": 45.5556140585133
+ },
+ "type": "org.apache.nifi.minifi.processors.GetFile",
+ "bundle": {
+ "group": "org.apache.nifi.minifi",
+ "artifact": "minifi-standard-processors",
+ "version": "0.15.0"
+ },
+ "properties": {
+ "Keep Source File": "false",
+ "Minimum File Age": "0 sec",
+ "Polling Interval": "0 sec",
+ "Input Directory": "/tmp/input",
+ "Maximum File Age": "0 sec",
+ "Batch Size": "10",
+ "Maximum File Size": "0 B",
+ "Minimum File Size": "0 B",
+ "Ignore Hidden Files": "true",
+ "Recurse Subdirectories": "true",
+ "File Filter": ".*"
+ },
+ "propertyDescriptors": {
+ "Keep Source File": {
+ "name": "Keep Source File",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Minimum File Age": {
+ "name": "Minimum File Age",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Polling Interval": {
+ "name": "Polling Interval",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Input Directory": {
+ "name": "Input Directory",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Maximum File Age": {
+ "name": "Maximum File Age",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Batch Size": {
+ "name": "Batch Size",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Maximum File Size": {
+ "name": "Maximum File Size",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Minimum File Size": {
+ "name": "Minimum File Size",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Ignore Hidden Files": {
+ "name": "Ignore Hidden Files",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "Recurse Subdirectories": {
+ "name": "Recurse Subdirectories",
+ "identifiesControllerService": false,
+ "sensitive": false
+ },
+ "File Filter": {
+ "name": "File Filter",
+ "identifiesControllerService": false,
+ "sensitive": false
+ }
+ },
+ "style": {},
+ "schedulingPeriod": "2 sec",
+ "schedulingStrategy": "TIMER_DRIVEN",
+ "executionNode": "ALL",
+ "penaltyDuration": "30000 ms",
+ "yieldDuration": "1000 ms",
+ "bulletinLevel": "WARN",
+ "runDurationMillis": 0,
+ "concurrentlySchedulableTaskCount": 1,
+ "autoTerminatedRelationships": [],
+ "componentType": "PROCESSOR",
+ "groupIdentifier": "1c2f34d2-ee57-448a-b150-8140f0f7643c"
+ }
+ ],
+ "inputPorts": [],
+ "outputPorts": [],
+ "connections": [
+ {
+ "identifier": "4f72d514-8d0d-4783-859a-945eb5a7ed0f",
+ "instanceIdentifier": "08b12177-6667-4e96-9f87-0896e39d9a22",
+ "name": "to_nifi",
+ "position": {
+ "x": 0.0,
+ "y": 0.0
+ },
+ "source": {
+ "id": "e974fc57-9bb5-42e1-a807-1da267e6b281",
+ "type": "PROCESSOR",
+ "groupId": "1c2f34d2-ee57-448a-b150-8140f0f7643c",
+ "name": "Get files from /tmp/input",
+ "instanceIdentifier": "9def2e01-4843-486d-9e87-a60af90b874c"
+ },
+ "destination": {
+ "id": "4222cd3a-746e-3497-abd0-b7695faa750c",
+ "type": "REMOTE_INPUT_PORT",
+ "groupId": "7b8d0cc8-b565-4b98-b5e5-da30e15e8cb1",
+ "name": "4222cd3a-746e-3497-abd0-b7695faa750c",
+ "instanceIdentifier": "4222cd3a-746e-3497-abd0-b7695faa750c"
+ },
+ "labelIndex": 1,
+ "zIndex": 0,
+ "selectedRelationships": [
+ "success"
+ ],
+ "backPressureObjectThreshold": 2000,
+ "backPressureDataSizeThreshold": "100 MB",
+ "flowFileExpiration": "0 seconds",
+ "prioritizers": [],
+ "bends": [],
+ "componentType": "CONNECTION",
+ "groupIdentifier": "1c2f34d2-ee57-448a-b150-8140f0f7643c"
+ }
+ ],
+ "labels": [],
+ "funnels": [],
+ "controllerServices": [],
+ "variables": {},
+ "componentType": "PROCESS_GROUP"
+ }
+}
diff --git a/examples/splittext_puttcp_config.json b/examples/splittext_puttcp_config.json
new file mode 100644
index 000000000..270e3d86f
--- /dev/null
+++ b/examples/splittext_puttcp_config.json
@@ -0,0 +1,84 @@
+{
+ "rootGroup": {
+ "name": "MiNiFi Flow",
+ "processors": [
+ {
+ "name": "Get files from /tmp/input",
+ "identifier": "a8fd96f8-6f3e-4ab8-9cc7-7dcb3304d60b",
+ "type": "org.apache.nifi.processors.standard.GetFile",
+ "schedulingStrategy": "TIMER_DRIVEN",
+ "schedulingPeriod": "2 sec",
+ "penaltyDuration": "30 sec",
+ "yieldDuration": "1 sec",
+ "properties": {
+ "Input Directory": "/tmp/input"
+ },
+ "autoTerminatedRelationships": []
+ },
+ {
+ "name": "Split text by lines skipping the first 3 header lines",
+ "identifier": "40b45f52-ae1d-443c-9465-7e070ce3bea0",
+ "type": "org.apache.nifi.processors.standard.SplitText",
+ "schedulingStrategy": "EVENT_DRIVEN",
+ "penaltyDuration": "30 sec",
+ "yieldDuration": "1 sec",
+ "properties": {
+ "Header Line Count": "3",
+ "Line Split Count": "1",
+ "Remove Trailing Newlines": "false"
+ },
+ "autoTerminatedRelationships": [
+ "failure",
+ "original"
+ ]
+ },
+ {
+ "name": "Put data to TCP server",
+ "identifier": "11167a2c-9778-48ba-be2d-d1f9f4b0433f",
+ "type": "org.apache.nifi.processors.standard.PutTCP",
+ "schedulingStrategy": "EVENT_DRIVEN",
+ "penaltyDuration": "30 sec",
+ "yieldDuration": "1 sec",
+ "properties": {
+ "Hostname": "192.168.1.5",
+ "Port": "8081"
+ },
+ "autoTerminatedRelationships": [
+ "success",
+ "failure"
+ ]
+ }
+ ],
+ "funnels": [],
+ "connections": [
+ {
+ "identifier": "76bc36d3-f484-49a4-ad69-bec47cd921c8",
+ "name": "GetFile/success/SplitText",
+ "source": {
+ "id": "a8fd96f8-6f3e-4ab8-9cc7-7dcb3304d60b"
+ },
+ "destination": {
+ "id": "40b45f52-ae1d-443c-9465-7e070ce3bea0"
+ },
+ "selectedRelationships": [
+ "success"
+ ]
+ },
+ {
+ "identifier": "25ee2e5c-fc74-4575-a510-db33e6c5683d",
+ "name": "SplitText/splits/PutTCP",
+ "source": {
+ "id": "40b45f52-ae1d-443c-9465-7e070ce3bea0"
+ },
+ "destination": {
+ "id": "11167a2c-9778-48ba-be2d-d1f9f4b0433f"
+ },
+ "selectedRelationships": [
+ "splits"
+ ]
+ }
+ ],
+ "remoteProcessGroups": [],
+ "controllerServices": []
+ }
+}
diff --git a/examples/splittext_puttcp_config.yml b/examples/splittext_puttcp_config.yml
new file mode 100644
index 000000000..8f56e9f2f
--- /dev/null
+++ b/examples/splittext_puttcp_config.yml
@@ -0,0 +1,53 @@
+MiNiFi Config Version: 3
+Flow Controller:
+ name: MiNiFi Flow
+Processors:
+ - name: Get files from /tmp/input
+ id: a8fd96f8-6f3e-4ab8-9cc7-7dcb3304d60b
+ class: org.apache.nifi.processors.standard.GetFile
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 2000 ms
+ penalization period: 30 sec
+ yield period: 1 sec
+ Properties:
+ Input Directory: /tmp/input
+ - name: Split text by lines skipping the first 3 header lines
+ id: 40b45f52-ae1d-443c-9465-7e070ce3bea0
+ class: org.apache.nifi.processors.standard.SplitText
+ scheduling strategy: EVENT_DRIVEN
+ penalization period: 30 sec
+ yield period: 1 sec
+ auto-terminated relationships list:
+ - failure
+ - original
+ Properties:
+ Header Line Count: 3
+ Line Split Count: 1
+ Remove Trailing Newlines: false
+ - name: Put data to TCP server
+ id: 11167a2c-9778-48ba-be2d-d1f9f4b0433f
+ class: org.apache.nifi.processors.standard.PutTCP
+ scheduling strategy: EVENT_DRIVEN
+ penalization period: 30 sec
+ yield period: 1 sec
+ auto-terminated relationships list:
+ - success
+ - failure
+ Properties:
+ Hostname: 192.168.1.5
+ Port: 8081
+Connections:
+ - name: GetFile/success/SplitText
+ id: 76bc36d3-f484-49a4-ad69-bec47cd921c8
+ source id: a8fd96f8-6f3e-4ab8-9cc7-7dcb3304d60b
+ source relationship names:
+ - success
+ destination id: 40b45f52-ae1d-443c-9465-7e070ce3bea0
+ - name: SplitText/splits/PutTCP
+ id: 25ee2e5c-fc74-4575-a510-db33e6c5683d
+ source id: 40b45f52-ae1d-443c-9465-7e070ce3bea0
+ source relationship names:
+ - splits
+ destination id: 11167a2c-9778-48ba-be2d-d1f9f4b0433f
+Controller Services: []
+Remote Process Groups: []
diff --git a/examples/tailfile_config.json b/examples/tailfile_config.json
new file mode 100644
index 000000000..350f46f9e
--- /dev/null
+++ b/examples/tailfile_config.json
@@ -0,0 +1,50 @@
+{
+ "rootGroup": {
+ "name": "MiNiFi Flow",
+ "processors": [
+ {
+ "name": "Log tailfile attributes",
+ "identifier": "41141af1-3c43-4fa4-8236-2b207b0fe9d8",
+ "type": "org.apache.nifi.minifi.processors.LogAttribute",
+ "schedulingStrategy": "EVENT_DRIVEN",
+ "properties": {
+ "FlowFiles To Log": "0"
+ },
+ "autoTerminatedRelationships": [
+ "success"
+ ]
+ },
+ {
+ "name": "Tail /tmp/test_file.log",
+ "identifier": "0ef22940-ac0a-4813-9ad9-fe41c2b51fff",
+ "type": "org.apache.nifi.minifi.processors.TailFile",
+ "schedulingStrategy": "TIMER_DRIVEN",
+ "schedulingPeriod": "1000 ms",
+ "properties": {
+ "File to Tail": "/tmp/test_file.log",
+ "Initial Start Position": "Beginning of File",
+ "Input Delimiter": "\\n",
+ "tail-mode": "Single file"
+ },
+ "autoTerminatedRelationships": []
+ }
+ ],
+ "connections": [
+ {
+ "identifier": "eda7f758-c548-46d6-aba0-faa6e9937d8c",
+ "name": "TailFile/success/LogAttribute",
+ "source": {
+ "id": "0ef22940-ac0a-4813-9ad9-fe41c2b51fff"
+ },
+ "destination": {
+ "id": "41141af1-3c43-4fa4-8236-2b207b0fe9d8"
+ },
+ "selectedRelationships": [
+ "success"
+ ]
+ }
+ ],
+ "remoteProcessGroups": [],
+ "controllerServices": []
+ }
+}
diff --git a/examples/tailfile_config.yml b/examples/tailfile_config.yml
index ada27ffb8..7dfd3dfd5 100644
--- a/examples/tailfile_config.yml
+++ b/examples/tailfile_config.yml
@@ -20,8 +20,7 @@ Processors:
- name: Log tailfile attributes
id: 41141af1-3c43-4fa4-8236-2b207b0fe9d8
class: org.apache.nifi.minifi.processors.LogAttribute
- scheduling strategy: TIMER_DRIVEN
- scheduling period: 1000 ms
+ scheduling strategy: EVENT_DRIVEN
auto-terminated relationships list:
- success
Properties: