You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by GitBox <gi...@apache.org> on 2021/07/08 14:40:36 UTC
[GitHub] [nifi-minifi-cpp] lordgamez opened a new pull request #1127: MINIFICPP-1601 Add example flow configurations
lordgamez opened a new pull request #1127:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1127
Add example flow configuration files with several common use cases for release testing purposes and for easier adaption of new users.
Jira ticket: https://issues.apache.org/jira/browse/MINIFICPP-1601
-----------------------------------------------------------------------------------------------
Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with MINIFICPP-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [ ] Has your PR been rebased against the latest commit within the target branch (typically main)?
- [ ] Is your initial contribution a single, squashed commit?
### For code changes:
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file?
- [ ] If applicable, have you updated the NOTICE file?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #1127: MINIFICPP-1601 Add example flow configurations
Posted by GitBox <gi...@apache.org>.
lordgamez commented on a change in pull request #1127:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1127#discussion_r672313737
##########
File path: examples/README.md
##########
@@ -12,6 +12,130 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-# Apache NiFi - MiNiFi - C++ Examples
+# Apache NiFi - MiNiFi - C++ Examples
-- [Site-2-Site Bi-directional Configuration](BidirectionalSiteToSite/README.md)
\ No newline at end of file
+The following examples show simple flow configurations for several common use cases. Use the path of these yaml files as the flow configuration set in the `nifi.flow.configuration.file` property of `conf/minifi.properties` file or replace the default `conf/config.yml` file to try them out.
+
+- [Filesystem Operations](#filesystem-operations)
+ - [Getting File Data and Putting It in an Output Directory](#getting-file-data-and-putting-it-in-an-output-directory)
+ - [Tailing a Single File](#tailing-a-single-file)
+- [Windows Specific Processors](#windows-specific-processors)
+ - [Consuming Windows Event Logs](#consuming-windows-event-logs)
+ - [Reading System Performance Data](#reading-system-performance-data)
+- [HTTP Operations](#http-operations)
+ - [HTTP POST Invocation](#http-post-invocation)
+- [Site to Site Operations](#site-to-site-operations)
+ - [Transfer Data to Remote Nifi Instance](#transfer-data-to-remote-nifi-instance)
+ - [Site-2-Site Bi-directional Configuration](#site-2-site-bi-directional-configuration)
+- [Kafka Operations](#kafka-operations)
+ - [Publish Message to Kafka Broker](#publish-message-to-kafka-broker)
+ - [Publish Message to Kafka Broker Through SSL](#publish-message-to-kafka-broker-through-ssl)
+ - [Consume Messages from Kafka](#consume-messages-from-kafka)
+- [Public Cloud Operations](#public-cloud-operations)
+ - [Upload Blob to Azure Storage](#upload-blob-to-azure-storage)
+ - [Put Object in AWS S3 Bucket](#put-object-in-aws-s3-bucket)
+ - [List and Fetch Content from AWS S3 Bucket](#list-and-fetch-content-from-aws-s3-bucket)
+- [SQL Operations](#sql-operations)
+ - [Query Database Table](#query-database-table)
+- [System Log Operations](#system-log-operations)
+ - [Consume Systemd-Journald System Journal Messages](#consume-systemd-journald-system-journal-messages)
+
+## Filesystem Operations
+
+### Getting File Data and Putting It in an Output Directory
+
+Using the [getfile_putfile_config.yml](getfile_putfile_config.yml) flow configuration MiNiFi gets all files of minimum 1MB size from the `/tmp/getfile_dir` directory and puts them in the `/tmp/out_dir` output directory.
+
+The flow: GetFile ➔ success ➔ PutFile
+
+### Tailing a Single File
+
+Using the [tailfile_config.yml](tailfile_config.yml) flow configuration MiNiFi tails a single file `/tmp/test_file.log` and creates flowfiles from every single line, then logs the flowfile attributes.
+
+The flow: TailFile ➔ success ➔ LogAttribute
+
+## Windows Specific Processors
+
+### Consuming Windows Event Logs
+
+Using the [cwel_config.yml](cwel_config.yml) flow configuration MiNiFi queries all Windows system events and puts them to the `C:\temp\` directory in flattened JSON format.
+
+The flow: ConsumeWindowsEventLog ➔ PutFile
+
+## Linux Specific Processors
+
+### Consume Systemd-Journald System Journal Messages
+
+Using the [consumejournald_config.yml](consumejournald_config.yml) flow configuration MiNiFi reads systemd-journald journal messages and logs them on `info` level.
+
+The flow: ConsumeJournald ➔ LogAttribute
+
+### Reading System Performance Data
+
+Using the [pdh_config.yml](pdh_config.yml) flow configuration MiNiFi reads CPU and Disk performance data through Windows' Performance Data Helper (PDH) component and puts the data to the `C:\temp\` directory in a compact JSON format.
+
+The flow: PerformanceDataMonitor ➔ PutFile
+
Review comment:
Good catch, fixed in d02f0d09ded60838efd5498b9627bc4e841e1db5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1127: MINIFICPP-1601 Add example flow configurations
Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #1127:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1127#discussion_r666937904
##########
File path: examples/README.md
##########
@@ -12,6 +12,130 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-# Apache NiFi - MiNiFi - C++ Examples
+# Apache NiFi - MiNiFi - C++ Examples
-- [Site-2-Site Bi-directional Configuration](BidirectionalSiteToSite/README.md)
\ No newline at end of file
+The following examples show simple flow configurations for several common use cases. Use the path of these yaml files as the flow configuration set in the `nifi.flow.configuration.file` property of `conf/minifi.properties` file or replace the default `conf/config.yml` file to try them out.
+
+- [Filesystem Operations](#filesystem-operations)
+ - [Getting File Data and Putting It in an Output Directory](#getting-file-data-and-putting-it-in-an-output-directory)
+ - [Tailing a Single File](#tailing-a-single-file)
+- [Windows Specific Processors](#windows-specific-processors)
+ - [Consuming Windows Event Logs](#consuming-windows-event-logs)
+ - [Reading System Performance Data](#reading-system-performance-data)
+- [HTTP Operations](#http-operations)
+ - [HTTP POST Invocation](#http-post-invocation)
+- [Site to Site Operations](#site-to-site-operations)
+ - [Transfer Data to Remote Nifi Instance](#transfer-data-to-remote-nifi-instance)
+ - [Site-2-Site Bi-directional Configuration](#site-2-site-bi-directional-configuration)
+- [Kafka Operations](#kafka-operations)
+ - [Publish Message to Kafka Broker](#publish-message-to-kafka-broker)
+ - [Publish Message to Kafka Broker Through SSL](#publish-message-to-kafka-broker-through-ssl)
+ - [Consume Messages from Kafka](#consume-messages-from-kafka)
+- [Public Cloud Operations](#public-cloud-operations)
+ - [Upload Blob to Azure Storage](#upload-blob-to-azure-storage)
+ - [Put Object in AWS S3 Bucket](#put-object-in-aws-s3-bucket)
+ - [List and Fetch Content from AWS S3 Bucket](#list-and-fetch-content-from-aws-s3-bucket)
+- [SQL Operations](#sql-operations)
+ - [Query Database Table](#query-database-table)
+- [System Log Operations](#system-log-operations)
+ - [Consume Journal Logs](#consume-journal-logs)
+
+## Filesystem Operations
+
+### Getting File Data and Putting It in an Output Directory
+
+Using the [getfile_putfile_config.yml](getfile_putfile_config.yml) flow configuration MiNiFi gets all files of minimum 1MB size from the `/tmp/getfile_dir` directory and puts them in the `/tmp/out_dir` output directory.
+
+The flow: GetFile ➔ success ➔ PutFile
+
+### Tailing a Single File
+
+Using the [tailfile_config.yml](tailfile_config.yml) flow configuration MiNiFi tails a single file `/tmp/test_file.log` and creates flowfiles from every single line, then logs the flowfile attributes.
+
+The flow: TailFile ➔ success ➔ LogAttribute
+
+## Windows Specific Processors
+
+### Consuming Windows Event Logs
+
+Using the [cwel_config.yml](cwel_config.yml) flow configuration MiNiFi queries all Windows system events and puts them to the `C:\temp\` directory in flattened JSON format.
+
+The flow: ConsumeWindowsEventLog ➔ PutFile
+
+## Linux Specific Processors
+
+### Consume Journal Messages
+
+Using the [consumejournald_config.yml](consumejournald_config.yml) flow configuration MiNiFi reads incoming journal messages and logs the attributes on `info` level.
Review comment:
They are not really incoming messages, as they are produced on the system. The contents are also logged, not only the attributes.
```suggestion
Using the [consumejournald_config.yml](consumejournald_config.yml) flow configuration MiNiFi reads systemd-journald journal messages and logs them on `info` level.
```
##########
File path: examples/consumekafka_config.yml
##########
@@ -0,0 +1,51 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+MiNiFi Config Version: 3
+Flow Controller:
+ name: MiNiFi Flow
+Processors:
+- name: Consume Kafka messages from ConsumeKafkaTest topic
+ id: 37d6e8e5-6284-423e-8620-c5168a71add7
+ class: org.apache.nifi.processors.standard.ConsumeKafka
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 1 sec
+ Properties:
+ Group ID: test_group
+ Honor Transactions: 'true'
+ Kafka Brokers: kafka-broker:9092
+ Key Attribute Encoding: UTF-8
+ Max Poll Time: 4 sec
+ Message Header Encoding: UTF-8
+ Offset Reset: earliest
+ Session Timeout: 60 sec
+ Topic Name Format: Names
+ Topic Names: ConsumeKafkaTest
Review comment:
Missing auto-terminated relationships list, but ConsumeKafka also has a failure relationship with no connections from it.
##########
File path: examples/http_post_config.yml
##########
@@ -0,0 +1,50 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+MiNiFi Config Version: 3
+Flow Controller:
+ name: MiNiFi Flow
+Processors:
+- name: Get files from /tmp/input
+ id: 962790e7-ea35-4096-9362-96f527288669
+ class: org.apache.nifi.processors.standard.GetFile
+ scheduling period: 2 sec
+ scheduling strategy: TIMER_DRIVEN
+ Properties:
+ Input Directory: /tmp/input
+ Keep Source File: 'true'
+- name: Invoke POST request on remote URL
+ id: 33b373b6-6f19-4194-b45a-1ef73c357a8e
+ class: org.apache.nifi.processors.standard.InvokeHTTP
+ scheduling strategy: EVENT_DRIVEN
+ auto-terminated relationships list:
+ - success
+ - response
+ - retry
+ - no retry
+ Properties:
+ HTTP Method: POST
+ Remote URL: http://1.2.3.4:8080/contentListener
+Connections:
+- name: GetFile/success/InvokeHTTP
Review comment:
It would be nice to implement retry here. Flow files at the retry relationship should be retried. Maybe we should also limit the number of retries.
##########
File path: examples/consumejournald_config.yml
##########
@@ -0,0 +1,44 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the \"License\"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an \"AS IS\" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+MiNiFi Config Version: 3
+Flow Controller:
+ name: MiNiFi Flow
+Processors:
+- name: Consume journal messages
+ id: 66fbb65d-8b11-43a9-aaad-746a07d2fe35
+ class: org.apache.nifi.minifi.extensions.systemd.ConsumeJournald
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 5 sec
+- name: Log attributes to info level
+ id: f7281113-9576-4680-9a87-a8e961101b5c
+ class: org.apache.nifi.minifi.processors.LogAttribute
+ scheduling strategy: EVENT_DRIVEN
+ auto-terminated relationships list: [success]
+ Properties:
+ FlowFiles To Log: 0
+ Log Level: info
+ Log Payload: true
+ Log Prefix: '====='
+ Maxumim Payload Line Length: 0
+Connections:
+- name: ConsumeJournald/success/LogAttribute
+ id: f57713d9-fa0b-4416-948d-4b316b1a1c0d
+ source name: ConsumeJournald
Review comment:
```suggestion
source name: Consume journal messages
```
I wonder if this could be omitted.
##########
File path: examples/README.md
##########
@@ -12,6 +12,130 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-# Apache NiFi - MiNiFi - C++ Examples
+# Apache NiFi - MiNiFi - C++ Examples
-- [Site-2-Site Bi-directional Configuration](BidirectionalSiteToSite/README.md)
\ No newline at end of file
+The following examples show simple flow configurations for several common use cases. Use the path of these yaml files as the flow configuration set in the `nifi.flow.configuration.file` property of `conf/minifi.properties` file or replace the default `conf/config.yml` file to try them out.
+
+- [Filesystem Operations](#filesystem-operations)
+ - [Getting File Data and Putting It in an Output Directory](#getting-file-data-and-putting-it-in-an-output-directory)
+ - [Tailing a Single File](#tailing-a-single-file)
+- [Windows Specific Processors](#windows-specific-processors)
+ - [Consuming Windows Event Logs](#consuming-windows-event-logs)
+ - [Reading System Performance Data](#reading-system-performance-data)
+- [HTTP Operations](#http-operations)
+ - [HTTP POST Invocation](#http-post-invocation)
+- [Site to Site Operations](#site-to-site-operations)
+ - [Transfer Data to Remote Nifi Instance](#transfer-data-to-remote-nifi-instance)
+ - [Site-2-Site Bi-directional Configuration](#site-2-site-bi-directional-configuration)
+- [Kafka Operations](#kafka-operations)
+ - [Publish Message to Kafka Broker](#publish-message-to-kafka-broker)
+ - [Publish Message to Kafka Broker Through SSL](#publish-message-to-kafka-broker-through-ssl)
+ - [Consume Messages from Kafka](#consume-messages-from-kafka)
+- [Public Cloud Operations](#public-cloud-operations)
+ - [Upload Blob to Azure Storage](#upload-blob-to-azure-storage)
+ - [Put Object in AWS S3 Bucket](#put-object-in-aws-s3-bucket)
+ - [List and Fetch Content from AWS S3 Bucket](#list-and-fetch-content-from-aws-s3-bucket)
+- [SQL Operations](#sql-operations)
+ - [Query Database Table](#query-database-table)
+- [System Log Operations](#system-log-operations)
+ - [Consume Journal Logs](#consume-journal-logs)
Review comment:
I would call this "Consume systemd-journald system journal messages" for clarity, since there are many things called journals, and this is only about the systemd one on linux.
##########
File path: examples/consumejournald_config.yml
##########
@@ -0,0 +1,44 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the \"License\"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an \"AS IS\" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+MiNiFi Config Version: 3
+Flow Controller:
+ name: MiNiFi Flow
+Processors:
+- name: Consume journal messages
+ id: 66fbb65d-8b11-43a9-aaad-746a07d2fe35
+ class: org.apache.nifi.minifi.extensions.systemd.ConsumeJournald
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 5 sec
+- name: Log attributes to info level
Review comment:
With `Log Payload: true`, it would be more accurate to name this "Log flow files" or "Log messages"
```suggestion
- name: Log messages to info level
```
##########
File path: examples/publishkafka_config.yml
##########
@@ -0,0 +1,46 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+MiNiFi Config Version: 3
+Flow Controller:
+ name: MiNiFi Flow
+Processors:
+- name: Get files from /tmp/input
+ id: 7fd166aa-0662-4c42-affa-88f6fb39807f
+ class: org.apache.nifi.processors.standard.GetFile
+ scheduling period: 2 sec
+ scheduling strategy: TIMER_DRIVEN
+ Properties:
+ Input Directory: /tmp/input
+- name: Publish messages to Kafka topic test
+ id: 8a534b4a-2b4a-4e1e-ab07-8a09fa08f848
+ class: org.apache.nifi.processors.standard.PublishKafka
+ scheduling strategy: EVENT_DRIVEN
+ Properties:
+ Batch Size: '10'
+ Client Name: test-client
+ Compress Codec: none
+ Delivery Guarantee: '1'
+ Known Brokers: kafka-broker:9092
+ Message Timeout: 12 sec
+ Request Timeout: 10 sec
+ Topic Name: test
Review comment:
This one is missing auto-terminated relationships list
##########
File path: examples/publishkafka_ssl_config.yml
##########
@@ -0,0 +1,55 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+MiNiFi Config Version: 3
+Flow Controller:
+ name: MiNiFi Flow
+Processors:
+- name: Get files from /tmp/input
+ id: a78f9d7d-cc68-4f30-aac1-ee716d42bf9d
+ class: org.apache.nifi.processors.standard.GetFile
+ scheduling period: 2 sec
+ scheduling strategy: TIMER_DRIVEN
+ Properties:
+ Input Directory: /tmp/input
+- name: Publish messages to Kafka topic test
+ id: 8c0791c7-920c-4541-bcde-f5d559f42ed2
+ class: org.apache.nifi.processors.standard.PublishKafka
+ scheduling period: 1 sec
+ scheduling strategy: EVENT_DRIVEN
+ auto-terminated relationships list:
+ - success
+ Properties:
Review comment:
What about `failure`? No connections starting from there and not auto-terminated either. I think you meant to have it either auto-terminated or retried.
auto-terminated (errors ignored):
```suggestion
auto-terminated relationships list:
- success
- failure
Properties:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #1127: MINIFICPP-1601 Add example flow configurations
Posted by GitBox <gi...@apache.org>.
lordgamez commented on a change in pull request #1127:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1127#discussion_r666418871
##########
File path: examples/querydbtable_config.yml
##########
@@ -0,0 +1,51 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+Flow Controller:
Review comment:
It seems it is not used currently as we are on version 0.10.0. I suppose it will be only checked after version 1.0 to handle config versions for backward compatibility. I added the config version to all yamls in f45f0ed787ea86742bbd093a97d11006201f26c6
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #1127: MINIFICPP-1601 Add example flow configurations
Posted by GitBox <gi...@apache.org>.
lordgamez commented on a change in pull request #1127:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1127#discussion_r672313737
##########
File path: examples/README.md
##########
@@ -12,6 +12,130 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-# Apache NiFi - MiNiFi - C++ Examples
+# Apache NiFi - MiNiFi - C++ Examples
-- [Site-2-Site Bi-directional Configuration](BidirectionalSiteToSite/README.md)
\ No newline at end of file
+The following examples show simple flow configurations for several common use cases. Use the path of these yaml files as the flow configuration set in the `nifi.flow.configuration.file` property of `conf/minifi.properties` file or replace the default `conf/config.yml` file to try them out.
+
+- [Filesystem Operations](#filesystem-operations)
+ - [Getting File Data and Putting It in an Output Directory](#getting-file-data-and-putting-it-in-an-output-directory)
+ - [Tailing a Single File](#tailing-a-single-file)
+- [Windows Specific Processors](#windows-specific-processors)
+ - [Consuming Windows Event Logs](#consuming-windows-event-logs)
+ - [Reading System Performance Data](#reading-system-performance-data)
+- [HTTP Operations](#http-operations)
+ - [HTTP POST Invocation](#http-post-invocation)
+- [Site to Site Operations](#site-to-site-operations)
+ - [Transfer Data to Remote Nifi Instance](#transfer-data-to-remote-nifi-instance)
+ - [Site-2-Site Bi-directional Configuration](#site-2-site-bi-directional-configuration)
+- [Kafka Operations](#kafka-operations)
+ - [Publish Message to Kafka Broker](#publish-message-to-kafka-broker)
+ - [Publish Message to Kafka Broker Through SSL](#publish-message-to-kafka-broker-through-ssl)
+ - [Consume Messages from Kafka](#consume-messages-from-kafka)
+- [Public Cloud Operations](#public-cloud-operations)
+ - [Upload Blob to Azure Storage](#upload-blob-to-azure-storage)
+ - [Put Object in AWS S3 Bucket](#put-object-in-aws-s3-bucket)
+ - [List and Fetch Content from AWS S3 Bucket](#list-and-fetch-content-from-aws-s3-bucket)
+- [SQL Operations](#sql-operations)
+ - [Query Database Table](#query-database-table)
+- [System Log Operations](#system-log-operations)
+ - [Consume Systemd-Journald System Journal Messages](#consume-systemd-journald-system-journal-messages)
+
+## Filesystem Operations
+
+### Getting File Data and Putting It in an Output Directory
+
+Using the [getfile_putfile_config.yml](getfile_putfile_config.yml) flow configuration MiNiFi gets all files of minimum 1MB size from the `/tmp/getfile_dir` directory and puts them in the `/tmp/out_dir` output directory.
+
+The flow: GetFile ➔ success ➔ PutFile
+
+### Tailing a Single File
+
+Using the [tailfile_config.yml](tailfile_config.yml) flow configuration MiNiFi tails a single file `/tmp/test_file.log` and creates flowfiles from every single line, then logs the flowfile attributes.
+
+The flow: TailFile ➔ success ➔ LogAttribute
+
+## Windows Specific Processors
+
+### Consuming Windows Event Logs
+
+Using the [cwel_config.yml](cwel_config.yml) flow configuration MiNiFi queries all Windows system events and puts them to the `C:\temp\` directory in flattened JSON format.
+
+The flow: ConsumeWindowsEventLog ➔ PutFile
+
+## Linux Specific Processors
+
+### Consume Systemd-Journald System Journal Messages
+
+Using the [consumejournald_config.yml](consumejournald_config.yml) flow configuration MiNiFi reads systemd-journald journal messages and logs them on `info` level.
+
+The flow: ConsumeJournald ➔ LogAttribute
+
+### Reading System Performance Data
+
+Using the [pdh_config.yml](pdh_config.yml) flow configuration MiNiFi reads CPU and Disk performance data through Windows' Performance Data Helper (PDH) component and puts the data to the `C:\temp\` directory in a compact JSON format.
+
+The flow: PerformanceDataMonitor ➔ PutFile
+
Review comment:
Good catch, fixed in d02f0d09ded60838efd5498b9627bc4e841e1db5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1127: MINIFICPP-1601 Add example flow configurations
Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #1127:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1127#discussion_r672308980
##########
File path: examples/README.md
##########
@@ -12,6 +12,130 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-# Apache NiFi - MiNiFi - C++ Examples
+# Apache NiFi - MiNiFi - C++ Examples
-- [Site-2-Site Bi-directional Configuration](BidirectionalSiteToSite/README.md)
\ No newline at end of file
+The following examples show simple flow configurations for several common use cases. Use the path of these yaml files as the flow configuration set in the `nifi.flow.configuration.file` property of `conf/minifi.properties` file or replace the default `conf/config.yml` file to try them out.
+
+- [Filesystem Operations](#filesystem-operations)
+ - [Getting File Data and Putting It in an Output Directory](#getting-file-data-and-putting-it-in-an-output-directory)
+ - [Tailing a Single File](#tailing-a-single-file)
+- [Windows Specific Processors](#windows-specific-processors)
+ - [Consuming Windows Event Logs](#consuming-windows-event-logs)
+ - [Reading System Performance Data](#reading-system-performance-data)
+- [HTTP Operations](#http-operations)
+ - [HTTP POST Invocation](#http-post-invocation)
+- [Site to Site Operations](#site-to-site-operations)
+ - [Transfer Data to Remote Nifi Instance](#transfer-data-to-remote-nifi-instance)
+ - [Site-2-Site Bi-directional Configuration](#site-2-site-bi-directional-configuration)
+- [Kafka Operations](#kafka-operations)
+ - [Publish Message to Kafka Broker](#publish-message-to-kafka-broker)
+ - [Publish Message to Kafka Broker Through SSL](#publish-message-to-kafka-broker-through-ssl)
+ - [Consume Messages from Kafka](#consume-messages-from-kafka)
+- [Public Cloud Operations](#public-cloud-operations)
+ - [Upload Blob to Azure Storage](#upload-blob-to-azure-storage)
+ - [Put Object in AWS S3 Bucket](#put-object-in-aws-s3-bucket)
+ - [List and Fetch Content from AWS S3 Bucket](#list-and-fetch-content-from-aws-s3-bucket)
+- [SQL Operations](#sql-operations)
+ - [Query Database Table](#query-database-table)
+- [System Log Operations](#system-log-operations)
+ - [Consume Systemd-Journald System Journal Messages](#consume-systemd-journald-system-journal-messages)
+
+## Filesystem Operations
+
+### Getting File Data and Putting It in an Output Directory
+
+Using the [getfile_putfile_config.yml](getfile_putfile_config.yml) flow configuration MiNiFi gets all files of minimum 1MB size from the `/tmp/getfile_dir` directory and puts them in the `/tmp/out_dir` output directory.
+
+The flow: GetFile ➔ success ➔ PutFile
+
+### Tailing a Single File
+
+Using the [tailfile_config.yml](tailfile_config.yml) flow configuration MiNiFi tails a single file `/tmp/test_file.log` and creates flowfiles from every single line, then logs the flowfile attributes.
+
+The flow: TailFile ➔ success ➔ LogAttribute
+
+## Windows Specific Processors
+
+### Consuming Windows Event Logs
+
+Using the [cwel_config.yml](cwel_config.yml) flow configuration MiNiFi queries all Windows system events and puts them to the `C:\temp\` directory in flattened JSON format.
+
+The flow: ConsumeWindowsEventLog ➔ PutFile
+
+## Linux Specific Processors
+
+### Consume Systemd-Journald System Journal Messages
+
+Using the [consumejournald_config.yml](consumejournald_config.yml) flow configuration MiNiFi reads systemd-journald journal messages and logs them on `info` level.
+
+The flow: ConsumeJournald ➔ LogAttribute
+
+### Reading System Performance Data
+
+Using the [pdh_config.yml](pdh_config.yml) flow configuration MiNiFi reads CPU and Disk performance data through Windows' Performance Data Helper (PDH) component and puts the data to the `C:\temp\` directory in a compact JSON format.
+
+The flow: PerformanceDataMonitor ➔ PutFile
+
Review comment:
Could you move this up to the Windows-specific section? It's now under Linux-specific processors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm closed pull request #1127: MINIFICPP-1601 Add example flow configurations
Posted by GitBox <gi...@apache.org>.
szaszm closed pull request #1127:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1127
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #1127: MINIFICPP-1601 Add example flow configurations
Posted by GitBox <gi...@apache.org>.
lordgamez commented on a change in pull request #1127:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1127#discussion_r667722592
##########
File path: examples/README.md
##########
@@ -12,6 +12,130 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-# Apache NiFi - MiNiFi - C++ Examples
+# Apache NiFi - MiNiFi - C++ Examples
-- [Site-2-Site Bi-directional Configuration](BidirectionalSiteToSite/README.md)
\ No newline at end of file
+The following examples show simple flow configurations for several common use cases. Use the path of these yaml files as the flow configuration set in the `nifi.flow.configuration.file` property of `conf/minifi.properties` file or replace the default `conf/config.yml` file to try them out.
+
+- [Filesystem Operations](#filesystem-operations)
+ - [Getting File Data and Putting It in an Output Directory](#getting-file-data-and-putting-it-in-an-output-directory)
+ - [Tailing a Single File](#tailing-a-single-file)
+- [Windows Specific Processors](#windows-specific-processors)
+ - [Consuming Windows Event Logs](#consuming-windows-event-logs)
+ - [Reading System Performance Data](#reading-system-performance-data)
+- [HTTP Operations](#http-operations)
+ - [HTTP POST Invocation](#http-post-invocation)
+- [Site to Site Operations](#site-to-site-operations)
+ - [Transfer Data to Remote Nifi Instance](#transfer-data-to-remote-nifi-instance)
+ - [Site-2-Site Bi-directional Configuration](#site-2-site-bi-directional-configuration)
+- [Kafka Operations](#kafka-operations)
+ - [Publish Message to Kafka Broker](#publish-message-to-kafka-broker)
+ - [Publish Message to Kafka Broker Through SSL](#publish-message-to-kafka-broker-through-ssl)
+ - [Consume Messages from Kafka](#consume-messages-from-kafka)
+- [Public Cloud Operations](#public-cloud-operations)
+ - [Upload Blob to Azure Storage](#upload-blob-to-azure-storage)
+ - [Put Object in AWS S3 Bucket](#put-object-in-aws-s3-bucket)
+ - [List and Fetch Content from AWS S3 Bucket](#list-and-fetch-content-from-aws-s3-bucket)
+- [SQL Operations](#sql-operations)
+ - [Query Database Table](#query-database-table)
+- [System Log Operations](#system-log-operations)
+ - [Consume Journal Logs](#consume-journal-logs)
Review comment:
Fixed in b7840cdf4f5ca58bacad19cc2986c3a325cf035e
##########
File path: examples/README.md
##########
@@ -12,6 +12,130 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-# Apache NiFi - MiNiFi - C++ Examples
+# Apache NiFi - MiNiFi - C++ Examples
-- [Site-2-Site Bi-directional Configuration](BidirectionalSiteToSite/README.md)
\ No newline at end of file
+The following examples show simple flow configurations for several common use cases. Use the path of these yaml files as the flow configuration set in the `nifi.flow.configuration.file` property of `conf/minifi.properties` file or replace the default `conf/config.yml` file to try them out.
+
+- [Filesystem Operations](#filesystem-operations)
+ - [Getting File Data and Putting It in an Output Directory](#getting-file-data-and-putting-it-in-an-output-directory)
+ - [Tailing a Single File](#tailing-a-single-file)
+- [Windows Specific Processors](#windows-specific-processors)
+ - [Consuming Windows Event Logs](#consuming-windows-event-logs)
+ - [Reading System Performance Data](#reading-system-performance-data)
+- [HTTP Operations](#http-operations)
+ - [HTTP POST Invocation](#http-post-invocation)
+- [Site to Site Operations](#site-to-site-operations)
+ - [Transfer Data to Remote Nifi Instance](#transfer-data-to-remote-nifi-instance)
+ - [Site-2-Site Bi-directional Configuration](#site-2-site-bi-directional-configuration)
+- [Kafka Operations](#kafka-operations)
+ - [Publish Message to Kafka Broker](#publish-message-to-kafka-broker)
+ - [Publish Message to Kafka Broker Through SSL](#publish-message-to-kafka-broker-through-ssl)
+ - [Consume Messages from Kafka](#consume-messages-from-kafka)
+- [Public Cloud Operations](#public-cloud-operations)
+ - [Upload Blob to Azure Storage](#upload-blob-to-azure-storage)
+ - [Put Object in AWS S3 Bucket](#put-object-in-aws-s3-bucket)
+ - [List and Fetch Content from AWS S3 Bucket](#list-and-fetch-content-from-aws-s3-bucket)
+- [SQL Operations](#sql-operations)
+ - [Query Database Table](#query-database-table)
+- [System Log Operations](#system-log-operations)
+ - [Consume Journal Logs](#consume-journal-logs)
+
+## Filesystem Operations
+
+### Getting File Data and Putting It in an Output Directory
+
+Using the [getfile_putfile_config.yml](getfile_putfile_config.yml) flow configuration MiNiFi gets all files of minimum 1MB size from the `/tmp/getfile_dir` directory and puts them in the `/tmp/out_dir` output directory.
+
+The flow: GetFile ➔ success ➔ PutFile
+
+### Tailing a Single File
+
+Using the [tailfile_config.yml](tailfile_config.yml) flow configuration MiNiFi tails a single file `/tmp/test_file.log` and creates flowfiles from every single line, then logs the flowfile attributes.
+
+The flow: TailFile ➔ success ➔ LogAttribute
+
+## Windows Specific Processors
+
+### Consuming Windows Event Logs
+
+Using the [cwel_config.yml](cwel_config.yml) flow configuration MiNiFi queries all Windows system events and puts them to the `C:\temp\` directory in flattened JSON format.
+
+The flow: ConsumeWindowsEventLog ➔ PutFile
+
+## Linux Specific Processors
+
+### Consume Journal Messages
+
+Using the [consumejournald_config.yml](consumejournald_config.yml) flow configuration MiNiFi reads incoming journal messages and logs the attributes on `info` level.
Review comment:
Fixed in b7840cdf4f5ca58bacad19cc2986c3a325cf035e
##########
File path: examples/consumejournald_config.yml
##########
@@ -0,0 +1,44 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the \"License\"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an \"AS IS\" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+MiNiFi Config Version: 3
+Flow Controller:
+ name: MiNiFi Flow
+Processors:
+- name: Consume journal messages
+ id: 66fbb65d-8b11-43a9-aaad-746a07d2fe35
+ class: org.apache.nifi.minifi.extensions.systemd.ConsumeJournald
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 5 sec
+- name: Log attributes to info level
Review comment:
Fixed in b7840cdf4f5ca58bacad19cc2986c3a325cf035e
##########
File path: examples/consumejournald_config.yml
##########
@@ -0,0 +1,44 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the \"License\"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an \"AS IS\" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+MiNiFi Config Version: 3
+Flow Controller:
+ name: MiNiFi Flow
+Processors:
+- name: Consume journal messages
+ id: 66fbb65d-8b11-43a9-aaad-746a07d2fe35
+ class: org.apache.nifi.minifi.extensions.systemd.ConsumeJournald
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 5 sec
+- name: Log attributes to info level
+ id: f7281113-9576-4680-9a87-a8e961101b5c
+ class: org.apache.nifi.minifi.processors.LogAttribute
+ scheduling strategy: EVENT_DRIVEN
+ auto-terminated relationships list: [success]
+ Properties:
+ FlowFiles To Log: 0
+ Log Level: info
+ Log Payload: true
+ Log Prefix: '====='
+ Maxumim Payload Line Length: 0
+Connections:
+- name: ConsumeJournald/success/LogAttribute
+ id: f57713d9-fa0b-4416-948d-4b316b1a1c0d
+ source name: ConsumeJournald
Review comment:
Fixed in b7840cdf4f5ca58bacad19cc2986c3a325cf035e
##########
File path: examples/publishkafka_ssl_config.yml
##########
@@ -0,0 +1,55 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+MiNiFi Config Version: 3
+Flow Controller:
+ name: MiNiFi Flow
+Processors:
+- name: Get files from /tmp/input
+ id: a78f9d7d-cc68-4f30-aac1-ee716d42bf9d
+ class: org.apache.nifi.processors.standard.GetFile
+ scheduling period: 2 sec
+ scheduling strategy: TIMER_DRIVEN
+ Properties:
+ Input Directory: /tmp/input
+- name: Publish messages to Kafka topic test
+ id: 8c0791c7-920c-4541-bcde-f5d559f42ed2
+ class: org.apache.nifi.processors.standard.PublishKafka
+ scheduling period: 1 sec
+ scheduling strategy: EVENT_DRIVEN
+ auto-terminated relationships list:
+ - success
+ Properties:
Review comment:
Fixed in b7840cdf4f5ca58bacad19cc2986c3a325cf035e
##########
File path: examples/publishkafka_config.yml
##########
@@ -0,0 +1,46 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+MiNiFi Config Version: 3
+Flow Controller:
+ name: MiNiFi Flow
+Processors:
+- name: Get files from /tmp/input
+ id: 7fd166aa-0662-4c42-affa-88f6fb39807f
+ class: org.apache.nifi.processors.standard.GetFile
+ scheduling period: 2 sec
+ scheduling strategy: TIMER_DRIVEN
+ Properties:
+ Input Directory: /tmp/input
+- name: Publish messages to Kafka topic test
+ id: 8a534b4a-2b4a-4e1e-ab07-8a09fa08f848
+ class: org.apache.nifi.processors.standard.PublishKafka
+ scheduling strategy: EVENT_DRIVEN
+ Properties:
+ Batch Size: '10'
+ Client Name: test-client
+ Compress Codec: none
+ Delivery Guarantee: '1'
+ Known Brokers: kafka-broker:9092
+ Message Timeout: 12 sec
+ Request Timeout: 10 sec
+ Topic Name: test
Review comment:
Fixed in b7840cdf4f5ca58bacad19cc2986c3a325cf035e
##########
File path: examples/consumekafka_config.yml
##########
@@ -0,0 +1,51 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+MiNiFi Config Version: 3
+Flow Controller:
+ name: MiNiFi Flow
+Processors:
+- name: Consume Kafka messages from ConsumeKafkaTest topic
+ id: 37d6e8e5-6284-423e-8620-c5168a71add7
+ class: org.apache.nifi.processors.standard.ConsumeKafka
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 1 sec
+ Properties:
+ Group ID: test_group
+ Honor Transactions: 'true'
+ Kafka Brokers: kafka-broker:9092
+ Key Attribute Encoding: UTF-8
+ Max Poll Time: 4 sec
+ Message Header Encoding: UTF-8
+ Offset Reset: earliest
+ Session Timeout: 60 sec
+ Topic Name Format: Names
+ Topic Names: ConsumeKafkaTest
Review comment:
Fixed in b7840cdf4f5ca58bacad19cc2986c3a325cf035e
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] martinzink commented on a change in pull request #1127: MINIFICPP-1601 Add example flow configurations
Posted by GitBox <gi...@apache.org>.
martinzink commented on a change in pull request #1127:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1127#discussion_r666394926
##########
File path: examples/publishkafka_ssl_config.yml
##########
@@ -0,0 +1,55 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+MiNiFi Config Version: 3
+Flow Controller:
+ name: MiNiFi Flow
+Processors:
+- name: Get files from /tmp/input
+ id: a78f9d7d-cc68-4f30-aac1-ee716d42bf9d
+ class: org.apache.nifi.processors.standard.GetFile
+ scheduling period: 2 sec
+ scheduling strategy: TIMER_DRIVEN
+ Properties:
+ Input Directory: /tmp/input
+- name: Publish messages to Kafka topic test
+ id: 8c0791c7-920c-4541-bcde-f5d559f42ed2
+ class: org.apache.nifi.processors.standard.PublishKafka
+ scheduling period: 1 sec
+ scheduling strategy: EVENT_DRIVEN
+ auto-terminated relationships list:
+ - success
+ Properties:
+ Batch Size: 10
+ Client Name: test_client
+ Compress Codec: none
+ Delivery Guarantee: 1
+ Known Brokers: kafka-broker:9093
+ Message Timeout: 12 sec
+ Message Timeout Phrase: 12 sec
+ Request Timeout: 10 sec
+ Security CA Key: /tmp/resources/certs/ca-cert
+ Security Cert: /tmp/resources/certs/client_test_client_client.pem
+ Security Pass Phrase: abcdefgh
+ Security Private Key: /tmp/resources/certs/client_test_client_client.key
+ Security Protocol: ssl
+ Topic Name: test
+Connections:
+- name: 54c70a6d-d9d9-4709-9747-1ab47edad3c5
Review comment:
We should also have a human readable name here similar to the others.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on pull request #1127: MINIFICPP-1601 Add example flow configurations
Posted by GitBox <gi...@apache.org>.
lordgamez commented on pull request #1127:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1127#issuecomment-876966835
> BTW I really like this PR , I think it makes the project much more approachable for new users.
I cannot take all the credit as it was @arpadboda's idea :slightly_smiling_face:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #1127: MINIFICPP-1601 Add example flow configurations
Posted by GitBox <gi...@apache.org>.
lordgamez commented on a change in pull request #1127:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1127#discussion_r666381189
##########
File path: examples/puts3_config.yml
##########
@@ -0,0 +1,72 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+MiNiFi Config Version: 3
+Flow Controller:
+ name: MiNiFi Flow
+Processors:
+- Properties:
+ Input Directory: /tmp/input
+ auto-terminated relationships list: []
+ class: org.apache.nifi.processors.standard.GetFile
+ id: bfd376a8-c845-453b-9f74-4cf6f3a95e74
+ name: GetFile
+ penalization period: 30 sec
+ run duration nanos: 0
+ scheduling period: 2 sec
+ scheduling strategy: TIMER_DRIVEN
+ yield period: 1 sec
+- Properties:
+ Access Key: test_access_key
+ Bucket: test_bucket
+ Secret Key: test_secret
+ auto-terminated relationships list:
+ - failure
+ class: org.apache.nifi.processors.standard.PutS3Object
+ id: 8c9e2512-e814-4b73-ac5c-f33753332128
+ name: PutS3Object
+ penalization period: 30 sec
+ run duration nanos: 0
+ scheduling period: 1 sec
+ scheduling strategy: TIMER_DRIVEN
+ yield period: 1 sec
+- Properties:
+ Directory: /tmp/output
+ Directory Permissions: '777'
+ Permissions: '777'
+ auto-terminated relationships list:
+ - success
+ - failure
+ class: org.apache.nifi.processors.standard.PutFile
Review comment:
Changed the files have a consistent layout in 369f1085d136a749731b067dad19c73cded52532. Also removed the optional properties that were just making noise.
I really liked your example, it's simple, on the point for anyone who wants to use ConsumeJournald processors, so I added that to the list of examples in 5583561941397e5605c339cb7e15148ac411f286
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] martinzink commented on a change in pull request #1127: MINIFICPP-1601 Add example flow configurations
Posted by GitBox <gi...@apache.org>.
martinzink commented on a change in pull request #1127:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1127#discussion_r666392703
##########
File path: examples/pdh_config.yml
##########
@@ -0,0 +1,49 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+MiNiFi Config Version: 3
+Flow Controller:
+ name: MiNiFi Flow
+Processors:
+- name: Read disk and CPU performance data
+ id: 2f2a3b47-f5ba-49f6-82b5-bc1c86b96e27
+ class: org.apache.nifi.minifi.processors.PerformanceDataMonitor
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 10000 ms
+ Properties:
+ Predefined Groups: "Disk,CPU"
+ Custom PDH Counters: "\\System\\Processes,\\Process(*)\\ID Process,\\Process(*)\\Private Bytes"
+ Output Format: "JSON"
+ Output Compactness: "Compact"
+- name: Put files to C:\temp\
+ id: e143601d-de4f-44ba-a6ec-d1f97d77ec94
+ class: org.apache.nifi.minifi.processors.PutFile
+ scheduling strategy: EVENT_DRIVEN
+ auto-terminated relationships list:
+ - failure
+ - success
+ Properties:
+ Conflict Resolution Strategy: fail
+ Create Missing Directories: 'true'
+ Directory: C:\temp\
+Connections:
+- name: ConsumeWindowsEventLog/success/PutFile
Review comment:
This connection name is inconsistent
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1127: MINIFICPP-1601 Add example flow configurations
Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #1127:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1127#discussion_r672308980
##########
File path: examples/README.md
##########
@@ -12,6 +12,130 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-# Apache NiFi - MiNiFi - C++ Examples
+# Apache NiFi - MiNiFi - C++ Examples
-- [Site-2-Site Bi-directional Configuration](BidirectionalSiteToSite/README.md)
\ No newline at end of file
+The following examples show simple flow configurations for several common use cases. Use the path of these yaml files as the flow configuration set in the `nifi.flow.configuration.file` property of `conf/minifi.properties` file or replace the default `conf/config.yml` file to try them out.
+
+- [Filesystem Operations](#filesystem-operations)
+ - [Getting File Data and Putting It in an Output Directory](#getting-file-data-and-putting-it-in-an-output-directory)
+ - [Tailing a Single File](#tailing-a-single-file)
+- [Windows Specific Processors](#windows-specific-processors)
+ - [Consuming Windows Event Logs](#consuming-windows-event-logs)
+ - [Reading System Performance Data](#reading-system-performance-data)
+- [HTTP Operations](#http-operations)
+ - [HTTP POST Invocation](#http-post-invocation)
+- [Site to Site Operations](#site-to-site-operations)
+ - [Transfer Data to Remote Nifi Instance](#transfer-data-to-remote-nifi-instance)
+ - [Site-2-Site Bi-directional Configuration](#site-2-site-bi-directional-configuration)
+- [Kafka Operations](#kafka-operations)
+ - [Publish Message to Kafka Broker](#publish-message-to-kafka-broker)
+ - [Publish Message to Kafka Broker Through SSL](#publish-message-to-kafka-broker-through-ssl)
+ - [Consume Messages from Kafka](#consume-messages-from-kafka)
+- [Public Cloud Operations](#public-cloud-operations)
+ - [Upload Blob to Azure Storage](#upload-blob-to-azure-storage)
+ - [Put Object in AWS S3 Bucket](#put-object-in-aws-s3-bucket)
+ - [List and Fetch Content from AWS S3 Bucket](#list-and-fetch-content-from-aws-s3-bucket)
+- [SQL Operations](#sql-operations)
+ - [Query Database Table](#query-database-table)
+- [System Log Operations](#system-log-operations)
+ - [Consume Systemd-Journald System Journal Messages](#consume-systemd-journald-system-journal-messages)
+
+## Filesystem Operations
+
+### Getting File Data and Putting It in an Output Directory
+
+Using the [getfile_putfile_config.yml](getfile_putfile_config.yml) flow configuration MiNiFi gets all files of minimum 1MB size from the `/tmp/getfile_dir` directory and puts them in the `/tmp/out_dir` output directory.
+
+The flow: GetFile ➔ success ➔ PutFile
+
+### Tailing a Single File
+
+Using the [tailfile_config.yml](tailfile_config.yml) flow configuration MiNiFi tails a single file `/tmp/test_file.log` and creates flowfiles from every single line, then logs the flowfile attributes.
+
+The flow: TailFile ➔ success ➔ LogAttribute
+
+## Windows Specific Processors
+
+### Consuming Windows Event Logs
+
+Using the [cwel_config.yml](cwel_config.yml) flow configuration MiNiFi queries all Windows system events and puts them to the `C:\temp\` directory in flattened JSON format.
+
+The flow: ConsumeWindowsEventLog ➔ PutFile
+
+## Linux Specific Processors
+
+### Consume Systemd-Journald System Journal Messages
+
+Using the [consumejournald_config.yml](consumejournald_config.yml) flow configuration MiNiFi reads systemd-journald journal messages and logs them on `info` level.
+
+The flow: ConsumeJournald ➔ LogAttribute
+
+### Reading System Performance Data
+
+Using the [pdh_config.yml](pdh_config.yml) flow configuration MiNiFi reads CPU and Disk performance data through Windows' Performance Data Helper (PDH) component and puts the data to the `C:\temp\` directory in a compact JSON format.
+
+The flow: PerformanceDataMonitor ➔ PutFile
+
Review comment:
Could you move this up to the Windows-specific section? It's now under Linux-specific processors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #1127: MINIFICPP-1601 Add example flow configurations
Posted by GitBox <gi...@apache.org>.
lordgamez commented on a change in pull request #1127:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1127#discussion_r666381580
##########
File path: examples/pdh_config.yml
##########
@@ -0,0 +1,62 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+MiNiFi Config Version: 3
+Flow Controller:
+ name: MiNiFi Flow
+Processors:
+- id: 2f2a3b47-f5ba-49f6-82b5-bc1c86b96e27
+ name: ConsumeWindowsEventLog
+ class: org.apache.nifi.minifi.processors.PerformanceDataMonitor
Review comment:
Fixed in 369f1085d136a749731b067dad19c73cded52532
##########
File path: examples/puts3_config.yml
##########
@@ -0,0 +1,72 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+MiNiFi Config Version: 3
+Flow Controller:
+ name: MiNiFi Flow
+Processors:
+- Properties:
+ Input Directory: /tmp/input
+ auto-terminated relationships list: []
+ class: org.apache.nifi.processors.standard.GetFile
+ id: bfd376a8-c845-453b-9f74-4cf6f3a95e74
+ name: GetFile
+ penalization period: 30 sec
+ run duration nanos: 0
+ scheduling period: 2 sec
+ scheduling strategy: TIMER_DRIVEN
+ yield period: 1 sec
+- Properties:
+ Access Key: test_access_key
+ Bucket: test_bucket
+ Secret Key: test_secret
+ auto-terminated relationships list:
+ - failure
+ class: org.apache.nifi.processors.standard.PutS3Object
+ id: 8c9e2512-e814-4b73-ac5c-f33753332128
+ name: PutS3Object
+ penalization period: 30 sec
+ run duration nanos: 0
+ scheduling period: 1 sec
+ scheduling strategy: TIMER_DRIVEN
+ yield period: 1 sec
+- Properties:
+ Directory: /tmp/output
+ Directory Permissions: '777'
+ Permissions: '777'
+ auto-terminated relationships list:
+ - success
+ - failure
+ class: org.apache.nifi.processors.standard.PutFile
+ id: d7dbc3af-ee24-4bc0-8c63-90e0f3185e0f
+ name: PutFile
+ penalization period: 30 sec
+ run duration nanos: 0
+ scheduling period: 1 sec
+ scheduling strategy: EVENT_DRIVEN
+ yield period: 1 sec
+Connections:
+- destination id: 8c9e2512-e814-4b73-ac5c-f33753332128
+ drop empty: 'false'
+ name: cc62a4a5-3a43-4725-8d05-d0b3e65d1ee6
Review comment:
Good point, especially for newcomers for easier understanding. Fixed in 369f1085d136a749731b067dad19c73cded52532
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #1127: MINIFICPP-1601 Add example flow configurations
Posted by GitBox <gi...@apache.org>.
lordgamez commented on a change in pull request #1127:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1127#discussion_r667723353
##########
File path: examples/http_post_config.yml
##########
@@ -0,0 +1,50 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+MiNiFi Config Version: 3
+Flow Controller:
+ name: MiNiFi Flow
+Processors:
+- name: Get files from /tmp/input
+ id: 962790e7-ea35-4096-9362-96f527288669
+ class: org.apache.nifi.processors.standard.GetFile
+ scheduling period: 2 sec
+ scheduling strategy: TIMER_DRIVEN
+ Properties:
+ Input Directory: /tmp/input
+ Keep Source File: 'true'
+- name: Invoke POST request on remote URL
+ id: 33b373b6-6f19-4194-b45a-1ef73c357a8e
+ class: org.apache.nifi.processors.standard.InvokeHTTP
+ scheduling strategy: EVENT_DRIVEN
+ auto-terminated relationships list:
+ - success
+ - response
+ - retry
+ - no retry
+ Properties:
+ HTTP Method: POST
+ Remote URL: http://1.2.3.4:8080/contentListener
+Connections:
+- name: GetFile/success/InvokeHTTP
Review comment:
Added retry connection in 5b2bb27b854cbe7e3a07ffad506c186b5257c378, but I could not find the way to limit the number of retries.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm closed pull request #1127: MINIFICPP-1601 Add example flow configurations
Posted by GitBox <gi...@apache.org>.
szaszm closed pull request #1127:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1127
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1127: MINIFICPP-1601 Add example flow configurations
Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #1127:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1127#discussion_r666274691
##########
File path: examples/puts3_config.yml
##########
@@ -0,0 +1,72 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+MiNiFi Config Version: 3
+Flow Controller:
+ name: MiNiFi Flow
+Processors:
+- Properties:
+ Input Directory: /tmp/input
+ auto-terminated relationships list: []
+ class: org.apache.nifi.processors.standard.GetFile
+ id: bfd376a8-c845-453b-9f74-4cf6f3a95e74
+ name: GetFile
+ penalization period: 30 sec
+ run duration nanos: 0
+ scheduling period: 2 sec
+ scheduling strategy: TIMER_DRIVEN
+ yield period: 1 sec
+- Properties:
+ Access Key: test_access_key
+ Bucket: test_bucket
+ Secret Key: test_secret
+ auto-terminated relationships list:
+ - failure
+ class: org.apache.nifi.processors.standard.PutS3Object
+ id: 8c9e2512-e814-4b73-ac5c-f33753332128
+ name: PutS3Object
+ penalization period: 30 sec
+ run duration nanos: 0
+ scheduling period: 1 sec
+ scheduling strategy: TIMER_DRIVEN
+ yield period: 1 sec
+- Properties:
+ Directory: /tmp/output
+ Directory Permissions: '777'
+ Permissions: '777'
+ auto-terminated relationships list:
+ - success
+ - failure
+ class: org.apache.nifi.processors.standard.PutFile
Review comment:
As an example, this is the config I was using to test ConsumeJournald: https://pastebin.com/svRPwyMh
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1127: MINIFICPP-1601 Add example flow configurations
Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #1127:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1127#discussion_r666266339
##########
File path: examples/puts3_config.yml
##########
@@ -0,0 +1,72 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+MiNiFi Config Version: 3
+Flow Controller:
+ name: MiNiFi Flow
+Processors:
+- Properties:
+ Input Directory: /tmp/input
+ auto-terminated relationships list: []
+ class: org.apache.nifi.processors.standard.GetFile
+ id: bfd376a8-c845-453b-9f74-4cf6f3a95e74
+ name: GetFile
+ penalization period: 30 sec
+ run duration nanos: 0
+ scheduling period: 2 sec
+ scheduling strategy: TIMER_DRIVEN
+ yield period: 1 sec
+- Properties:
+ Access Key: test_access_key
+ Bucket: test_bucket
+ Secret Key: test_secret
+ auto-terminated relationships list:
+ - failure
+ class: org.apache.nifi.processors.standard.PutS3Object
+ id: 8c9e2512-e814-4b73-ac5c-f33753332128
+ name: PutS3Object
+ penalization period: 30 sec
+ run duration nanos: 0
+ scheduling period: 1 sec
+ scheduling strategy: TIMER_DRIVEN
+ yield period: 1 sec
+- Properties:
+ Directory: /tmp/output
+ Directory Permissions: '777'
+ Permissions: '777'
+ auto-terminated relationships list:
+ - success
+ - failure
+ class: org.apache.nifi.processors.standard.PutFile
+ id: d7dbc3af-ee24-4bc0-8c63-90e0f3185e0f
+ name: PutFile
+ penalization period: 30 sec
+ run duration nanos: 0
+ scheduling period: 1 sec
+ scheduling strategy: EVENT_DRIVEN
+ yield period: 1 sec
+Connections:
+- destination id: 8c9e2512-e814-4b73-ac5c-f33753332128
+ drop empty: 'false'
+ name: cc62a4a5-3a43-4725-8d05-d0b3e65d1ee6
Review comment:
More descriptive names are a good practice. For connections, the usual source-name/relationship/destination-name is fine. For processors, something like "Upload to S3 test_bucket" is preferrable over "PutS3Object".
##########
File path: examples/puts3_config.yml
##########
@@ -0,0 +1,72 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+MiNiFi Config Version: 3
+Flow Controller:
+ name: MiNiFi Flow
+Processors:
+- Properties:
+ Input Directory: /tmp/input
+ auto-terminated relationships list: []
+ class: org.apache.nifi.processors.standard.GetFile
+ id: bfd376a8-c845-453b-9f74-4cf6f3a95e74
+ name: GetFile
+ penalization period: 30 sec
+ run duration nanos: 0
+ scheduling period: 2 sec
+ scheduling strategy: TIMER_DRIVEN
+ yield period: 1 sec
+- Properties:
+ Access Key: test_access_key
+ Bucket: test_bucket
+ Secret Key: test_secret
+ auto-terminated relationships list:
+ - failure
+ class: org.apache.nifi.processors.standard.PutS3Object
+ id: 8c9e2512-e814-4b73-ac5c-f33753332128
+ name: PutS3Object
+ penalization period: 30 sec
+ run duration nanos: 0
+ scheduling period: 1 sec
+ scheduling strategy: TIMER_DRIVEN
+ yield period: 1 sec
+- Properties:
+ Directory: /tmp/output
+ Directory Permissions: '777'
+ Permissions: '777'
+ auto-terminated relationships list:
+ - success
+ - failure
+ class: org.apache.nifi.processors.standard.PutFile
Review comment:
I find this layout confusing. Consider following the format of `CONFIGURE.md`:
```
- name: GetFile
id: 471deef6-2a6e-4a7d-912a-81cc17e3a206
class: org.apache.nifi.processors.standard.GetFile
max concurrent tasks: 1
scheduling strategy: TIMER_DRIVEN
scheduling period: 1 sec
penalization period: 30 sec
yield period: 1 sec
run duration nanos: 0
auto-terminated relationships list:
Properties:
Input Directory: /tmp/getfile
Keep Source File: true
```
Also, I would exclude unnecessary settings from the example config, like "scheduling period" in case of EVENT_DRIVEN strategy. Or yield period, unless it has some significance.
##########
File path: examples/pdh_config.yml
##########
@@ -0,0 +1,62 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+MiNiFi Config Version: 3
+Flow Controller:
+ name: MiNiFi Flow
+Processors:
+- id: 2f2a3b47-f5ba-49f6-82b5-bc1c86b96e27
+ name: ConsumeWindowsEventLog
+ class: org.apache.nifi.minifi.processors.PerformanceDataMonitor
Review comment:
There is some inconsistency here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1127: MINIFICPP-1601 Add example flow configurations
Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #1127:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1127#discussion_r667735503
##########
File path: examples/http_post_config.yml
##########
@@ -0,0 +1,50 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+MiNiFi Config Version: 3
+Flow Controller:
+ name: MiNiFi Flow
+Processors:
+- name: Get files from /tmp/input
+ id: 962790e7-ea35-4096-9362-96f527288669
+ class: org.apache.nifi.processors.standard.GetFile
+ scheduling period: 2 sec
+ scheduling strategy: TIMER_DRIVEN
+ Properties:
+ Input Directory: /tmp/input
+ Keep Source File: 'true'
+- name: Invoke POST request on remote URL
+ id: 33b373b6-6f19-4194-b45a-1ef73c357a8e
+ class: org.apache.nifi.processors.standard.InvokeHTTP
+ scheduling strategy: EVENT_DRIVEN
+ auto-terminated relationships list:
+ - success
+ - response
+ - retry
+ - no retry
+ Properties:
+ HTTP Method: POST
+ Remote URL: http://1.2.3.4:8080/contentListener
+Connections:
+- name: GetFile/success/InvokeHTTP
Review comment:
One can limit the number or retries using the [RetryFlowFile](https://github.com/apache/nifi-minifi-cpp/blob/main/PROCESSORS.md#retryflowfile) processor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] martinzink commented on a change in pull request #1127: MINIFICPP-1601 Add example flow configurations
Posted by GitBox <gi...@apache.org>.
martinzink commented on a change in pull request #1127:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1127#discussion_r666394274
##########
File path: examples/publishkafka_config.yml
##########
@@ -0,0 +1,46 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+MiNiFi Config Version: 3
+Flow Controller:
+ name: MiNiFi Flow
+Processors:
+- name: Get files from /tmp/input
+ id: 7fd166aa-0662-4c42-affa-88f6fb39807f
+ class: org.apache.nifi.processors.standard.GetFile
+ scheduling period: 2 sec
+ scheduling strategy: TIMER_DRIVEN
+ Properties:
+ Input Directory: /tmp/input
+- name: Publish messages to Kafka topic test
+ id: 8a534b4a-2b4a-4e1e-ab07-8a09fa08f848
+ class: org.apache.nifi.processors.standard.PublishKafka
+ scheduling strategy: EVENT_DRIVEN
+ Properties:
+ Batch Size: '10'
+ Client Name: test-client
+ Compress Codec: none
+ Delivery Guarantee: '1'
+ Known Brokers: kafka-broker:9092
+ Message Timeout: 12 sec
+ Request Timeout: 10 sec
+ Topic Name: test
+Connections:
+- name: d61e509d-03a7-4c70-a7cf-6676d8047825
Review comment:
We should also have a human readable name here similar to the others.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #1127: MINIFICPP-1601 Add example flow configurations
Posted by GitBox <gi...@apache.org>.
lordgamez commented on a change in pull request #1127:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1127#discussion_r672313737
##########
File path: examples/README.md
##########
@@ -12,6 +12,130 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-# Apache NiFi - MiNiFi - C++ Examples
+# Apache NiFi - MiNiFi - C++ Examples
-- [Site-2-Site Bi-directional Configuration](BidirectionalSiteToSite/README.md)
\ No newline at end of file
+The following examples show simple flow configurations for several common use cases. Use the path of these yaml files as the flow configuration set in the `nifi.flow.configuration.file` property of `conf/minifi.properties` file or replace the default `conf/config.yml` file to try them out.
+
+- [Filesystem Operations](#filesystem-operations)
+ - [Getting File Data and Putting It in an Output Directory](#getting-file-data-and-putting-it-in-an-output-directory)
+ - [Tailing a Single File](#tailing-a-single-file)
+- [Windows Specific Processors](#windows-specific-processors)
+ - [Consuming Windows Event Logs](#consuming-windows-event-logs)
+ - [Reading System Performance Data](#reading-system-performance-data)
+- [HTTP Operations](#http-operations)
+ - [HTTP POST Invocation](#http-post-invocation)
+- [Site to Site Operations](#site-to-site-operations)
+ - [Transfer Data to Remote Nifi Instance](#transfer-data-to-remote-nifi-instance)
+ - [Site-2-Site Bi-directional Configuration](#site-2-site-bi-directional-configuration)
+- [Kafka Operations](#kafka-operations)
+ - [Publish Message to Kafka Broker](#publish-message-to-kafka-broker)
+ - [Publish Message to Kafka Broker Through SSL](#publish-message-to-kafka-broker-through-ssl)
+ - [Consume Messages from Kafka](#consume-messages-from-kafka)
+- [Public Cloud Operations](#public-cloud-operations)
+ - [Upload Blob to Azure Storage](#upload-blob-to-azure-storage)
+ - [Put Object in AWS S3 Bucket](#put-object-in-aws-s3-bucket)
+ - [List and Fetch Content from AWS S3 Bucket](#list-and-fetch-content-from-aws-s3-bucket)
+- [SQL Operations](#sql-operations)
+ - [Query Database Table](#query-database-table)
+- [System Log Operations](#system-log-operations)
+ - [Consume Systemd-Journald System Journal Messages](#consume-systemd-journald-system-journal-messages)
+
+## Filesystem Operations
+
+### Getting File Data and Putting It in an Output Directory
+
+Using the [getfile_putfile_config.yml](getfile_putfile_config.yml) flow configuration MiNiFi gets all files of minimum 1MB size from the `/tmp/getfile_dir` directory and puts them in the `/tmp/out_dir` output directory.
+
+The flow: GetFile ➔ success ➔ PutFile
+
+### Tailing a Single File
+
+Using the [tailfile_config.yml](tailfile_config.yml) flow configuration MiNiFi tails a single file `/tmp/test_file.log` and creates flowfiles from every single line, then logs the flowfile attributes.
+
+The flow: TailFile ➔ success ➔ LogAttribute
+
+## Windows Specific Processors
+
+### Consuming Windows Event Logs
+
+Using the [cwel_config.yml](cwel_config.yml) flow configuration MiNiFi queries all Windows system events and puts them to the `C:\temp\` directory in flattened JSON format.
+
+The flow: ConsumeWindowsEventLog ➔ PutFile
+
+## Linux Specific Processors
+
+### Consume Systemd-Journald System Journal Messages
+
+Using the [consumejournald_config.yml](consumejournald_config.yml) flow configuration MiNiFi reads systemd-journald journal messages and logs them on `info` level.
+
+The flow: ConsumeJournald ➔ LogAttribute
+
+### Reading System Performance Data
+
+Using the [pdh_config.yml](pdh_config.yml) flow configuration MiNiFi reads CPU and Disk performance data through Windows' Performance Data Helper (PDH) component and puts the data to the `C:\temp\` directory in a compact JSON format.
+
+The flow: PerformanceDataMonitor ➔ PutFile
+
Review comment:
Good catch, fixed in d02f0d09ded60838efd5498b9627bc4e841e1db5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1127: MINIFICPP-1601 Add example flow configurations
Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #1127:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1127#discussion_r672308980
##########
File path: examples/README.md
##########
@@ -12,6 +12,130 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-# Apache NiFi - MiNiFi - C++ Examples
+# Apache NiFi - MiNiFi - C++ Examples
-- [Site-2-Site Bi-directional Configuration](BidirectionalSiteToSite/README.md)
\ No newline at end of file
+The following examples show simple flow configurations for several common use cases. Use the path of these yaml files as the flow configuration set in the `nifi.flow.configuration.file` property of `conf/minifi.properties` file or replace the default `conf/config.yml` file to try them out.
+
+- [Filesystem Operations](#filesystem-operations)
+ - [Getting File Data and Putting It in an Output Directory](#getting-file-data-and-putting-it-in-an-output-directory)
+ - [Tailing a Single File](#tailing-a-single-file)
+- [Windows Specific Processors](#windows-specific-processors)
+ - [Consuming Windows Event Logs](#consuming-windows-event-logs)
+ - [Reading System Performance Data](#reading-system-performance-data)
+- [HTTP Operations](#http-operations)
+ - [HTTP POST Invocation](#http-post-invocation)
+- [Site to Site Operations](#site-to-site-operations)
+ - [Transfer Data to Remote Nifi Instance](#transfer-data-to-remote-nifi-instance)
+ - [Site-2-Site Bi-directional Configuration](#site-2-site-bi-directional-configuration)
+- [Kafka Operations](#kafka-operations)
+ - [Publish Message to Kafka Broker](#publish-message-to-kafka-broker)
+ - [Publish Message to Kafka Broker Through SSL](#publish-message-to-kafka-broker-through-ssl)
+ - [Consume Messages from Kafka](#consume-messages-from-kafka)
+- [Public Cloud Operations](#public-cloud-operations)
+ - [Upload Blob to Azure Storage](#upload-blob-to-azure-storage)
+ - [Put Object in AWS S3 Bucket](#put-object-in-aws-s3-bucket)
+ - [List and Fetch Content from AWS S3 Bucket](#list-and-fetch-content-from-aws-s3-bucket)
+- [SQL Operations](#sql-operations)
+ - [Query Database Table](#query-database-table)
+- [System Log Operations](#system-log-operations)
+ - [Consume Systemd-Journald System Journal Messages](#consume-systemd-journald-system-journal-messages)
+
+## Filesystem Operations
+
+### Getting File Data and Putting It in an Output Directory
+
+Using the [getfile_putfile_config.yml](getfile_putfile_config.yml) flow configuration MiNiFi gets all files of minimum 1MB size from the `/tmp/getfile_dir` directory and puts them in the `/tmp/out_dir` output directory.
+
+The flow: GetFile ➔ success ➔ PutFile
+
+### Tailing a Single File
+
+Using the [tailfile_config.yml](tailfile_config.yml) flow configuration MiNiFi tails a single file `/tmp/test_file.log` and creates flowfiles from every single line, then logs the flowfile attributes.
+
+The flow: TailFile ➔ success ➔ LogAttribute
+
+## Windows Specific Processors
+
+### Consuming Windows Event Logs
+
+Using the [cwel_config.yml](cwel_config.yml) flow configuration MiNiFi queries all Windows system events and puts them to the `C:\temp\` directory in flattened JSON format.
+
+The flow: ConsumeWindowsEventLog ➔ PutFile
+
+## Linux Specific Processors
+
+### Consume Systemd-Journald System Journal Messages
+
+Using the [consumejournald_config.yml](consumejournald_config.yml) flow configuration MiNiFi reads systemd-journald journal messages and logs them on `info` level.
+
+The flow: ConsumeJournald ➔ LogAttribute
+
+### Reading System Performance Data
+
+Using the [pdh_config.yml](pdh_config.yml) flow configuration MiNiFi reads CPU and Disk performance data through Windows' Performance Data Helper (PDH) component and puts the data to the `C:\temp\` directory in a compact JSON format.
+
+The flow: PerformanceDataMonitor ➔ PutFile
+
Review comment:
Could you move this up to the Windows-specific section? It's now under Linux-specific processors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #1127: MINIFICPP-1601 Add example flow configurations
Posted by GitBox <gi...@apache.org>.
lordgamez commented on a change in pull request #1127:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1127#discussion_r666398304
##########
File path: examples/pdh_config.yml
##########
@@ -0,0 +1,49 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+MiNiFi Config Version: 3
+Flow Controller:
+ name: MiNiFi Flow
+Processors:
+- name: Read disk and CPU performance data
+ id: 2f2a3b47-f5ba-49f6-82b5-bc1c86b96e27
+ class: org.apache.nifi.minifi.processors.PerformanceDataMonitor
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 10000 ms
+ Properties:
+ Predefined Groups: "Disk,CPU"
+ Custom PDH Counters: "\\System\\Processes,\\Process(*)\\ID Process,\\Process(*)\\Private Bytes"
+ Output Format: "JSON"
+ Output Compactness: "Compact"
+- name: Put files to C:\temp\
+ id: e143601d-de4f-44ba-a6ec-d1f97d77ec94
+ class: org.apache.nifi.minifi.processors.PutFile
+ scheduling strategy: EVENT_DRIVEN
+ auto-terminated relationships list:
+ - failure
+ - success
+ Properties:
+ Conflict Resolution Strategy: fail
+ Create Missing Directories: 'true'
+ Directory: C:\temp\
+Connections:
+- name: ConsumeWindowsEventLog/success/PutFile
Review comment:
Fixed in 43b08aec7d021fff6b00a89176576d5ff5eefdb9
##########
File path: examples/publishkafka_config.yml
##########
@@ -0,0 +1,46 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+MiNiFi Config Version: 3
+Flow Controller:
+ name: MiNiFi Flow
+Processors:
+- name: Get files from /tmp/input
+ id: 7fd166aa-0662-4c42-affa-88f6fb39807f
+ class: org.apache.nifi.processors.standard.GetFile
+ scheduling period: 2 sec
+ scheduling strategy: TIMER_DRIVEN
+ Properties:
+ Input Directory: /tmp/input
+- name: Publish messages to Kafka topic test
+ id: 8a534b4a-2b4a-4e1e-ab07-8a09fa08f848
+ class: org.apache.nifi.processors.standard.PublishKafka
+ scheduling strategy: EVENT_DRIVEN
+ Properties:
+ Batch Size: '10'
+ Client Name: test-client
+ Compress Codec: none
+ Delivery Guarantee: '1'
+ Known Brokers: kafka-broker:9092
+ Message Timeout: 12 sec
+ Request Timeout: 10 sec
+ Topic Name: test
+Connections:
+- name: d61e509d-03a7-4c70-a7cf-6676d8047825
Review comment:
Fixed in 43b08aec7d021fff6b00a89176576d5ff5eefdb9
##########
File path: examples/publishkafka_ssl_config.yml
##########
@@ -0,0 +1,55 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+MiNiFi Config Version: 3
+Flow Controller:
+ name: MiNiFi Flow
+Processors:
+- name: Get files from /tmp/input
+ id: a78f9d7d-cc68-4f30-aac1-ee716d42bf9d
+ class: org.apache.nifi.processors.standard.GetFile
+ scheduling period: 2 sec
+ scheduling strategy: TIMER_DRIVEN
+ Properties:
+ Input Directory: /tmp/input
+- name: Publish messages to Kafka topic test
+ id: 8c0791c7-920c-4541-bcde-f5d559f42ed2
+ class: org.apache.nifi.processors.standard.PublishKafka
+ scheduling period: 1 sec
+ scheduling strategy: EVENT_DRIVEN
+ auto-terminated relationships list:
+ - success
+ Properties:
+ Batch Size: 10
+ Client Name: test_client
+ Compress Codec: none
+ Delivery Guarantee: 1
+ Known Brokers: kafka-broker:9093
+ Message Timeout: 12 sec
+ Message Timeout Phrase: 12 sec
+ Request Timeout: 10 sec
+ Security CA Key: /tmp/resources/certs/ca-cert
+ Security Cert: /tmp/resources/certs/client_test_client_client.pem
+ Security Pass Phrase: abcdefgh
+ Security Private Key: /tmp/resources/certs/client_test_client_client.key
+ Security Protocol: ssl
+ Topic Name: test
+Connections:
+- name: 54c70a6d-d9d9-4709-9747-1ab47edad3c5
Review comment:
Fixed in 43b08aec7d021fff6b00a89176576d5ff5eefdb9
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm closed pull request #1127: MINIFICPP-1601 Add example flow configurations
Posted by GitBox <gi...@apache.org>.
szaszm closed pull request #1127:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1127
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] martinzink commented on a change in pull request #1127: MINIFICPP-1601 Add example flow configurations
Posted by GitBox <gi...@apache.org>.
martinzink commented on a change in pull request #1127:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1127#discussion_r666402516
##########
File path: examples/querydbtable_config.yml
##########
@@ -0,0 +1,51 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+Flow Controller:
Review comment:
I am not even sure what it does but in the previous examples the files started with
` MiNiFi Config Version: 3`
We could be consistent with that and include it everywhere, or nowhere?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1127: MINIFICPP-1601 Add example flow configurations
Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #1127:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1127#discussion_r667748881
##########
File path: examples/http_post_config.yml
##########
@@ -0,0 +1,54 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+MiNiFi Config Version: 3
+Flow Controller:
+ name: MiNiFi Flow
+Processors:
+- name: Get files from /tmp/input
+ id: 962790e7-ea35-4096-9362-96f527288669
+ class: org.apache.nifi.processors.standard.GetFile
+ scheduling period: 2 sec
+ scheduling strategy: TIMER_DRIVEN
+ Properties:
+ Input Directory: /tmp/input
+ Keep Source File: 'true'
+- name: Invoke POST request on remote URL
+ id: 33b373b6-6f19-4194-b45a-1ef73c357a8e
+ class: org.apache.nifi.processors.standard.InvokeHTTP
+ scheduling strategy: EVENT_DRIVEN
+ auto-terminated relationships list:
+ - success
+ - response
+ - no retry
+ Properties:
+ HTTP Method: POST
+ Remote URL: http://1.2.3.4:8080/contentListener
Review comment:
```suggestion
- name: Invoke POST request on remote URL
id: 33b373b6-6f19-4194-b45a-1ef73c357a8e
class: org.apache.nifi.processors.standard.InvokeHTTP
scheduling strategy: EVENT_DRIVEN
auto-terminated relationships list:
- success
- response
Properties:
HTTP Method: POST
Remote URL: http://1.2.3.4:8080/contentListener
- name: Retry on HTTP 5xx status codes
id: 3b1c6971-541b-4530-9f72-175757500699
class: org.apache.nifi.processors.standard.RetryFlowFile
scheduling strategy: EVENT_DRIVEN
auto-terminated relationships list:
- failure
Properties:
Maximum Retries: 5
Remote URL: http://1.2.3.4:8080/contentListener
- name: Log failed flow files
id: 1f669bb0-6d50-41e6-9230-90c1cc3e21f9
class: org.apache.nifi.minifi.processors.LogAttribute
scheduling strategy: EVENT_DRIVEN
auto-terminated relationships list: [success]
Properties:
FlowFiles To Log: 0
Log Level: warn
Log Prefix: '=== Failed HTTP request ==='
Maxumim Payload Line Length: 0
```
##########
File path: examples/http_post_config.yml
##########
@@ -0,0 +1,54 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+MiNiFi Config Version: 3
+Flow Controller:
+ name: MiNiFi Flow
+Processors:
+- name: Get files from /tmp/input
+ id: 962790e7-ea35-4096-9362-96f527288669
+ class: org.apache.nifi.processors.standard.GetFile
+ scheduling period: 2 sec
+ scheduling strategy: TIMER_DRIVEN
+ Properties:
+ Input Directory: /tmp/input
+ Keep Source File: 'true'
+- name: Invoke POST request on remote URL
+ id: 33b373b6-6f19-4194-b45a-1ef73c357a8e
+ class: org.apache.nifi.processors.standard.InvokeHTTP
+ scheduling strategy: EVENT_DRIVEN
+ auto-terminated relationships list:
+ - success
+ - response
+ - no retry
+ Properties:
+ HTTP Method: POST
+ Remote URL: http://1.2.3.4:8080/contentListener
+Connections:
+- name: GetFile/success/InvokeHTTP
+ source id: 962790e7-ea35-4096-9362-96f527288669
+ source relationship name: success
+ destination id: 33b373b6-6f19-4194-b45a-1ef73c357a8e
+- name: InvokeHTTP/failure/InvokeHTTP
+ source id: 33b373b6-6f19-4194-b45a-1ef73c357a8e
+ source relationship name: failure
+ destination id: 33b373b6-6f19-4194-b45a-1ef73c357a8e
+- name: InvokeHTTP/retry/InvokeHTTP
+ id: c8a96f1d-48cd-4ee3-8a2f-081c3e3c7bcd
+ source id: 33b373b6-6f19-4194-b45a-1ef73c357a8e
+ source relationship names: retry
+ destination id: 33b373b6-6f19-4194-b45a-1ef73c357a8e
Review comment:
```suggestion
- name: InvokeHTTP/retry/RetryFlowFile
id: c8a96f1d-48cd-4ee3-8a2f-081c3e3c7bcd
source id: 33b373b6-6f19-4194-b45a-1ef73c357a8e
source relationship names: retry
destination id: 3b1c6971-541b-4530-9f72-175757500699
- name: RetryFlowFile/retry/InvokeHTTP
id: cadf51f0-3468-423a-9d12-5c40270aa7ca
source id: 3b1c6971-541b-4530-9f72-175757500699
source relationship names: retry
destination id: 33b373b6-6f19-4194-b45a-1ef73c357a8e
- name: RetryFlowFile/retries_exceeded/LogAttribute
id: 160fc0f4-6efe-4274-8bc7-2a416570bdeb
source id: 3b1c6971-541b-4530-9f72-175757500699
source relationship names: retries_exceeded
destination id: 1f669bb0-6d50-41e6-9230-90c1cc3e21f9
- name: InvokeHTTP/no retry/LogAttribute
source id: 33b373b6-6f19-4194-b45a-1ef73c357a8e
source relationship name: no retry
destination id: 1f669bb0-6d50-41e6-9230-90c1cc3e21f9
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #1127: MINIFICPP-1601 Add example flow configurations
Posted by GitBox <gi...@apache.org>.
lordgamez commented on a change in pull request #1127:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1127#discussion_r667795165
##########
File path: examples/http_post_config.yml
##########
@@ -0,0 +1,50 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+MiNiFi Config Version: 3
+Flow Controller:
+ name: MiNiFi Flow
+Processors:
+- name: Get files from /tmp/input
+ id: 962790e7-ea35-4096-9362-96f527288669
+ class: org.apache.nifi.processors.standard.GetFile
+ scheduling period: 2 sec
+ scheduling strategy: TIMER_DRIVEN
+ Properties:
+ Input Directory: /tmp/input
+ Keep Source File: 'true'
+- name: Invoke POST request on remote URL
+ id: 33b373b6-6f19-4194-b45a-1ef73c357a8e
+ class: org.apache.nifi.processors.standard.InvokeHTTP
+ scheduling strategy: EVENT_DRIVEN
+ auto-terminated relationships list:
+ - success
+ - response
+ - retry
+ - no retry
+ Properties:
+ HTTP Method: POST
+ Remote URL: http://1.2.3.4:8080/contentListener
+Connections:
+- name: GetFile/success/InvokeHTTP
Review comment:
Thanks, I was only thinking about any retry functionality in the `InvokeHTTP` processor. I tried your suggestion and it works, I only removed the `Remote URL` property in the `RetryFlowFile` processor. Added in ae393afd2fa15f0433a073255b49b8b0f8aa66ec.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #1127: MINIFICPP-1601 Add example flow configurations
Posted by GitBox <gi...@apache.org>.
lordgamez commented on a change in pull request #1127:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1127#discussion_r666381189
##########
File path: examples/puts3_config.yml
##########
@@ -0,0 +1,72 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+MiNiFi Config Version: 3
+Flow Controller:
+ name: MiNiFi Flow
+Processors:
+- Properties:
+ Input Directory: /tmp/input
+ auto-terminated relationships list: []
+ class: org.apache.nifi.processors.standard.GetFile
+ id: bfd376a8-c845-453b-9f74-4cf6f3a95e74
+ name: GetFile
+ penalization period: 30 sec
+ run duration nanos: 0
+ scheduling period: 2 sec
+ scheduling strategy: TIMER_DRIVEN
+ yield period: 1 sec
+- Properties:
+ Access Key: test_access_key
+ Bucket: test_bucket
+ Secret Key: test_secret
+ auto-terminated relationships list:
+ - failure
+ class: org.apache.nifi.processors.standard.PutS3Object
+ id: 8c9e2512-e814-4b73-ac5c-f33753332128
+ name: PutS3Object
+ penalization period: 30 sec
+ run duration nanos: 0
+ scheduling period: 1 sec
+ scheduling strategy: TIMER_DRIVEN
+ yield period: 1 sec
+- Properties:
+ Directory: /tmp/output
+ Directory Permissions: '777'
+ Permissions: '777'
+ auto-terminated relationships list:
+ - success
+ - failure
+ class: org.apache.nifi.processors.standard.PutFile
Review comment:
Changed the files have a consistent layout in 369f1085d136a749731b067dad19c73cded52532. Also removed the optional properties that were just making noise.
I really liked your example, it's simple, on the point for anyone who wants to use the ConsumeJournald processors, so I added that to the list of examples in 5583561941397e5605c339cb7e15148ac411f286
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org