You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@nifi.apache.org by al...@apache.org on 2016/12/28 01:46:37 UTC
nifi-minifi-cpp git commit: MINIFI-174
Repository: nifi-minifi-cpp
Updated Branches:
refs/heads/master 628687280 -> ee4bb1353
MINIFI-174
Documentation for bidirectional communication configuration
This closes #33.
Signed-off-by: Aldrin Piri <al...@apache.org>
Project: http://git-wip-us.apache.org/repos/asf/nifi-minifi-cpp/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi-minifi-cpp/commit/ee4bb135
Tree: http://git-wip-us.apache.org/repos/asf/nifi-minifi-cpp/tree/ee4bb135
Diff: http://git-wip-us.apache.org/repos/asf/nifi-minifi-cpp/diff/ee4bb135
Branch: refs/heads/master
Commit: ee4bb13535783abda26677bc13652a1a54ff12a0
Parents: 6286872
Author: Jeremy Dyer <jd...@gmail.com>
Authored: Tue Dec 27 16:57:10 2016 -0500
Committer: Aldrin Piri <al...@apache.org>
Committed: Tue Dec 27 20:37:04 2016 -0500
----------------------------------------------------------------------
examples/BidirectionalSiteToSite/README.md | 108 ++++++++++++
.../assets/Apache_NiFi_Incoming_Data.png | Bin 0 -> 96992 bytes
.../assets/Apache_NiFi_Outgoing_Data.png | Bin 0 -> 97630 bytes
.../assets/Apache_NiFi_Workflow.png | Bin 0 -> 97556 bytes
.../assets/BidirectionalSite2SiteExample.xml | 169 +++++++++++++++++++
.../assets/BidirectionalSiteToSite.png | Bin 0 -> 81229 bytes
examples/README.md | 17 ++
7 files changed, 294 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/nifi-minifi-cpp/blob/ee4bb135/examples/BidirectionalSiteToSite/README.md
----------------------------------------------------------------------
diff --git a/examples/BidirectionalSiteToSite/README.md b/examples/BidirectionalSiteToSite/README.md
new file mode 100644
index 0000000..78c0c58
--- /dev/null
+++ b/examples/BidirectionalSiteToSite/README.md
@@ -0,0 +1,108 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+# Bidirectional Site-To-Site Configuration
+
+## Overview
+
+Site-To-Site is a cornerstone feature to the Apache NiFi platform. Currently configuring Site-To-Site with MiNiFi-CPP can be a little tricky however. There are plans in the roadmap to improve this configuration flow but given what we have today this example will demonstrate how you can configure Apache MiNiFi-C++ for bidirectional communication with Apache NiFi.
+
+Below is a very high level diagram illustrating our end goal in this example.
+
+
+
+## Apache NiFi Setup
+
+Before we start setting up our MiNiFi-CPP instance we need to first setup our Apache NiFi instance that the MiNiFi instance will be communicating with. If we recall from the deployment diagram above our example will have a simple environment consisting of a single NiFi instance. While we are using a small environment here the same principals would apply for larger deployments.
+
+First we must [download and install NiFi](https://nifi.apache.org/docs/nifi-docs/html/getting-started.html#downloading-and-installing-nifi) if you haven't done so already.
+
+As you can see from the screenshot below our contrived NiFi workflow example simply accepts data from MiNiFi, logs that data, and also writes that data back to MiNiFi.
+
+
+
+You can download this workflow from the screenshot [here](assets/BidirectionalSite2SiteExample.xml)
+
+After installing NiFi, uploading the workflow, and ensuring site-to-site is enabled there are a few environment specific values that you should take note of as you will need them when configuring MiNiFi.
+
+- ```{NIFI_HOST_OR_IP}``` - The hostname or IP of your NiFi and MiNiFi instances.
+ - ```nifi.dev``` and ```minifi.dev``` are the hostnames for this example instance but your instance will vary.
+- ```{NIFI_OUTPUT_PORT_ID}``` and ```{NIFI_INPUT_PORT_ID}``` - The ID of the Input and Output ports from your NiFi workflow.
+ - Even if you download and use the same workflow from [here](assets/BidirectionalSite2SiteExample.xml) these ID values are autogenerated and will be different in your environment.
+ - You can find these values by highlighting the Input or Output port in the NiFi workflow and copying the ID value from the ```Operate``` tile in the NiFi UI.
+- ```{NIFI_SITE_TO_SITE_PORT}``` - Ensure Apache NiFi site-to-site is enabled by following [this guide](https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#site_to_site_properties). Note the port you specify for ```nifi.remote.input.socket.port```
+
+## Apache MiNiFi-CPP Setup
+
+Setup for the MiNiFi instance really only involves properly configuring your flow yml file which by default is located at ```$MINIFI_HOME/conf/config.yml```. The below configuration can be used after replacing the ```{}``` values with those called out above in the NiFi configuration. After updating this config.yml it can be placed in your ```$MINIFI_HOME/conf/config.yml``` and then MiNiFi can be restarted. After restarting MiNiFi you should start seeing flowfiles accumulate in the output connection from the ```MiNiFiData``` input port in your NiFi UI when you place or touch a file in the directory ```/tmp/getfile/``` on the MiNiFi host.
+
+```
+Flow Controller:
+ name: MiNiFi Bidirectional SiteToSite Example
+
+Processors:
+ - name: GetFile
+ class: org.apache.nifi.processors.standard.GetFile
+ max concurrent tasks: 1
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 1 sec
+ penalization period: 30 sec
+ yield period: 1 sec
+ run duration nanos: 0
+ auto-terminated relationships list:
+ Properties:
+ Input Directory: /tmp/getfile
+ Keep Source File: true
+
+Connections:
+ - name: TransferFilesToRPG
+ source name: GetFile
+ source relationship name: success
+ destination name: {NIFI_INPUT_PORT_ID}
+ max work queue size: 0
+ max work queue data size: 1 MB
+ flowfile expiration: 60 sec
+
+Remote Processing Groups:
+ - name: NiFi Flow
+ url: http://{NIFI_HOST_OR_IP}:{NIFI_WEB_UI_PORT}/nifi
+ timeout: 30 secs
+ yield period: 10 sec
+ Input Ports:
+ - id: {NIFI_INPUT_PORT_ID}
+ name: MiNiFiData
+ max concurrent tasks: 1
+ Properties:
+ Port: {NIFI_SITE_TO_SITE_PORT}
+ Host Name: {NIFI_HOST_OR_IP}
+ Output Ports:
+ - id: {NIFI_OUTPUT_PORT_ID}
+ name: ToMiNiFi
+ max concurrent tasks: 1
+ Properties:
+ Port: {NIFI_SITE_TO_SITE_PORT}
+ Host Name: {NIFI_HOST_OR_IP}
+```
+
+## Conclusion
+
+After starting MiNiFi we can see that the connections going to both ```LogAttribute``` and ```ToMiNiFi``` both contain 24 flowfiles. Since both of the processors are stopped this makes perfect sense and validates that the data is flowing from MiNiFi to NiFi as expected.
+
+
+
+Now to validate that the flowfiles will flow back to MiNiFi we can start the ```ToMiNiFi``` output port and the connection queue should drain its flowfiles.
+
+
+
+As you can see from the screenshot after starting ```ToMiNiFi``` the connection successfully drained and the flowfiles were successfully transfered back to MiNiFi completing the bidirectional lifecycle.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/nifi-minifi-cpp/blob/ee4bb135/examples/BidirectionalSiteToSite/assets/Apache_NiFi_Incoming_Data.png
----------------------------------------------------------------------
diff --git a/examples/BidirectionalSiteToSite/assets/Apache_NiFi_Incoming_Data.png b/examples/BidirectionalSiteToSite/assets/Apache_NiFi_Incoming_Data.png
new file mode 100644
index 0000000..63cf3be
Binary files /dev/null and b/examples/BidirectionalSiteToSite/assets/Apache_NiFi_Incoming_Data.png differ
http://git-wip-us.apache.org/repos/asf/nifi-minifi-cpp/blob/ee4bb135/examples/BidirectionalSiteToSite/assets/Apache_NiFi_Outgoing_Data.png
----------------------------------------------------------------------
diff --git a/examples/BidirectionalSiteToSite/assets/Apache_NiFi_Outgoing_Data.png b/examples/BidirectionalSiteToSite/assets/Apache_NiFi_Outgoing_Data.png
new file mode 100644
index 0000000..c206e80
Binary files /dev/null and b/examples/BidirectionalSiteToSite/assets/Apache_NiFi_Outgoing_Data.png differ
http://git-wip-us.apache.org/repos/asf/nifi-minifi-cpp/blob/ee4bb135/examples/BidirectionalSiteToSite/assets/Apache_NiFi_Workflow.png
----------------------------------------------------------------------
diff --git a/examples/BidirectionalSiteToSite/assets/Apache_NiFi_Workflow.png b/examples/BidirectionalSiteToSite/assets/Apache_NiFi_Workflow.png
new file mode 100644
index 0000000..b467c35
Binary files /dev/null and b/examples/BidirectionalSiteToSite/assets/Apache_NiFi_Workflow.png differ
http://git-wip-us.apache.org/repos/asf/nifi-minifi-cpp/blob/ee4bb135/examples/BidirectionalSiteToSite/assets/BidirectionalSite2SiteExample.xml
----------------------------------------------------------------------
diff --git a/examples/BidirectionalSiteToSite/assets/BidirectionalSite2SiteExample.xml b/examples/BidirectionalSiteToSite/assets/BidirectionalSite2SiteExample.xml
new file mode 100644
index 0000000..186231f
--- /dev/null
+++ b/examples/BidirectionalSiteToSite/assets/BidirectionalSite2SiteExample.xml
@@ -0,0 +1,169 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<?xml version="1.0" ?>
+<template encoding-version="1.0">
+ <description></description>
+ <groupId>cfed8382-0158-1000-1395-192d30f8b560</groupId>
+ <name>BidirectionSite2SiteExample</name>
+ <snippet>
+ <connections>
+ <id>08757b0a-0159-1000-0000-000000000000</id>
+ <parentGroupId>cfed8382-0158-1000-0000-000000000000</parentGroupId>
+ <backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>
+ <backPressureObjectThreshold>10000</backPressureObjectThreshold>
+ <destination>
+ <groupId>cfed8382-0158-1000-0000-000000000000</groupId>
+ <id>08756e38-0159-1000-0000-000000000000</id>
+ <type>PROCESSOR</type>
+ </destination>
+ <flowFileExpiration>0 sec</flowFileExpiration>
+ <labelIndex>1</labelIndex>
+ <name></name>
+ <source>
+ <groupId>cfed8382-0158-1000-0000-000000000000</groupId>
+ <id>08755c4f-0159-1000-0000-000000000000</id>
+ <type>INPUT_PORT</type>
+ </source>
+ <zIndex>0</zIndex>
+ </connections>
+ <connections>
+ <id>4152969d-0159-1000-0000-000000000000</id>
+ <parentGroupId>cfed8382-0158-1000-0000-000000000000</parentGroupId>
+ <backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>
+ <backPressureObjectThreshold>10000</backPressureObjectThreshold>
+ <destination>
+ <groupId>cfed8382-0158-1000-0000-000000000000</groupId>
+ <id>37052b39-0159-1000-0000-000000000000</id>
+ <type>OUTPUT_PORT</type>
+ </destination>
+ <flowFileExpiration>0 sec</flowFileExpiration>
+ <labelIndex>1</labelIndex>
+ <name></name>
+ <source>
+ <groupId>cfed8382-0158-1000-0000-000000000000</groupId>
+ <id>08755c4f-0159-1000-0000-000000000000</id>
+ <type>INPUT_PORT</type>
+ </source>
+ <zIndex>0</zIndex>
+ </connections>
+ <inputPorts>
+ <id>08755c4f-0159-1000-0000-000000000000</id>
+ <parentGroupId>cfed8382-0158-1000-0000-000000000000</parentGroupId>
+ <position>
+ <x>0.0</x>
+ <y>6.696388244628906</y>
+ </position>
+ <comments></comments>
+ <concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount>
+ <name>MiNiFiData</name>
+ <state>RUNNING</state>
+ <transmitting>false</transmitting>
+ <type>INPUT_PORT</type>
+ </inputPorts>
+ <outputPorts>
+ <id>37052b39-0159-1000-0000-000000000000</id>
+ <parentGroupId>cfed8382-0158-1000-0000-000000000000</parentGroupId>
+ <position>
+ <x>478.088623046875</x>
+ <y>0.0</y>
+ </position>
+ <comments></comments>
+ <concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount>
+ <name>ToMiNiFi</name>
+ <state>RUNNING</state>
+ <transmitting>false</transmitting>
+ <type>OUTPUT_PORT</type>
+ </outputPorts>
+ <processors>
+ <id>08756e38-0159-1000-0000-000000000000</id>
+ <parentGroupId>cfed8382-0158-1000-0000-000000000000</parentGroupId>
+ <position>
+ <x>179.394287109375</x>
+ <y>151.79802703857422</y>
+ </position>
+ <config>
+ <bulletinLevel>WARN</bulletinLevel>
+ <comments></comments>
+ <concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount>
+ <descriptors>
+ <entry>
+ <key>Log Level</key>
+ <value>
+ <name>Log Level</name>
+ </value>
+ </entry>
+ <entry>
+ <key>Log Payload</key>
+ <value>
+ <name>Log Payload</name>
+ </value>
+ </entry>
+ <entry>
+ <key>Attributes to Log</key>
+ <value>
+ <name>Attributes to Log</name>
+ </value>
+ </entry>
+ <entry>
+ <key>Attributes to Ignore</key>
+ <value>
+ <name>Attributes to Ignore</name>
+ </value>
+ </entry>
+ <entry>
+ <key>Log prefix</key>
+ <value>
+ <name>Log prefix</name>
+ </value>
+ </entry>
+ </descriptors>
+ <executionNode>ALL</executionNode>
+ <lossTolerant>false</lossTolerant>
+ <penaltyDuration>30 sec</penaltyDuration>
+ <properties>
+ <entry>
+ <key>Log Level</key>
+ <value>info</value>
+ </entry>
+ <entry>
+ <key>Log Payload</key>
+ <value>true</value>
+ </entry>
+ <entry>
+ <key>Attributes to Log</key>
+ </entry>
+ <entry>
+ <key>Attributes to Ignore</key>
+ </entry>
+ <entry>
+ <key>Log prefix</key>
+ </entry>
+ </properties>
+ <runDurationMillis>0</runDurationMillis>
+ <schedulingPeriod>0 sec</schedulingPeriod>
+ <schedulingStrategy>TIMER_DRIVEN</schedulingStrategy>
+ <yieldDuration>1 sec</yieldDuration>
+ </config>
+ <name>Log Incoming Data</name>
+ <relationships>
+ <autoTerminate>true</autoTerminate>
+ <name>success</name>
+ </relationships>
+ <style></style>
+ <type>org.apache.nifi.processors.standard.LogAttribute</type>
+ </processors>
+ </snippet>
+ <timestamp>12/27/2016 17:29:28 UTC</timestamp>
+</template>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/nifi-minifi-cpp/blob/ee4bb135/examples/BidirectionalSiteToSite/assets/BidirectionalSiteToSite.png
----------------------------------------------------------------------
diff --git a/examples/BidirectionalSiteToSite/assets/BidirectionalSiteToSite.png b/examples/BidirectionalSiteToSite/assets/BidirectionalSiteToSite.png
new file mode 100644
index 0000000..51bf1f4
Binary files /dev/null and b/examples/BidirectionalSiteToSite/assets/BidirectionalSiteToSite.png differ
http://git-wip-us.apache.org/repos/asf/nifi-minifi-cpp/blob/ee4bb135/examples/README.md
----------------------------------------------------------------------
diff --git a/examples/README.md b/examples/README.md
new file mode 100644
index 0000000..f12935a
--- /dev/null
+++ b/examples/README.md
@@ -0,0 +1,17 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+# Apache NiFi - MiNiFi - C++ Examples
+
+- [Site-2-Site Bi-directional Configuration](BidirectionalSiteToSite/README.md)
\ No newline at end of file