You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@nifi.apache.org by randerzander <gi...@git.apache.org> on 2017/04/01 19:34:23 UTC

[GitHub] nifi-site pull request #17: Added MiNiFi getting-started page, formatting tw...

GitHub user randerzander opened a pull request:

    https://github.com/apache/nifi-site/pull/17

    Added MiNiFi getting-started page, formatting tweaks to minifi index.html

    

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/randerzander/nifi-site master

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/nifi-site/pull/17.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #17
    
----
commit 6d72649b508af800e0b753fab2057f8f241ef87f
Author: Randy Gelhausen <rg...@gmail.com>
Date:   2017-04-01T19:29:42Z

    Added MiNiFi getting-started page, formatting tweaks to the minifi index.html page

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-site pull request #17: Added MiNiFi getting-started page, formatting tw...

Posted by randerzander <gi...@git.apache.org>.

Github user randerzander commented on a diff in the pull request:

https://github.com/apache/nifi-site/pull/17#discussion_r109748735

--- Diff: src/pages/markdown/minifi/getting-started.md ---
@@ -0,0 +1,120 @@
+---
+title: Apache NiFi MiNiFi: Getting Started
+---
+
+# Getting started with MiNiFi
+
+This page explains how to configure and deploy MiNiFi agents.
+
+The Java agent is able to run most of [NiFi's available processors](http://nifi.apache.org/docs.html), but is a larger binary distribution (49MB) and consumes greater system resources (24MB max JVM heapsize by default). If you need maximum flexibility to make routing and processing decisions at your data's point of origin, the Java agent is a good fit.
+
+The C++ agent is a smaller binary (3.2MB), consumes low system memory (about 5MB at idle) but has [a limited subset of processors](https://github.com/apache/nifi-minifi-cpp#caveats). If your primary concern is gathering and pushing data to downstream consumers and minimizing system impact, the C++ agent is a good fit.
+
+
+1. Install the appropriate OS level dependencies:
+
+ #### MiNiFi Java:
+
+ - Java 1.8+
+
+ #### MiNiFi C++:
+ ###### RHEL/CentOS:
+ - yum install -y epel-release
+ - yum install -y leveldb
+
+ ###### Debian/Ubuntu:
+ - apt install -y libleveldb-dev
+ - apt install -y libxml2
+2. Download the relevant compressed binary from the [Downloads](download.html) page
+3. Copy and decompress the binary to your target deployment environment
+4. Set the MINIFI_HOME environment variable to your decompressed binary directory
+5. From $MINIFI_HOME, ./bin/minifi.sh {start|stop|run|restart|status|flowStatus|dump|install}
+
+*flowStatus* and *dump* are available only for MiNiFi Java.
+
+For MiNiFi Java, the agent logs to $MINIFI_HOME/logs/minifi-app.log.
+
+For MiNiFi C++, the agent logs to $MINIFI_HOME/minifi-app.log.txt
+
+# Configuring MiNiFi Agent Flows
+
+Agents execute the flow defined in $MINIFI_HOME/conf/config.yml. The shipped config.yml file has an "empty" flow.
+
+While it's possible to create and modify config.yml programmatically or by hand, the more common scenario is to:
+1. Use NiFi's web interface to define a flow
+2. Save the flow as a template and download its XML file
+3. Use the [Converter Toolkit](minifi-toolkit.html) to convert the template XML to an equivalent config.yml
+4. Put the config.yml file in $MINIFI_HOME/conf and start the agent.
+
+# Pushing Data From MiNiFi to NiFi
+
+What follows is an illustrated walkthrough of configuring MiNiFi and NiFi to generate and collect log data.
+
+If you're new to NiFi, watch the [NiFi introductory video series](http://nifi.apache.org/videos.html) before proceeding. If you don't already have NiFi, you can download it from [here](http://nifi.apache.org/download.html).
+
+## Configuring NiFi to Receive Data
+
+First, check that $NIFI_HOME/conf/nifi.properties has a Site to Site input socket port specified:
+```
+# Site to Site properties
+nifi.remote.input.host=
+nifi.remote.input.secure=false
+nifi.remote.input.socket.port=1026
+nifi.remote.input.http.enabled=true
+nifi.remote.input.http.transaction.ttl=30 sec
+```
+
+Restart NiFi so changes take effect.
+
+On NiFi's web UI, place an Input Port "From MiNiFi" and connect it to a downstream processor:
+![](../assets/images/minifi/nifi-receive.png)
+
+## Creating the MiNiFi Flow
+
+Create a new process group, and double click on it to enter it. Working inside this new processor group will let us create a clean template that includes flow components meant to run in MiNiFi.
+
+Inside the process group, create a Remote Process Group (RPG) and give it the URL of your NiFi instance. The RPG will connect to NiFi and discover the input port created earlier.
+
+Create a TailFile processor and connect it to the RPG, selecting the "From MiNiFi" input port.
+![](../assets/images/minifi/minifi-flow.png)
+
+Now create a template, download it as an XML file, and download the [Converter ToolKit](download.html).
+
+From the toolkit directory:
+```
+[root@host minifi-toolkit-0.0.1]# ./bin/config.sh transform /path/to/template.xml /output/path/config.yml
+
+Java home: /usr/lib/jvm/jre-1.8.0-openjdk
+MiNiFi Toolkit home: /minifi-toolkit-0.0.1
+
+
+No validation errors found in template.
--- End diff --

Not sure how to address this one. Are there common validation errors one might expect?

---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-site pull request #17: MiNiFi-260: Create a "Getting Started" Guide for...

Posted by asfgit <gi...@git.apache.org>.

Github user asfgit closed the pull request at:

    https://github.com/apache/nifi-site/pull/17


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-site pull request #17: MiNiFi-260: Create a "Getting Started" Guide for...

Posted by apiri <gi...@git.apache.org>.

Github user apiri commented on a diff in the pull request:

    https://github.com/apache/nifi-site/pull/17#discussion_r110799319
  
    --- Diff: src/includes/minifi/minifi-topbar.hbs ---
    @@ -26,16 +26,17 @@
                     <li class="has-dropdown">
                         <a href="#">Documentation</a>
                         <ul class="dropdown">
    -                        <li><a href="faq.html"><i class="fa"></i>FAQ</a></li>
    +                        <li><a href="getting-started.html"><i class="fa"></i>Getting Started</a></li>
                             <li><a href="system-admin-guide.html"><i class="fa"></i>Administrator's Guide</a></li>
                             <li><a href="minifi-toolkit.html"><i class="fa"></i>Converter Toolkit</a></li>
    +                        <li><a href="faq.html"><i class="fa"></i>FAQ</a></li>
                             <li><a href="https://cwiki.apache.org/confluence/display/MiNiFi"><i class="fa fa-external-link external-link"></i>Wiki</a></li>
                         </ul>
                     </li>
                     <li class="has-dropdown">
    -                    <a href="#">Download</a>
    --- End diff --
    
    Great, thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-site pull request #17: Added MiNiFi getting-started page, formatting tw...

Posted by phrocker <gi...@git.apache.org>.

Github user phrocker commented on a diff in the pull request:

    https://github.com/apache/nifi-site/pull/17#discussion_r109715147
  
    --- Diff: src/pages/html/minifi/index.hbs ---
    @@ -34,10 +34,11 @@ title: Apache NiFi - MiNiFi
             <p class="description">
                 Specific goals for the initial thrust of the MiNiFi effort comprise:
                 <ul>
    -                <li>small and lightweight footprint,</li>
    -                <li>central management of agents,</li>
    -                <li>generation of data provenance, and</li>
    -                <li>integration with NiFi for follow-on dataflow management and full chain of custody of information.</li>
    +                <li>Small and lightweight footprint</li>
    --- End diff --
    
    Can we have a Small and Heavyweight footprint? Is that possible?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-site pull request #17: Added MiNiFi getting-started page, formatting tw...

Posted by phrocker <gi...@git.apache.org>.

Github user phrocker commented on a diff in the pull request:

    https://github.com/apache/nifi-site/pull/17#discussion_r109715336
  
    --- Diff: src/pages/html/minifi/index.hbs ---
    @@ -34,10 +34,11 @@ title: Apache NiFi - MiNiFi
             <p class="description">
                 Specific goals for the initial thrust of the MiNiFi effort comprise:
                 <ul>
    -                <li>small and lightweight footprint,</li>
    -                <li>central management of agents,</li>
    -                <li>generation of data provenance, and</li>
    -                <li>integration with NiFi for follow-on dataflow management and full chain of custody of information.</li>
    +                <li>Small and lightweight footprint</li>
    +                <li>Central management of agents</li>
    +                <li>Generation of data provenance</li>
    +                <li>Full chain of custody of information</li>
    --- End diff --
    
    Provenance implies CoC.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-site pull request #17: MiNiFi-260: Create a "Getting Started" Guide for...

Posted by randerzander <gi...@git.apache.org>.

Github user randerzander commented on a diff in the pull request:

    https://github.com/apache/nifi-site/pull/17#discussion_r110705173
  
    --- Diff: src/includes/minifi/minifi-topbar.hbs ---
    @@ -26,16 +26,17 @@
                     <li class="has-dropdown">
                         <a href="#">Documentation</a>
                         <ul class="dropdown">
    -                        <li><a href="faq.html"><i class="fa"></i>FAQ</a></li>
    +                        <li><a href="getting-started.html"><i class="fa"></i>Getting Started</a></li>
                             <li><a href="system-admin-guide.html"><i class="fa"></i>Administrator's Guide</a></li>
                             <li><a href="minifi-toolkit.html"><i class="fa"></i>Converter Toolkit</a></li>
    +                        <li><a href="faq.html"><i class="fa"></i>FAQ</a></li>
                             <li><a href="https://cwiki.apache.org/confluence/display/MiNiFi"><i class="fa fa-external-link external-link"></i>Wiki</a></li>
                         </ul>
                     </li>
                     <li class="has-dropdown">
    -                    <a href="#">Download</a>
    --- End diff --
    
    I changed to Download_s_ for MiNiFi since there are 3 distinct binaries for different purposes. 
    
    For NiFi you typically download only the latest binary release, but there are several different links. I will change NiFi's topbar to "Downloads" as well.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-site pull request #17: MiNiFi-260: Create a "Getting Started" Guide for...

Posted by apiri <gi...@git.apache.org>.

Github user apiri commented on a diff in the pull request:

https://github.com/apache/nifi-site/pull/17#discussion_r111404358

--- Diff: src/pages/markdown/minifi/getting-started.md ---
@@ -0,0 +1,122 @@
+---
+title: Apache NiFi MiNiFi: Getting Started
+---
+
+# Getting started with MiNiFi
+
+This page explains how to configure and deploy MiNiFi agents.
+
+The Java agent is able to run most of [NiFi's available processors](http://nifi.apache.org/docs.html), but is a larger binary distribution (49MB) and consumes greater system resources (24MB max JVM heapsize by default). If you need maximum flexibility to make routing and processing decisions at your data's point of origin, the Java agent is a good fit.
+
+The C++ agent is a smaller binary (3.2MB), consumes low system memory (about 5MB at idle) but has [a limited subset of processors](https://github.com/apache/nifi-minifi-cpp#caveats). If your primary concern is gathering and pushing data to downstream consumers and minimizing system impact, the C++ agent is a good fit.
+
+
+1. Install the appropriate OS level dependencies:
+
+ #### MiNiFi Java:
+
+ - Java 1.8+
+
+ #### MiNiFi C++:
--- End diff --

@randerzander totally fair point. certainly messed up mapping items between the current release and master. From that standpoint, I am on board. We (as in the community at large) should think about how to make this process a bit smoother in terms of releases and the site, but that is a concern for another issue and effort.

[GitHub] nifi-site pull request #17: Added MiNiFi getting-started page, formatting tw...

Posted by phrocker <gi...@git.apache.org>.

Github user phrocker commented on a diff in the pull request:

https://github.com/apache/nifi-site/pull/17#discussion_r109721955

Any troubleshooting help if there are validation errors?

[GitHub] nifi-site pull request #17: MiNiFi-260: Create a "Getting Started" Guide for...

Posted by apiri <gi...@git.apache.org>.

Github user apiri commented on a diff in the pull request:

https://github.com/apache/nifi-site/pull/17#discussion_r110800166

I did indeed conflate those, apologies. We do have some considerations in the current build in terms of dynamically linked items though, such as the libboost items. I do not believe the current build makes these static.

libxml2 should also no longer be needed. I noticed this was a remnant in our Travis build and created MINIFI-264 to clean that up.

[GitHub] nifi-site pull request #17: MiNiFi-260: Create a "Getting Started" Guide for...

Posted by apiri <gi...@git.apache.org>.

Github user apiri commented on a diff in the pull request:

    https://github.com/apache/nifi-site/pull/17#discussion_r110674033
  
    --- Diff: src/includes/minifi/minifi-topbar.hbs ---
    @@ -26,16 +26,17 @@
                     <li class="has-dropdown">
                         <a href="#">Documentation</a>
                         <ul class="dropdown">
    -                        <li><a href="faq.html"><i class="fa"></i>FAQ</a></li>
    +                        <li><a href="getting-started.html"><i class="fa"></i>Getting Started</a></li>
                             <li><a href="system-admin-guide.html"><i class="fa"></i>Administrator's Guide</a></li>
                             <li><a href="minifi-toolkit.html"><i class="fa"></i>Converter Toolkit</a></li>
    +                        <li><a href="faq.html"><i class="fa"></i>FAQ</a></li>
                             <li><a href="https://cwiki.apache.org/confluence/display/MiNiFi"><i class="fa fa-external-link external-link"></i>Wiki</a></li>
                         </ul>
                     </li>
                     <li class="has-dropdown">
    -                    <a href="#">Download</a>
    --- End diff --
    
    Pretty minor but would like some consistency between NiFi's nav and MiNiFi's.  Could we either roll these back or update NIFi's to be similar?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-site pull request #17: MiNiFi-260: Create a "Getting Started" Guide for...

Posted by apiri <gi...@git.apache.org>.

Github user apiri commented on a diff in the pull request:

https://github.com/apache/nifi-site/pull/17#discussion_r110673346

We likely have some room for improvement on this front (from a tooling perspective, less so this doc patch).

[GitHub] nifi-site pull request #17: Added MiNiFi getting-started page, formatting tw...

Posted by randerzander <gi...@git.apache.org>.

Github user randerzander closed the pull request at:

    https://github.com/apache/nifi-site/pull/17


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-site pull request #17: Added MiNiFi getting-started page, formatting tw...

Posted by phrocker <gi...@git.apache.org>.

Github user phrocker commented on a diff in the pull request:

https://github.com/apache/nifi-site/pull/17#discussion_r109721860

Is root needed or was that just an artifact of your environment?

[GitHub] nifi-site pull request #17: Added MiNiFi getting-started page, formatting tw...

Posted by randerzander <gi...@git.apache.org>.

GitHub user randerzander reopened a pull request:

    https://github.com/apache/nifi-site/pull/17

    Added MiNiFi getting-started page, formatting tweaks to minifi index.html

    

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/randerzander/nifi-site master

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/nifi-site/pull/17.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #17
    
----
commit 5c012249c0dac5386420ab360988776f714d2304
Author: Randy Gelhausen <rg...@gmail.com>
Date:   2017-04-01T19:29:42Z

    Added MiNiFi getting-started page, formatting tweaks to the minifi index.html page

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-site pull request #17: MiNiFi-260: Create a "Getting Started" Guide for...

Posted by apiri <gi...@git.apache.org>.

Github user apiri commented on a diff in the pull request:

    https://github.com/apache/nifi-site/pull/17#discussion_r110674818
  
    --- Diff: src/pages/markdown/minifi/getting-started.md ---
    @@ -0,0 +1,122 @@
    +---
    +title:     Apache NiFi MiNiFi: Getting Started
    +---
    +
    +# Getting started with MiNiFi
    +
    +This page explains how to configure and deploy MiNiFi agents.
    +
    +The Java agent is able to run most of [NiFi's available processors](http://nifi.apache.org/docs.html), but is a larger binary distribution (49MB) and consumes greater system resources (24MB max JVM heapsize by default). If you need maximum flexibility to make routing and processing decisions at your data's point of origin, the Java agent is a good fit.
    +
    +The C++ agent is a smaller binary (3.2MB), consumes low system memory (about 5MB at idle) but has [a limited subset of processors](https://github.com/apache/nifi-minifi-cpp#caveats). If your primary concern is gathering and pushing data to downstream consumers and minimizing system impact, the C++ agent is a good fit.
    +
    +
    +1. Install the appropriate OS level dependencies:
    +
    +  #### MiNiFi Java:
    +
    +  - Java 1.8+
    +
    +  #### MiNiFi C++:
    --- End diff --
    
    Should also note C++ 11 needed 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-site pull request #17: MiNiFi-260: Create a "Getting Started" Guide for...

Posted by randerzander <gi...@git.apache.org>.

Github user randerzander commented on a diff in the pull request:

https://github.com/apache/nifi-site/pull/17#discussion_r110802366

Two items of note here, both in the context of the as-is, currently released minifi-cpp binary:

1. I tested in Docker "FROM centos", which is CentOS 7.3.1611, and only installed the two listed yum packages, after which the binary runs successfully. Perhaps the boost libraries were installed as deps of the leveldb package?

2. I tested the binary in Docker "FROM ubuntu" which is Ubuntu 16.04.2. While libxml2 may not be used as a dependency, attempting to run the binary without having it installed yielded errors about missing that library. That said, the binary still segfaults, albeit not with errors about libxml2

I was under the impression that docs are intended to reflect how released artifacts work. If I'm mistaken, I'll update the PR to show how things work from a recent master branch build. Which would you suggest?

[GitHub] nifi-site pull request #17: MiNiFi-260: Create a "Getting Started" Guide for...

Posted by randerzander <gi...@git.apache.org>.

Github user randerzander commented on a diff in the pull request:

https://github.com/apache/nifi-site/pull/17#discussion_r110702518

It's only needed if you plan to do a build from source, not if you're just running the released binary.