You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by andrewmlim <gi...@git.apache.org> on 2018/11/01 18:37:57 UTC
[GitHub] nifi pull request #3124: NIFI-5767 Added NiFi Toolkit Guide to docs
GitHub user andrewmlim opened a pull request:
https://github.com/apache/nifi/pull/3124
NIFI-5767 Added NiFi Toolkit Guide to docs
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/andrewmlim/nifi NIFI-5767
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/3124.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3124
----
commit 060a4c5ccaf1b30f01dc3374edb2b69bb0f8833a
Author: Andrew Lim <an...@...>
Date: 2018-11-01T18:36:38Z
NIFI-5767 Added NiFi Toolkit Guide to docs
----
---
[GitHub] nifi issue #3124: NIFI-5767 Added NiFi Toolkit Guide to docs
Posted by andrewmlim <gi...@git.apache.org>.
Github user andrewmlim commented on the issue:
https://github.com/apache/nifi/pull/3124
@pvillard31 I think with my latest changes, ready to merge unless you see any other issues. Thanks!
---
[GitHub] nifi pull request #3124: NIFI-5767 Added NiFi Toolkit Guide to docs
Posted by pvillard31 <gi...@git.apache.org>.
Github user pvillard31 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3124#discussion_r230325510
--- Diff: nifi-docs/src/main/asciidoc/toolkit-guide.adoc ---
@@ -0,0 +1,1257 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+= Apache NiFi Toolkit Guide
+Apache NiFi Team <de...@nifi.apache.org>
+:homepage: http://nifi.apache.org
+:linkattrs:
+
+== Overview
+The NiFi Toolkit contains several command line utilities to setup and support NiFi in standalone and clustered environments. The utilities include:
+
+* CLI -- The `cli` tool enables administrators to interact with NiFi and NiFi Registry instances to automate tasks such as deploying versioned flows and managing process groups and cluster nodes.
+* Encrypt Config -- The `encrypt-config` tool encrypts the sensitive keys in the _nifi.properties_ file to facilitate the setup of a secure NiFi instance.
+* File Manager -- The `file-manager` tool enables administrators to backup, install or restore a NiFi installation from backup.
+* Flow Analyzer -- The `flow-analyzer` tool produces a report that helps administrators understand the max amount of data which can be stored in backpressure for a given flow.
+* Node Manager -- The `node-manager` tool enables administrators to perform status checks on nodes as well as the ability to connect, disconnect, or remove nodes from the cluster.
+* Notify -- The `notify` tool enables administrators to send bulletins to the NiFi UI.
+* S2S -- The `s2s` tool enables administrators to send data into or out of NiFi flows over site-to-site.
+* TLS Toolkit -- The `tls-toolkit` utility generates the required keystores, truststore, and relevant configuration files to facilitate the setup of a secure NiFi instance.
+* ZooKeeper Migrator -- The `zk-migrator` tool enables administrators to:
+** move ZooKeeper information from one ZooKeeper cluster to another
+** migrate ZooKeeper node ownership
+
+The utilities are executed with scripts found in the `bin` folder of your NiFi Toolkit installation.
+
+NOTE: The NiFi Toolkit is downloaded separately from NiFi (see the link:https://nifi.apache.org/download.html[Apache NiFi downloads page^]).
+
+=== Prerequisites for Running in a Secure Environment
+For secured nodes and clusters, two policies should be configured in advance:
+
+* Access the controller – A user that will have access to these utilities should be authorized in NiFi by creating an “access the controller” policy (`/controller`) with both view and modify rights
+* Proxy user request – If not previously set, node’s identity (the DN value of the node’s certificate) should be authorized to proxy requests on behalf of a user
+
+When executing either the Notify or Node Manager tools in a secured environment the `proxyDN` flag option should be used in order to properly identify the user that was authorized to execute these commands. In non-secure environments, or if running the status operation on the Node Manager tool, the flag is ignored.
+
+== NiFi CLI
+This tool offers a CLI focused on interacting with NiFi and NiFi Registry in order to automate tasks, such as deploying flows from a NIFi Registy to a NiFi instance or managing process groups and cluster nodes.
+
+=== Usage
+The CLI toolkit can be executed in standalone mode to execute a single command, or interactive mode to enter an interactive shell.
+
+To execute a single command:
+
+ ./bin/cli.sh <command> <args>
+
+To launch the interactive shell:
+
+ ./bin/cli.sh
+
+To show help:
+
+ cli.sh -h
+
+The following are available options:
+
+ demo quick-import
+ nifi current-user
+ nifi cluster-summary
+ nifi connect-node
+ nifi delete-node
+ nifi disconnect-node
+ nifi get-root-id
+ nifi get-node
+ nifi get-nodes
+ nifi offload-node
+ nifi list-reg-clients
+ nifi create-reg-client
+ nifi update-reg-client
+ nifi get-reg-client-id
+ nifi pg-import
+ nifi pg-start
+ nifi pg-stop
+ nifi pg-get-vars
+ nifi pg-set-var
+ nifi pg-get-version
+ nifi pg-change-version
+ nifi pg-get-all-versions
+ nifi pg-list
+ nifi pg-status
+ nifi pg-get-services
+ nifi pg-enable-services
+ nifi pg-disable-services
+ registry current-user
+ registry list-buckets
+ registry create-bucket
+ registry delete-bucket
+ registry list-flows
+ registry create-flow
+ registry delete-flow
+ registry list-flow-versions
+ registry export-flow-version
+ registry import-flow-version
+ registry sync-flow-versions
+ registry transfer-flow-version
+ session keys
+ session show
+ session get
+ session set
+ session remove
+ session clear
+ exit
+ help
+
+=== Property/Argument Handling
+Most commands will require specifying a baseUrl for the NiFi or NiFi Registry instance.
+
+An example command to list the buckets in a NiFi Registry instance would be the following:
+
+ ./bin/cli.sh registry list-buckets -u http://localhost:18080
+
+In order to avoid specifying the URL (and possibly other optional arguments for TLS) on every command, you can define a properties file containing the repetitive arguments.
+
+An example properties file for a local NiFi Registry instance would look like the following:
+
+```
+ baseUrl=http://localhost:18080
+ keystore=
+ keystoreType=
+ keystorePasswd=
+ keyPasswd=
+ truststore=
+ truststoreType=
+ truststorePasswd=
+ proxiedEntity=
+```
+
+This properties file can then be used on a command by specifying `-p`:
+
+ ./bin/cli.sh registry list-buckets -p /path/to/local-nifi-registry.properties
+
+You could then maintain a properties file for each environment you plan to interact with, such as Dev, QA, and Prod.
+
+In addition to specifying a properties file on each command, you can setup a default properties file to be used in the event that no properties file is specified.
+
+The default properties file is specified using the `session` concept, which persists to the users home directory in a file called _.nifi-cli.config_.
+
+An example of setting the default property files for NiFi would be the following:
+
+ ./bin/cli.sh session set nifi.props /path/to/local-nifi.properties
+
+An example for NiFi Registry would be the following:
+
+ ./bin/cli.sh session set nifi.reg.props /path/to/local-nifi-registry.properties
+
+This will write the above properties into the _.nifi-cli.config_ in the user's home directory and will allow commands to be executed without specifying a URL or properties file:
+
+ ./bin/cli.sh registry list-buckets
+
+The above command will now use the `baseUrl` from _local-nifi-registry.properties_.
+
+The order of resolving an argument is the following:
+
+* A direct argument overrides anything in a properties file or session
+* A properties file argument (`-p`) overrides the session
+* The session is used when nothing else is specified
+
+=== Security Configuration
+If NiFi and NiFi Registry are secured, then commands executed from the CLI will need to make a TLS connection and authenticate as a user with permissions to perform the desired action.
+
+Currently the CLI supports authenticating with a client certificate and an optional proxied-entity. A common scenario would be running the CLI from one of the nodes where NiFi or NiFi Registry is installed, which allows the CLI to use the same keystore and truststore as the NiFi/NiFi Registry instance.
+
+The security configuration can be specified per-command, or in one of the properties files described in the previous section.
+
+The examples below are for NiFi Registry, but the same concept applies for NiFi commands.
+
+==== Example - Secure NiFi Registry without Proxied-Entity
+Assuming we have a keystore containing the certificate for "CN=user1, OU=NIFI", an example properties file would be the following:
+
+```
+ baseUrl=https://localhost:18443
+ keystore=/path/to/keystore.jks
+ keystoreType=JKS
+ keystorePasswd=changeme
+ keyPasswd=changeme
+ truststore=/path/to/truststore.jks
+ truststoreType=JKS
+ truststorePasswd=changeme
+```
+
+In this example, commands will be executed as "CN=user1, OU=NIFI". This user would need to be a user in NiFi Registry, and commands accessing buckets would be restricted to buckets this user has access to.
+
+==== Example - Secure NiFi Registry with Proxied-Entity
+Assuming we have access to the keystore of NiFi Registry itself, and that NiFi Registry is also configured to allow Kerberos or LDAP authentication, an example properties file would be the following:
+
+```
+ baseUrl=https://localhost:18443
+ keystore=/path/to/keystore.jks
+ keystoreType=JKS
+ keystorePasswd=changeme
+ keyPasswd=changeme
+ truststore=/path/to/truststore.jks
+ truststoreType=JKS
+ truststorePasswd=changeme
+ proxiedEntity=user1@NIFI.COM
+```
+
+In this example, the certificate in _keystore.jks_ would be for the NiFi Registry server, for example "CN=localhost, OU=NIFI". This identity would need to be defined as a user in NiFi Registry and given permissions to 'Proxy'.
+
+"CN=localhost, OU=NIFI" would be proxying commands to be executed as user1@NIFI.COM.
+
+=== Interactive Usage
+In interactive mode the tab key can be used to perform auto-completion.
+
+For example, typing tab at an empty prompt should display possible commands for the first argument:
+
+ #>
+ demo exit help nifi registry session
+
+Typing "nifi " and then a tab will show the sub-commands for NiFi:
+
+ #> nifi
+ cluster-summary get-nodes pg-enable-services pg-set-var
+ connect-node get-reg-client-id pg-get-all-versions pg-start
+ create-reg-client get-root-id pg-get-services pg-status
+ current-user list-reg-clients pg-get-vars pg-stop
+ delete-node offload-node pg-get-version update-reg-client
+ disconnect-node pg-change-version pg-import
+ get-node pg-disable-services pg-list
+
+Arguments that represent a path to a file, such as `-p` or when setting a properties file in the session, will auto-complete the path being typed:
+
+ #> session set nifi.props /tmp/
+ dir1/ dir2/ dir3/
+
+=== Output
+Most commands support the ability to specify an `--outputType` argument, or `-ot` for short.
+
+Currently the output type may be simple or json.
+
+The default output type in interactive mode is simple, and the default output type in standalone mode is json.
+
+Example of simple output for `list-buckets`:
+
+ #> registry list-buckets -ot simple
+ My Bucket - 3c7b7467-0012-4d8f-a918-6aa42b6b9d39
+
+Example of json output for `list-buckets`:
+
+ #> registry list-buckets -ot json
+ [ {
+ "identifier" : "3c7b7467-0012-4d8f-a918-6aa42b6b9d39",
+ "name" : "My Bucket",
+ "createdTimestamp" : 1516718733854,
+ "permissions" : {
+ "canRead" : true,
+ "canWrite" : true,
+ "canDelete" : true
+ },
+ "link" : {
+ "params" : {
+ "rel" : "self"
+ },
+ "href" : "buckets/3c7b7467-0012-4d8f-a918-6aa42b6b9d39"
+ }
+ } ]
+
+=== Back-Referencing
+When using the interactive CLI, a common scenario will be using an id from a previous result as the input to the next command. Back-referencing provides a shortcut for referencing a result from the previous command via a positional reference.
+
+NOTE: Not every command produces back-references. To determine if a command supports back-referencing, check the usage.
+
+ #> registry list-buckets help
+ Lists the buckets that the current user has access to.
+ PRODUCES BACK-REFERENCES
+
+A common scenario for utilizing back-references would be the following:
+
+1. User starts by exploring the available buckets in a registry instance
+
+ #> registry list-buckets
+ # Name Id Description
+ - ------------ ------------------------------------ -----------
+ 1 My Bucket 3c7b7467-0012-4d8f-a918-6aa42b6b9d39 (empty)
+ 2 Other Bucket 175fb557-43a2-4abb-871f-81a354f47bc2 (empty)
+
+2. User then views the flows in one of the buckets using a back-reference to the bucket id from the previous result in position 1
+
+ #> registry list-flows -b &1
+ Using a positional back-reference for 'My Bucket'
+ # Name Id Description
+ - ------- ------------------------------------ ----------------
+ 1 My Flow 06acb207-d2f1-447f-85ed-9b8672fe6d30 This is my flow.
+
+3. User then views the version of the flow using a back-reference to the flow id from the previous result in position 1
+
+ #> registry list-flow-versions -f &1
+ Using a positional back-reference for 'My Flow'
+ Ver Date Author Message
+ --- -------------------------- ------------------------ -------------------------------------
+ 1 Tue, Jan 23 2018 09:48 EST anonymous This is the first version of my flow.
+
+4. User deploys version 1 of the flow using back-references to the bucket and flow id from step 2
+
+ #> nifi pg-import -b &1 -f &1 -fv 1
+ Using a positional back-reference for 'My Bucket'
+ Using a positional back-reference for 'My Flow'
+ 9bd157d4-0161-1000-b946-c1f9b1832efd
+
+The reason step 4 was able to reference the results from step 2, is because the `list-flow-versions` command in step 3 does not produce back-references, so the results from step 2 are still available.
+
+=== Adding Commands
+To add a NiFi command, create a new class that extends `AbstractNiFiCommand`:
+
+```
+public class MyCommand extends AbstractNiFiCommand {
+
+ public MyCommand() {
+ super("my-command");
+ }
+
+ @Override
+ protected void doExecute(NiFiClient client, Properties properties)
+ throws NiFiClientException, IOException, MissingOptionException, CommandException {
+ // TODO implement
+ }
+
+ @Override
+ public String getDescription() {
+ return "This is my new command";
+ }
+}
+```
+
+Add the new command to `NiFiCommandGroup`:
+
+```
+commands.add(new MyCommand());
+```
+
+To add a NiFi Registry command, perform the same steps, but extend from `AbstractNiFiRegistryCommand`, and add the command to `NiFiRegistryCommandGroup`.
+
+== Encrypt-Config Tool
+The `encrypt-config` command line tool (invoked as `./bin/encrypt-config.sh` or `bin\encrypt-config.bat`) reads from a _nifi.properties_ file with plaintext sensitive configuration values, prompts for a master password or raw hexadecimal key, and encrypts each value. It replaces the plain values with the protected value in the same file, or writes to a new _nifi.properties_ file if specified.
+
+The default encryption algorithm utilized is AES/GCM 128/256-bit. 128-bit is used if the JCE Unlimited Strength Cryptographic Jurisdiction Policy files are not installed, and 256-bit is used if they are installed.
+
+=== Usage
+To show help:
+
+ encrypt-config.sh -h
+
+The following are available options:
+
+ * `-h`,`--help` Prints this usage message
+ * `-v`,`--verbose` Sets verbose mode (default false)
+ * `-n`,`--niFiProperties <arg>` The _nifi.properties_ file containing unprotected config values (will be overwritten)
+ * `-l`,`--loginIdentityProviders <arg>` The _login-identity-providers.xml_ file containing unprotected config values (will be overwritten)
+ * `-a`,`--authorizers <arg>` The _authorizers.xml_ file containing unprotected config values (will be overwritten)
+ * `-f`,`--flowXml <arg>` The _flow.xml.gz_ file currently protected with old password (will be overwritten)
+ * `-b`,`--bootstrapConf <arg>` The _bootstrap.conf_ file to persist master key
+ * `-o`,`--outputNiFiProperties <arg>` The destination _nifi.properties_ file containing protected config values (will not modify input _nifi.properties_)
+ * `-i`,`--outputLoginIdentityProviders <arg>` The destination _login-identity-providers.xml_ file containing protected config values (will not modify input _login-identity-providers.xml_)
+ * `-u`,`--outputAuthorizers <arg>` The destination _authorizers.xml_ file containing protected config values (will not modify input _authorizers.xml_)
+ * `-g`,`--outputFlowXml <arg>` The destination _flow.xml.gz_ file containing protected config values (will not modify input _flow.xml.gz_)
+ * `-k`,`--key <arg>` The raw hexadecimal key to use to encrypt the sensitive properties
+ * `-e`,`--oldKey <arg>` The old raw hexadecimal key to use during key migration
+ * `-p`,`--password <arg>` The password from which to derive the key to use to encrypt the sensitive properties
+ * `-w`,`--oldPassword <arg>` The old password from which to derive the key during migration
+ * `-r`,`--useRawKey` If provided, the secure console will prompt for the raw key value in hexadecimal form
+ * `-m`,`--migrate` If provided, the _nifi.properties_ and/or _login-identity-providers.xml_ sensitive properties will be re-encrypted with a new key
+ * `-x`,`--encryptFlowXmlOnly` If provided, the properties in _flow.xml.gz_ will be re-encrypted with a new key but the _nifi.properties_ and/or _login-identity-providers.xml_ files will not be modified
+ * `-s`,`--propsKey <arg>` The password or key to use to encrypt the sensitive processor properties in _flow.xml.gz_
+ * `-A`,`--newFlowAlgorithm <arg>` The algorithm to use to encrypt the sensitive processor properties in _flow.xml.gz_
+ * `-P`,`--newFlowProvider <arg>` The security provider to use to encrypt the sensitive processor properties in _flow.xml.gz_
+
+As an example of how the tool works, assume that you have installed the tool on a machine supporting 256-bit encryption and with the following existing values in the _nifi.properties_ file:
+
+```
+# security properties #
+nifi.sensitive.props.key=thisIsABadSensitiveKeyPassword
+nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
+nifi.sensitive.props.provider=BC
+nifi.sensitive.props.additional.keys=
+
+nifi.security.keystore=/path/to/keystore.jks
+nifi.security.keystoreType=JKS
+nifi.security.keystorePasswd=thisIsABadKeystorePassword
+nifi.security.keyPasswd=thisIsABadKeyPassword
+nifi.security.truststore=
+nifi.security.truststoreType=
+nifi.security.truststorePasswd=
+```
+
+Enter the following arguments when using the tool:
+
+----
+encrypt-config.sh
+-b bootstrap.conf
+-k 0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210
+-n nifi.properties
+----
+
+As a result, the _nifi.properties_ file is overwritten with protected properties and sibling encryption identifiers (`aes/gcm/256`, the currently supported algorithm):
+
+```
+# security properties #
+nifi.sensitive.props.key=n2z+tTTbHuZ4V4V2||uWhdasyDXD4ZG2lMAes/vqh6u4vaz4xgL4aEbF4Y/dXevqk3ulRcOwf1vc4RDQ==
+nifi.sensitive.props.key.protected=aes/gcm/256
+nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
+nifi.sensitive.props.provider=BC
+nifi.sensitive.props.additional.keys=
+
+nifi.security.keystore=/path/to/keystore.jks
+nifi.security.keystoreType=JKS
+nifi.security.keystorePasswd=oBjT92hIGRElIGOh||MZ6uYuWNBrOA6usq/Jt3DaD2e4otNirZDytac/w/KFe0HOkrJR03vcbo
+nifi.security.keystorePasswd.protected=aes/gcm/256
+nifi.security.keyPasswd=ac/BaE35SL/esLiJ||+ULRvRLYdIDA2VqpE0eQXDEMjaLBMG2kbKOdOwBk/hGebDKlVg==
+nifi.security.keyPasswd.protected=aes/gcm/256
+nifi.security.truststore=
+nifi.security.truststoreType=
+nifi.security.truststorePasswd=
+```
+
+Additionally, the _bootstrap.conf_ file is updated with the encryption key as follows:
+
+```
+# Master key in hexadecimal format for encrypted sensitive configuration values
+nifi.bootstrap.sensitive.key=0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210
+```
+
+Sensitive configuration values are encrypted by the tool by default, however you can encrypt any additional properties, if desired. To encrypt additional properties, specify them as comma-separated values in the `nifi.sensitive.props.additional.keys` property.
+
+If the _nifi.properties_ file already has valid protected values, those property values are not modified by the tool.
+
+When applied to _login-identity-providers.xml_ and _authorizers.xml_, the property elements are updated with an `encryption` attribute:
+
+Example of protected _login-identity-providers.xml_:
+
+```
+ <!-- LDAP Provider -->
+ <provider>
+ <identifier>ldap-provider</identifier>
+ <class>org.apache.nifi.ldap.LdapProvider</class>
+ <property name="Authentication Strategy">START_TLS</property>
+ <property name="Manager DN">someuser</property>
+ <property name="Manager Password" encryption="aes/gcm/128">q4r7WIgN0MaxdAKM||SGgdCTPGSFEcuH4RraMYEdeyVbOx93abdWTVSWvh1w+klA</property>
+ <property name="TLS - Keystore"></property>
+ <property name="TLS - Keystore Password" encryption="aes/gcm/128">Uah59TWX+Ru5GY5p||B44RT/LJtC08QWA5ehQf01JxIpf0qSJUzug25UwkF5a50g</property>
+ <property name="TLS - Keystore Type"></property>
+ ...
+ </provider>
+```
+
+Example of protected _authorizers.xml_:
+
+```
+ <!-- LDAP User Group Provider -->
+ <userGroupProvider>
+ <identifier>ldap-user-group-provider</identifier>
+ <class>org.apache.nifi.ldap.tenants.LdapUserGroupProvider</class>
+ <property name="Authentication Strategy">START_TLS</property>
+ <property name="Manager DN">someuser</property>
+ <property name="Manager Password" encryption="aes/gcm/128">q4r7WIgN0MaxdAKM||SGgdCTPGSFEcuH4RraMYEdeyVbOx93abdWTVSWvh1w+klA</property>
+ <property name="TLS - Keystore"></property>
+ <property name="TLS - Keystore Password" encryption="aes/gcm/128">Uah59TWX+Ru5GY5p||B44RT/LJtC08QWA5ehQf01JxIpf0qSJUzug25UwkF5a50g</property>
+ <property name="TLS - Keystore Type"></property>
+ ...
+ </userGroupProvider>
+```
+
+== File Manager
+The File Manager utility (invoked as `./bin/file-manager.sh` or `bin\file-manager.bat`) allows system administrators to take a backup of an existing NiFi installation, install a new version of NiFi in a designated location (while migrating any previous configuration settings) or restore an installation from a previous backup. File Manager supports NiFi version 1.0.0 and higher.
+
+=== Usage
+To show help:
+
+ file-manager.sh -h
+
+The following are available options:
+
+* `-b`,`--backupDir <arg>` Backup NiFi Directory (used with backup or restore operation)
+* `-c`,`--nifiCurrentDir <arg>` Current NiFi Installation Directory (used optionally with install or restore operation)
+* `-d`,`--nifiInstallDir <arg>` NiFi Installation Directory (used with install or restore operation)
+* `-h`,`--help` Print help info (optional)
+* `-i`,`--installFile <arg>` NiFi Install File (used with install operation)
+* `-m`,`--moveRepositories` Allow repositories to be moved to new/restored nifi directory from existing installation, if available (used optionally with install or restore operation)
+* `-o`,`--operation <arg>` File operation (install | backup | restore)
+* `-r`,`--nifiRollbackDir <arg>` NiFi Installation Directory (used with install or restore operation)
+* `-t`,`--bootstrapConf <arg>` Current NiFi Bootstrap Configuration File (used optionally)
+* `-v`,`--verbose` Verbose messaging (optional)
+* `-x`,`--overwriteConfigs` Overwrite existing configuration directory with upgrade changes (used optionally with install or restore operation)
+
+Example usage on Linux:
+
+ # backup NiFi installation
+ # option -t may be provided to ensure backup of external boostrap.conf file
+ ./file-manager.sh
+ -o backup
+ –b /tmp/nifi_bak
+ –c /usr/nifi_old
+ -v
+
+ # install NiFi using compressed tar file into /usr/nifi directory (should install as /usr/nifi/nifi-1.3.0).
+ # migrate existing configurations with location determined by external bootstrap.conf and move over repositories from nifi_old
+ # options -t and -c should both be provided if migration of configurations, state and repositories are required
+ ./file-manager.sh
+ -o install
+ –i nifi-1.3.0.tar.gz
+ –d /usr/nifi
+ –c /usr/nifi/nifi_old
+ -t /usr/nifi/old_conf/bootstrap.conf
+ -v
+ -m
+
+ # restore NiFi installation from backup directory and move back repositories
+ # option -t may be provided to ensure bootstrap.conf is restored to the file path provided, otherwise it is placed in the
+ # default directory under the rollback path (e.g. /usr/nifi_old/conf)
+ ./file-manager.sh
+ -o restore
+ –b /tmp/nifi_bak
+ –r /usr/nifi_old
+ –c /usr/nifi
+ -m
+ -v
+
+=== Expected Behavior
+
+==== Backup
+During the backup operation a backup directory is created in a designated location for an existing NiFi installation. Backups will capture all critical files (including any internal or external configurations, libraries, scripts and documents) however it excludes backing up repositories and logs due to potential size. If configuration/library files are external from the existing installation folder the backup operation will capture those as well.
+
+==== Install
+During the install operation File Manager will perform installation using the designated NiFi binary file (either tar.gz or zip file) to create a new installation or migrate an existing nifi installation to a new one. Installation can optionally move repositories (if located within the configuration folder of the current installation) to the new installation as well as migrate configuration files to the newer installation.
+
+==== Restore
+The restore operation allows an existing installation to revert back to a previous installation. Using an existing backup directory (created from the backup operation) the FileManager utility will restore libraries, scripts and documents as well as revert to previous configurations.
+
+NOTE: If repositories were changed due to the installation of a newer version of NiFi these may no longer be compatible during restore. In that scenario exclude the `-m` option to ensure new repositories will be created or, if repositories live outside of the NiFi directory, remove them so they can be recreated on startup after restore.
+
+== Flow Analyzer
+The `flow-analyzer` tool (invoked as `./bin/flow-analyzer.sh` or `bin\flow-analyzer.bat`) analyzes the _flow.xml.gz_ file and reports:
+
+* Total Bytes Utilized by the System
+* Min/Max Back Pressure Size
+* Average Back Pressure Size
+* Min/Max Flowfile Queue Size
+* Average Flowfile Queue Size
+
+=== Usage
+To execute the `flow-analyzer` tool:
+
+ flow-analyzer.sh <path to flow.xml.gz>
+
+Example:
+
+ $ ./flow-analyzer.sh /Users/nifiuser/nifi-1.8.0/conf/flow.xml.gz
+ Using flow=/Users/nifiuser/nifi-1.8.0/conf/flow.xml.gz
+ Total Bytes Utilized by System=1518 GB
+ Max Back Pressure Size=1 GB
+ Min Back Pressure Size=1 GB
+ Average Back Pressure Size=2.504950495 GB
+ Max Flowfile Queue Size=10000
+ Min Flowfile Queue Size=10000
+ Avg Flowfile Queue Size=10000.000000000
+
+== Node Manager
+Node manager (invoked as `./bin/node-manager.sh` or `bin\node-manager.bat`) supports connecting, disconnecting and removing a node when in a cluster (an error message displays if the node is not part of a cluster) as well as obtaining the status of a node. When nodes are disconnected from a cluster and need to be connected or removed, a list of urls of connected nodes should be provided to send the required command to the active cluster. Node Manager supports NiFi version 1.0.0 and higher.
+
+=== Usage
+To show help:
+
+ node-manager.sh -h
+
+The following are available options:
+
+* `-b`,`--bootstrapConf <arg>` Existing Bootstrap Configuration file (required)
+* `-d`,`--nifiInstallDir <arg>` NiFi Root Folder (required)
+* `-h`,`--help` Help Text (optional)
+* `-o`, `--operation <arg>` Operations supported: status, connect (cluster), disconnect (cluster), remove (cluster)
+* `-p`,`--proxyDN <arg>` Proxy or User DN (required for secured nodes doing connect, disconnect and remove operations)
+* `-u`,`--clusterUrls <arg>` Comma delimited list of active urls for cluster (optional). Not required for disconnecting a node yet will be needed when connecting or removing from a cluster
+* `-v`,`--verbose` Verbose messaging (optional)
+
+
+To connect, disconnect, or remove a node from a cluster:
+
+ node-manager.sh -d {$NIFI_HOME} –b { nifi bootstrap file path}
+ -o {remove|disconnect|connect|status} [-u {url list}] [-p {proxy name}] [-v]
--- End diff --
Not related to the PR itself but wondering if it'd make sense to add 'offload' operation (cc @jtstorck) even though it's available in the CLI.
---
[GitHub] nifi pull request #3124: NIFI-5767 Added NiFi Toolkit Guide to docs
Posted by pvillard31 <gi...@git.apache.org>.
Github user pvillard31 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3124#discussion_r230324180
--- Diff: nifi-docs/src/main/asciidoc/toolkit-guide.adoc ---
@@ -0,0 +1,1257 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+= Apache NiFi Toolkit Guide
+Apache NiFi Team <de...@nifi.apache.org>
+:homepage: http://nifi.apache.org
+:linkattrs:
+
+== Overview
+The NiFi Toolkit contains several command line utilities to setup and support NiFi in standalone and clustered environments. The utilities include:
+
+* CLI -- The `cli` tool enables administrators to interact with NiFi and NiFi Registry instances to automate tasks such as deploying versioned flows and managing process groups and cluster nodes.
+* Encrypt Config -- The `encrypt-config` tool encrypts the sensitive keys in the _nifi.properties_ file to facilitate the setup of a secure NiFi instance.
+* File Manager -- The `file-manager` tool enables administrators to backup, install or restore a NiFi installation from backup.
+* Flow Analyzer -- The `flow-analyzer` tool produces a report that helps administrators understand the max amount of data which can be stored in backpressure for a given flow.
+* Node Manager -- The `node-manager` tool enables administrators to perform status checks on nodes as well as the ability to connect, disconnect, or remove nodes from the cluster.
+* Notify -- The `notify` tool enables administrators to send bulletins to the NiFi UI.
+* S2S -- The `s2s` tool enables administrators to send data into or out of NiFi flows over site-to-site.
+* TLS Toolkit -- The `tls-toolkit` utility generates the required keystores, truststore, and relevant configuration files to facilitate the setup of a secure NiFi instance.
+* ZooKeeper Migrator -- The `zk-migrator` tool enables administrators to:
+** move ZooKeeper information from one ZooKeeper cluster to another
+** migrate ZooKeeper node ownership
+
+The utilities are executed with scripts found in the `bin` folder of your NiFi Toolkit installation.
+
+NOTE: The NiFi Toolkit is downloaded separately from NiFi (see the link:https://nifi.apache.org/download.html[Apache NiFi downloads page^]).
+
+=== Prerequisites for Running in a Secure Environment
+For secured nodes and clusters, two policies should be configured in advance:
+
+* Access the controller – A user that will have access to these utilities should be authorized in NiFi by creating an “access the controller” policy (`/controller`) with both view and modify rights
+* Proxy user request – If not previously set, node’s identity (the DN value of the node’s certificate) should be authorized to proxy requests on behalf of a user
+
+When executing either the Notify or Node Manager tools in a secured environment the `proxyDN` flag option should be used in order to properly identify the user that was authorized to execute these commands. In non-secure environments, or if running the status operation on the Node Manager tool, the flag is ignored.
+
+== NiFi CLI
+This tool offers a CLI focused on interacting with NiFi and NiFi Registry in order to automate tasks, such as deploying flows from a NIFi Registy to a NiFi instance or managing process groups and cluster nodes.
+
+=== Usage
+The CLI toolkit can be executed in standalone mode to execute a single command, or interactive mode to enter an interactive shell.
+
+To execute a single command:
+
+ ./bin/cli.sh <command> <args>
+
+To launch the interactive shell:
+
+ ./bin/cli.sh
+
+To show help:
+
+ cli.sh -h
+
+The following are available options:
+
+ demo quick-import
+ nifi current-user
+ nifi cluster-summary
+ nifi connect-node
+ nifi delete-node
+ nifi disconnect-node
+ nifi get-root-id
+ nifi get-node
+ nifi get-nodes
+ nifi offload-node
+ nifi list-reg-clients
+ nifi create-reg-client
+ nifi update-reg-client
+ nifi get-reg-client-id
+ nifi pg-import
+ nifi pg-start
+ nifi pg-stop
+ nifi pg-get-vars
+ nifi pg-set-var
+ nifi pg-get-version
+ nifi pg-change-version
+ nifi pg-get-all-versions
+ nifi pg-list
+ nifi pg-status
+ nifi pg-get-services
+ nifi pg-enable-services
+ nifi pg-disable-services
+ registry current-user
+ registry list-buckets
+ registry create-bucket
+ registry delete-bucket
+ registry list-flows
+ registry create-flow
+ registry delete-flow
+ registry list-flow-versions
+ registry export-flow-version
+ registry import-flow-version
+ registry sync-flow-versions
+ registry transfer-flow-version
+ session keys
+ session show
+ session get
+ session set
+ session remove
+ session clear
+ exit
+ help
--- End diff --
Maybe add that ``./bin/cli.sh <command> -h`` will display extensive help usage of the command
---
[GitHub] nifi pull request #3124: NIFI-5767 Added NiFi Toolkit Guide to docs
Posted by andrewmlim <gi...@git.apache.org>.
Github user andrewmlim commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3124#discussion_r230381765
--- Diff: nifi-docs/src/main/asciidoc/toolkit-guide.adoc ---
@@ -0,0 +1,1257 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+= Apache NiFi Toolkit Guide
+Apache NiFi Team <de...@nifi.apache.org>
+:homepage: http://nifi.apache.org
+:linkattrs:
+
+== Overview
+The NiFi Toolkit contains several command line utilities to setup and support NiFi in standalone and clustered environments. The utilities include:
+
+* CLI -- The `cli` tool enables administrators to interact with NiFi and NiFi Registry instances to automate tasks such as deploying versioned flows and managing process groups and cluster nodes.
+* Encrypt Config -- The `encrypt-config` tool encrypts the sensitive keys in the _nifi.properties_ file to facilitate the setup of a secure NiFi instance.
+* File Manager -- The `file-manager` tool enables administrators to backup, install or restore a NiFi installation from backup.
+* Flow Analyzer -- The `flow-analyzer` tool produces a report that helps administrators understand the max amount of data which can be stored in backpressure for a given flow.
+* Node Manager -- The `node-manager` tool enables administrators to perform status checks on nodes as well as the ability to connect, disconnect, or remove nodes from the cluster.
+* Notify -- The `notify` tool enables administrators to send bulletins to the NiFi UI.
+* S2S -- The `s2s` tool enables administrators to send data into or out of NiFi flows over site-to-site.
+* TLS Toolkit -- The `tls-toolkit` utility generates the required keystores, truststore, and relevant configuration files to facilitate the setup of a secure NiFi instance.
+* ZooKeeper Migrator -- The `zk-migrator` tool enables administrators to:
+** move ZooKeeper information from one ZooKeeper cluster to another
+** migrate ZooKeeper node ownership
+
+The utilities are executed with scripts found in the `bin` folder of your NiFi Toolkit installation.
+
+NOTE: The NiFi Toolkit is downloaded separately from NiFi (see the link:https://nifi.apache.org/download.html[Apache NiFi downloads page^]).
+
+=== Prerequisites for Running in a Secure Environment
+For secured nodes and clusters, two policies should be configured in advance:
+
+* Access the controller – A user that will have access to these utilities should be authorized in NiFi by creating an “access the controller” policy (`/controller`) with both view and modify rights
+* Proxy user request – If not previously set, node’s identity (the DN value of the node’s certificate) should be authorized to proxy requests on behalf of a user
+
+When executing either the Notify or Node Manager tools in a secured environment the `proxyDN` flag option should be used in order to properly identify the user that was authorized to execute these commands. In non-secure environments, or if running the status operation on the Node Manager tool, the flag is ignored.
+
+== NiFi CLI
+This tool offers a CLI focused on interacting with NiFi and NiFi Registry in order to automate tasks, such as deploying flows from a NIFi Registy to a NiFi instance or managing process groups and cluster nodes.
+
+=== Usage
+The CLI toolkit can be executed in standalone mode to execute a single command, or interactive mode to enter an interactive shell.
+
+To execute a single command:
+
+ ./bin/cli.sh <command> <args>
+
+To launch the interactive shell:
+
+ ./bin/cli.sh
+
+To show help:
+
+ cli.sh -h
+
+The following are available options:
+
+ demo quick-import
+ nifi current-user
+ nifi cluster-summary
+ nifi connect-node
+ nifi delete-node
+ nifi disconnect-node
+ nifi get-root-id
+ nifi get-node
+ nifi get-nodes
+ nifi offload-node
+ nifi list-reg-clients
+ nifi create-reg-client
+ nifi update-reg-client
+ nifi get-reg-client-id
+ nifi pg-import
+ nifi pg-start
+ nifi pg-stop
+ nifi pg-get-vars
+ nifi pg-set-var
+ nifi pg-get-version
+ nifi pg-change-version
+ nifi pg-get-all-versions
+ nifi pg-list
+ nifi pg-status
+ nifi pg-get-services
+ nifi pg-enable-services
+ nifi pg-disable-services
+ registry current-user
+ registry list-buckets
+ registry create-bucket
+ registry delete-bucket
+ registry list-flows
+ registry create-flow
+ registry delete-flow
+ registry list-flow-versions
+ registry export-flow-version
+ registry import-flow-version
+ registry sync-flow-versions
+ registry transfer-flow-version
+ session keys
+ session show
+ session get
+ session set
+ session remove
+ session clear
+ exit
+ help
--- End diff --
Will do.
---
[GitHub] nifi pull request #3124: NIFI-5767 Added NiFi Toolkit Guide to docs
Posted by andrewmlim <gi...@git.apache.org>.
Github user andrewmlim commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3124#discussion_r230381451
--- Diff: nifi-docs/src/main/asciidoc/toolkit-guide.adoc ---
@@ -0,0 +1,1257 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+= Apache NiFi Toolkit Guide
+Apache NiFi Team <de...@nifi.apache.org>
+:homepage: http://nifi.apache.org
+:linkattrs:
+
+== Overview
+The NiFi Toolkit contains several command line utilities to setup and support NiFi in standalone and clustered environments. The utilities include:
+
+* CLI -- The `cli` tool enables administrators to interact with NiFi and NiFi Registry instances to automate tasks such as deploying versioned flows and managing process groups and cluster nodes.
+* Encrypt Config -- The `encrypt-config` tool encrypts the sensitive keys in the _nifi.properties_ file to facilitate the setup of a secure NiFi instance.
+* File Manager -- The `file-manager` tool enables administrators to backup, install or restore a NiFi installation from backup.
+* Flow Analyzer -- The `flow-analyzer` tool produces a report that helps administrators understand the max amount of data which can be stored in backpressure for a given flow.
+* Node Manager -- The `node-manager` tool enables administrators to perform status checks on nodes as well as the ability to connect, disconnect, or remove nodes from the cluster.
+* Notify -- The `notify` tool enables administrators to send bulletins to the NiFi UI.
+* S2S -- The `s2s` tool enables administrators to send data into or out of NiFi flows over site-to-site.
+* TLS Toolkit -- The `tls-toolkit` utility generates the required keystores, truststore, and relevant configuration files to facilitate the setup of a secure NiFi instance.
+* ZooKeeper Migrator -- The `zk-migrator` tool enables administrators to:
+** move ZooKeeper information from one ZooKeeper cluster to another
+** migrate ZooKeeper node ownership
+
+The utilities are executed with scripts found in the `bin` folder of your NiFi Toolkit installation.
+
+NOTE: The NiFi Toolkit is downloaded separately from NiFi (see the link:https://nifi.apache.org/download.html[Apache NiFi downloads page^]).
+
+=== Prerequisites for Running in a Secure Environment
+For secured nodes and clusters, two policies should be configured in advance:
+
+* Access the controller – A user that will have access to these utilities should be authorized in NiFi by creating an “access the controller” policy (`/controller`) with both view and modify rights
+* Proxy user request – If not previously set, node’s identity (the DN value of the node’s certificate) should be authorized to proxy requests on behalf of a user
+
+When executing either the Notify or Node Manager tools in a secured environment the `proxyDN` flag option should be used in order to properly identify the user that was authorized to execute these commands. In non-secure environments, or if running the status operation on the Node Manager tool, the flag is ignored.
+
+== NiFi CLI
+This tool offers a CLI focused on interacting with NiFi and NiFi Registry in order to automate tasks, such as deploying flows from a NIFi Registy to a NiFi instance or managing process groups and cluster nodes.
+
+=== Usage
+The CLI toolkit can be executed in standalone mode to execute a single command, or interactive mode to enter an interactive shell.
+
+To execute a single command:
+
+ ./bin/cli.sh <command> <args>
+
+To launch the interactive shell:
+
+ ./bin/cli.sh
+
+To show help:
+
+ cli.sh -h
--- End diff --
Do you think we should preface every help command example in the doc with "./bin/"? I can make those changes.
---
[GitHub] nifi pull request #3124: NIFI-5767 Added NiFi Toolkit Guide to docs
Posted by andrewmlim <gi...@git.apache.org>.
Github user andrewmlim commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3124#discussion_r230881754
--- Diff: nifi-docs/src/main/asciidoc/toolkit-guide.adoc ---
@@ -0,0 +1,1257 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+= Apache NiFi Toolkit Guide
+Apache NiFi Team <de...@nifi.apache.org>
+:homepage: http://nifi.apache.org
+:linkattrs:
+
+== Overview
+The NiFi Toolkit contains several command line utilities to setup and support NiFi in standalone and clustered environments. The utilities include:
+
+* CLI -- The `cli` tool enables administrators to interact with NiFi and NiFi Registry instances to automate tasks such as deploying versioned flows and managing process groups and cluster nodes.
+* Encrypt Config -- The `encrypt-config` tool encrypts the sensitive keys in the _nifi.properties_ file to facilitate the setup of a secure NiFi instance.
+* File Manager -- The `file-manager` tool enables administrators to backup, install or restore a NiFi installation from backup.
+* Flow Analyzer -- The `flow-analyzer` tool produces a report that helps administrators understand the max amount of data which can be stored in backpressure for a given flow.
+* Node Manager -- The `node-manager` tool enables administrators to perform status checks on nodes as well as the ability to connect, disconnect, or remove nodes from the cluster.
+* Notify -- The `notify` tool enables administrators to send bulletins to the NiFi UI.
+* S2S -- The `s2s` tool enables administrators to send data into or out of NiFi flows over site-to-site.
+* TLS Toolkit -- The `tls-toolkit` utility generates the required keystores, truststore, and relevant configuration files to facilitate the setup of a secure NiFi instance.
+* ZooKeeper Migrator -- The `zk-migrator` tool enables administrators to:
+** move ZooKeeper information from one ZooKeeper cluster to another
+** migrate ZooKeeper node ownership
+
+The utilities are executed with scripts found in the `bin` folder of your NiFi Toolkit installation.
+
+NOTE: The NiFi Toolkit is downloaded separately from NiFi (see the link:https://nifi.apache.org/download.html[Apache NiFi downloads page^]).
+
+=== Prerequisites for Running in a Secure Environment
+For secured nodes and clusters, two policies should be configured in advance:
+
+* Access the controller – A user that will have access to these utilities should be authorized in NiFi by creating an “access the controller” policy (`/controller`) with both view and modify rights
+* Proxy user request – If not previously set, node’s identity (the DN value of the node’s certificate) should be authorized to proxy requests on behalf of a user
+
+When executing either the Notify or Node Manager tools in a secured environment the `proxyDN` flag option should be used in order to properly identify the user that was authorized to execute these commands. In non-secure environments, or if running the status operation on the Node Manager tool, the flag is ignored.
+
+== NiFi CLI
+This tool offers a CLI focused on interacting with NiFi and NiFi Registry in order to automate tasks, such as deploying flows from a NIFi Registy to a NiFi instance or managing process groups and cluster nodes.
+
+=== Usage
+The CLI toolkit can be executed in standalone mode to execute a single command, or interactive mode to enter an interactive shell.
+
+To execute a single command:
+
+ ./bin/cli.sh <command> <args>
+
+To launch the interactive shell:
+
+ ./bin/cli.sh
+
+To show help:
+
+ cli.sh -h
+
+The following are available options:
+
+ demo quick-import
+ nifi current-user
+ nifi cluster-summary
+ nifi connect-node
+ nifi delete-node
+ nifi disconnect-node
+ nifi get-root-id
+ nifi get-node
+ nifi get-nodes
+ nifi offload-node
+ nifi list-reg-clients
+ nifi create-reg-client
+ nifi update-reg-client
+ nifi get-reg-client-id
+ nifi pg-import
+ nifi pg-start
+ nifi pg-stop
+ nifi pg-get-vars
+ nifi pg-set-var
+ nifi pg-get-version
+ nifi pg-change-version
+ nifi pg-get-all-versions
+ nifi pg-list
+ nifi pg-status
+ nifi pg-get-services
+ nifi pg-enable-services
+ nifi pg-disable-services
+ registry current-user
+ registry list-buckets
+ registry create-bucket
+ registry delete-bucket
+ registry list-flows
+ registry create-flow
+ registry delete-flow
+ registry list-flow-versions
+ registry export-flow-version
+ registry import-flow-version
+ registry sync-flow-versions
+ registry transfer-flow-version
+ session keys
+ session show
+ session get
+ session set
+ session remove
+ session clear
+ exit
+ help
--- End diff --
Done.
---
[GitHub] nifi pull request #3124: NIFI-5767 Added NiFi Toolkit Guide to docs
Posted by pvillard31 <gi...@git.apache.org>.
Github user pvillard31 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3124#discussion_r230322139
--- Diff: nifi-docs/src/main/asciidoc/toolkit-guide.adoc ---
@@ -0,0 +1,1257 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+= Apache NiFi Toolkit Guide
+Apache NiFi Team <de...@nifi.apache.org>
+:homepage: http://nifi.apache.org
+:linkattrs:
+
+== Overview
+The NiFi Toolkit contains several command line utilities to setup and support NiFi in standalone and clustered environments. The utilities include:
+
+* CLI -- The `cli` tool enables administrators to interact with NiFi and NiFi Registry instances to automate tasks such as deploying versioned flows and managing process groups and cluster nodes.
+* Encrypt Config -- The `encrypt-config` tool encrypts the sensitive keys in the _nifi.properties_ file to facilitate the setup of a secure NiFi instance.
+* File Manager -- The `file-manager` tool enables administrators to backup, install or restore a NiFi installation from backup.
+* Flow Analyzer -- The `flow-analyzer` tool produces a report that helps administrators understand the max amount of data which can be stored in backpressure for a given flow.
+* Node Manager -- The `node-manager` tool enables administrators to perform status checks on nodes as well as the ability to connect, disconnect, or remove nodes from the cluster.
+* Notify -- The `notify` tool enables administrators to send bulletins to the NiFi UI.
+* S2S -- The `s2s` tool enables administrators to send data into or out of NiFi flows over site-to-site.
+* TLS Toolkit -- The `tls-toolkit` utility generates the required keystores, truststore, and relevant configuration files to facilitate the setup of a secure NiFi instance.
+* ZooKeeper Migrator -- The `zk-migrator` tool enables administrators to:
+** move ZooKeeper information from one ZooKeeper cluster to another
+** migrate ZooKeeper node ownership
+
+The utilities are executed with scripts found in the `bin` folder of your NiFi Toolkit installation.
+
+NOTE: The NiFi Toolkit is downloaded separately from NiFi (see the link:https://nifi.apache.org/download.html[Apache NiFi downloads page^]).
+
+=== Prerequisites for Running in a Secure Environment
+For secured nodes and clusters, two policies should be configured in advance:
+
+* Access the controller – A user that will have access to these utilities should be authorized in NiFi by creating an “access the controller” policy (`/controller`) with both view and modify rights
+* Proxy user request – If not previously set, node’s identity (the DN value of the node’s certificate) should be authorized to proxy requests on behalf of a user
+
+When executing either the Notify or Node Manager tools in a secured environment the `proxyDN` flag option should be used in order to properly identify the user that was authorized to execute these commands. In non-secure environments, or if running the status operation on the Node Manager tool, the flag is ignored.
+
+== NiFi CLI
+This tool offers a CLI focused on interacting with NiFi and NiFi Registry in order to automate tasks, such as deploying flows from a NIFi Registy to a NiFi instance or managing process groups and cluster nodes.
+
+=== Usage
+The CLI toolkit can be executed in standalone mode to execute a single command, or interactive mode to enter an interactive shell.
+
+To execute a single command:
+
+ ./bin/cli.sh <command> <args>
+
+To launch the interactive shell:
+
+ ./bin/cli.sh
+
+To show help:
+
+ cli.sh -h
--- End diff --
```suggestion
./bin/cli.sh -h
```
---
[GitHub] nifi issue #3124: NIFI-5767 Added NiFi Toolkit Guide to docs
Posted by andrewmlim <gi...@git.apache.org>.
Github user andrewmlim commented on the issue:
https://github.com/apache/nifi/pull/3124
I will file a Jira to edit the other docs that have duplicate content. The other docs can refer to the Toolkit Guide as needed.
---
[GitHub] nifi issue #3124: NIFI-5767 Added NiFi Toolkit Guide to docs
Posted by bbende <gi...@git.apache.org>.
Github user bbende commented on the issue:
https://github.com/apache/nifi/pull/3124
I'm good with removing the README once 1.9.0 is released, or changing it to a link to the new toolkit docs.
---
[GitHub] nifi pull request #3124: NIFI-5767 Added NiFi Toolkit Guide to docs
Posted by andrewmlim <gi...@git.apache.org>.
Github user andrewmlim commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3124#discussion_r230881702
--- Diff: nifi-docs/src/main/asciidoc/toolkit-guide.adoc ---
@@ -0,0 +1,1257 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+= Apache NiFi Toolkit Guide
+Apache NiFi Team <de...@nifi.apache.org>
+:homepage: http://nifi.apache.org
+:linkattrs:
+
+== Overview
+The NiFi Toolkit contains several command line utilities to setup and support NiFi in standalone and clustered environments. The utilities include:
+
+* CLI -- The `cli` tool enables administrators to interact with NiFi and NiFi Registry instances to automate tasks such as deploying versioned flows and managing process groups and cluster nodes.
+* Encrypt Config -- The `encrypt-config` tool encrypts the sensitive keys in the _nifi.properties_ file to facilitate the setup of a secure NiFi instance.
+* File Manager -- The `file-manager` tool enables administrators to backup, install or restore a NiFi installation from backup.
+* Flow Analyzer -- The `flow-analyzer` tool produces a report that helps administrators understand the max amount of data which can be stored in backpressure for a given flow.
+* Node Manager -- The `node-manager` tool enables administrators to perform status checks on nodes as well as the ability to connect, disconnect, or remove nodes from the cluster.
+* Notify -- The `notify` tool enables administrators to send bulletins to the NiFi UI.
+* S2S -- The `s2s` tool enables administrators to send data into or out of NiFi flows over site-to-site.
+* TLS Toolkit -- The `tls-toolkit` utility generates the required keystores, truststore, and relevant configuration files to facilitate the setup of a secure NiFi instance.
+* ZooKeeper Migrator -- The `zk-migrator` tool enables administrators to:
+** move ZooKeeper information from one ZooKeeper cluster to another
+** migrate ZooKeeper node ownership
+
+The utilities are executed with scripts found in the `bin` folder of your NiFi Toolkit installation.
+
+NOTE: The NiFi Toolkit is downloaded separately from NiFi (see the link:https://nifi.apache.org/download.html[Apache NiFi downloads page^]).
+
+=== Prerequisites for Running in a Secure Environment
+For secured nodes and clusters, two policies should be configured in advance:
+
+* Access the controller – A user that will have access to these utilities should be authorized in NiFi by creating an “access the controller” policy (`/controller`) with both view and modify rights
+* Proxy user request – If not previously set, node’s identity (the DN value of the node’s certificate) should be authorized to proxy requests on behalf of a user
+
+When executing either the Notify or Node Manager tools in a secured environment the `proxyDN` flag option should be used in order to properly identify the user that was authorized to execute these commands. In non-secure environments, or if running the status operation on the Node Manager tool, the flag is ignored.
+
+== NiFi CLI
+This tool offers a CLI focused on interacting with NiFi and NiFi Registry in order to automate tasks, such as deploying flows from a NIFi Registy to a NiFi instance or managing process groups and cluster nodes.
+
+=== Usage
+The CLI toolkit can be executed in standalone mode to execute a single command, or interactive mode to enter an interactive shell.
+
+To execute a single command:
+
+ ./bin/cli.sh <command> <args>
+
+To launch the interactive shell:
+
+ ./bin/cli.sh
+
+To show help:
+
+ cli.sh -h
--- End diff --
Updated all help examples accordingly.
---
[GitHub] nifi pull request #3124: NIFI-5767 Added NiFi Toolkit Guide to docs
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/nifi/pull/3124
---
[GitHub] nifi issue #3124: NIFI-5767 Added NiFi Toolkit Guide to docs
Posted by pvillard31 <gi...@git.apache.org>.
Github user pvillard31 commented on the issue:
https://github.com/apache/nifi/pull/3124
Hey @andrewmlim - it looks good to me, thanks for taking care of it, it's going to be very useful!
@bbende - do you think it still makes sense to keep the CLI README file? we don't want to remove it until 1.9.0 is released (with the doc changes) but I think it'd be better to keep things in one place after that. I can file a JIRA if you agree with it.
---