You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by andrewmlim <gi...@git.apache.org> on 2016/11/18 20:45:01 UTC
[GitHub] nifi pull request #1246: NIFI-3058 Add documentation for ZooKeeper Migrator ...
GitHub user andrewmlim opened a pull request:
https://github.com/apache/nifi/pull/1246
NIFI-3058 Add documentation for ZooKeeper Migrator to Admin Guide
Also, corrected some incorrectly capitalized references to ZooKeeper.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/andrewmlim/nifi NIFI-3058
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/1246.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1246
----
commit c3e65387650cc1d37cbf0b870fe02b66726f2d10
Author: Andrew Lim <an...@gmail.com>
Date: 2016-11-18T20:41:04Z
NIFI-3058 Add documentation for ZooKeeper Migrator to Admin Guide
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1246: NIFI-3058 Add documentation for ZooKeeper Migrator ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/nifi/pull/1246
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1246: NIFI-3058 Add documentation for ZooKeeper Migrator ...
Posted by andrewmlim <gi...@git.apache.org>.
Github user andrewmlim commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1246#discussion_r88743768
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -1599,7 +1599,97 @@ java.arg.16=-Dsun.security.krb5.debug=true
This will cause the debug output to be written to the NiFi Bootstrap log file. By default, this is located at _$NIFI_HOME/logs/nifi-bootstrap.log_.
This output can be rather verbose but provides extremely valuable information for troubleshooting Kerberos failures.
+[[zookeeper_migrator]]
+=== ZooKeeper Migrator
+You can use the NiFi ZooKeeper Migrator to perform the following tasks:
+* Move ZooKeeper information from one ZooKeeper cluster to another
--- End diff --
thanks...will do.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1246: NIFI-3058 Add documentation for ZooKeeper Migrator ...
Posted by andrewmlim <gi...@git.apache.org>.
Github user andrewmlim commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1246#discussion_r88743950
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -1599,7 +1599,97 @@ java.arg.16=-Dsun.security.krb5.debug=true
This will cause the debug output to be written to the NiFi Bootstrap log file. By default, this is located at _$NIFI_HOME/logs/nifi-bootstrap.log_.
This output can be rather verbose but provides extremely valuable information for troubleshooting Kerberos failures.
+[[zookeeper_migrator]]
+=== ZooKeeper Migrator
+You can use the NiFi ZooKeeper Migrator to perform the following tasks:
+* Move ZooKeeper information from one ZooKeeper cluster to another
+* Migrating ZooKeeper node ownership
+
+For example, you may want to use the ZooKeeper Migrator when you are:
+
+* Upgrading from NiFi 0.x to NiFi 1.x in which embedded ZooKeepers are used
+* Migrating from an embedded ZooKeeper in NiFi 0.x or 1.x to an external ZooKeeper
+* Upgrading from NiFi 0.x with an external ZooKeeper to NiFi 1.x with the same external ZooKeeper
+* Migrating from an external ZooKeeper to an embedded ZooKeeper in NiFi 1.x
+
+The NiFi ZooKeeper Migrator (`zk-migrator.sh`) is part of the NiFi Toolkit and is downloaded separately from the
+link:https://nifi.apache.org/download.html[Apache NiFi download page].
+
+[[zk_migrator_command_line_parameters]]
+==== zk-migrator.sh Command Line Parameters
+
+You can use the following command line options with the ZooKeeper Migrator:
+
+* `-a,--auth <username:password>` Allows the specification of a username and password for authentication with ZooKeeper. This option is mutually exclusive with the `-k,--krb-conf` option.
+* `-f,--file <filename>` The file to be used for JSON data. When used with the `-r,--receive` option, data read from ZooKeeper will be stored in the given filename. When used with the `-s,--send` option, the data in the file will be sent to ZooKeeper.
+* `-h,--help` Prints help, displays available parameters with descriptions
+* `--ignore-source` Allows the ZooKeeper Migrator to write to the ZooKeeper and path from which the data was obtained.
+* `-k,--krb-conf <jaas-filename>` Allows the specification of a JAAS configuration file to allow authentication with a ZooKeeper configured to use Kerberos. This option is mutually exclusive with the `-a,--auth` option.
+* `-r,--receive` Receives data from ZooKeeper and writes to the given filename (if the `-f,--file` option is provided) or standard output. The data received will contain the full path to each node read from ZooKeeper. This option is mutually exclusive with the `-s,--send` option.
+* `-s,--send` Sends data to ZooKeeper that is read from the given filename (if the `-f,--file` option is provided) or standard input. The paths for each node in the data being sent to ZooKeeper are absolute paths, and will be stored in ZooKeeper under the *path* portion of the `-z,--zookeeper` argument. Typically, the *path* portion of the argument can be omitted, which will store the nodes at their absolute paths. This option is mutually exclusive with the `-r,--receive` option.
+* `-z,--zookeeper <zookeeper-endpoint>` The ZooKeeper server to use, specified by a connection string with path, in the format of _host:port/znode/path_.
+
+[[migrating_between_source_destination_zookeepers]]
+==== Migrating Between Source and Destination ZooKeepers
+
+Before you begin, confirm that:
+
+* You have installed the destination ZooKeeper cluster.
+* You have installed and configured a NiFi cluster to use the destination ZooKeeper cluster.
+* If you are migrating ZooKeepers due to upgrading NiFi from 0.x to 1.x,, you have already followed appropriate NiFi upgrade steps.
+* You have configured Kerberos as needed.
+* You have not started processing any dataflow (to avoid duplicate data processing).
+* If one of the ZooKeeper clusters you are using is configured with Kerberos, you are running the ZooKeeper Migrator from a host that has access to NiFi\u2019s ZooKeeper client jaas configuration file (see <<zk_kerberos_client>> for more information).
+
+1. Collect the following information:
+
+|====
+|*Required Information*|*Description*
+|Source ZooKeeper hostname (*sourceHostname*)|The hostname must be one of the hosts running in the ZooKeeper ensemble, which can be found in <NiFi installation dir>/conf/zookeeper.properties. Any of the hostnames declared in the *server.N* properties can be used.
+|Destination ZooKeeper hostname (*destinationHostname*)|The hostname must be one of the hosts running in the ZooKeeper ensemble, which can be found in <NiFi installation dir>/conf/zookeeper.properties. Any of the hostnames declared in the *server.N* properties can be used.
+|Source ZooKeeper port (*sourceClientPort*)|This can be found in *zookeeper.properties* of the <NiFi installation dir>/conf/zookeeper.properties. The port is specified in the *clientPort* property.
+|Destination ZooKeeper port (*destinationClientPort*)|This can be found in *zookeeper.properties* of the <NiFi installation dir>/conf/zookeeper.properties. The port is specified in the *clientPort* property.
+|Export data path|Determine the path that will store a json file containing the export of data from ZooKeeper. It must be readable and writable by the user running the zk-migrator tool.
+|Source ZooKeeper Authentication Information|This information is in <NiFi installation dir>/conf/state-management.xml. For NiFI 0.x, if Creator Only is specified in state-management.xml, you need to supply authentication information using the `-a,--auth` argument with the values from the Username and Password properties in state-management.xml. For NiFi 1.x, supply authentication information using the `-k,--krb-conf` argument.
--- End diff --
Will go through and grep for these.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1246: NIFI-3058 Add documentation for ZooKeeper Migrator ...
Posted by markap14 <gi...@git.apache.org>.
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1246#discussion_r88740952
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -1432,7 +1432,7 @@ If Kerberos is not already setup in your environment, you can find information o
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Managing_Smart_Cards/Configuring_a_Kerberos_5_Server.html[_https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Managing_Smart_Cards/Configuring_a_Kerberos_5_Server.html_]
. This guide assumes that Kerberos already has been installed in the environment in which NiFi is running.
-Note, the following procedures for kerberizing an Embedded Zookeeper server in your NiFI Node and kerberizing a zookeeper NiFI client will require that
+Note, the following procedures for kerberizing an Embedded ZooKeeper server in your NiFI Node and kerberizing a ZooKeeper NiFI client will require that
--- End diff --
NiFi is spelled with a capital I twice here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1246: NIFI-3058 Add documentation for ZooKeeper Migrator to Admi...
Posted by jtstorck <gi...@git.apache.org>.
Github user jtstorck commented on the issue:
https://github.com/apache/nifi/pull/1246
+1, I've gone over the documentation, looks good to me at this point.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1246: NIFI-3058 Add documentation for ZooKeeper Migrator ...
Posted by jtstorck <gi...@git.apache.org>.
Github user jtstorck commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1246#discussion_r88740528
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -1599,7 +1599,97 @@ java.arg.16=-Dsun.security.krb5.debug=true
This will cause the debug output to be written to the NiFi Bootstrap log file. By default, this is located at _$NIFI_HOME/logs/nifi-bootstrap.log_.
This output can be rather verbose but provides extremely valuable information for troubleshooting Kerberos failures.
+[[zookeeper_migrator]]
+=== ZooKeeper Migrator
+You can use the NiFi ZooKeeper Migrator to perform the following tasks:
+* Move ZooKeeper information from one ZooKeeper cluster to another
+* Migrating ZooKeeper node ownership
+
+For example, you may want to use the ZooKeeper Migrator when you are:
+
+* Upgrading from NiFi 0.x to NiFi 1.x in which embedded ZooKeepers are used
+* Migrating from an embedded ZooKeeper in NiFi 0.x or 1.x to an external ZooKeeper
+* Upgrading from NiFi 0.x with an external ZooKeeper to NiFi 1.x with the same external ZooKeeper
+* Migrating from an external ZooKeeper to an embedded ZooKeeper in NiFi 1.x
+
+The NiFi ZooKeeper Migrator (`zk-migrator.sh`) is part of the NiFi Toolkit and is downloaded separately from the
--- End diff --
Might want to remove the mention of zk-migrator.sh here, since the tool provites a .bat file for Windows as well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1246: NIFI-3058 Add documentation for ZooKeeper Migrator ...
Posted by andrewmlim <gi...@git.apache.org>.
Github user andrewmlim commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1246#discussion_r88743855
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -1599,7 +1599,97 @@ java.arg.16=-Dsun.security.krb5.debug=true
This will cause the debug output to be written to the NiFi Bootstrap log file. By default, this is located at _$NIFI_HOME/logs/nifi-bootstrap.log_.
This output can be rather verbose but provides extremely valuable information for troubleshooting Kerberos failures.
+[[zookeeper_migrator]]
+=== ZooKeeper Migrator
+You can use the NiFi ZooKeeper Migrator to perform the following tasks:
+* Move ZooKeeper information from one ZooKeeper cluster to another
+* Migrating ZooKeeper node ownership
+
+For example, you may want to use the ZooKeeper Migrator when you are:
+
+* Upgrading from NiFi 0.x to NiFi 1.x in which embedded ZooKeepers are used
+* Migrating from an embedded ZooKeeper in NiFi 0.x or 1.x to an external ZooKeeper
+* Upgrading from NiFi 0.x with an external ZooKeeper to NiFi 1.x with the same external ZooKeeper
+* Migrating from an external ZooKeeper to an embedded ZooKeeper in NiFi 1.x
+
+The NiFi ZooKeeper Migrator (`zk-migrator.sh`) is part of the NiFi Toolkit and is downloaded separately from the
+link:https://nifi.apache.org/download.html[Apache NiFi download page].
+
+[[zk_migrator_command_line_parameters]]
+==== zk-migrator.sh Command Line Parameters
+
+You can use the following command line options with the ZooKeeper Migrator:
+
+* `-a,--auth <username:password>` Allows the specification of a username and password for authentication with ZooKeeper. This option is mutually exclusive with the `-k,--krb-conf` option.
+* `-f,--file <filename>` The file to be used for JSON data. When used with the `-r,--receive` option, data read from ZooKeeper will be stored in the given filename. When used with the `-s,--send` option, the data in the file will be sent to ZooKeeper.
--- End diff --
will do.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1246: NIFI-3058 Add documentation for ZooKeeper Migrator ...
Posted by markap14 <gi...@git.apache.org>.
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1246#discussion_r88741041
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -1599,7 +1599,97 @@ java.arg.16=-Dsun.security.krb5.debug=true
This will cause the debug output to be written to the NiFi Bootstrap log file. By default, this is located at _$NIFI_HOME/logs/nifi-bootstrap.log_.
This output can be rather verbose but provides extremely valuable information for troubleshooting Kerberos failures.
+[[zookeeper_migrator]]
+=== ZooKeeper Migrator
+You can use the NiFi ZooKeeper Migrator to perform the following tasks:
+* Move ZooKeeper information from one ZooKeeper cluster to another
+* Migrating ZooKeeper node ownership
+
+For example, you may want to use the ZooKeeper Migrator when you are:
+
+* Upgrading from NiFi 0.x to NiFi 1.x in which embedded ZooKeepers are used
+* Migrating from an embedded ZooKeeper in NiFi 0.x or 1.x to an external ZooKeeper
+* Upgrading from NiFi 0.x with an external ZooKeeper to NiFi 1.x with the same external ZooKeeper
+* Migrating from an external ZooKeeper to an embedded ZooKeeper in NiFi 1.x
+
+The NiFi ZooKeeper Migrator (`zk-migrator.sh`) is part of the NiFi Toolkit and is downloaded separately from the
+link:https://nifi.apache.org/download.html[Apache NiFi download page].
+
+[[zk_migrator_command_line_parameters]]
+==== zk-migrator.sh Command Line Parameters
+
+You can use the following command line options with the ZooKeeper Migrator:
+
+* `-a,--auth <username:password>` Allows the specification of a username and password for authentication with ZooKeeper. This option is mutually exclusive with the `-k,--krb-conf` option.
+* `-f,--file <filename>` The file to be used for JSON data. When used with the `-r,--receive` option, data read from ZooKeeper will be stored in the given filename. When used with the `-s,--send` option, the data in the file will be sent to ZooKeeper.
+* `-h,--help` Prints help, displays available parameters with descriptions
+* `--ignore-source` Allows the ZooKeeper Migrator to write to the ZooKeeper and path from which the data was obtained.
+* `-k,--krb-conf <jaas-filename>` Allows the specification of a JAAS configuration file to allow authentication with a ZooKeeper configured to use Kerberos. This option is mutually exclusive with the `-a,--auth` option.
+* `-r,--receive` Receives data from ZooKeeper and writes to the given filename (if the `-f,--file` option is provided) or standard output. The data received will contain the full path to each node read from ZooKeeper. This option is mutually exclusive with the `-s,--send` option.
+* `-s,--send` Sends data to ZooKeeper that is read from the given filename (if the `-f,--file` option is provided) or standard input. The paths for each node in the data being sent to ZooKeeper are absolute paths, and will be stored in ZooKeeper under the *path* portion of the `-z,--zookeeper` argument. Typically, the *path* portion of the argument can be omitted, which will store the nodes at their absolute paths. This option is mutually exclusive with the `-r,--receive` option.
+* `-z,--zookeeper <zookeeper-endpoint>` The ZooKeeper server to use, specified by a connection string with path, in the format of _host:port/znode/path_.
+
+[[migrating_between_source_destination_zookeepers]]
+==== Migrating Between Source and Destination ZooKeepers
+
+Before you begin, confirm that:
+
+* You have installed the destination ZooKeeper cluster.
+* You have installed and configured a NiFi cluster to use the destination ZooKeeper cluster.
+* If you are migrating ZooKeepers due to upgrading NiFi from 0.x to 1.x,, you have already followed appropriate NiFi upgrade steps.
+* You have configured Kerberos as needed.
+* You have not started processing any dataflow (to avoid duplicate data processing).
+* If one of the ZooKeeper clusters you are using is configured with Kerberos, you are running the ZooKeeper Migrator from a host that has access to NiFi\u2019s ZooKeeper client jaas configuration file (see <<zk_kerberos_client>> for more information).
+
+1. Collect the following information:
+
+|====
+|*Required Information*|*Description*
+|Source ZooKeeper hostname (*sourceHostname*)|The hostname must be one of the hosts running in the ZooKeeper ensemble, which can be found in <NiFi installation dir>/conf/zookeeper.properties. Any of the hostnames declared in the *server.N* properties can be used.
+|Destination ZooKeeper hostname (*destinationHostname*)|The hostname must be one of the hosts running in the ZooKeeper ensemble, which can be found in <NiFi installation dir>/conf/zookeeper.properties. Any of the hostnames declared in the *server.N* properties can be used.
+|Source ZooKeeper port (*sourceClientPort*)|This can be found in *zookeeper.properties* of the <NiFi installation dir>/conf/zookeeper.properties. The port is specified in the *clientPort* property.
+|Destination ZooKeeper port (*destinationClientPort*)|This can be found in *zookeeper.properties* of the <NiFi installation dir>/conf/zookeeper.properties. The port is specified in the *clientPort* property.
+|Export data path|Determine the path that will store a json file containing the export of data from ZooKeeper. It must be readable and writable by the user running the zk-migrator tool.
+|Source ZooKeeper Authentication Information|This information is in <NiFi installation dir>/conf/state-management.xml. For NiFI 0.x, if Creator Only is specified in state-management.xml, you need to supply authentication information using the `-a,--auth` argument with the values from the Username and Password properties in state-management.xml. For NiFi 1.x, supply authentication information using the `-k,--krb-conf` argument.
+
+If the state-management.xml specifies Open, no authentication is required.
+|Destination ZooKeeper Authentication Information|This information is in <NiFi installation dir>/conf/state-management.xml. For NiFI 0.x, if Creator Only is specified in state-management.xml, you need to supply authentication information using the `-a,--auth` argument with the values from the Username and Password properties in state-management.xml. For NiFi 1.x, supply authentication information using the `-k,--krb-conf` argument.
--- End diff --
Capitalization wrong here again on NiFi
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1246: NIFI-3058 Add documentation for ZooKeeper Migrator ...
Posted by jtstorck <gi...@git.apache.org>.
Github user jtstorck commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1246#discussion_r88741590
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -1599,7 +1599,97 @@ java.arg.16=-Dsun.security.krb5.debug=true
This will cause the debug output to be written to the NiFi Bootstrap log file. By default, this is located at _$NIFI_HOME/logs/nifi-bootstrap.log_.
This output can be rather verbose but provides extremely valuable information for troubleshooting Kerberos failures.
+[[zookeeper_migrator]]
+=== ZooKeeper Migrator
+You can use the NiFi ZooKeeper Migrator to perform the following tasks:
+* Move ZooKeeper information from one ZooKeeper cluster to another
+* Migrating ZooKeeper node ownership
+
+For example, you may want to use the ZooKeeper Migrator when you are:
+
+* Upgrading from NiFi 0.x to NiFi 1.x in which embedded ZooKeepers are used
+* Migrating from an embedded ZooKeeper in NiFi 0.x or 1.x to an external ZooKeeper
+* Upgrading from NiFi 0.x with an external ZooKeeper to NiFi 1.x with the same external ZooKeeper
+* Migrating from an external ZooKeeper to an embedded ZooKeeper in NiFi 1.x
+
+The NiFi ZooKeeper Migrator (`zk-migrator.sh`) is part of the NiFi Toolkit and is downloaded separately from the
+link:https://nifi.apache.org/download.html[Apache NiFi download page].
+
+[[zk_migrator_command_line_parameters]]
+==== zk-migrator.sh Command Line Parameters
+
+You can use the following command line options with the ZooKeeper Migrator:
+
+* `-a,--auth <username:password>` Allows the specification of a username and password for authentication with ZooKeeper. This option is mutually exclusive with the `-k,--krb-conf` option.
+* `-f,--file <filename>` The file to be used for JSON data. When used with the `-r,--receive` option, data read from ZooKeeper will be stored in the given filename. When used with the `-s,--send` option, the data in the file will be sent to ZooKeeper.
+* `-h,--help` Prints help, displays available parameters with descriptions
+* `--ignore-source` Allows the ZooKeeper Migrator to write to the ZooKeeper and path from which the data was obtained.
+* `-k,--krb-conf <jaas-filename>` Allows the specification of a JAAS configuration file to allow authentication with a ZooKeeper configured to use Kerberos. This option is mutually exclusive with the `-a,--auth` option.
+* `-r,--receive` Receives data from ZooKeeper and writes to the given filename (if the `-f,--file` option is provided) or standard output. The data received will contain the full path to each node read from ZooKeeper. This option is mutually exclusive with the `-s,--send` option.
+* `-s,--send` Sends data to ZooKeeper that is read from the given filename (if the `-f,--file` option is provided) or standard input. The paths for each node in the data being sent to ZooKeeper are absolute paths, and will be stored in ZooKeeper under the *path* portion of the `-z,--zookeeper` argument. Typically, the *path* portion of the argument can be omitted, which will store the nodes at their absolute paths. This option is mutually exclusive with the `-r,--receive` option.
+* `-z,--zookeeper <zookeeper-endpoint>` The ZooKeeper server to use, specified by a connection string with path, in the format of _host:port/znode/path_.
+
+[[migrating_between_source_destination_zookeepers]]
+==== Migrating Between Source and Destination ZooKeepers
+
+Before you begin, confirm that:
+
+* You have installed the destination ZooKeeper cluster.
+* You have installed and configured a NiFi cluster to use the destination ZooKeeper cluster.
+* If you are migrating ZooKeepers due to upgrading NiFi from 0.x to 1.x,, you have already followed appropriate NiFi upgrade steps.
+* You have configured Kerberos as needed.
+* You have not started processing any dataflow (to avoid duplicate data processing).
+* If one of the ZooKeeper clusters you are using is configured with Kerberos, you are running the ZooKeeper Migrator from a host that has access to NiFi\u2019s ZooKeeper client jaas configuration file (see <<zk_kerberos_client>> for more information).
+
+1. Collect the following information:
--- End diff --
When the document is previewed (at least in Atom), the first step looks like it's indented along with the previouss section of the "Before you begin" steps. Perhaps adding a ZooKeeper Migration Steps header for this section?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1246: NIFI-3058 Add documentation for ZooKeeper Migrator ...
Posted by markap14 <gi...@git.apache.org>.
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1246#discussion_r88741022
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -1599,7 +1599,97 @@ java.arg.16=-Dsun.security.krb5.debug=true
This will cause the debug output to be written to the NiFi Bootstrap log file. By default, this is located at _$NIFI_HOME/logs/nifi-bootstrap.log_.
This output can be rather verbose but provides extremely valuable information for troubleshooting Kerberos failures.
+[[zookeeper_migrator]]
+=== ZooKeeper Migrator
+You can use the NiFi ZooKeeper Migrator to perform the following tasks:
+* Move ZooKeeper information from one ZooKeeper cluster to another
+* Migrating ZooKeeper node ownership
+
+For example, you may want to use the ZooKeeper Migrator when you are:
+
+* Upgrading from NiFi 0.x to NiFi 1.x in which embedded ZooKeepers are used
+* Migrating from an embedded ZooKeeper in NiFi 0.x or 1.x to an external ZooKeeper
+* Upgrading from NiFi 0.x with an external ZooKeeper to NiFi 1.x with the same external ZooKeeper
+* Migrating from an external ZooKeeper to an embedded ZooKeeper in NiFi 1.x
+
+The NiFi ZooKeeper Migrator (`zk-migrator.sh`) is part of the NiFi Toolkit and is downloaded separately from the
+link:https://nifi.apache.org/download.html[Apache NiFi download page].
+
+[[zk_migrator_command_line_parameters]]
+==== zk-migrator.sh Command Line Parameters
+
+You can use the following command line options with the ZooKeeper Migrator:
+
+* `-a,--auth <username:password>` Allows the specification of a username and password for authentication with ZooKeeper. This option is mutually exclusive with the `-k,--krb-conf` option.
+* `-f,--file <filename>` The file to be used for JSON data. When used with the `-r,--receive` option, data read from ZooKeeper will be stored in the given filename. When used with the `-s,--send` option, the data in the file will be sent to ZooKeeper.
+* `-h,--help` Prints help, displays available parameters with descriptions
+* `--ignore-source` Allows the ZooKeeper Migrator to write to the ZooKeeper and path from which the data was obtained.
+* `-k,--krb-conf <jaas-filename>` Allows the specification of a JAAS configuration file to allow authentication with a ZooKeeper configured to use Kerberos. This option is mutually exclusive with the `-a,--auth` option.
+* `-r,--receive` Receives data from ZooKeeper and writes to the given filename (if the `-f,--file` option is provided) or standard output. The data received will contain the full path to each node read from ZooKeeper. This option is mutually exclusive with the `-s,--send` option.
+* `-s,--send` Sends data to ZooKeeper that is read from the given filename (if the `-f,--file` option is provided) or standard input. The paths for each node in the data being sent to ZooKeeper are absolute paths, and will be stored in ZooKeeper under the *path* portion of the `-z,--zookeeper` argument. Typically, the *path* portion of the argument can be omitted, which will store the nodes at their absolute paths. This option is mutually exclusive with the `-r,--receive` option.
+* `-z,--zookeeper <zookeeper-endpoint>` The ZooKeeper server to use, specified by a connection string with path, in the format of _host:port/znode/path_.
+
+[[migrating_between_source_destination_zookeepers]]
+==== Migrating Between Source and Destination ZooKeepers
+
+Before you begin, confirm that:
+
+* You have installed the destination ZooKeeper cluster.
+* You have installed and configured a NiFi cluster to use the destination ZooKeeper cluster.
+* If you are migrating ZooKeepers due to upgrading NiFi from 0.x to 1.x,, you have already followed appropriate NiFi upgrade steps.
+* You have configured Kerberos as needed.
+* You have not started processing any dataflow (to avoid duplicate data processing).
+* If one of the ZooKeeper clusters you are using is configured with Kerberos, you are running the ZooKeeper Migrator from a host that has access to NiFi\u2019s ZooKeeper client jaas configuration file (see <<zk_kerberos_client>> for more information).
+
+1. Collect the following information:
+
+|====
+|*Required Information*|*Description*
+|Source ZooKeeper hostname (*sourceHostname*)|The hostname must be one of the hosts running in the ZooKeeper ensemble, which can be found in <NiFi installation dir>/conf/zookeeper.properties. Any of the hostnames declared in the *server.N* properties can be used.
+|Destination ZooKeeper hostname (*destinationHostname*)|The hostname must be one of the hosts running in the ZooKeeper ensemble, which can be found in <NiFi installation dir>/conf/zookeeper.properties. Any of the hostnames declared in the *server.N* properties can be used.
+|Source ZooKeeper port (*sourceClientPort*)|This can be found in *zookeeper.properties* of the <NiFi installation dir>/conf/zookeeper.properties. The port is specified in the *clientPort* property.
+|Destination ZooKeeper port (*destinationClientPort*)|This can be found in *zookeeper.properties* of the <NiFi installation dir>/conf/zookeeper.properties. The port is specified in the *clientPort* property.
+|Export data path|Determine the path that will store a json file containing the export of data from ZooKeeper. It must be readable and writable by the user running the zk-migrator tool.
+|Source ZooKeeper Authentication Information|This information is in <NiFi installation dir>/conf/state-management.xml. For NiFI 0.x, if Creator Only is specified in state-management.xml, you need to supply authentication information using the `-a,--auth` argument with the values from the Username and Password properties in state-management.xml. For NiFi 1.x, supply authentication information using the `-k,--krb-conf` argument.
--- End diff --
Capitalization wrong here again on NiFi
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1246: NIFI-3058 Add documentation for ZooKeeper Migrator ...
Posted by andrewmlim <gi...@git.apache.org>.
Github user andrewmlim commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1246#discussion_r88743987
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -1599,7 +1599,97 @@ java.arg.16=-Dsun.security.krb5.debug=true
This will cause the debug output to be written to the NiFi Bootstrap log file. By default, this is located at _$NIFI_HOME/logs/nifi-bootstrap.log_.
This output can be rather verbose but provides extremely valuable information for troubleshooting Kerberos failures.
+[[zookeeper_migrator]]
+=== ZooKeeper Migrator
+You can use the NiFi ZooKeeper Migrator to perform the following tasks:
+* Move ZooKeeper information from one ZooKeeper cluster to another
+* Migrating ZooKeeper node ownership
+
+For example, you may want to use the ZooKeeper Migrator when you are:
+
+* Upgrading from NiFi 0.x to NiFi 1.x in which embedded ZooKeepers are used
+* Migrating from an embedded ZooKeeper in NiFi 0.x or 1.x to an external ZooKeeper
+* Upgrading from NiFi 0.x with an external ZooKeeper to NiFi 1.x with the same external ZooKeeper
+* Migrating from an external ZooKeeper to an embedded ZooKeeper in NiFi 1.x
+
+The NiFi ZooKeeper Migrator (`zk-migrator.sh`) is part of the NiFi Toolkit and is downloaded separately from the
+link:https://nifi.apache.org/download.html[Apache NiFi download page].
+
+[[zk_migrator_command_line_parameters]]
+==== zk-migrator.sh Command Line Parameters
+
+You can use the following command line options with the ZooKeeper Migrator:
+
+* `-a,--auth <username:password>` Allows the specification of a username and password for authentication with ZooKeeper. This option is mutually exclusive with the `-k,--krb-conf` option.
+* `-f,--file <filename>` The file to be used for JSON data. When used with the `-r,--receive` option, data read from ZooKeeper will be stored in the given filename. When used with the `-s,--send` option, the data in the file will be sent to ZooKeeper.
+* `-h,--help` Prints help, displays available parameters with descriptions
+* `--ignore-source` Allows the ZooKeeper Migrator to write to the ZooKeeper and path from which the data was obtained.
+* `-k,--krb-conf <jaas-filename>` Allows the specification of a JAAS configuration file to allow authentication with a ZooKeeper configured to use Kerberos. This option is mutually exclusive with the `-a,--auth` option.
+* `-r,--receive` Receives data from ZooKeeper and writes to the given filename (if the `-f,--file` option is provided) or standard output. The data received will contain the full path to each node read from ZooKeeper. This option is mutually exclusive with the `-s,--send` option.
+* `-s,--send` Sends data to ZooKeeper that is read from the given filename (if the `-f,--file` option is provided) or standard input. The paths for each node in the data being sent to ZooKeeper are absolute paths, and will be stored in ZooKeeper under the *path* portion of the `-z,--zookeeper` argument. Typically, the *path* portion of the argument can be omitted, which will store the nodes at their absolute paths. This option is mutually exclusive with the `-r,--receive` option.
+* `-z,--zookeeper <zookeeper-endpoint>` The ZooKeeper server to use, specified by a connection string with path, in the format of _host:port/znode/path_.
+
+[[migrating_between_source_destination_zookeepers]]
+==== Migrating Between Source and Destination ZooKeepers
+
+Before you begin, confirm that:
+
+* You have installed the destination ZooKeeper cluster.
+* You have installed and configured a NiFi cluster to use the destination ZooKeeper cluster.
+* If you are migrating ZooKeepers due to upgrading NiFi from 0.x to 1.x,, you have already followed appropriate NiFi upgrade steps.
+* You have configured Kerberos as needed.
+* You have not started processing any dataflow (to avoid duplicate data processing).
+* If one of the ZooKeeper clusters you are using is configured with Kerberos, you are running the ZooKeeper Migrator from a host that has access to NiFi\u2019s ZooKeeper client jaas configuration file (see <<zk_kerberos_client>> for more information).
+
+1. Collect the following information:
+
+|====
+|*Required Information*|*Description*
+|Source ZooKeeper hostname (*sourceHostname*)|The hostname must be one of the hosts running in the ZooKeeper ensemble, which can be found in <NiFi installation dir>/conf/zookeeper.properties. Any of the hostnames declared in the *server.N* properties can be used.
+|Destination ZooKeeper hostname (*destinationHostname*)|The hostname must be one of the hosts running in the ZooKeeper ensemble, which can be found in <NiFi installation dir>/conf/zookeeper.properties. Any of the hostnames declared in the *server.N* properties can be used.
+|Source ZooKeeper port (*sourceClientPort*)|This can be found in *zookeeper.properties* of the <NiFi installation dir>/conf/zookeeper.properties. The port is specified in the *clientPort* property.
+|Destination ZooKeeper port (*destinationClientPort*)|This can be found in *zookeeper.properties* of the <NiFi installation dir>/conf/zookeeper.properties. The port is specified in the *clientPort* property.
+|Export data path|Determine the path that will store a json file containing the export of data from ZooKeeper. It must be readable and writable by the user running the zk-migrator tool.
+|Source ZooKeeper Authentication Information|This information is in <NiFi installation dir>/conf/state-management.xml. For NiFI 0.x, if Creator Only is specified in state-management.xml, you need to supply authentication information using the `-a,--auth` argument with the values from the Username and Password properties in state-management.xml. For NiFi 1.x, supply authentication information using the `-k,--krb-conf` argument.
+
+If the state-management.xml specifies Open, no authentication is required.
+|Destination ZooKeeper Authentication Information|This information is in <NiFi installation dir>/conf/state-management.xml. For NiFI 0.x, if Creator Only is specified in state-management.xml, you need to supply authentication information using the `-a,--auth` argument with the values from the Username and Password properties in state-management.xml. For NiFi 1.x, supply authentication information using the `-k,--krb-conf` argument.
--- End diff --
thanks
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1246: NIFI-3058 Add documentation for ZooKeeper Migrator ...
Posted by andrewmlim <gi...@git.apache.org>.
Github user andrewmlim commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1246#discussion_r88743900
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -1432,7 +1432,7 @@ If Kerberos is not already setup in your environment, you can find information o
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Managing_Smart_Cards/Configuring_a_Kerberos_5_Server.html[_https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Managing_Smart_Cards/Configuring_a_Kerberos_5_Server.html_]
. This guide assumes that Kerberos already has been installed in the environment in which NiFi is running.
-Note, the following procedures for kerberizing an Embedded Zookeeper server in your NiFI Node and kerberizing a zookeeper NiFI client will require that
+Note, the following procedures for kerberizing an Embedded ZooKeeper server in your NiFI Node and kerberizing a ZooKeeper NiFI client will require that
--- End diff --
Good catch.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1246: NIFI-3058 Add documentation for ZooKeeper Migrator ...
Posted by jtstorck <gi...@git.apache.org>.
Github user jtstorck commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1246#discussion_r88740943
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -1599,7 +1599,97 @@ java.arg.16=-Dsun.security.krb5.debug=true
This will cause the debug output to be written to the NiFi Bootstrap log file. By default, this is located at _$NIFI_HOME/logs/nifi-bootstrap.log_.
This output can be rather verbose but provides extremely valuable information for troubleshooting Kerberos failures.
+[[zookeeper_migrator]]
+=== ZooKeeper Migrator
+You can use the NiFi ZooKeeper Migrator to perform the following tasks:
+* Move ZooKeeper information from one ZooKeeper cluster to another
+* Migrating ZooKeeper node ownership
+
+For example, you may want to use the ZooKeeper Migrator when you are:
+
+* Upgrading from NiFi 0.x to NiFi 1.x in which embedded ZooKeepers are used
+* Migrating from an embedded ZooKeeper in NiFi 0.x or 1.x to an external ZooKeeper
+* Upgrading from NiFi 0.x with an external ZooKeeper to NiFi 1.x with the same external ZooKeeper
+* Migrating from an external ZooKeeper to an embedded ZooKeeper in NiFi 1.x
+
+The NiFi ZooKeeper Migrator (`zk-migrator.sh`) is part of the NiFi Toolkit and is downloaded separately from the
+link:https://nifi.apache.org/download.html[Apache NiFi download page].
+
+[[zk_migrator_command_line_parameters]]
+==== zk-migrator.sh Command Line Parameters
+
+You can use the following command line options with the ZooKeeper Migrator:
+
+* `-a,--auth <username:password>` Allows the specification of a username and password for authentication with ZooKeeper. This option is mutually exclusive with the `-k,--krb-conf` option.
+* `-f,--file <filename>` The file to be used for JSON data. When used with the `-r,--receive` option, data read from ZooKeeper will be stored in the given filename. When used with the `-s,--send` option, the data in the file will be sent to ZooKeeper.
--- End diff --
Suggest changing the first sentence to "The file used for ZooKeeper data serialized as JSON."
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1246: NIFI-3058 Add documentation for ZooKeeper Migrator ...
Posted by jtstorck <gi...@git.apache.org>.
Github user jtstorck commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1246#discussion_r88740216
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -1599,7 +1599,97 @@ java.arg.16=-Dsun.security.krb5.debug=true
This will cause the debug output to be written to the NiFi Bootstrap log file. By default, this is located at _$NIFI_HOME/logs/nifi-bootstrap.log_.
This output can be rather verbose but provides extremely valuable information for troubleshooting Kerberos failures.
+[[zookeeper_migrator]]
+=== ZooKeeper Migrator
+You can use the NiFi ZooKeeper Migrator to perform the following tasks:
+* Move ZooKeeper information from one ZooKeeper cluster to another
--- End diff --
Suggest changing to "Moving" instead of "Move"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1246: NIFI-3058 Add documentation for ZooKeeper Migrator ...
Posted by andrewmlim <gi...@git.apache.org>.
Github user andrewmlim commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1246#discussion_r88744920
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -1599,7 +1599,97 @@ java.arg.16=-Dsun.security.krb5.debug=true
This will cause the debug output to be written to the NiFi Bootstrap log file. By default, this is located at _$NIFI_HOME/logs/nifi-bootstrap.log_.
This output can be rather verbose but provides extremely valuable information for troubleshooting Kerberos failures.
+[[zookeeper_migrator]]
+=== ZooKeeper Migrator
+You can use the NiFi ZooKeeper Migrator to perform the following tasks:
+* Move ZooKeeper information from one ZooKeeper cluster to another
+* Migrating ZooKeeper node ownership
+
+For example, you may want to use the ZooKeeper Migrator when you are:
+
+* Upgrading from NiFi 0.x to NiFi 1.x in which embedded ZooKeepers are used
+* Migrating from an embedded ZooKeeper in NiFi 0.x or 1.x to an external ZooKeeper
+* Upgrading from NiFi 0.x with an external ZooKeeper to NiFi 1.x with the same external ZooKeeper
+* Migrating from an external ZooKeeper to an embedded ZooKeeper in NiFi 1.x
+
+The NiFi ZooKeeper Migrator (`zk-migrator.sh`) is part of the NiFi Toolkit and is downloaded separately from the
+link:https://nifi.apache.org/download.html[Apache NiFi download page].
+
+[[zk_migrator_command_line_parameters]]
+==== zk-migrator.sh Command Line Parameters
+
+You can use the following command line options with the ZooKeeper Migrator:
+
+* `-a,--auth <username:password>` Allows the specification of a username and password for authentication with ZooKeeper. This option is mutually exclusive with the `-k,--krb-conf` option.
+* `-f,--file <filename>` The file to be used for JSON data. When used with the `-r,--receive` option, data read from ZooKeeper will be stored in the given filename. When used with the `-s,--send` option, the data in the file will be sent to ZooKeeper.
+* `-h,--help` Prints help, displays available parameters with descriptions
+* `--ignore-source` Allows the ZooKeeper Migrator to write to the ZooKeeper and path from which the data was obtained.
+* `-k,--krb-conf <jaas-filename>` Allows the specification of a JAAS configuration file to allow authentication with a ZooKeeper configured to use Kerberos. This option is mutually exclusive with the `-a,--auth` option.
+* `-r,--receive` Receives data from ZooKeeper and writes to the given filename (if the `-f,--file` option is provided) or standard output. The data received will contain the full path to each node read from ZooKeeper. This option is mutually exclusive with the `-s,--send` option.
+* `-s,--send` Sends data to ZooKeeper that is read from the given filename (if the `-f,--file` option is provided) or standard input. The paths for each node in the data being sent to ZooKeeper are absolute paths, and will be stored in ZooKeeper under the *path* portion of the `-z,--zookeeper` argument. Typically, the *path* portion of the argument can be omitted, which will store the nodes at their absolute paths. This option is mutually exclusive with the `-r,--receive` option.
+* `-z,--zookeeper <zookeeper-endpoint>` The ZooKeeper server to use, specified by a connection string with path, in the format of _host:port/znode/path_.
+
+[[migrating_between_source_destination_zookeepers]]
+==== Migrating Between Source and Destination ZooKeepers
+
+Before you begin, confirm that:
+
+* You have installed the destination ZooKeeper cluster.
+* You have installed and configured a NiFi cluster to use the destination ZooKeeper cluster.
+* If you are migrating ZooKeepers due to upgrading NiFi from 0.x to 1.x,, you have already followed appropriate NiFi upgrade steps.
+* You have configured Kerberos as needed.
+* You have not started processing any dataflow (to avoid duplicate data processing).
+* If one of the ZooKeeper clusters you are using is configured with Kerberos, you are running the ZooKeeper Migrator from a host that has access to NiFi\u2019s ZooKeeper client jaas configuration file (see <<zk_kerberos_client>> for more information).
+
+1. Collect the following information:
--- End diff --
Good suggestion
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---