You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@nifi.apache.org by andrewmlim <gi...@git.apache.org> on 2018/10/09 20:33:55 UTC

[GitHub] nifi pull request #3056: NIFI-5659 Add documentation for Offloading Nodes fu...

GitHub user andrewmlim opened a pull request:

    https://github.com/apache/nifi/pull/3056

    NIFI-5659 Add documentation for Offloading Nodes functionality and ne…

    …w Node related CLI commands

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/andrewmlim/nifi NIFI-5659

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/nifi/pull/3056.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3056
    
----
commit 80d49e2f15b84fd59ae83229bc05af5c9d8d1f5d
Author: Andrew Lim <an...@...>
Date:   2018-10-09T20:31:56Z

    NIFI-5659 Add documentation for Offloading Nodes functionality and new Node related CLI commands

----


---

[GitHub] nifi pull request #3056: NIFI-5659 Add documentation for Offloading Nodes fu...

Posted by asfgit <gi...@git.apache.org>.

Github user asfgit closed the pull request at:

    https://github.com/apache/nifi/pull/3056


---

[GitHub] nifi pull request #3056: NIFI-5659 Add documentation for Offloading Nodes fu...

Posted by andrewmlim <gi...@git.apache.org>.

Github user andrewmlim commented on a diff in the pull request:

https://github.com/apache/nifi/pull/3056#discussion_r224134708

--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -2393,19 +2395,53 @@ When the DFM makes changes to the dataflow, the node that receives the request t
nodes and waits for each node to respond, indicating that it has made the change on its local flow.

Good points. My intent was to present the available choices provided by the UI for a disconnected node. Perhaps we can add a note to the end of this section that provides more of this error handling/troubleshooting information.

NOTE: Not all nodes in a "Disconnected" state can be offloaded. If the node is disconnected because the node died/lack of heartbeat, the offload operation will be unable to reach the node to start the offloading. Additionally, offloading may also be interrupted or prevented due to firewall issues (e.g., if the load balancing port is blocked).

---

[GitHub] nifi pull request #3056: NIFI-5659 Add documentation for Offloading Nodes fu...

Posted by markap14 <gi...@git.apache.org>.

Github user markap14 commented on a diff in the pull request:

    https://github.com/apache/nifi/pull/3056#discussion_r224463328
  
    --- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
    @@ -3939,8 +3976,7 @@ to the cluster. It provides an additional layer of security. This value is blank
     |`nifi.cluster.flow.election.max.candidates`|Specifies the number of Nodes required in the cluster to cause early election of Flows. This allows the Nodes in the cluster to avoid having to wait a
     long time before starting processing if we reach at least this number of nodes in the cluster.
     |`nifi.cluster.load.balance.port`|Specifies the port to listen on for incoming connections for load balancing data across the cluster. The default value is `6342`.
    --- End diff --
    
    @jtstorck oops. I did mean for that to be 6432 in all places. I started with 6432, then when coding things up I remembered the value incorrectly, using 7430. I realized this and thought that I had changed all of them to 6432 but I must have missed a spot.


---

[GitHub] nifi pull request #3056: NIFI-5659 Add documentation for Offloading Nodes fu...

Posted by andrewmlim <gi...@git.apache.org>.

Github user andrewmlim commented on a diff in the pull request:

https://github.com/apache/nifi/pull/3056#discussion_r224156171

OK. How about this then for the note:

---

[GitHub] nifi pull request #3056: NIFI-5659 Add documentation for Offloading Nodes fu...

Posted by andrewmlim <gi...@git.apache.org>.

Github user andrewmlim commented on a diff in the pull request:

    https://github.com/apache/nifi/pull/3056#discussion_r225181107
  
    --- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
    @@ -3939,8 +3976,7 @@ to the cluster. It provides an additional layer of security. This value is blank
     |`nifi.cluster.flow.election.max.candidates`|Specifies the number of Nodes required in the cluster to cause early election of Flows. This allows the Nodes in the cluster to avoid having to wait a
     long time before starting processing if we reach at least this number of nodes in the cluster.
     |`nifi.cluster.load.balance.port`|Specifies the port to listen on for incoming connections for load balancing data across the cluster. The default value is `6342`.
    --- End diff --
    
    Just noting that the default value is 6342 not 6432.


---

[GitHub] nifi pull request #3056: NIFI-5659 Add documentation for Offloading Nodes fu...

Posted by jtstorck <gi...@git.apache.org>.

Github user jtstorck commented on a diff in the pull request:

https://github.com/apache/nifi/pull/3056#discussion_r224150506

I misspoke on the load balance port being blocked. That would prevent other nodes from sending flowfiles through load balancing to that particular node. If the disconnected node is unreachable, it would not be able to receive the offload request. I agree with moving the error scenarios to the end of the section, or to a separate error handling/troubleshooting section.

---

[GitHub] nifi pull request #3056: NIFI-5659 Add documentation for Offloading Nodes fu...

Posted by andrewmlim <gi...@git.apache.org>.

Github user andrewmlim commented on a diff in the pull request:

https://github.com/apache/nifi/pull/3056#discussion_r224161590

I like your version. Let's run with that.

---

[GitHub] nifi pull request #3056: NIFI-5659 Add documentation for Offloading Nodes fu...

Posted by jtstorck <gi...@git.apache.org>.

Github user jtstorck commented on a diff in the pull request:

https://github.com/apache/nifi/pull/3056#discussion_r223855961

For a node that’s disconnected due to lack of heartbeat, it can't offload in all disconnected scenarios. If it gets disconnected because the node died, obviously it won’t be able to be offloaded, but if the operation can’t reach the node to start the offloading, it won’t be able to start it and the UI should reflect that error. If it is disconnected due to firewall issues, that might effect offloading as well, if the load balancing port is also blocked by the firewall.

---

[GitHub] nifi pull request #3056: NIFI-5659 Add documentation for Offloading Nodes fu...

Posted by andrewmlim <gi...@git.apache.org>.

Github user andrewmlim commented on a diff in the pull request:

https://github.com/apache/nifi/pull/3056#discussion_r224099820

-*Dealing with Disconnected Nodes* +
+=== Managing Nodes
+
+==== Disconnect Nodes
+
+A DFM may manually disconnect a node from the cluster. A node may also become disconnected for other reasons, such as due to a lack of heartbeat. The Cluster Coordinator will show a bulletin on the User Interface when a node is disconnected. The DFM will not be able to make any changes to the dataflow until the issue of the disconnected node is resolved. The DFM or the Administrator will need to troubleshoot the issue with the node and resolve it before any new changes can be made to the dataflow. However, it is worth noting that just because a node is disconnected does not mean that it is not working. This may happen for a few reasons, for example when the node is unable to communicate with the Cluster Coordinator due to network problems.
+
+To manually disconnect a node, select the "Disconnect" icon (image:iconDisconnect.png["Disconnect Icon"]) from the node's row.
+
+image::disconnected-node-cluster-mgt.png["Disconnected Node in Cluster Management UI"]
+
+A disconnected node can be connected (image:iconConnect.png["Connect Icon"]), offloaded (image:iconOffload.png["Offload Icon"]) or deleted (image:iconDelete.png["Delete Icon"]).
+
+==== Offload Nodes
+
+Flowfiles that remain on a disconnected node can be rebalanced to other active nodes in the cluster via offloading. In the Cluster Management dialog, select the "Offload" icon (image:iconOffload.png["Offload Icon"]) for a Disconnected node. This will stop all processors, terminate all processors, stop transmitting on all remote process groups and rebalance flowfiles to the other connected nodes in the cluster.
+
+image::offloading-node-cluster-mgt.png["Offloading Node in Cluster Management UI"]
+
+Nodes that remain in "Offloading" state due to errors encountered (out of memory, no network connection, etc.) can be reconnected to the cluster by restarting NiFi on the node. Offloaded nodes can be either reconnected to the cluster (by selecting Connect or restarting NiFi on the node) or deleted from the cluster.
+
+image::offloaded-node-cluster-mgt.png["Offloaded Node in Cluster Management UI"]
+
+==== Delete Nodes
+
+There are cases where a DFM may wish to continue making changes to the flow, even though a node is not connected to the cluster. In this case, the DFM may elect to delete the node from the cluster entirely. In the Cluster Management dialog, select the "Delete" icon (image:iconDelete.png["Delete Icon"]) for a Disconnected or Offloaded node. Once deleted, the node cannot be rejoined to the cluster until it has been restarted.
+
+==== Decommission Nodes
+
+The steps to decommission a node and remove it from a cluster are as follows:
+
+1. Disconnect the node.
+2. Once disconnect completes, offload the node.
+3. Once offload completes, delete the node.
+4. Once the delete request has finished, stop/remove the NiFi service on the host.
+
+==== NiFi Toolkit Node Commands

-A DFM may manually disconnect a node from the cluster. But if a node becomes disconnected for any other reason (such as due to lack of heartbeat),
-the Cluster Coordinator will show a bulletin on the User Interface. The DFM will not be able to make any changes to the dataflow until the issue
-of the disconnected node is resolved. The DFM or the Administrator will need to troubleshoot the issue with the node and resolve it before any
-new changes may be made to the dataflow. However, it is worth noting that just because a node is disconnected does not mean that it is not working;
-this may happen for a few reasons, including that the node is unable to communicate with the Cluster Coordinator due to network problems.
+As an alternative to the UI, the following NiFi Toolkit CLI commands can be used for retrieving a single node, retrieving a list of nodes, and connecting/disconnecting/offloading/deleting nodes:

-There are cases where a DFM may wish to continue making changes to the flow, even though a node is not connected to the cluster.
-In this case, they DFM may elect to remove the node from the cluster entirely through the Cluster Management dialog. Once removed,
-the node cannot be rejoined to the cluster until it has been restarted.
+* `nifi get-node`
--- End diff --

Sounds good. Will definitely link commands covered in the Admin Guide to their corresponding sections in a Toolkit Guide.

---

[GitHub] nifi pull request #3056: NIFI-5659 Add documentation for Offloading Nodes fu...

Posted by jtstorck <gi...@git.apache.org>.

Github user jtstorck commented on a diff in the pull request:

https://github.com/apache/nifi/pull/3056#discussion_r223912939

Should these commands be more verbose? Adding examples including parameters for using the TLS Toolkit CLI commands securely would be beneficial. Also, talking about the response types would be useful, since the simple response type is more of a human readable summary, and the json response is the complete entity that's returned from the API endpoint.

---

[GitHub] nifi pull request #3056: NIFI-5659 Add documentation for Offloading Nodes fu...

Posted by jtstorck <gi...@git.apache.org>.

Github user jtstorck commented on a diff in the pull request:

https://github.com/apache/nifi/pull/3056#discussion_r223919080

I'm fine with having those details covered in the NiFi Toolkit Guide, especially if the commands mentioned here have links to their respective sections in the NiFi Toolkit Guide itself.

---

[GitHub] nifi pull request #3056: NIFI-5659 Add documentation for Offloading Nodes fu...

Posted by jtstorck <gi...@git.apache.org>.

Github user jtstorck commented on a diff in the pull request:

https://github.com/apache/nifi/pull/3056#discussion_r224164285

---

[GitHub] nifi pull request #3056: NIFI-5659 Add documentation for Offloading Nodes fu...

Posted by andrewmlim <gi...@git.apache.org>.

Github user andrewmlim commented on a diff in the pull request:

https://github.com/apache/nifi/pull/3056#discussion_r223917872

I see where you are coming from. As I noted in the related Jira (NIFI-5659), was planning on adding more in depth coverage of these specific commands and other CLI commands in a separate NiFI Toolkit Guide. Currently, no examples exist in the Admin Guide for the CLI commands and was trying to be consistent. Examples do exist in the ReadMe (https://github.com/apache/nifi/blob/master/nifi-toolkit/nifi-toolkit-cli/README.md) where the simple and json output types are also covered.

---

[GitHub] nifi pull request #3056: NIFI-5659 Add documentation for Offloading Nodes fu...

Posted by jtstorck <gi...@git.apache.org>.

Github user jtstorck commented on a diff in the pull request:

https://github.com/apache/nifi/pull/3056#discussion_r224160990

Maybe it would be better to be less explicit:

NOTE: Not all nodes in a "Disconnected" state can be offloaded. If the node is disconnected and unreachable, the offload request can not be received by the node to start the offloading. Additionally, offloading may be interrupted or prevented due to firewall rules.

---

[GitHub] nifi pull request #3056: NIFI-5659 Add documentation for Offloading Nodes fu...

Posted by jtstorck <gi...@git.apache.org>.

Github user jtstorck commented on a diff in the pull request:

    https://github.com/apache/nifi/pull/3056#discussion_r223914705
  
    --- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
    @@ -3939,8 +3976,7 @@ to the cluster. It provides an additional layer of security. This value is blank
     |`nifi.cluster.flow.election.max.candidates`|Specifies the number of Nodes required in the cluster to cause early election of Flows. This allows the Nodes in the cluster to avoid having to wait a
     long time before starting processing if we reach at least this number of nodes in the cluster.
     |`nifi.cluster.load.balance.port`|Specifies the port to listen on for incoming connections for load balancing data across the cluster. The default value is `6342`.
    --- End diff --
    
    This wasn't part of your PR, but I noticed that the default value of 6432 for nifi.cluster.load.balance.port is not technically correct.  There's maven filtering occurring in nifi-framework/nifi-resources/pom.xml that set the property during build-time to 7430. The resulting nifi.properties in nifi-assembly/target/nifi-1.8.0-SNAPSHOT-bin/nifi-1.8.0-SNAPSHOT/conf has:
    ```properties
    nifi.cluster.load.balance.port=7430
    ```
    The following code (from nifi-properties/src/main/java/org/apache/nifi/util/NiFiProperties.java) that reads this property will use the default of 6432 if the property is missing from nifi.properties:
    ```java
    public InetSocketAddress getClusterLoadBalanceAddress() {
        try {
            String address = getProperty(LOAD_BALANCE_ADDRESS);
            if (StringUtils.isBlank(address)) {
                address = getProperty(CLUSTER_NODE_ADDRESS);
            }
            if (StringUtils.isBlank(address)) {
                address = "localhost";
            }
    
            final int port = getIntegerProperty(LOAD_BALANCE_PORT, DEFAULT_LOAD_BALANCE_PORT);
            return InetSocketAddress.createUnresolved(address, port);
        } catch (final Exception e) {
            throw new RuntimeException("Invalid load balance address/port due to: " + e, e);
        }
    }
    ```


---

[GitHub] nifi pull request #3056: NIFI-5659 Add documentation for Offloading Nodes fu...

Posted by jtstorck <gi...@git.apache.org>.

Github user jtstorck commented on a diff in the pull request:

    https://github.com/apache/nifi/pull/3056#discussion_r223919361
  
    --- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
    @@ -3939,8 +3976,7 @@ to the cluster. It provides an additional layer of security. This value is blank
     |`nifi.cluster.flow.election.max.candidates`|Specifies the number of Nodes required in the cluster to cause early election of Flows. This allows the Nodes in the cluster to avoid having to wait a
     long time before starting processing if we reach at least this number of nodes in the cluster.
     |`nifi.cluster.load.balance.port`|Specifies the port to listen on for incoming connections for load balancing data across the cluster. The default value is `6342`.
    --- End diff --
    
    @markap14 Given the difference of the two values that are used for the default `nifi.cluster.load.balance.port` property, should one value be used in both places, or was the difference intentional?


---