You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Phillip Lord <ph...@gmail.com> on 2023/03/14 13:46:28 UTC
Cluster modification
Nifi Guardians,
Can someone explain the motivation behind the somewhat recent change(I
believe 1.17ish) that now allows users to make canvas changes to a cluster
that is missing a node. For instance a cluster that is normally 3/3 is for
whtvr reason now 2/3. Previously you would get a warning and you were
unable to make changes to the 2/3 clustered canvas. Which certainly
prevents future headaches.
Now you're allowed to make changes... meaning the node that was removed
from the cluster now has a flow that is no longer in-sync with the cluster
and you have to remove that node's flow before it's able to rejoin the
cluster. Was this intentional? I know there are a lot of things coming
down the pipe with Nifi 2.0 but looking to understand the thought process
behind this...
Thanks,
Phil
Re: Cluster modification
Posted by Joe Witt <jo...@gmail.com>.
Phil
Yes it should happen then too.
Thanks
On Tue, Mar 14, 2023 at 7:05 AM Phillip Lord <ph...@gmail.com> wrote:
> I see... Thanks Pierre. This all makes sense... I'm going to assume the
> reconnecting node uses the Cluster Coordinator to retrieve the latest
> flow-version. In Mark's video he demonstrated reconnecting a node through
> the UI. Should the same process occur when a node is not just
> disconnected, just the nifi on that node is actually stopped and requires
> restarting the node? So that on start-up the node ensures it's utilizing
> the "up-to-date" cluster flow?
>
> Thanks,
> Phil
>
> On Tue, Mar 14, 2023 at 9:56 AM Pierre Villard <
> pierre.villard.fr@gmail.com> wrote:
>
>> Hi Phil,
>>
>>
>>> Now you're allowed to make changes... meaning the node that was removed
>>> from the cluster now has a flow that is no longer in-sync with the cluster
>>> and you have to remove that node's flow before it's able to rejoin the
>>> cluster. Was this intentional? I know there are a lot of things coming
>>> down the pipe with Nifi 2.0 but looking to understand the thought process
>>> behind this...
>>>
>>
>> This is not completely correct. The node should be able to rejoin the
>> cluster and NiFi will automatically apply the flow changes to the rejoining
>> cluster so that the flow definition is in-sync again across the cluster.
>> This is definitely intentional to provide a better user experience and not
>> switch to read-only when a node is disconnected.
>>
>> Note that there are cases where the node would not be able to rejoin the
>> cluster in order to not cause any unwanted data loss: for example if a
>> connection is deleted in the cluster, and there is data in this connection
>> when the node tries to rejoin the cluster, we would not delete the data.
>>
>> I'm sure others can provide more details, but I also recommend watching
>> Mark's video on this topic:
>> https://youtu.be/8G6niPKntTc?t=709
>>
>> Thanks,
>> Pierre
>>
>> Le mar. 14 mars 2023 à 14:46, Phillip Lord <ph...@gmail.com> a
>> écrit :
>>
>>> Nifi Guardians,
>>>
>>> Can someone explain the motivation behind the somewhat recent change(I
>>> believe 1.17ish) that now allows users to make canvas changes to a cluster
>>> that is missing a node. For instance a cluster that is normally 3/3 is for
>>> whtvr reason now 2/3. Previously you would get a warning and you were
>>> unable to make changes to the 2/3 clustered canvas. Which certainly
>>> prevents future headaches.
>>>
>>> Now you're allowed to make changes... meaning the node that was removed
>>> from the cluster now has a flow that is no longer in-sync with the cluster
>>> and you have to remove that node's flow before it's able to rejoin the
>>> cluster. Was this intentional? I know there are a lot of things coming
>>> down the pipe with Nifi 2.0 but looking to understand the thought process
>>> behind this...
>>>
>>> Thanks,
>>> Phil
>>>
>>
Re: Cluster modification
Posted by Phillip Lord <ph...@gmail.com>.
I see... Thanks Pierre. This all makes sense... I'm going to assume the
reconnecting node uses the Cluster Coordinator to retrieve the latest
flow-version. In Mark's video he demonstrated reconnecting a node through
the UI. Should the same process occur when a node is not just
disconnected, just the nifi on that node is actually stopped and requires
restarting the node? So that on start-up the node ensures it's utilizing
the "up-to-date" cluster flow?
Thanks,
Phil
On Tue, Mar 14, 2023 at 9:56 AM Pierre Villard <pi...@gmail.com>
wrote:
> Hi Phil,
>
>
>> Now you're allowed to make changes... meaning the node that was removed
>> from the cluster now has a flow that is no longer in-sync with the cluster
>> and you have to remove that node's flow before it's able to rejoin the
>> cluster. Was this intentional? I know there are a lot of things coming
>> down the pipe with Nifi 2.0 but looking to understand the thought process
>> behind this...
>>
>
> This is not completely correct. The node should be able to rejoin the
> cluster and NiFi will automatically apply the flow changes to the rejoining
> cluster so that the flow definition is in-sync again across the cluster.
> This is definitely intentional to provide a better user experience and not
> switch to read-only when a node is disconnected.
>
> Note that there are cases where the node would not be able to rejoin the
> cluster in order to not cause any unwanted data loss: for example if a
> connection is deleted in the cluster, and there is data in this connection
> when the node tries to rejoin the cluster, we would not delete the data.
>
> I'm sure others can provide more details, but I also recommend watching
> Mark's video on this topic:
> https://youtu.be/8G6niPKntTc?t=709
>
> Thanks,
> Pierre
>
> Le mar. 14 mars 2023 à 14:46, Phillip Lord <ph...@gmail.com> a
> écrit :
>
>> Nifi Guardians,
>>
>> Can someone explain the motivation behind the somewhat recent change(I
>> believe 1.17ish) that now allows users to make canvas changes to a cluster
>> that is missing a node. For instance a cluster that is normally 3/3 is for
>> whtvr reason now 2/3. Previously you would get a warning and you were
>> unable to make changes to the 2/3 clustered canvas. Which certainly
>> prevents future headaches.
>>
>> Now you're allowed to make changes... meaning the node that was removed
>> from the cluster now has a flow that is no longer in-sync with the cluster
>> and you have to remove that node's flow before it's able to rejoin the
>> cluster. Was this intentional? I know there are a lot of things coming
>> down the pipe with Nifi 2.0 but looking to understand the thought process
>> behind this...
>>
>> Thanks,
>> Phil
>>
>
Re: Cluster modification
Posted by Pierre Villard <pi...@gmail.com>.
Hi Phil,
> Now you're allowed to make changes... meaning the node that was removed
> from the cluster now has a flow that is no longer in-sync with the cluster
> and you have to remove that node's flow before it's able to rejoin the
> cluster. Was this intentional? I know there are a lot of things coming
> down the pipe with Nifi 2.0 but looking to understand the thought process
> behind this...
>
This is not completely correct. The node should be able to rejoin the
cluster and NiFi will automatically apply the flow changes to the rejoining
cluster so that the flow definition is in-sync again across the cluster.
This is definitely intentional to provide a better user experience and not
switch to read-only when a node is disconnected.
Note that there are cases where the node would not be able to rejoin the
cluster in order to not cause any unwanted data loss: for example if a
connection is deleted in the cluster, and there is data in this connection
when the node tries to rejoin the cluster, we would not delete the data.
I'm sure others can provide more details, but I also recommend watching
Mark's video on this topic:
https://youtu.be/8G6niPKntTc?t=709
Thanks,
Pierre
Le mar. 14 mars 2023 à 14:46, Phillip Lord <ph...@gmail.com> a
écrit :
> Nifi Guardians,
>
> Can someone explain the motivation behind the somewhat recent change(I
> believe 1.17ish) that now allows users to make canvas changes to a cluster
> that is missing a node. For instance a cluster that is normally 3/3 is for
> whtvr reason now 2/3. Previously you would get a warning and you were
> unable to make changes to the 2/3 clustered canvas. Which certainly
> prevents future headaches.
>
> Now you're allowed to make changes... meaning the node that was removed
> from the cluster now has a flow that is no longer in-sync with the cluster
> and you have to remove that node's flow before it's able to rejoin the
> cluster. Was this intentional? I know there are a lot of things coming
> down the pipe with Nifi 2.0 but looking to understand the thought process
> behind this...
>
> Thanks,
> Phil
>