You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Alexis Sarda-Espinosa <sa...@gmail.com> on 2024/08/17 09:02:37 UTC
Draining queues during NiFi cluster restart in Kubernetes
Hello,
We use NiFi in cluster mode in Kubernetes via a StatefulSet. Sometimes, we
need to restart the cluster, so the StatefulSet restarts one pod at a time.
In this scenario, will a given node attempt to process all "in-flight"
FlowFiles to drain the queues as much as possible before shutting down?
I know it's probably impossible to have a completely graceful offload in
these cases, but since the process still receives a termination signal and
has some time before it must stop, it could try to prevent FlowFiles from
staying in the system for too long, but I don't know if there are any
mechanisms to handle this in NiFi.
Regards,
Alexis.
Re: Draining queues during NiFi cluster restart in Kubernetes
Posted by Alexis Sarda-Espinosa <sa...@gmail.com>.
The main scenarios for us are upgrading NiFi and updating (and consequently
restarting) the Kubernetes workers for security updates. Both of these make
manual offloading basically impossible - the StatefulSet handles pod
restarts by itself, and the Kubernetes control plane drains and restarts
worker machines automatically.
Regards,
Alexis.
On Sat, 17 Aug 2024, 17:40 Joe Witt, <jo...@gmail.com> wrote:
> Alexis
>
> Yep we totally understand.
>
> What scenarios are you trying to plan/design for that require restarts?
>
> Thanks
>
> On Sat, Aug 17, 2024 at 8:24 AM Alexis Sarda-Espinosa <
> sarda.espinosa@gmail.com> wrote:
>
>> Hi Joe,
>>
>> I agree the system can recover after the restart, but we want to deliver
>> data with as little latency as possible, so if FlowFiles stay queued during
>> the restart time window, by the time the node is up and running again the
>> data will be very late when time it finally leaves the NiFi cluster.
>>
>> Regards,
>> Alexis.
>>
>> On Sat, 17 Aug 2024, 16:59 Joe Witt, <jo...@gmail.com> wrote:
>>
>>> Alexis
>>>
>>>
>>> If you are simply restarting nodes you dont need to ensure everything is
>>> processed first. It will recover.
>>>
>>> If you want to scale down by removing a node there is an api to invoke
>>> offloading so you could scale down to a single node even if needed.
>>>
>>> Thanks
>>>
>>> On Sat, Aug 17, 2024 at 2:02 AM Alexis Sarda-Espinosa <
>>> sarda.espinosa@gmail.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> We use NiFi in cluster mode in Kubernetes via a StatefulSet. Sometimes,
>>>> we need to restart the cluster, so the StatefulSet restarts one pod at a
>>>> time. In this scenario, will a given node attempt to process all
>>>> "in-flight" FlowFiles to drain the queues as much as possible before
>>>> shutting down?
>>>>
>>>> I know it's probably impossible to have a completely graceful offload
>>>> in these cases, but since the process still receives a termination signal
>>>> and has some time before it must stop, it could try to prevent FlowFiles
>>>> from staying in the system for too long, but I don't know if there are any
>>>> mechanisms to handle this in NiFi.
>>>>
>>>> Regards,
>>>> Alexis.
>>>>
>>>
Re: Draining queues during NiFi cluster restart in Kubernetes
Posted by Joe Witt <jo...@gmail.com>.
Alexis
Yep we totally understand.
What scenarios are you trying to plan/design for that require restarts?
Thanks
On Sat, Aug 17, 2024 at 8:24 AM Alexis Sarda-Espinosa <
sarda.espinosa@gmail.com> wrote:
> Hi Joe,
>
> I agree the system can recover after the restart, but we want to deliver
> data with as little latency as possible, so if FlowFiles stay queued during
> the restart time window, by the time the node is up and running again the
> data will be very late when time it finally leaves the NiFi cluster.
>
> Regards,
> Alexis.
>
> On Sat, 17 Aug 2024, 16:59 Joe Witt, <jo...@gmail.com> wrote:
>
>> Alexis
>>
>>
>> If you are simply restarting nodes you dont need to ensure everything is
>> processed first. It will recover.
>>
>> If you want to scale down by removing a node there is an api to invoke
>> offloading so you could scale down to a single node even if needed.
>>
>> Thanks
>>
>> On Sat, Aug 17, 2024 at 2:02 AM Alexis Sarda-Espinosa <
>> sarda.espinosa@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> We use NiFi in cluster mode in Kubernetes via a StatefulSet. Sometimes,
>>> we need to restart the cluster, so the StatefulSet restarts one pod at a
>>> time. In this scenario, will a given node attempt to process all
>>> "in-flight" FlowFiles to drain the queues as much as possible before
>>> shutting down?
>>>
>>> I know it's probably impossible to have a completely graceful offload in
>>> these cases, but since the process still receives a termination signal and
>>> has some time before it must stop, it could try to prevent FlowFiles from
>>> staying in the system for too long, but I don't know if there are any
>>> mechanisms to handle this in NiFi.
>>>
>>> Regards,
>>> Alexis.
>>>
>>
Re: Draining queues during NiFi cluster restart in Kubernetes
Posted by Alexis Sarda-Espinosa <sa...@gmail.com>.
Hi Joe,
I agree the system can recover after the restart, but we want to deliver
data with as little latency as possible, so if FlowFiles stay queued during
the restart time window, by the time the node is up and running again the
data will be very late when time it finally leaves the NiFi cluster.
Regards,
Alexis.
On Sat, 17 Aug 2024, 16:59 Joe Witt, <jo...@gmail.com> wrote:
> Alexis
>
>
> If you are simply restarting nodes you dont need to ensure everything is
> processed first. It will recover.
>
> If you want to scale down by removing a node there is an api to invoke
> offloading so you could scale down to a single node even if needed.
>
> Thanks
>
> On Sat, Aug 17, 2024 at 2:02 AM Alexis Sarda-Espinosa <
> sarda.espinosa@gmail.com> wrote:
>
>> Hello,
>>
>> We use NiFi in cluster mode in Kubernetes via a StatefulSet. Sometimes,
>> we need to restart the cluster, so the StatefulSet restarts one pod at a
>> time. In this scenario, will a given node attempt to process all
>> "in-flight" FlowFiles to drain the queues as much as possible before
>> shutting down?
>>
>> I know it's probably impossible to have a completely graceful offload in
>> these cases, but since the process still receives a termination signal and
>> has some time before it must stop, it could try to prevent FlowFiles from
>> staying in the system for too long, but I don't know if there are any
>> mechanisms to handle this in NiFi.
>>
>> Regards,
>> Alexis.
>>
>
Re: Draining queues during NiFi cluster restart in Kubernetes
Posted by Joe Witt <jo...@gmail.com>.
Alexis
If you are simply restarting nodes you dont need to ensure everything is
processed first. It will recover.
If you want to scale down by removing a node there is an api to invoke
offloading so you could scale down to a single node even if needed.
Thanks
On Sat, Aug 17, 2024 at 2:02 AM Alexis Sarda-Espinosa <
sarda.espinosa@gmail.com> wrote:
> Hello,
>
> We use NiFi in cluster mode in Kubernetes via a StatefulSet. Sometimes, we
> need to restart the cluster, so the StatefulSet restarts one pod at a time.
> In this scenario, will a given node attempt to process all "in-flight"
> FlowFiles to drain the queues as much as possible before shutting down?
>
> I know it's probably impossible to have a completely graceful offload in
> these cases, but since the process still receives a termination signal and
> has some time before it must stop, it could try to prevent FlowFiles from
> staying in the system for too long, but I don't know if there are any
> mechanisms to handle this in NiFi.
>
> Regards,
> Alexis.
>