You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@nifi.apache.org by Darren Govoni <da...@ontrenet.com> on 2021/08/24 14:23:56 UTC

PYFI At-Scale Design

Hi!
  I've added a diagram[1] detailing the at-scale properties of PYFI for those who are interested. In a nutshell, PYFI scales at the processor level (vs. the flow), allowing you to have say, 50 processors of the same type running on 50 CPUs, whereas another (less intensive) processor in your flow, might only require 10 CPUs.

At-Scale means there is a 1-1 correspondence between logical and physical compute units.

[1] https://github.com/radiantone/pyfi#at-scale-design

If a data ingress or queue is particularly demanding, then scale only that processor. Furthermore, there are auto-scaling elastic qualities as well where PYFI will scale a processor based on temporary demand.

This design is inherently load-balanced at the processor level and has the following additional qualities:

  *   hardware redundancy
  *   high-availability
  *   fault-tolerance
  *   fail-over
  *   performance
  *   ease of maintenance

[cid:0375d7ee-31cf-439f-9273-2312d2325801]

Cheers!
Darren