[DISCUSS] Future Support for Python Processors?

[David Handermann <ex...@apache.org>]

Team,

The introduction of support for Python Processors was one of the major
new features of Apache NiFi 2. In light of experience, however, I
believe it is time to reconsider future support for this extension
strategy. The idea of removing this feature may raise some concerns,
but it is important to consider the options based on scalability,
security, and supportability.

One of the greatest benefits of Python-based Processors is rapid
iteration. Although this is a strength during the development phase,
it can become a serious weakness when moving to production.

NiFi is fundamentally a Java-based application, thus tuning resource
allocation is a key part of maintaining a production system. Running a
Python Processor requires one or more independent Python processes,
each with their own memory consumption and CPU usage patterns. What
might work at a small scale on a development machine breaks down when
attempting to run at production scale, without careful tuning. This is
challenging enough with Java-based components, but global settings
like heap and non-heap memory provide clear boundaries. Python
Processors, as currently implemented, do not support this level of
control. Beyond process control, operations that require communicating
between the NiFi JVM and Python processes involve socket
communication, which can be expensive for frequent calls. Thus an
apparent initial advantage of Python Processors becomes a major
problem when moving beyond limited types of operations.

When it comes to security, Python Processors can involve more serious
risks than bundled Java extensions.

With NiFi support for downloading dependencies at runtime, it is
essential for an administrator to provide a secure supply chain
strategy for retrieving these dependencies. Although the NiFi
framework also supports packaging Python Processors as NAR bundles,
this packaging capability is not currently part of the Apache NiFi
project. The project could build on third party work, but as Python
packaging is less opinionated, this is an additional maintenance
question that quickly becomes more involved. This combination of
supply chain security and lack of packaging support raises security
challenges.

Supportability is always a question for any feature in an open source
project, but Python Processors are a particular point of concern.
Providing NiFi framework support requires a strong knowledge of the
NiFi project itself, and a strong knowledge of Python. Project
maintainers and contributors usually have expertise in particular
areas, but it is rare to find both. More to the point, however, the
Py4J project, which NiFi depends on for integration, has seen limited
maintenance, with the last tagged version dating to January 2025.
Without robust architecture support, the NiFi project cannot provide a
strong framework for future implementation.

Taking these issues together, I believe the project should consider
deprecating support for Python Processors. Similar to other
deprecations, actual removal would not occur until a major release
version, and that would be a separate conversation.

Rather than continuing to extend the life of Python Processor support,
it seems better to invest limited project resources into maintaining
and improving core functionality.

Regards,
David Handermann
Apache NiFi PMC Chair

Re: [DISCUSS] Future Support for Python Processors?

[Mike Thomsen <mi...@gmail.com>]

Last night, I did an audit of the Python support with Claude, and one of
the top recommendations it made was to refactor the current implementation
so that it deliberately uses as few calls back to NiFi as possible. It
highlighted an example where the attributes map was built iteratively
rather than massing a dict type back to Java for an equivalent of
putAllAttributes.

It also suggested that we pass back the complete updated state of the
flowfile all at once like this:

{
  "relationship": "success",
  "attributes": { "mime.type": "application/json", ... },

  # transform left content alone — keep the input claim, no bytes moved:
  "content": { "kind": "unchanged" }

  # small rewritten content — the bytes getContents() returns today,
  # folded into this one reply:
  # "content": { "kind": "inline", "data": "<base64>" }

  # large rewritten content — Python wrote it to the spool dir; the JVM
  # imports the file and unlinks it:
  # "content": { "kind": "spooled",
  #              "path": "<spoolDir>/<invocationId>", "length": 5242880 }
}

That would likely require breaking API changes, but it seems like a viable
way forward to bring the Python support much closer to the Java side.

On Tue, Aug 18, 2026 at 10:51 PM Ryan Hendrickson <
ryan.andrew.hendrickson@gmail.com> wrote:

> The team I work with uses Python processors when (1) the libraries are
> unique to python, (2) the developer prefers Python over Java, and (3) raw
> data processing speed is not a factor.  When speed is a factor, we have:
> (1) Swapped to side-car containers deployed next to NiFi hosting the python
> code with a rest interface for interaction [1]; OR (2) Converted the
> processor to either Groovy or pure Java.
>
> Looking back, it seemed developer preference for Python (with the rapid
> development cycle of python code) and unique libraries in Python drove the
> desire to use Python in NiFi and the adoption of Python-based data
> processing capabilities within our team.  While the integration isn't
> perfect in NiFi, it does check a box when evaluating the flexibility of
> major programming languages supported in the core NiFi project/ecosystem.
>
> We have long looked at ways to performantly call non-Java code (typically
> Python) from NiFi, using NiFi as the over-arching dataflow governance
> mechanism.  Marc P's comment on "governed NiFi dataflow" is accurate and
> desired [2][3].  The dataflow UI is beautiful, highly functional, and very
> simple.  We've also looked at ways to scale specific processors
> independently of the total resources provided to NiFi, notably to isolate
> resources directly to a processor or process group to adjust for less
> performant code on vertically scaled infrastructure.
>
> My thoughts have diverged from the Python topic, but two common themes
> involving Python tend to be (1) dataflows governed by NiFi and (2)
> performance.
>
> [1] https://issues.apache.org/jira/browse/NIFI-9026 | Integration with
> auto-scaling container environments
> [2] https://issues.apache.org/jira/browse/NIFI-9028 | Large Canvas
> Management
> [3] https://issues.apache.org/jira/browse/NIFI-9027 | Integration of S3
> buckets with Remote Process Groups for outage events and data overruns
>
> I appreciate the community's hard work in maintaining NiFi and transparent
> communication style.  I look forward to the direction NiFi moves.
>
> Best,
> Ryan
>
> On Mon, Aug 17, 2026 at 10:14 AM David Handermann <
> exceptionfactory@apache.org> wrote:
>
> > Gabor,
> >
> > Thanks for the reply and additional perspective from the MiNiFi C++ work.
> >
> > As you highlighted, the scalability and security issues became more
> > apparent after release and initial adoption, which is a natural part
> > of the development process. I agree with you that the Python Processor
> > API presents a simplified structure, which has a lot of advantages for
> > basic use cases. Deprecating the API certainly would discourage use of
> > Python Processors, and that is part of the purpose in setting a
> > maintainable direction. The scalability concern is not something that
> > can be solved through documentation, given the inherent problems with
> > independent process management. That is the reason the current
> > approach needs to be deprecated, not documented as a limitation.
> >
> > As mentioned earlier, deprecating the current implementation does not
> > preclude future alternatives. Instead, it provides a clear signal to
> > users of project direction. It will cause challenges for current
> > adopters, but the standard deprecation and removal process supplies
> > that transitional notification strategy.
> >
> > It is notable that the MiNiFi C++ integration approach provides a
> > different avenue, given the fundamentally different architecture. It
> > is also worth clarifying that the deprecation would apply to the main
> > NiFi project, not necessarily to the MiNiFi C++ application.
> >
> > To be clear, this discussion is not intended to minimize the
> > usefulness of Python Processors in selected situations. The main
> > purpose is to be clear among project maintainers about what is
> > sustainable, both from a development perspective, and from a
> > production quality perspective. As everyone thus far has acknowledged
> > the severe scaling limitations of the current architecture, that
> > highlights the importance of moving in a different direction.
> >
> > Based on this discussion thus far, I'm evaluating some related work on
> > a different scripted Processor architecture. The current scripted
> > Processors use the generic Java ScriptEngine, but the structure
> > requires supporting languages within the project itself. I'm
> > considering an approach that would allow users to bring their own
> > ScriptEngine implementation. This kind of decoupling may provide a
> > more supportable strategy for the project itself, while enabling users
> > to build their own extensions based on desired language. Even GraalPy,
> > which supports Python 3 and some native libraries, has limitations,
> > but it might provide a way forward beyond simple deprecation. I'm
> > still exploring the options.
> >
> > With that background, unmaintained capabilities and problematic
> > scaling considerations are something the project must address, so I
> > appreciate the interaction on the topic from multiple perspectives.
> >
> > Regards,
> > David Handermann
> >
> > On Mon, Aug 17, 2026 at 3:36 AM Gábor Gyimesi <lo...@apache.org>
> > wrote:
> > >
> > > Hi,
> > >
> > > I also agree with Marc on this. From the user interactions, I got the
> > > feeling that Python processors are quite valuable to the community,
> > > and I would be hesitant to deprecate and remove them when the main
> > > problems are related to the underlying implementation.
> > >
> > > I agree with several of the problems David mentioned, particularly
> > > around scalability and dependency installation. These issues were not
> > > foreseen with the initial release of NiFi 2.0 and only became apparent
> > > as these processors started being used more extensively by the
> > > community, especially with heavy AI libraries. The problem is that,
> > > after the initial release of 2.0, there has been very little
> > > development on the Python side, so these issues were never addressed.
> > >
> > > From a user's point of view, I think the Python processor API is
> > > simple and easy to use. Even with the current implementation, there
> > > are several use cases for which it is sufficient, so I would not
> > > remove the option to use Python processors. Deprecating them would
> > > discourage people from creating and using Python processors, even for
> > > use cases where they make sense. In my opinion, the documentation for
> > > Python processors should instead be extended to clearly explain the
> > > caveats of using them, including what problems users can expect in
> > > terms of dependency management, scalability, etc., and which use cases
> > > are recommended for these processors.
> > >
> > > In the future, I would support developing an alternative
> > > implementation under the hood, replacing Py4J, if we decide to pursue
> > > IPC communication, use an embedded interpreter, or adopt any of the
> > > other solutions Matt mentioned. In MiNiFi C++, we went with an
> > > embedded interpreter solution, which provides much better performance,
> > > with the caveat that only one interpreter can be embedded in a MiNiFi
> > > process and is therefore shared between processors. The dependency
> > > management issue is still a challenge in MiNiFi C++ as well, and I
> > > would support creating a packaging solution that could work for both
> > > projects.
> > >
> > > As Marc said, from a user's point of view, in many cases it is not
> > > really an option to rewrite a processor in Java, as many libraries,
> > > especially AI libraries, are only available in Python. If the decision
> > > is made to deprecate and eventually remove Python processor support
> > > from NiFi, forcing users to create a separate Python microservice
> > > based on Marc's suggestion of a "Apache NiFi -> HTTP/gRPC -> Python
> > > microservice -> NiFi" architecture, we could consider officially
> > > recommending MiNiFi C++ as an interim solution, since it retains
> > > Python support. The same approach could potentially be used for
> > > anything requiring C/C++ or Rust support as well.
> > >
> > > Regards,
> > > Gabor
> > >
> > > On Thu, 13 Aug 2026 at 21:51, Mark Payne <ma...@hotmail.com> wrote:
> > > >
> > > > Kevin,
> > > >
> > > > Py4J is the glue that bridges the NiFi JVM to the python process that
> > runs python processors. So there’s a py4j server that is run in a
> separate
> > python process. Then a py4j client that runs in the JVM. Py4J handles the
> > RPC to make it possible to call the Python processors as if they were
> Java
> > objects.
> > > >
> > > > Thank
> > > > -Makr
> > > >
> > > >
> > > > > On Aug 13, 2026, at 3:37 PM, Kevin Doran <kd...@apache.org>
> wrote:
> > > > >
> > > > > Can someone clarify where Py4J is used today vs where subprocesses
> > > > > that invoke a configurable python executable are used? My
> > > > > understanding was that the Py4J was mostly restricted to the
> > scripting
> > > > > processors, but if that is not the case I would be interested in
> > > > > hearing more.
> > > > >
> > > > > On Thu, Aug 13, 2026 at 1:42 PM David Handermann
> > > > > <ex...@apache.org> wrote:
> > > > >>
> > > > >> Dan,
> > > > >>
> > > > >> Thanks for the substantive reply and consideration of the various
> > > > >> issues surrounding support for native Python Processors.
> > > > >>
> > > > >> As you highlighted, the scalability issue is foundational.
> Although
> > > > >> Processor authors can work with self-imposed constraints, the NiFi
> > > > >> framework does not provide the same level of safeguards. Any
> future
> > > > >> solution would need to account for these types of production-level
> > > > >> concerns.
> > > > >>
> > > > >> Regarding supply chain security, you are correct that there are
> > > > >> potential paths forward on the packaging strategy. The framework
> > could
> > > > >> remove support for runtime loading as a safety measure. Right now,
> > > > >> lack of pip or uv effectively disables runtime loading. Adjusting
> > the
> > > > >> default project container configuration would strengthen the
> > standard
> > > > >> position, and thus require Python Processors to be packaged using
> a
> > > > >> NAR. The project could provide tooling, similar to the NiFi NAR
> > Maven
> > > > >> Plugin, but this broadens the maintenance scope in a meaningful
> way.
> > > > >> Thus before going in this direction, the future support strategy
> > needs
> > > > >> to be considered.
> > > > >>
> > > > >> The Py4J architecture remains a critical concern and high risk for
> > > > >> future maintainability. Considering an alternative architecture
> > would
> > > > >> require a substantial redesign and rewrite of Python Processor
> > > > >> support, abandoning a majority of the current implementation.
> > Current
> > > > >> tooling makes this more plausible in theory, but is a significant
> > > > >> undertaking that is difficult to justify without serious
> commitment
> > on
> > > > >> the part of multiple project maintainers. To be clear, deprecating
> > the
> > > > >> current implementation for removal does not preclude future
> > redesign.
> > > > >> These two questions should be considered independently.
> > > > >>
> > > > >> Deprecation and removal without replacement will have a different
> > > > >> impact depending on the current scope of usage. Simpler Python
> > > > >> Processors should be easier to convert to Java, but more complex
> > > > >> implementations would likely require something outside of NiFi, as
> > you
> > > > >> mentioned. Although these are difficult questions, it ultimately
> > comes
> > > > >> down to project focus and available maintenance cycles. Apache
> NiFi
> > > > >> has historically done many things. NiFi 2 served as a focusing
> > > > >> function. Going forward, maintaining project health and stability
> is
> > > > >> paramount, which may at times involve removing certain
> capabilities.
> > > > >>
> > > > >> Regards,
> > > > >> David Handermann
> > > > >>
> > > > >> On Wed, Aug 12, 2026 at 3:27 PM Daniel Weidman via dev
> > > > >> <de...@nifi.apache.org> wrote:
> > > > >>>
> > > > >>> David,
> > > > >>>
> > > > >>> Thanks for raising this directly rather than letting it drift,
> and
> > for laying
> > > > >>> out the reasoning rather than just the conclusion.
> > > > >>>
> > > > >>> On scalability, your point lands. In my experience the memory
> > behavior of a
> > > > >>> Python Processor is largely a function of how it is written, and
> > disciplined
> > > > >>> implementations can be well behaved. But I recognize that is an
> > argument about
> > > > >>> good citizenship, not about enforcement, and it does not give an
> > operator the
> > > > >>> equivalent of a heap ceiling. That is a real gap. It does seem
> > addressable at
> > > > >>> the process supervision layer, through cgroup or rlimit bounds on
> > spawned
> > > > >>> interpreters, rather than being inherent to Python as an
> extension
> > language.
> > > > >>>
> > > > >>> On security, I would offer a friendly amendment. The meaningful
> > difference does
> > > > >>> not seem to be Python versus Java, but when and where third party
> > dependencies
> > > > >>> land. A NAR resolves its tree at build time and ships self
> > contained, so it can
> > > > >>> be scanned, signed, and deployed into an air gapped environment,
> > and unpacking
> > > > >>> it executes nothing. The current Python path resolves
> dependencies
> > at runtime
> > > > >>> via pip on the NiFi host, and pip installation does execute
> > packaging code.
> > > > >>> That is a real governance gap, and I will concede PyPI carries
> > risks Maven
> > > > >>> Central does not.
> > > > >>>
> > > > >>> But as you noted, the framework already supports loading Python
> > Processors
> > > > >>> packaged as NARs. What is missing is project-provided tooling to
> > build them.
> > > > >>> Sean's note about relying on a blog post to produce one seems
> > telling. Closing
> > > > >>> that gap looks like a smaller undertaking than removing the
> > extension point,
> > > > >>> and it would address the supply chain concern directly rather
> than
> > by
> > > > >>> subtraction.
> > > > >>>
> > > > >>> I will admit I had not realized the current implementation still
> > rests on Py4J.
> > > > >>> That reframes a good deal of this for me, and it tracks with
> > Matt's read that
> > > > >>> Py4J is the long pole rather than the capability itself.
> > > > >>>
> > > > >>> For what it is worth as an operator, the practical value of
> Python
> > Processors
> > > > >>> is not just iteration speed. It is that the people writing our
> > transforms are
> > > > >>> Python native. Removal without a replacement does not convert
> that
> > work to
> > > > >>> Java, it pushes it out of NiFi entirely, to sidecars or external
> > services,
> > > > >>> where, as Marc noted, we keep every one of the concerns you
> listed
> > but lose
> > > > >>> NiFi's visibility into them.
> > > > >>>
> > > > >>> I would rather see the runtime architecture reconsidered than the
> > capability
> > > > >>> retired. If the deciding constraint is maintainer cycles, it
> would
> > be useful to
> > > > >>> know what a replacement would need in terms of effort and
> > expertise. That is a
> > > > >>> more actionable ask of the community than a deprecation vote.
> > > > >>>
> > > > >>> Regards,
> > > > >>> Dan
> > > > >>>
> > > > >>>
> > > > >>> From: David Handermann <ex...@apache.org>
> > > > >>> Date: Wednesday, August 12, 2026 at 9:10 AM
> > > > >>> To: dev@nifi.apache.org <de...@nifi.apache.org>
> > > > >>> Subject: Re: [DISCUSS] Future Support for Python Processors?
> > > > >>>
> > > > >>> Thanks to everyone for the replies thus far!
> > > > >>>
> > > > >>> The question about alternatives gets at the core issue, and the
> > ideas
> > > > >>> around a more loosely coupled architecture provide some helpful
> > > > >>> possibilities.
> > > > >>>
> > > > >>> At a basic level, simple deprecation and removal would leave the
> > > > >>> current set of options, including standard Java, Groovy
> scripting,
> > or
> > > > >>> externalized Python scripts executed as a stream. None of these
> > > > >>> provide the same level of features for Python itself, which
> > prompted
> > > > >>> the original native Python Processor implementation. Without a
> > > > >>> replacement, these alternatives would remain. With the rise of
> > > > >>> AI-enabled development, it is now much easier to go from Python
> to
> > > > >>> Java, but work is still required.
> > > > >>>
> > > > >>> As to future strategies for Python integration, the current
> > discussion
> > > > >>> around deprecation is more narrowly focused on the existing
> > > > >>> framework-level extension. Removal of the current implementation
> is
> > > > >>> not intended to halt future innovation. Although removal without
> an
> > > > >>> initial replacement might not be the perfect solution, it may be
> > > > >>> necessary given available maintainer cycles, and the substantive
> > > > >>> issues I described in the opening.
> > > > >>>
> > > > >>> If there is sufficient interest and effort to propose and build a
> > > > >>> different solution, that would be a great outcome of this
> > discussion.
> > > > >>> I've thought at different points about reworking the Script
> Engine
> > > > >>> integration to provide a cleaner path to external extension, so
> > there
> > > > >>> are definitely opportunities in that direction.
> > > > >>>
> > > > >>> With that, my primary concerns with the current implementation
> > remain,
> > > > >>> but glad to consider future alternatives.
> > > > >>>
> > > > >>> Regards,
> > > > >>> David Handermann
> > > > >>>
> > > > >>>
> > > > >>> On Wed, Aug 12, 2026 at 7:52 AM Matt Burgess <
> mattyb149@apache.org>
> > wrote:
> > > > >>>>
> > > > >>>> I agree with Marc, the concerns of the implementation of the
> > capability
> > > > >>>> should not warrant deprecation and future removal of the
> > capability, but
> > > > >>>> certainly we want to ensure that anything we implement for NiFi
> > doesn't
> > > > >>>> negatively the reputation of Apache NiFi as a secure, robust,
> > reliable
> > > > >>>> system.
> > > > >>>>
> > > > >>>> I think we should instead consider a new architecture, one
> > > > >>>> fully extensible so if the community decides to deprecate the
> > capability,
> > > > >>>> it could still be restored using some extension point. This kind
> > of generic
> > > > >>>> approach could allow other extensions not sanctioned by the
> > Apache NiFi
> > > > >>>> community to coexist in the "NiFi+" deployment. Not to get
> > off-topic but it
> > > > >>>> is somewhat analogous to exposing the Spring context such that
> > alternate
> > > > >>>> implementations of the beans could be provided.
> > > > >>>>
> > > > >>>> There are options to consider, I'm sure each with its own set of
> > pros and
> > > > >>>> cons. Off the top of my head:
> > > > >>>>
> > > > >>>> - Sidecar server responsible for executing Python scripts with
> > FlowFile
> > > > >>>> input(s) and returning FlowFile output(s) directly from/to
> > processors, like
> > > > >>>> Marc's suggestion but managed by NiFi
> > > > >>>> - Jep [1] and it's OS-independent counterpart jep-distro [2]
> > > > >>>> - Require Java bindings calling out to an embedded interpreter,
> > such as
> > > > >>>> WrapPy4J (which uses Jep under the hood) [3]
> > > > >>>>
> > > > >>>> I am far from a Python expert and I know there are several
> places
> > in the
> > > > >>>> framework where the Python stuff has landed, so I don't expect
> > any other
> > > > >>>> implementation to be any easier or take less time than the
> > original, which
> > > > >>>> I'm sure took quite a lot of time and effort which is very much
> > > > >>>> appreciated. It seems that Py4J is the long pole in the tent for
> > the
> > > > >>>> concerns, not the capability itself.
> > > > >>>>
> > > > >>>> At the end of the day I think a discussion like this clearly
> > shows the
> > > > >>>> commitment of the whole community to ensure Apache NiFi is all
> > that we want
> > > > >>>> it to be and that we can introspect, improve, and collaborate to
> > keep NiFi
> > > > >>>> awesome :)
> > > > >>>>
> > > > >>>> Regards,
> > > > >>>> Matt
> > > > >>>>
> > > > >>>> [1]
> >
> https://urldefense.com/v3/__https://github.com/ninia/JEP?tab=readme-ov-file__;!!FtR4BK4x7WL3xYs!6mIniDSbJ62B5zII3b9g972tiJiM1K8M1E-1R8rvwlwl__KEfyNqtqWjM3xwIPbVawsv_VEsYzoNRToUxhft8Q3Rnnw$
> > > > >>>> [2]
> >
> https://urldefense.com/v3/__https://github.com/icemachined/jep-distro?tab=readme-ov-file__;!!FtR4BK4x7WL3xYs!6mIniDSbJ62B5zII3b9g972tiJiM1K8M1E-1R8rvwlwl__KEfyNqtqWjM3xwIPbVawsv_VEsYzoNRToUxhft703BJKY$
> > > > >>>> [3]
> >
> https://urldefense.com/v3/__https://github.com/313hemant313/wrappyj__;!!FtR4BK4x7WL3xYs!6mIniDSbJ62B5zII3b9g972tiJiM1K8M1E-1R8rvwlwl__KEfyNqtqWjM3xwIPbVawsv_VEsYzoNRToUxhftYE_fqvo$
> > > > >>>>
> > > > >>>> On Wed, Aug 12, 2026 at 8:19 AM Marc P. <ma...@gmail.com>
> > wrote:
> > > > >>>>
> > > > >>>>> This is a feature I find personally valuable. I agree with
> > several
> > > > >>>>> underlying issues, particularly resource management, Py4J,
> > dependency
> > > > >>>>> handling, and the difficulty of maintaining a framework that
> > spans both the
> > > > >>>>> JVM and Python ecosystems.
> > > > >>>>>
> > > > >>>>> Where I disagree is with the conclusion that these issues
> > necessarily point
> > > > >>>>> toward deprecating Python Processor support itself.
> > > > >>>>>
> > > > >>>>> I think there is an important distinction between the value of
> > Python as an
> > > > >>>>> extension model and the architecture currently used to
> implement
> > it.
> > > > >>>>>
> > > > >>>>> To be solution oriented one way to look at this is to consider
> > what users
> > > > >>>>> would do in the absence of Python Processor support. In many
> > cases, the
> > > > >>>>> alternative is not to rewrite the functionality in Java. It is
> > likely to
> > > > >>>>> become something like:
> > > > >>>>>
> > > > >>>>> Apache NiFi -> HTTP/gRPC -> Python microservice -> NiFi
> > > > >>>>>
> > > > >>>>> That architecture still has independent Python processes,
> Python
> > dependency
> > > > >>>>> management, memory and CPU consumption outside the JVM, and a
> > Python
> > > > >>>>> software supply chain. We have not eliminated any of those
> > concerns; we
> > > > >>>>> have moved them outside NiFi and made every operator build and
> > manage that
> > > > >>>>> integration themselves.
> > > > >>>>>
> > > > >>>>> There are also costs to that approach: additional network
> > boundaries,
> > > > >>>>> deployment artifacts, authentication, service discovery,
> > observability,
> > > > >>>>> retry semantics, and operational complexity. From a governance
> > perspective,
> > > > >>>>> NiFi may actually have less visibility and control over that
> > external
> > > > >>>>> Python workload than it would if Python execution remained part
> > of a
> > > > >>>>> defined NiFi extension model.
> > > > >>>>>
> > > > >>>>> That makes me wonder whether the better direction is to
> > reconsider the
> > > > >>>>> Python runtime architecture rather than Python support itself.
> > > > >>>>>
> > > > >>>>> For example, instead of treating the current Py4J-based
> > implementation as
> > > > >>>>> something that must be preserved indefinitely, we could define
> a
> > stronger
> > > > >>>>> process boundary around Python execution. A future
> > implementation could use
> > > > >>>>> a well-defined IPC or RPC contract, potentially based on
> > gRPC/protobuf,
> > > > >>>>> Unix domain sockets, shared memory, Arrow, or another mechanism
> > designed
> > > > >>>>> around coarse-grained operations.
> > > > >>>>>
> > > > >>>>> NiFi does not need to become a Python integrated application,
> > but I think
> > > > >>>>> there is substantial value in allowing those workloads to
> > participate
> > > > >>>>> directly in a governed NiFi dataflow without requiring users to
> > construct a
> > > > >>>>> separate microservice architecture for every Python
> integration.
> > > > >>>>>
> > > > >>>>> For that reason, I would favor separating two questions:
> > > > >>>>>
> > > > >>>>>   1. Is the current Python Processor implementation the
> > architecture we
> > > > >>>>>   want to support long term?
> > > > >>>>>   2. Should NiFi continue to provide a first-class Python
> > extension
> > > > >>>>>   capability?
> > > > >>>>>
> > > > >>>>> I could see a strong argument that the answer to the first
> > question is no,
> > > > >>>>> particularly if Py4J is becoming a maintenance risk. I am much
> > less
> > > > >>>>> convinced that the answer to the second question should also be
> > no.
> > > > >>>>>
> > > > >>>>> With that said I distinctly understand that time is limited and
> > we would
> > > > >>>>> all develop the coolest of things with infinite resources, so I
> > yield to
> > > > >>>>> those with more influence, but would happy to contribute to a
> > better
> > > > >>>>> architecture if others find it valuable.
> > > > >>>>>
> > > > >>>>> Thanks,
> > > > >>>>>
> > > > >>>>> Marc
> > > > >>>>>
> > > > >>>>> On Wed, Aug 12, 2026 at 12:10 AM Joe Witt <jo...@gmail.com>
> > wrote:
> > > > >>>>>
> > > > >>>>>> You are totally fine to engage on this thread Sean. We
> > appreciate it
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>> On Tue, Aug 11, 2026 at 8:17 PM Sean Hunter via dev <
> > dev@nifi.apache.org
> > > > >>>>>>
> > > > >>>>>> wrote:
> > > > >>>>>>
> > > > >>>>>>> Hi David,
> > > > >>>>>>>
> > > > >>>>>>> As a user, rather than contributor, I think my primary
> concern
> > is one
> > > > >>>>> of
> > > > >>>>>>> operations and implementation. What alternative would be
> > available? We
> > > > >>>>>> used
> > > > >>>>>>> the 1.x version and were quite excited to move to 2.0 with
> its
> > more
> > > > >>>>>> native
> > > > >>>>>>> integration that was easier to troubleshoot. We also do build
> > a NAR as
> > > > >>>>>> part
> > > > >>>>>>> of our pipeline process, and I suspect since we used a blog
> as
> > guidance
> > > > >>>>>> to
> > > > >>>>>>> develop that process, that others likely do as well. Our use
> > case is a
> > > > >>>>>>> particularly nasty data transform (Windows logs of some sort,
> > > > >>>>> naturally)
> > > > >>>>>>> that’s reasonable to write and execute in Python. It’s
> > readable, while
> > > > >>>>>>> still being performant enough for our needs, and seemingly
> not
> > causing
> > > > >>>>>>> system stability issues for us.
> > > > >>>>>>>
> > > > >>>>>>> I do understand and take your points. What I’d like to better
> > > > >>>>> understand
> > > > >>>>>>> are the options users have. Do you have any thoughts?
> > > > >>>>>>>
> > > > >>>>>>> Thank you and have a great day!
> > > > >>>>>>> Sean
> > > > >>>>>>>
> > > > >>>>>>> PS - Please let me know if this is too far off topic for the
> > dev
> > > > >>>>> mailing
> > > > >>>>>>> list and I’ll take it elsewhere.
> > > > >>>>>>>
> > > > >>>>>>> From: David Handermann <ex...@apache.org>
> > > > >>>>>>> Date: Tuesday, August 11, 2026 at 8:38 PM
> > > > >>>>>>> To: dev@nifi.apache.org <de...@nifi.apache.org>
> > > > >>>>>>> Subject: [DISCUSS] Future Support for Python Processors?
> > > > >>>>>>>
> > > > >>>>>>> Be Careful: Email is from an External Source
> > > > >>>>>>> Use caution when opening emails and do not click on links or
> > > > >>>>> attachments
> > > > >>>>>>> from unknown senders. For more resources, visit
> > > > >>>>>> http://security.cat.com/phishing .
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>> Team,
> > > > >>>>>>>
> > > > >>>>>>> The introduction of support for Python Processors was one of
> > the major
> > > > >>>>>>> new features of Apache NiFi 2. In light of experience,
> > however, I
> > > > >>>>>>> believe it is time to reconsider future support for this
> > extension
> > > > >>>>>>> strategy. The idea of removing this feature may raise some
> > concerns,
> > > > >>>>>>> but it is important to consider the options based on
> > scalability,
> > > > >>>>>>> security, and supportability.
> > > > >>>>>>>
> > > > >>>>>>> One of the greatest benefits of Python-based Processors is
> > rapid
> > > > >>>>>>> iteration. Although this is a strength during the development
> > phase,
> > > > >>>>>>> it can become a serious weakness when moving to production.
> > > > >>>>>>>
> > > > >>>>>>> NiFi is fundamentally a Java-based application, thus tuning
> > resource
> > > > >>>>>>> allocation is a key part of maintaining a production system.
> > Running a
> > > > >>>>>>> Python Processor requires one or more independent Python
> > processes,
> > > > >>>>>>> each with their own memory consumption and CPU usage
> patterns.
> > What
> > > > >>>>>>> might work at a small scale on a development machine breaks
> > down when
> > > > >>>>>>> attempting to run at production scale, without careful
> tuning.
> > This is
> > > > >>>>>>> challenging enough with Java-based components, but global
> > settings
> > > > >>>>>>> like heap and non-heap memory provide clear boundaries.
> Python
> > > > >>>>>>> Processors, as currently implemented, do not support this
> > level of
> > > > >>>>>>> control. Beyond process control, operations that require
> > communicating
> > > > >>>>>>> between the NiFi JVM and Python processes involve socket
> > > > >>>>>>> communication, which can be expensive for frequent calls.
> Thus
> > an
> > > > >>>>>>> apparent initial advantage of Python Processors becomes a
> major
> > > > >>>>>>> problem when moving beyond limited types of operations.
> > > > >>>>>>>
> > > > >>>>>>> When it comes to security, Python Processors can involve more
> > serious
> > > > >>>>>>> risks than bundled Java extensions.
> > > > >>>>>>>
> > > > >>>>>>> With NiFi support for downloading dependencies at runtime, it
> > is
> > > > >>>>>>> essential for an administrator to provide a secure supply
> chain
> > > > >>>>>>> strategy for retrieving these dependencies. Although the NiFi
> > > > >>>>>>> framework also supports packaging Python Processors as NAR
> > bundles,
> > > > >>>>>>> this packaging capability is not currently part of the Apache
> > NiFi
> > > > >>>>>>> project. The project could build on third party work, but as
> > Python
> > > > >>>>>>> packaging is less opinionated, this is an additional
> > maintenance
> > > > >>>>>>> question that quickly becomes more involved. This combination
> > of
> > > > >>>>>>> supply chain security and lack of packaging support raises
> > security
> > > > >>>>>>> challenges.
> > > > >>>>>>>
> > > > >>>>>>> Supportability is always a question for any feature in an
> open
> > source
> > > > >>>>>>> project, but Python Processors are a particular point of
> > concern.
> > > > >>>>>>> Providing NiFi framework support requires a strong knowledge
> > of the
> > > > >>>>>>> NiFi project itself, and a strong knowledge of Python.
> Project
> > > > >>>>>>> maintainers and contributors usually have expertise in
> > particular
> > > > >>>>>>> areas, but it is rare to find both. More to the point,
> > however, the
> > > > >>>>>>> Py4J project, which NiFi depends on for integration, has seen
> > limited
> > > > >>>>>>> maintenance, with the last tagged version dating to January
> > 2025.
> > > > >>>>>>> Without robust architecture support, the NiFi project cannot
> > provide a
> > > > >>>>>>> strong framework for future implementation.
> > > > >>>>>>>
> > > > >>>>>>> Taking these issues together, I believe the project should
> > consider
> > > > >>>>>>> deprecating support for Python Processors. Similar to other
> > > > >>>>>>> deprecations, actual removal would not occur until a major
> > release
> > > > >>>>>>> version, and that would be a separate conversation.
> > > > >>>>>>>
> > > > >>>>>>> Rather than continuing to extend the life of Python Processor
> > support,
> > > > >>>>>>> it seems better to invest limited project resources into
> > maintaining
> > > > >>>>>>> and improving core functionality.
> > > > >>>>>>>
> > > > >>>>>>> Regards,
> > > > >>>>>>> David Handermann
> > > > >>>>>>> Apache NiFi PMC Chair
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>>
> > > > >>>
> > > >
> >
>

Re: [DISCUSS] Future Support for Python Processors?

[Ryan Hendrickson <ry...@gmail.com>]

The team I work with uses Python processors when (1) the libraries are
unique to python, (2) the developer prefers Python over Java, and (3) raw
data processing speed is not a factor.  When speed is a factor, we have:
(1) Swapped to side-car containers deployed next to NiFi hosting the python
code with a rest interface for interaction [1]; OR (2) Converted the
processor to either Groovy or pure Java.

Looking back, it seemed developer preference for Python (with the rapid
development cycle of python code) and unique libraries in Python drove the
desire to use Python in NiFi and the adoption of Python-based data
processing capabilities within our team.  While the integration isn't
perfect in NiFi, it does check a box when evaluating the flexibility of
major programming languages supported in the core NiFi project/ecosystem.

We have long looked at ways to performantly call non-Java code (typically
Python) from NiFi, using NiFi as the over-arching dataflow governance
mechanism.  Marc P's comment on "governed NiFi dataflow" is accurate and
desired [2][3].  The dataflow UI is beautiful, highly functional, and very
simple.  We've also looked at ways to scale specific processors
independently of the total resources provided to NiFi, notably to isolate
resources directly to a processor or process group to adjust for less
performant code on vertically scaled infrastructure.

My thoughts have diverged from the Python topic, but two common themes
involving Python tend to be (1) dataflows governed by NiFi and (2)
performance.

[1] https://issues.apache.org/jira/browse/NIFI-9026 | Integration with
auto-scaling container environments
[2] https://issues.apache.org/jira/browse/NIFI-9028 | Large Canvas
Management
[3] https://issues.apache.org/jira/browse/NIFI-9027 | Integration of S3
buckets with Remote Process Groups for outage events and data overruns

I appreciate the community's hard work in maintaining NiFi and transparent
communication style.  I look forward to the direction NiFi moves.

Best,
Ryan

On Mon, Aug 17, 2026 at 10:14 AM David Handermann <
exceptionfactory@apache.org> wrote:

> Gabor,
>
> Thanks for the reply and additional perspective from the MiNiFi C++ work.
>
> As you highlighted, the scalability and security issues became more
> apparent after release and initial adoption, which is a natural part
> of the development process. I agree with you that the Python Processor
> API presents a simplified structure, which has a lot of advantages for
> basic use cases. Deprecating the API certainly would discourage use of
> Python Processors, and that is part of the purpose in setting a
> maintainable direction. The scalability concern is not something that
> can be solved through documentation, given the inherent problems with
> independent process management. That is the reason the current
> approach needs to be deprecated, not documented as a limitation.
>
> As mentioned earlier, deprecating the current implementation does not
> preclude future alternatives. Instead, it provides a clear signal to
> users of project direction. It will cause challenges for current
> adopters, but the standard deprecation and removal process supplies
> that transitional notification strategy.
>
> It is notable that the MiNiFi C++ integration approach provides a
> different avenue, given the fundamentally different architecture. It
> is also worth clarifying that the deprecation would apply to the main
> NiFi project, not necessarily to the MiNiFi C++ application.
>
> To be clear, this discussion is not intended to minimize the
> usefulness of Python Processors in selected situations. The main
> purpose is to be clear among project maintainers about what is
> sustainable, both from a development perspective, and from a
> production quality perspective. As everyone thus far has acknowledged
> the severe scaling limitations of the current architecture, that
> highlights the importance of moving in a different direction.
>
> Based on this discussion thus far, I'm evaluating some related work on
> a different scripted Processor architecture. The current scripted
> Processors use the generic Java ScriptEngine, but the structure
> requires supporting languages within the project itself. I'm
> considering an approach that would allow users to bring their own
> ScriptEngine implementation. This kind of decoupling may provide a
> more supportable strategy for the project itself, while enabling users
> to build their own extensions based on desired language. Even GraalPy,
> which supports Python 3 and some native libraries, has limitations,
> but it might provide a way forward beyond simple deprecation. I'm
> still exploring the options.
>
> With that background, unmaintained capabilities and problematic
> scaling considerations are something the project must address, so I
> appreciate the interaction on the topic from multiple perspectives.
>
> Regards,
> David Handermann
>
> On Mon, Aug 17, 2026 at 3:36 AM Gábor Gyimesi <lo...@apache.org>
> wrote:
> >
> > Hi,
> >
> > I also agree with Marc on this. From the user interactions, I got the
> > feeling that Python processors are quite valuable to the community,
> > and I would be hesitant to deprecate and remove them when the main
> > problems are related to the underlying implementation.
> >
> > I agree with several of the problems David mentioned, particularly
> > around scalability and dependency installation. These issues were not
> > foreseen with the initial release of NiFi 2.0 and only became apparent
> > as these processors started being used more extensively by the
> > community, especially with heavy AI libraries. The problem is that,
> > after the initial release of 2.0, there has been very little
> > development on the Python side, so these issues were never addressed.
> >
> > From a user's point of view, I think the Python processor API is
> > simple and easy to use. Even with the current implementation, there
> > are several use cases for which it is sufficient, so I would not
> > remove the option to use Python processors. Deprecating them would
> > discourage people from creating and using Python processors, even for
> > use cases where they make sense. In my opinion, the documentation for
> > Python processors should instead be extended to clearly explain the
> > caveats of using them, including what problems users can expect in
> > terms of dependency management, scalability, etc., and which use cases
> > are recommended for these processors.
> >
> > In the future, I would support developing an alternative
> > implementation under the hood, replacing Py4J, if we decide to pursue
> > IPC communication, use an embedded interpreter, or adopt any of the
> > other solutions Matt mentioned. In MiNiFi C++, we went with an
> > embedded interpreter solution, which provides much better performance,
> > with the caveat that only one interpreter can be embedded in a MiNiFi
> > process and is therefore shared between processors. The dependency
> > management issue is still a challenge in MiNiFi C++ as well, and I
> > would support creating a packaging solution that could work for both
> > projects.
> >
> > As Marc said, from a user's point of view, in many cases it is not
> > really an option to rewrite a processor in Java, as many libraries,
> > especially AI libraries, are only available in Python. If the decision
> > is made to deprecate and eventually remove Python processor support
> > from NiFi, forcing users to create a separate Python microservice
> > based on Marc's suggestion of a "Apache NiFi -> HTTP/gRPC -> Python
> > microservice -> NiFi" architecture, we could consider officially
> > recommending MiNiFi C++ as an interim solution, since it retains
> > Python support. The same approach could potentially be used for
> > anything requiring C/C++ or Rust support as well.
> >
> > Regards,
> > Gabor
> >
> > On Thu, 13 Aug 2026 at 21:51, Mark Payne <ma...@hotmail.com> wrote:
> > >
> > > Kevin,
> > >
> > > Py4J is the glue that bridges the NiFi JVM to the python process that
> runs python processors. So there’s a py4j server that is run in a separate
> python process. Then a py4j client that runs in the JVM. Py4J handles the
> RPC to make it possible to call the Python processors as if they were Java
> objects.
> > >
> > > Thank
> > > -Makr
> > >
> > >
> > > > On Aug 13, 2026, at 3:37 PM, Kevin Doran <kd...@apache.org> wrote:
> > > >
> > > > Can someone clarify where Py4J is used today vs where subprocesses
> > > > that invoke a configurable python executable are used? My
> > > > understanding was that the Py4J was mostly restricted to the
> scripting
> > > > processors, but if that is not the case I would be interested in
> > > > hearing more.
> > > >
> > > > On Thu, Aug 13, 2026 at 1:42 PM David Handermann
> > > > <ex...@apache.org> wrote:
> > > >>
> > > >> Dan,
> > > >>
> > > >> Thanks for the substantive reply and consideration of the various
> > > >> issues surrounding support for native Python Processors.
> > > >>
> > > >> As you highlighted, the scalability issue is foundational. Although
> > > >> Processor authors can work with self-imposed constraints, the NiFi
> > > >> framework does not provide the same level of safeguards. Any future
> > > >> solution would need to account for these types of production-level
> > > >> concerns.
> > > >>
> > > >> Regarding supply chain security, you are correct that there are
> > > >> potential paths forward on the packaging strategy. The framework
> could
> > > >> remove support for runtime loading as a safety measure. Right now,
> > > >> lack of pip or uv effectively disables runtime loading. Adjusting
> the
> > > >> default project container configuration would strengthen the
> standard
> > > >> position, and thus require Python Processors to be packaged using a
> > > >> NAR. The project could provide tooling, similar to the NiFi NAR
> Maven
> > > >> Plugin, but this broadens the maintenance scope in a meaningful way.
> > > >> Thus before going in this direction, the future support strategy
> needs
> > > >> to be considered.
> > > >>
> > > >> The Py4J architecture remains a critical concern and high risk for
> > > >> future maintainability. Considering an alternative architecture
> would
> > > >> require a substantial redesign and rewrite of Python Processor
> > > >> support, abandoning a majority of the current implementation.
> Current
> > > >> tooling makes this more plausible in theory, but is a significant
> > > >> undertaking that is difficult to justify without serious commitment
> on
> > > >> the part of multiple project maintainers. To be clear, deprecating
> the
> > > >> current implementation for removal does not preclude future
> redesign.
> > > >> These two questions should be considered independently.
> > > >>
> > > >> Deprecation and removal without replacement will have a different
> > > >> impact depending on the current scope of usage. Simpler Python
> > > >> Processors should be easier to convert to Java, but more complex
> > > >> implementations would likely require something outside of NiFi, as
> you
> > > >> mentioned. Although these are difficult questions, it ultimately
> comes
> > > >> down to project focus and available maintenance cycles. Apache NiFi
> > > >> has historically done many things. NiFi 2 served as a focusing
> > > >> function. Going forward, maintaining project health and stability is
> > > >> paramount, which may at times involve removing certain capabilities.
> > > >>
> > > >> Regards,
> > > >> David Handermann
> > > >>
> > > >> On Wed, Aug 12, 2026 at 3:27 PM Daniel Weidman via dev
> > > >> <de...@nifi.apache.org> wrote:
> > > >>>
> > > >>> David,
> > > >>>
> > > >>> Thanks for raising this directly rather than letting it drift, and
> for laying
> > > >>> out the reasoning rather than just the conclusion.
> > > >>>
> > > >>> On scalability, your point lands. In my experience the memory
> behavior of a
> > > >>> Python Processor is largely a function of how it is written, and
> disciplined
> > > >>> implementations can be well behaved. But I recognize that is an
> argument about
> > > >>> good citizenship, not about enforcement, and it does not give an
> operator the
> > > >>> equivalent of a heap ceiling. That is a real gap. It does seem
> addressable at
> > > >>> the process supervision layer, through cgroup or rlimit bounds on
> spawned
> > > >>> interpreters, rather than being inherent to Python as an extension
> language.
> > > >>>
> > > >>> On security, I would offer a friendly amendment. The meaningful
> difference does
> > > >>> not seem to be Python versus Java, but when and where third party
> dependencies
> > > >>> land. A NAR resolves its tree at build time and ships self
> contained, so it can
> > > >>> be scanned, signed, and deployed into an air gapped environment,
> and unpacking
> > > >>> it executes nothing. The current Python path resolves dependencies
> at runtime
> > > >>> via pip on the NiFi host, and pip installation does execute
> packaging code.
> > > >>> That is a real governance gap, and I will concede PyPI carries
> risks Maven
> > > >>> Central does not.
> > > >>>
> > > >>> But as you noted, the framework already supports loading Python
> Processors
> > > >>> packaged as NARs. What is missing is project-provided tooling to
> build them.
> > > >>> Sean's note about relying on a blog post to produce one seems
> telling. Closing
> > > >>> that gap looks like a smaller undertaking than removing the
> extension point,
> > > >>> and it would address the supply chain concern directly rather than
> by
> > > >>> subtraction.
> > > >>>
> > > >>> I will admit I had not realized the current implementation still
> rests on Py4J.
> > > >>> That reframes a good deal of this for me, and it tracks with
> Matt's read that
> > > >>> Py4J is the long pole rather than the capability itself.
> > > >>>
> > > >>> For what it is worth as an operator, the practical value of Python
> Processors
> > > >>> is not just iteration speed. It is that the people writing our
> transforms are
> > > >>> Python native. Removal without a replacement does not convert that
> work to
> > > >>> Java, it pushes it out of NiFi entirely, to sidecars or external
> services,
> > > >>> where, as Marc noted, we keep every one of the concerns you listed
> but lose
> > > >>> NiFi's visibility into them.
> > > >>>
> > > >>> I would rather see the runtime architecture reconsidered than the
> capability
> > > >>> retired. If the deciding constraint is maintainer cycles, it would
> be useful to
> > > >>> know what a replacement would need in terms of effort and
> expertise. That is a
> > > >>> more actionable ask of the community than a deprecation vote.
> > > >>>
> > > >>> Regards,
> > > >>> Dan
> > > >>>
> > > >>>
> > > >>> From: David Handermann <ex...@apache.org>
> > > >>> Date: Wednesday, August 12, 2026 at 9:10 AM
> > > >>> To: dev@nifi.apache.org <de...@nifi.apache.org>
> > > >>> Subject: Re: [DISCUSS] Future Support for Python Processors?
> > > >>>
> > > >>> Thanks to everyone for the replies thus far!
> > > >>>
> > > >>> The question about alternatives gets at the core issue, and the
> ideas
> > > >>> around a more loosely coupled architecture provide some helpful
> > > >>> possibilities.
> > > >>>
> > > >>> At a basic level, simple deprecation and removal would leave the
> > > >>> current set of options, including standard Java, Groovy scripting,
> or
> > > >>> externalized Python scripts executed as a stream. None of these
> > > >>> provide the same level of features for Python itself, which
> prompted
> > > >>> the original native Python Processor implementation. Without a
> > > >>> replacement, these alternatives would remain. With the rise of
> > > >>> AI-enabled development, it is now much easier to go from Python to
> > > >>> Java, but work is still required.
> > > >>>
> > > >>> As to future strategies for Python integration, the current
> discussion
> > > >>> around deprecation is more narrowly focused on the existing
> > > >>> framework-level extension. Removal of the current implementation is
> > > >>> not intended to halt future innovation. Although removal without an
> > > >>> initial replacement might not be the perfect solution, it may be
> > > >>> necessary given available maintainer cycles, and the substantive
> > > >>> issues I described in the opening.
> > > >>>
> > > >>> If there is sufficient interest and effort to propose and build a
> > > >>> different solution, that would be a great outcome of this
> discussion.
> > > >>> I've thought at different points about reworking the Script Engine
> > > >>> integration to provide a cleaner path to external extension, so
> there
> > > >>> are definitely opportunities in that direction.
> > > >>>
> > > >>> With that, my primary concerns with the current implementation
> remain,
> > > >>> but glad to consider future alternatives.
> > > >>>
> > > >>> Regards,
> > > >>> David Handermann
> > > >>>
> > > >>>
> > > >>> On Wed, Aug 12, 2026 at 7:52 AM Matt Burgess <ma...@apache.org>
> wrote:
> > > >>>>
> > > >>>> I agree with Marc, the concerns of the implementation of the
> capability
> > > >>>> should not warrant deprecation and future removal of the
> capability, but
> > > >>>> certainly we want to ensure that anything we implement for NiFi
> doesn't
> > > >>>> negatively the reputation of Apache NiFi as a secure, robust,
> reliable
> > > >>>> system.
> > > >>>>
> > > >>>> I think we should instead consider a new architecture, one
> > > >>>> fully extensible so if the community decides to deprecate the
> capability,
> > > >>>> it could still be restored using some extension point. This kind
> of generic
> > > >>>> approach could allow other extensions not sanctioned by the
> Apache NiFi
> > > >>>> community to coexist in the "NiFi+" deployment. Not to get
> off-topic but it
> > > >>>> is somewhat analogous to exposing the Spring context such that
> alternate
> > > >>>> implementations of the beans could be provided.
> > > >>>>
> > > >>>> There are options to consider, I'm sure each with its own set of
> pros and
> > > >>>> cons. Off the top of my head:
> > > >>>>
> > > >>>> - Sidecar server responsible for executing Python scripts with
> FlowFile
> > > >>>> input(s) and returning FlowFile output(s) directly from/to
> processors, like
> > > >>>> Marc's suggestion but managed by NiFi
> > > >>>> - Jep [1] and it's OS-independent counterpart jep-distro [2]
> > > >>>> - Require Java bindings calling out to an embedded interpreter,
> such as
> > > >>>> WrapPy4J (which uses Jep under the hood) [3]
> > > >>>>
> > > >>>> I am far from a Python expert and I know there are several places
> in the
> > > >>>> framework where the Python stuff has landed, so I don't expect
> any other
> > > >>>> implementation to be any easier or take less time than the
> original, which
> > > >>>> I'm sure took quite a lot of time and effort which is very much
> > > >>>> appreciated. It seems that Py4J is the long pole in the tent for
> the
> > > >>>> concerns, not the capability itself.
> > > >>>>
> > > >>>> At the end of the day I think a discussion like this clearly
> shows the
> > > >>>> commitment of the whole community to ensure Apache NiFi is all
> that we want
> > > >>>> it to be and that we can introspect, improve, and collaborate to
> keep NiFi
> > > >>>> awesome :)
> > > >>>>
> > > >>>> Regards,
> > > >>>> Matt
> > > >>>>
> > > >>>> [1]
> https://urldefense.com/v3/__https://github.com/ninia/JEP?tab=readme-ov-file__;!!FtR4BK4x7WL3xYs!6mIniDSbJ62B5zII3b9g972tiJiM1K8M1E-1R8rvwlwl__KEfyNqtqWjM3xwIPbVawsv_VEsYzoNRToUxhft8Q3Rnnw$
> > > >>>> [2]
> https://urldefense.com/v3/__https://github.com/icemachined/jep-distro?tab=readme-ov-file__;!!FtR4BK4x7WL3xYs!6mIniDSbJ62B5zII3b9g972tiJiM1K8M1E-1R8rvwlwl__KEfyNqtqWjM3xwIPbVawsv_VEsYzoNRToUxhft703BJKY$
> > > >>>> [3]
> https://urldefense.com/v3/__https://github.com/313hemant313/wrappyj__;!!FtR4BK4x7WL3xYs!6mIniDSbJ62B5zII3b9g972tiJiM1K8M1E-1R8rvwlwl__KEfyNqtqWjM3xwIPbVawsv_VEsYzoNRToUxhftYE_fqvo$
> > > >>>>
> > > >>>> On Wed, Aug 12, 2026 at 8:19 AM Marc P. <ma...@gmail.com>
> wrote:
> > > >>>>
> > > >>>>> This is a feature I find personally valuable. I agree with
> several
> > > >>>>> underlying issues, particularly resource management, Py4J,
> dependency
> > > >>>>> handling, and the difficulty of maintaining a framework that
> spans both the
> > > >>>>> JVM and Python ecosystems.
> > > >>>>>
> > > >>>>> Where I disagree is with the conclusion that these issues
> necessarily point
> > > >>>>> toward deprecating Python Processor support itself.
> > > >>>>>
> > > >>>>> I think there is an important distinction between the value of
> Python as an
> > > >>>>> extension model and the architecture currently used to implement
> it.
> > > >>>>>
> > > >>>>> To be solution oriented one way to look at this is to consider
> what users
> > > >>>>> would do in the absence of Python Processor support. In many
> cases, the
> > > >>>>> alternative is not to rewrite the functionality in Java. It is
> likely to
> > > >>>>> become something like:
> > > >>>>>
> > > >>>>> Apache NiFi -> HTTP/gRPC -> Python microservice -> NiFi
> > > >>>>>
> > > >>>>> That architecture still has independent Python processes, Python
> dependency
> > > >>>>> management, memory and CPU consumption outside the JVM, and a
> Python
> > > >>>>> software supply chain. We have not eliminated any of those
> concerns; we
> > > >>>>> have moved them outside NiFi and made every operator build and
> manage that
> > > >>>>> integration themselves.
> > > >>>>>
> > > >>>>> There are also costs to that approach: additional network
> boundaries,
> > > >>>>> deployment artifacts, authentication, service discovery,
> observability,
> > > >>>>> retry semantics, and operational complexity. From a governance
> perspective,
> > > >>>>> NiFi may actually have less visibility and control over that
> external
> > > >>>>> Python workload than it would if Python execution remained part
> of a
> > > >>>>> defined NiFi extension model.
> > > >>>>>
> > > >>>>> That makes me wonder whether the better direction is to
> reconsider the
> > > >>>>> Python runtime architecture rather than Python support itself.
> > > >>>>>
> > > >>>>> For example, instead of treating the current Py4J-based
> implementation as
> > > >>>>> something that must be preserved indefinitely, we could define a
> stronger
> > > >>>>> process boundary around Python execution. A future
> implementation could use
> > > >>>>> a well-defined IPC or RPC contract, potentially based on
> gRPC/protobuf,
> > > >>>>> Unix domain sockets, shared memory, Arrow, or another mechanism
> designed
> > > >>>>> around coarse-grained operations.
> > > >>>>>
> > > >>>>> NiFi does not need to become a Python integrated application,
> but I think
> > > >>>>> there is substantial value in allowing those workloads to
> participate
> > > >>>>> directly in a governed NiFi dataflow without requiring users to
> construct a
> > > >>>>> separate microservice architecture for every Python integration.
> > > >>>>>
> > > >>>>> For that reason, I would favor separating two questions:
> > > >>>>>
> > > >>>>>   1. Is the current Python Processor implementation the
> architecture we
> > > >>>>>   want to support long term?
> > > >>>>>   2. Should NiFi continue to provide a first-class Python
> extension
> > > >>>>>   capability?
> > > >>>>>
> > > >>>>> I could see a strong argument that the answer to the first
> question is no,
> > > >>>>> particularly if Py4J is becoming a maintenance risk. I am much
> less
> > > >>>>> convinced that the answer to the second question should also be
> no.
> > > >>>>>
> > > >>>>> With that said I distinctly understand that time is limited and
> we would
> > > >>>>> all develop the coolest of things with infinite resources, so I
> yield to
> > > >>>>> those with more influence, but would happy to contribute to a
> better
> > > >>>>> architecture if others find it valuable.
> > > >>>>>
> > > >>>>> Thanks,
> > > >>>>>
> > > >>>>> Marc
> > > >>>>>
> > > >>>>> On Wed, Aug 12, 2026 at 12:10 AM Joe Witt <jo...@gmail.com>
> wrote:
> > > >>>>>
> > > >>>>>> You are totally fine to engage on this thread Sean. We
> appreciate it
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> On Tue, Aug 11, 2026 at 8:17 PM Sean Hunter via dev <
> dev@nifi.apache.org
> > > >>>>>>
> > > >>>>>> wrote:
> > > >>>>>>
> > > >>>>>>> Hi David,
> > > >>>>>>>
> > > >>>>>>> As a user, rather than contributor, I think my primary concern
> is one
> > > >>>>> of
> > > >>>>>>> operations and implementation. What alternative would be
> available? We
> > > >>>>>> used
> > > >>>>>>> the 1.x version and were quite excited to move to 2.0 with its
> more
> > > >>>>>> native
> > > >>>>>>> integration that was easier to troubleshoot. We also do build
> a NAR as
> > > >>>>>> part
> > > >>>>>>> of our pipeline process, and I suspect since we used a blog as
> guidance
> > > >>>>>> to
> > > >>>>>>> develop that process, that others likely do as well. Our use
> case is a
> > > >>>>>>> particularly nasty data transform (Windows logs of some sort,
> > > >>>>> naturally)
> > > >>>>>>> that’s reasonable to write and execute in Python. It’s
> readable, while
> > > >>>>>>> still being performant enough for our needs, and seemingly not
> causing
> > > >>>>>>> system stability issues for us.
> > > >>>>>>>
> > > >>>>>>> I do understand and take your points. What I’d like to better
> > > >>>>> understand
> > > >>>>>>> are the options users have. Do you have any thoughts?
> > > >>>>>>>
> > > >>>>>>> Thank you and have a great day!
> > > >>>>>>> Sean
> > > >>>>>>>
> > > >>>>>>> PS - Please let me know if this is too far off topic for the
> dev
> > > >>>>> mailing
> > > >>>>>>> list and I’ll take it elsewhere.
> > > >>>>>>>
> > > >>>>>>> From: David Handermann <ex...@apache.org>
> > > >>>>>>> Date: Tuesday, August 11, 2026 at 8:38 PM
> > > >>>>>>> To: dev@nifi.apache.org <de...@nifi.apache.org>
> > > >>>>>>> Subject: [DISCUSS] Future Support for Python Processors?
> > > >>>>>>>
> > > >>>>>>> Be Careful: Email is from an External Source
> > > >>>>>>> Use caution when opening emails and do not click on links or
> > > >>>>> attachments
> > > >>>>>>> from unknown senders. For more resources, visit
> > > >>>>>> http://security.cat.com/phishing .
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> Team,
> > > >>>>>>>
> > > >>>>>>> The introduction of support for Python Processors was one of
> the major
> > > >>>>>>> new features of Apache NiFi 2. In light of experience,
> however, I
> > > >>>>>>> believe it is time to reconsider future support for this
> extension
> > > >>>>>>> strategy. The idea of removing this feature may raise some
> concerns,
> > > >>>>>>> but it is important to consider the options based on
> scalability,
> > > >>>>>>> security, and supportability.
> > > >>>>>>>
> > > >>>>>>> One of the greatest benefits of Python-based Processors is
> rapid
> > > >>>>>>> iteration. Although this is a strength during the development
> phase,
> > > >>>>>>> it can become a serious weakness when moving to production.
> > > >>>>>>>
> > > >>>>>>> NiFi is fundamentally a Java-based application, thus tuning
> resource
> > > >>>>>>> allocation is a key part of maintaining a production system.
> Running a
> > > >>>>>>> Python Processor requires one or more independent Python
> processes,
> > > >>>>>>> each with their own memory consumption and CPU usage patterns.
> What
> > > >>>>>>> might work at a small scale on a development machine breaks
> down when
> > > >>>>>>> attempting to run at production scale, without careful tuning.
> This is
> > > >>>>>>> challenging enough with Java-based components, but global
> settings
> > > >>>>>>> like heap and non-heap memory provide clear boundaries. Python
> > > >>>>>>> Processors, as currently implemented, do not support this
> level of
> > > >>>>>>> control. Beyond process control, operations that require
> communicating
> > > >>>>>>> between the NiFi JVM and Python processes involve socket
> > > >>>>>>> communication, which can be expensive for frequent calls. Thus
> an
> > > >>>>>>> apparent initial advantage of Python Processors becomes a major
> > > >>>>>>> problem when moving beyond limited types of operations.
> > > >>>>>>>
> > > >>>>>>> When it comes to security, Python Processors can involve more
> serious
> > > >>>>>>> risks than bundled Java extensions.
> > > >>>>>>>
> > > >>>>>>> With NiFi support for downloading dependencies at runtime, it
> is
> > > >>>>>>> essential for an administrator to provide a secure supply chain
> > > >>>>>>> strategy for retrieving these dependencies. Although the NiFi
> > > >>>>>>> framework also supports packaging Python Processors as NAR
> bundles,
> > > >>>>>>> this packaging capability is not currently part of the Apache
> NiFi
> > > >>>>>>> project. The project could build on third party work, but as
> Python
> > > >>>>>>> packaging is less opinionated, this is an additional
> maintenance
> > > >>>>>>> question that quickly becomes more involved. This combination
> of
> > > >>>>>>> supply chain security and lack of packaging support raises
> security
> > > >>>>>>> challenges.
> > > >>>>>>>
> > > >>>>>>> Supportability is always a question for any feature in an open
> source
> > > >>>>>>> project, but Python Processors are a particular point of
> concern.
> > > >>>>>>> Providing NiFi framework support requires a strong knowledge
> of the
> > > >>>>>>> NiFi project itself, and a strong knowledge of Python. Project
> > > >>>>>>> maintainers and contributors usually have expertise in
> particular
> > > >>>>>>> areas, but it is rare to find both. More to the point,
> however, the
> > > >>>>>>> Py4J project, which NiFi depends on for integration, has seen
> limited
> > > >>>>>>> maintenance, with the last tagged version dating to January
> 2025.
> > > >>>>>>> Without robust architecture support, the NiFi project cannot
> provide a
> > > >>>>>>> strong framework for future implementation.
> > > >>>>>>>
> > > >>>>>>> Taking these issues together, I believe the project should
> consider
> > > >>>>>>> deprecating support for Python Processors. Similar to other
> > > >>>>>>> deprecations, actual removal would not occur until a major
> release
> > > >>>>>>> version, and that would be a separate conversation.
> > > >>>>>>>
> > > >>>>>>> Rather than continuing to extend the life of Python Processor
> support,
> > > >>>>>>> it seems better to invest limited project resources into
> maintaining
> > > >>>>>>> and improving core functionality.
> > > >>>>>>>
> > > >>>>>>> Regards,
> > > >>>>>>> David Handermann
> > > >>>>>>> Apache NiFi PMC Chair
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>
> > >
>

Re: [DISCUSS] Future Support for Python Processors?

[David Handermann <ex...@apache.org>]

Gabor,

Thanks for the reply and additional perspective from the MiNiFi C++ work.

As you highlighted, the scalability and security issues became more
apparent after release and initial adoption, which is a natural part
of the development process. I agree with you that the Python Processor
API presents a simplified structure, which has a lot of advantages for
basic use cases. Deprecating the API certainly would discourage use of
Python Processors, and that is part of the purpose in setting a
maintainable direction. The scalability concern is not something that
can be solved through documentation, given the inherent problems with
independent process management. That is the reason the current
approach needs to be deprecated, not documented as a limitation.

As mentioned earlier, deprecating the current implementation does not
preclude future alternatives. Instead, it provides a clear signal to
users of project direction. It will cause challenges for current
adopters, but the standard deprecation and removal process supplies
that transitional notification strategy.

It is notable that the MiNiFi C++ integration approach provides a
different avenue, given the fundamentally different architecture. It
is also worth clarifying that the deprecation would apply to the main
NiFi project, not necessarily to the MiNiFi C++ application.

To be clear, this discussion is not intended to minimize the
usefulness of Python Processors in selected situations. The main
purpose is to be clear among project maintainers about what is
sustainable, both from a development perspective, and from a
production quality perspective. As everyone thus far has acknowledged
the severe scaling limitations of the current architecture, that
highlights the importance of moving in a different direction.

Based on this discussion thus far, I'm evaluating some related work on
a different scripted Processor architecture. The current scripted
Processors use the generic Java ScriptEngine, but the structure
requires supporting languages within the project itself. I'm
considering an approach that would allow users to bring their own
ScriptEngine implementation. This kind of decoupling may provide a
more supportable strategy for the project itself, while enabling users
to build their own extensions based on desired language. Even GraalPy,
which supports Python 3 and some native libraries, has limitations,
but it might provide a way forward beyond simple deprecation. I'm
still exploring the options.

With that background, unmaintained capabilities and problematic
scaling considerations are something the project must address, so I
appreciate the interaction on the topic from multiple perspectives.

Regards,
David Handermann

On Mon, Aug 17, 2026 at 3:36 AM Gábor Gyimesi <lo...@apache.org> wrote:
>
> Hi,
>
> I also agree with Marc on this. From the user interactions, I got the
> feeling that Python processors are quite valuable to the community,
> and I would be hesitant to deprecate and remove them when the main
> problems are related to the underlying implementation.
>
> I agree with several of the problems David mentioned, particularly
> around scalability and dependency installation. These issues were not
> foreseen with the initial release of NiFi 2.0 and only became apparent
> as these processors started being used more extensively by the
> community, especially with heavy AI libraries. The problem is that,
> after the initial release of 2.0, there has been very little
> development on the Python side, so these issues were never addressed.
>
> From a user's point of view, I think the Python processor API is
> simple and easy to use. Even with the current implementation, there
> are several use cases for which it is sufficient, so I would not
> remove the option to use Python processors. Deprecating them would
> discourage people from creating and using Python processors, even for
> use cases where they make sense. In my opinion, the documentation for
> Python processors should instead be extended to clearly explain the
> caveats of using them, including what problems users can expect in
> terms of dependency management, scalability, etc., and which use cases
> are recommended for these processors.
>
> In the future, I would support developing an alternative
> implementation under the hood, replacing Py4J, if we decide to pursue
> IPC communication, use an embedded interpreter, or adopt any of the
> other solutions Matt mentioned. In MiNiFi C++, we went with an
> embedded interpreter solution, which provides much better performance,
> with the caveat that only one interpreter can be embedded in a MiNiFi
> process and is therefore shared between processors. The dependency
> management issue is still a challenge in MiNiFi C++ as well, and I
> would support creating a packaging solution that could work for both
> projects.
>
> As Marc said, from a user's point of view, in many cases it is not
> really an option to rewrite a processor in Java, as many libraries,
> especially AI libraries, are only available in Python. If the decision
> is made to deprecate and eventually remove Python processor support
> from NiFi, forcing users to create a separate Python microservice
> based on Marc's suggestion of a "Apache NiFi -> HTTP/gRPC -> Python
> microservice -> NiFi" architecture, we could consider officially
> recommending MiNiFi C++ as an interim solution, since it retains
> Python support. The same approach could potentially be used for
> anything requiring C/C++ or Rust support as well.
>
> Regards,
> Gabor
>
> On Thu, 13 Aug 2026 at 21:51, Mark Payne <ma...@hotmail.com> wrote:
> >
> > Kevin,
> >
> > Py4J is the glue that bridges the NiFi JVM to the python process that runs python processors. So there’s a py4j server that is run in a separate python process. Then a py4j client that runs in the JVM. Py4J handles the RPC to make it possible to call the Python processors as if they were Java objects.
> >
> > Thank
> > -Makr
> >
> >
> > > On Aug 13, 2026, at 3:37 PM, Kevin Doran <kd...@apache.org> wrote:
> > >
> > > Can someone clarify where Py4J is used today vs where subprocesses
> > > that invoke a configurable python executable are used? My
> > > understanding was that the Py4J was mostly restricted to the scripting
> > > processors, but if that is not the case I would be interested in
> > > hearing more.
> > >
> > > On Thu, Aug 13, 2026 at 1:42 PM David Handermann
> > > <ex...@apache.org> wrote:
> > >>
> > >> Dan,
> > >>
> > >> Thanks for the substantive reply and consideration of the various
> > >> issues surrounding support for native Python Processors.
> > >>
> > >> As you highlighted, the scalability issue is foundational. Although
> > >> Processor authors can work with self-imposed constraints, the NiFi
> > >> framework does not provide the same level of safeguards. Any future
> > >> solution would need to account for these types of production-level
> > >> concerns.
> > >>
> > >> Regarding supply chain security, you are correct that there are
> > >> potential paths forward on the packaging strategy. The framework could
> > >> remove support for runtime loading as a safety measure. Right now,
> > >> lack of pip or uv effectively disables runtime loading. Adjusting the
> > >> default project container configuration would strengthen the standard
> > >> position, and thus require Python Processors to be packaged using a
> > >> NAR. The project could provide tooling, similar to the NiFi NAR Maven
> > >> Plugin, but this broadens the maintenance scope in a meaningful way.
> > >> Thus before going in this direction, the future support strategy needs
> > >> to be considered.
> > >>
> > >> The Py4J architecture remains a critical concern and high risk for
> > >> future maintainability. Considering an alternative architecture would
> > >> require a substantial redesign and rewrite of Python Processor
> > >> support, abandoning a majority of the current implementation. Current
> > >> tooling makes this more plausible in theory, but is a significant
> > >> undertaking that is difficult to justify without serious commitment on
> > >> the part of multiple project maintainers. To be clear, deprecating the
> > >> current implementation for removal does not preclude future redesign.
> > >> These two questions should be considered independently.
> > >>
> > >> Deprecation and removal without replacement will have a different
> > >> impact depending on the current scope of usage. Simpler Python
> > >> Processors should be easier to convert to Java, but more complex
> > >> implementations would likely require something outside of NiFi, as you
> > >> mentioned. Although these are difficult questions, it ultimately comes
> > >> down to project focus and available maintenance cycles. Apache NiFi
> > >> has historically done many things. NiFi 2 served as a focusing
> > >> function. Going forward, maintaining project health and stability is
> > >> paramount, which may at times involve removing certain capabilities.
> > >>
> > >> Regards,
> > >> David Handermann
> > >>
> > >> On Wed, Aug 12, 2026 at 3:27 PM Daniel Weidman via dev
> > >> <de...@nifi.apache.org> wrote:
> > >>>
> > >>> David,
> > >>>
> > >>> Thanks for raising this directly rather than letting it drift, and for laying
> > >>> out the reasoning rather than just the conclusion.
> > >>>
> > >>> On scalability, your point lands. In my experience the memory behavior of a
> > >>> Python Processor is largely a function of how it is written, and disciplined
> > >>> implementations can be well behaved. But I recognize that is an argument about
> > >>> good citizenship, not about enforcement, and it does not give an operator the
> > >>> equivalent of a heap ceiling. That is a real gap. It does seem addressable at
> > >>> the process supervision layer, through cgroup or rlimit bounds on spawned
> > >>> interpreters, rather than being inherent to Python as an extension language.
> > >>>
> > >>> On security, I would offer a friendly amendment. The meaningful difference does
> > >>> not seem to be Python versus Java, but when and where third party dependencies
> > >>> land. A NAR resolves its tree at build time and ships self contained, so it can
> > >>> be scanned, signed, and deployed into an air gapped environment, and unpacking
> > >>> it executes nothing. The current Python path resolves dependencies at runtime
> > >>> via pip on the NiFi host, and pip installation does execute packaging code.
> > >>> That is a real governance gap, and I will concede PyPI carries risks Maven
> > >>> Central does not.
> > >>>
> > >>> But as you noted, the framework already supports loading Python Processors
> > >>> packaged as NARs. What is missing is project-provided tooling to build them.
> > >>> Sean's note about relying on a blog post to produce one seems telling. Closing
> > >>> that gap looks like a smaller undertaking than removing the extension point,
> > >>> and it would address the supply chain concern directly rather than by
> > >>> subtraction.
> > >>>
> > >>> I will admit I had not realized the current implementation still rests on Py4J.
> > >>> That reframes a good deal of this for me, and it tracks with Matt's read that
> > >>> Py4J is the long pole rather than the capability itself.
> > >>>
> > >>> For what it is worth as an operator, the practical value of Python Processors
> > >>> is not just iteration speed. It is that the people writing our transforms are
> > >>> Python native. Removal without a replacement does not convert that work to
> > >>> Java, it pushes it out of NiFi entirely, to sidecars or external services,
> > >>> where, as Marc noted, we keep every one of the concerns you listed but lose
> > >>> NiFi's visibility into them.
> > >>>
> > >>> I would rather see the runtime architecture reconsidered than the capability
> > >>> retired. If the deciding constraint is maintainer cycles, it would be useful to
> > >>> know what a replacement would need in terms of effort and expertise. That is a
> > >>> more actionable ask of the community than a deprecation vote.
> > >>>
> > >>> Regards,
> > >>> Dan
> > >>>
> > >>>
> > >>> From: David Handermann <ex...@apache.org>
> > >>> Date: Wednesday, August 12, 2026 at 9:10 AM
> > >>> To: dev@nifi.apache.org <de...@nifi.apache.org>
> > >>> Subject: Re: [DISCUSS] Future Support for Python Processors?
> > >>>
> > >>> Thanks to everyone for the replies thus far!
> > >>>
> > >>> The question about alternatives gets at the core issue, and the ideas
> > >>> around a more loosely coupled architecture provide some helpful
> > >>> possibilities.
> > >>>
> > >>> At a basic level, simple deprecation and removal would leave the
> > >>> current set of options, including standard Java, Groovy scripting, or
> > >>> externalized Python scripts executed as a stream. None of these
> > >>> provide the same level of features for Python itself, which prompted
> > >>> the original native Python Processor implementation. Without a
> > >>> replacement, these alternatives would remain. With the rise of
> > >>> AI-enabled development, it is now much easier to go from Python to
> > >>> Java, but work is still required.
> > >>>
> > >>> As to future strategies for Python integration, the current discussion
> > >>> around deprecation is more narrowly focused on the existing
> > >>> framework-level extension. Removal of the current implementation is
> > >>> not intended to halt future innovation. Although removal without an
> > >>> initial replacement might not be the perfect solution, it may be
> > >>> necessary given available maintainer cycles, and the substantive
> > >>> issues I described in the opening.
> > >>>
> > >>> If there is sufficient interest and effort to propose and build a
> > >>> different solution, that would be a great outcome of this discussion.
> > >>> I've thought at different points about reworking the Script Engine
> > >>> integration to provide a cleaner path to external extension, so there
> > >>> are definitely opportunities in that direction.
> > >>>
> > >>> With that, my primary concerns with the current implementation remain,
> > >>> but glad to consider future alternatives.
> > >>>
> > >>> Regards,
> > >>> David Handermann
> > >>>
> > >>>
> > >>> On Wed, Aug 12, 2026 at 7:52 AM Matt Burgess <ma...@apache.org> wrote:
> > >>>>
> > >>>> I agree with Marc, the concerns of the implementation of the capability
> > >>>> should not warrant deprecation and future removal of the capability, but
> > >>>> certainly we want to ensure that anything we implement for NiFi doesn't
> > >>>> negatively the reputation of Apache NiFi as a secure, robust, reliable
> > >>>> system.
> > >>>>
> > >>>> I think we should instead consider a new architecture, one
> > >>>> fully extensible so if the community decides to deprecate the capability,
> > >>>> it could still be restored using some extension point. This kind of generic
> > >>>> approach could allow other extensions not sanctioned by the Apache NiFi
> > >>>> community to coexist in the "NiFi+" deployment. Not to get off-topic but it
> > >>>> is somewhat analogous to exposing the Spring context such that alternate
> > >>>> implementations of the beans could be provided.
> > >>>>
> > >>>> There are options to consider, I'm sure each with its own set of pros and
> > >>>> cons. Off the top of my head:
> > >>>>
> > >>>> - Sidecar server responsible for executing Python scripts with FlowFile
> > >>>> input(s) and returning FlowFile output(s) directly from/to processors, like
> > >>>> Marc's suggestion but managed by NiFi
> > >>>> - Jep [1] and it's OS-independent counterpart jep-distro [2]
> > >>>> - Require Java bindings calling out to an embedded interpreter, such as
> > >>>> WrapPy4J (which uses Jep under the hood) [3]
> > >>>>
> > >>>> I am far from a Python expert and I know there are several places in the
> > >>>> framework where the Python stuff has landed, so I don't expect any other
> > >>>> implementation to be any easier or take less time than the original, which
> > >>>> I'm sure took quite a lot of time and effort which is very much
> > >>>> appreciated. It seems that Py4J is the long pole in the tent for the
> > >>>> concerns, not the capability itself.
> > >>>>
> > >>>> At the end of the day I think a discussion like this clearly shows the
> > >>>> commitment of the whole community to ensure Apache NiFi is all that we want
> > >>>> it to be and that we can introspect, improve, and collaborate to keep NiFi
> > >>>> awesome :)
> > >>>>
> > >>>> Regards,
> > >>>> Matt
> > >>>>
> > >>>> [1] https://urldefense.com/v3/__https://github.com/ninia/JEP?tab=readme-ov-file__;!!FtR4BK4x7WL3xYs!6mIniDSbJ62B5zII3b9g972tiJiM1K8M1E-1R8rvwlwl__KEfyNqtqWjM3xwIPbVawsv_VEsYzoNRToUxhft8Q3Rnnw$
> > >>>> [2] https://urldefense.com/v3/__https://github.com/icemachined/jep-distro?tab=readme-ov-file__;!!FtR4BK4x7WL3xYs!6mIniDSbJ62B5zII3b9g972tiJiM1K8M1E-1R8rvwlwl__KEfyNqtqWjM3xwIPbVawsv_VEsYzoNRToUxhft703BJKY$
> > >>>> [3] https://urldefense.com/v3/__https://github.com/313hemant313/wrappyj__;!!FtR4BK4x7WL3xYs!6mIniDSbJ62B5zII3b9g972tiJiM1K8M1E-1R8rvwlwl__KEfyNqtqWjM3xwIPbVawsv_VEsYzoNRToUxhftYE_fqvo$
> > >>>>
> > >>>> On Wed, Aug 12, 2026 at 8:19 AM Marc P. <ma...@gmail.com> wrote:
> > >>>>
> > >>>>> This is a feature I find personally valuable. I agree with several
> > >>>>> underlying issues, particularly resource management, Py4J, dependency
> > >>>>> handling, and the difficulty of maintaining a framework that spans both the
> > >>>>> JVM and Python ecosystems.
> > >>>>>
> > >>>>> Where I disagree is with the conclusion that these issues necessarily point
> > >>>>> toward deprecating Python Processor support itself.
> > >>>>>
> > >>>>> I think there is an important distinction between the value of Python as an
> > >>>>> extension model and the architecture currently used to implement it.
> > >>>>>
> > >>>>> To be solution oriented one way to look at this is to consider what users
> > >>>>> would do in the absence of Python Processor support. In many cases, the
> > >>>>> alternative is not to rewrite the functionality in Java. It is likely to
> > >>>>> become something like:
> > >>>>>
> > >>>>> Apache NiFi -> HTTP/gRPC -> Python microservice -> NiFi
> > >>>>>
> > >>>>> That architecture still has independent Python processes, Python dependency
> > >>>>> management, memory and CPU consumption outside the JVM, and a Python
> > >>>>> software supply chain. We have not eliminated any of those concerns; we
> > >>>>> have moved them outside NiFi and made every operator build and manage that
> > >>>>> integration themselves.
> > >>>>>
> > >>>>> There are also costs to that approach: additional network boundaries,
> > >>>>> deployment artifacts, authentication, service discovery, observability,
> > >>>>> retry semantics, and operational complexity. From a governance perspective,
> > >>>>> NiFi may actually have less visibility and control over that external
> > >>>>> Python workload than it would if Python execution remained part of a
> > >>>>> defined NiFi extension model.
> > >>>>>
> > >>>>> That makes me wonder whether the better direction is to reconsider the
> > >>>>> Python runtime architecture rather than Python support itself.
> > >>>>>
> > >>>>> For example, instead of treating the current Py4J-based implementation as
> > >>>>> something that must be preserved indefinitely, we could define a stronger
> > >>>>> process boundary around Python execution. A future implementation could use
> > >>>>> a well-defined IPC or RPC contract, potentially based on gRPC/protobuf,
> > >>>>> Unix domain sockets, shared memory, Arrow, or another mechanism designed
> > >>>>> around coarse-grained operations.
> > >>>>>
> > >>>>> NiFi does not need to become a Python integrated application, but I think
> > >>>>> there is substantial value in allowing those workloads to participate
> > >>>>> directly in a governed NiFi dataflow without requiring users to construct a
> > >>>>> separate microservice architecture for every Python integration.
> > >>>>>
> > >>>>> For that reason, I would favor separating two questions:
> > >>>>>
> > >>>>>   1. Is the current Python Processor implementation the architecture we
> > >>>>>   want to support long term?
> > >>>>>   2. Should NiFi continue to provide a first-class Python extension
> > >>>>>   capability?
> > >>>>>
> > >>>>> I could see a strong argument that the answer to the first question is no,
> > >>>>> particularly if Py4J is becoming a maintenance risk. I am much less
> > >>>>> convinced that the answer to the second question should also be no.
> > >>>>>
> > >>>>> With that said I distinctly understand that time is limited and we would
> > >>>>> all develop the coolest of things with infinite resources, so I yield to
> > >>>>> those with more influence, but would happy to contribute to a better
> > >>>>> architecture if others find it valuable.
> > >>>>>
> > >>>>> Thanks,
> > >>>>>
> > >>>>> Marc
> > >>>>>
> > >>>>> On Wed, Aug 12, 2026 at 12:10 AM Joe Witt <jo...@gmail.com> wrote:
> > >>>>>
> > >>>>>> You are totally fine to engage on this thread Sean. We appreciate it
> > >>>>>>
> > >>>>>>
> > >>>>>> On Tue, Aug 11, 2026 at 8:17 PM Sean Hunter via dev <dev@nifi.apache.org
> > >>>>>>
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> Hi David,
> > >>>>>>>
> > >>>>>>> As a user, rather than contributor, I think my primary concern is one
> > >>>>> of
> > >>>>>>> operations and implementation. What alternative would be available? We
> > >>>>>> used
> > >>>>>>> the 1.x version and were quite excited to move to 2.0 with its more
> > >>>>>> native
> > >>>>>>> integration that was easier to troubleshoot. We also do build a NAR as
> > >>>>>> part
> > >>>>>>> of our pipeline process, and I suspect since we used a blog as guidance
> > >>>>>> to
> > >>>>>>> develop that process, that others likely do as well. Our use case is a
> > >>>>>>> particularly nasty data transform (Windows logs of some sort,
> > >>>>> naturally)
> > >>>>>>> that’s reasonable to write and execute in Python. It’s readable, while
> > >>>>>>> still being performant enough for our needs, and seemingly not causing
> > >>>>>>> system stability issues for us.
> > >>>>>>>
> > >>>>>>> I do understand and take your points. What I’d like to better
> > >>>>> understand
> > >>>>>>> are the options users have. Do you have any thoughts?
> > >>>>>>>
> > >>>>>>> Thank you and have a great day!
> > >>>>>>> Sean
> > >>>>>>>
> > >>>>>>> PS - Please let me know if this is too far off topic for the dev
> > >>>>> mailing
> > >>>>>>> list and I’ll take it elsewhere.
> > >>>>>>>
> > >>>>>>> From: David Handermann <ex...@apache.org>
> > >>>>>>> Date: Tuesday, August 11, 2026 at 8:38 PM
> > >>>>>>> To: dev@nifi.apache.org <de...@nifi.apache.org>
> > >>>>>>> Subject: [DISCUSS] Future Support for Python Processors?
> > >>>>>>>
> > >>>>>>> Be Careful: Email is from an External Source
> > >>>>>>> Use caution when opening emails and do not click on links or
> > >>>>> attachments
> > >>>>>>> from unknown senders. For more resources, visit
> > >>>>>> http://security.cat.com/phishing .
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> Team,
> > >>>>>>>
> > >>>>>>> The introduction of support for Python Processors was one of the major
> > >>>>>>> new features of Apache NiFi 2. In light of experience, however, I
> > >>>>>>> believe it is time to reconsider future support for this extension
> > >>>>>>> strategy. The idea of removing this feature may raise some concerns,
> > >>>>>>> but it is important to consider the options based on scalability,
> > >>>>>>> security, and supportability.
> > >>>>>>>
> > >>>>>>> One of the greatest benefits of Python-based Processors is rapid
> > >>>>>>> iteration. Although this is a strength during the development phase,
> > >>>>>>> it can become a serious weakness when moving to production.
> > >>>>>>>
> > >>>>>>> NiFi is fundamentally a Java-based application, thus tuning resource
> > >>>>>>> allocation is a key part of maintaining a production system. Running a
> > >>>>>>> Python Processor requires one or more independent Python processes,
> > >>>>>>> each with their own memory consumption and CPU usage patterns. What
> > >>>>>>> might work at a small scale on a development machine breaks down when
> > >>>>>>> attempting to run at production scale, without careful tuning. This is
> > >>>>>>> challenging enough with Java-based components, but global settings
> > >>>>>>> like heap and non-heap memory provide clear boundaries. Python
> > >>>>>>> Processors, as currently implemented, do not support this level of
> > >>>>>>> control. Beyond process control, operations that require communicating
> > >>>>>>> between the NiFi JVM and Python processes involve socket
> > >>>>>>> communication, which can be expensive for frequent calls. Thus an
> > >>>>>>> apparent initial advantage of Python Processors becomes a major
> > >>>>>>> problem when moving beyond limited types of operations.
> > >>>>>>>
> > >>>>>>> When it comes to security, Python Processors can involve more serious
> > >>>>>>> risks than bundled Java extensions.
> > >>>>>>>
> > >>>>>>> With NiFi support for downloading dependencies at runtime, it is
> > >>>>>>> essential for an administrator to provide a secure supply chain
> > >>>>>>> strategy for retrieving these dependencies. Although the NiFi
> > >>>>>>> framework also supports packaging Python Processors as NAR bundles,
> > >>>>>>> this packaging capability is not currently part of the Apache NiFi
> > >>>>>>> project. The project could build on third party work, but as Python
> > >>>>>>> packaging is less opinionated, this is an additional maintenance
> > >>>>>>> question that quickly becomes more involved. This combination of
> > >>>>>>> supply chain security and lack of packaging support raises security
> > >>>>>>> challenges.
> > >>>>>>>
> > >>>>>>> Supportability is always a question for any feature in an open source
> > >>>>>>> project, but Python Processors are a particular point of concern.
> > >>>>>>> Providing NiFi framework support requires a strong knowledge of the
> > >>>>>>> NiFi project itself, and a strong knowledge of Python. Project
> > >>>>>>> maintainers and contributors usually have expertise in particular
> > >>>>>>> areas, but it is rare to find both. More to the point, however, the
> > >>>>>>> Py4J project, which NiFi depends on for integration, has seen limited
> > >>>>>>> maintenance, with the last tagged version dating to January 2025.
> > >>>>>>> Without robust architecture support, the NiFi project cannot provide a
> > >>>>>>> strong framework for future implementation.
> > >>>>>>>
> > >>>>>>> Taking these issues together, I believe the project should consider
> > >>>>>>> deprecating support for Python Processors. Similar to other
> > >>>>>>> deprecations, actual removal would not occur until a major release
> > >>>>>>> version, and that would be a separate conversation.
> > >>>>>>>
> > >>>>>>> Rather than continuing to extend the life of Python Processor support,
> > >>>>>>> it seems better to invest limited project resources into maintaining
> > >>>>>>> and improving core functionality.
> > >>>>>>>
> > >>>>>>> Regards,
> > >>>>>>> David Handermann
> > >>>>>>> Apache NiFi PMC Chair
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>
> >

Re: [DISCUSS] Future Support for Python Processors?

[Gábor Gyimesi <lo...@apache.org>]

Hi,

I also agree with Marc on this. From the user interactions, I got the
feeling that Python processors are quite valuable to the community,
and I would be hesitant to deprecate and remove them when the main
problems are related to the underlying implementation.

I agree with several of the problems David mentioned, particularly
around scalability and dependency installation. These issues were not
foreseen with the initial release of NiFi 2.0 and only became apparent
as these processors started being used more extensively by the
community, especially with heavy AI libraries. The problem is that,
after the initial release of 2.0, there has been very little
development on the Python side, so these issues were never addressed.

From a user's point of view, I think the Python processor API is
simple and easy to use. Even with the current implementation, there
are several use cases for which it is sufficient, so I would not
remove the option to use Python processors. Deprecating them would
discourage people from creating and using Python processors, even for
use cases where they make sense. In my opinion, the documentation for
Python processors should instead be extended to clearly explain the
caveats of using them, including what problems users can expect in
terms of dependency management, scalability, etc., and which use cases
are recommended for these processors.

In the future, I would support developing an alternative
implementation under the hood, replacing Py4J, if we decide to pursue
IPC communication, use an embedded interpreter, or adopt any of the
other solutions Matt mentioned. In MiNiFi C++, we went with an
embedded interpreter solution, which provides much better performance,
with the caveat that only one interpreter can be embedded in a MiNiFi
process and is therefore shared between processors. The dependency
management issue is still a challenge in MiNiFi C++ as well, and I
would support creating a packaging solution that could work for both
projects.

As Marc said, from a user's point of view, in many cases it is not
really an option to rewrite a processor in Java, as many libraries,
especially AI libraries, are only available in Python. If the decision
is made to deprecate and eventually remove Python processor support
from NiFi, forcing users to create a separate Python microservice
based on Marc's suggestion of a "Apache NiFi -> HTTP/gRPC -> Python
microservice -> NiFi" architecture, we could consider officially
recommending MiNiFi C++ as an interim solution, since it retains
Python support. The same approach could potentially be used for
anything requiring C/C++ or Rust support as well.

Regards,
Gabor

On Thu, 13 Aug 2026 at 21:51, Mark Payne <ma...@hotmail.com> wrote:
>
> Kevin,
>
> Py4J is the glue that bridges the NiFi JVM to the python process that runs python processors. So there’s a py4j server that is run in a separate python process. Then a py4j client that runs in the JVM. Py4J handles the RPC to make it possible to call the Python processors as if they were Java objects.
>
> Thank
> -Makr
>
>
> > On Aug 13, 2026, at 3:37 PM, Kevin Doran <kd...@apache.org> wrote:
> >
> > Can someone clarify where Py4J is used today vs where subprocesses
> > that invoke a configurable python executable are used? My
> > understanding was that the Py4J was mostly restricted to the scripting
> > processors, but if that is not the case I would be interested in
> > hearing more.
> >
> > On Thu, Aug 13, 2026 at 1:42 PM David Handermann
> > <ex...@apache.org> wrote:
> >>
> >> Dan,
> >>
> >> Thanks for the substantive reply and consideration of the various
> >> issues surrounding support for native Python Processors.
> >>
> >> As you highlighted, the scalability issue is foundational. Although
> >> Processor authors can work with self-imposed constraints, the NiFi
> >> framework does not provide the same level of safeguards. Any future
> >> solution would need to account for these types of production-level
> >> concerns.
> >>
> >> Regarding supply chain security, you are correct that there are
> >> potential paths forward on the packaging strategy. The framework could
> >> remove support for runtime loading as a safety measure. Right now,
> >> lack of pip or uv effectively disables runtime loading. Adjusting the
> >> default project container configuration would strengthen the standard
> >> position, and thus require Python Processors to be packaged using a
> >> NAR. The project could provide tooling, similar to the NiFi NAR Maven
> >> Plugin, but this broadens the maintenance scope in a meaningful way.
> >> Thus before going in this direction, the future support strategy needs
> >> to be considered.
> >>
> >> The Py4J architecture remains a critical concern and high risk for
> >> future maintainability. Considering an alternative architecture would
> >> require a substantial redesign and rewrite of Python Processor
> >> support, abandoning a majority of the current implementation. Current
> >> tooling makes this more plausible in theory, but is a significant
> >> undertaking that is difficult to justify without serious commitment on
> >> the part of multiple project maintainers. To be clear, deprecating the
> >> current implementation for removal does not preclude future redesign.
> >> These two questions should be considered independently.
> >>
> >> Deprecation and removal without replacement will have a different
> >> impact depending on the current scope of usage. Simpler Python
> >> Processors should be easier to convert to Java, but more complex
> >> implementations would likely require something outside of NiFi, as you
> >> mentioned. Although these are difficult questions, it ultimately comes
> >> down to project focus and available maintenance cycles. Apache NiFi
> >> has historically done many things. NiFi 2 served as a focusing
> >> function. Going forward, maintaining project health and stability is
> >> paramount, which may at times involve removing certain capabilities.
> >>
> >> Regards,
> >> David Handermann
> >>
> >> On Wed, Aug 12, 2026 at 3:27 PM Daniel Weidman via dev
> >> <de...@nifi.apache.org> wrote:
> >>>
> >>> David,
> >>>
> >>> Thanks for raising this directly rather than letting it drift, and for laying
> >>> out the reasoning rather than just the conclusion.
> >>>
> >>> On scalability, your point lands. In my experience the memory behavior of a
> >>> Python Processor is largely a function of how it is written, and disciplined
> >>> implementations can be well behaved. But I recognize that is an argument about
> >>> good citizenship, not about enforcement, and it does not give an operator the
> >>> equivalent of a heap ceiling. That is a real gap. It does seem addressable at
> >>> the process supervision layer, through cgroup or rlimit bounds on spawned
> >>> interpreters, rather than being inherent to Python as an extension language.
> >>>
> >>> On security, I would offer a friendly amendment. The meaningful difference does
> >>> not seem to be Python versus Java, but when and where third party dependencies
> >>> land. A NAR resolves its tree at build time and ships self contained, so it can
> >>> be scanned, signed, and deployed into an air gapped environment, and unpacking
> >>> it executes nothing. The current Python path resolves dependencies at runtime
> >>> via pip on the NiFi host, and pip installation does execute packaging code.
> >>> That is a real governance gap, and I will concede PyPI carries risks Maven
> >>> Central does not.
> >>>
> >>> But as you noted, the framework already supports loading Python Processors
> >>> packaged as NARs. What is missing is project-provided tooling to build them.
> >>> Sean's note about relying on a blog post to produce one seems telling. Closing
> >>> that gap looks like a smaller undertaking than removing the extension point,
> >>> and it would address the supply chain concern directly rather than by
> >>> subtraction.
> >>>
> >>> I will admit I had not realized the current implementation still rests on Py4J.
> >>> That reframes a good deal of this for me, and it tracks with Matt's read that
> >>> Py4J is the long pole rather than the capability itself.
> >>>
> >>> For what it is worth as an operator, the practical value of Python Processors
> >>> is not just iteration speed. It is that the people writing our transforms are
> >>> Python native. Removal without a replacement does not convert that work to
> >>> Java, it pushes it out of NiFi entirely, to sidecars or external services,
> >>> where, as Marc noted, we keep every one of the concerns you listed but lose
> >>> NiFi's visibility into them.
> >>>
> >>> I would rather see the runtime architecture reconsidered than the capability
> >>> retired. If the deciding constraint is maintainer cycles, it would be useful to
> >>> know what a replacement would need in terms of effort and expertise. That is a
> >>> more actionable ask of the community than a deprecation vote.
> >>>
> >>> Regards,
> >>> Dan
> >>>
> >>>
> >>> From: David Handermann <ex...@apache.org>
> >>> Date: Wednesday, August 12, 2026 at 9:10 AM
> >>> To: dev@nifi.apache.org <de...@nifi.apache.org>
> >>> Subject: Re: [DISCUSS] Future Support for Python Processors?
> >>>
> >>> Thanks to everyone for the replies thus far!
> >>>
> >>> The question about alternatives gets at the core issue, and the ideas
> >>> around a more loosely coupled architecture provide some helpful
> >>> possibilities.
> >>>
> >>> At a basic level, simple deprecation and removal would leave the
> >>> current set of options, including standard Java, Groovy scripting, or
> >>> externalized Python scripts executed as a stream. None of these
> >>> provide the same level of features for Python itself, which prompted
> >>> the original native Python Processor implementation. Without a
> >>> replacement, these alternatives would remain. With the rise of
> >>> AI-enabled development, it is now much easier to go from Python to
> >>> Java, but work is still required.
> >>>
> >>> As to future strategies for Python integration, the current discussion
> >>> around deprecation is more narrowly focused on the existing
> >>> framework-level extension. Removal of the current implementation is
> >>> not intended to halt future innovation. Although removal without an
> >>> initial replacement might not be the perfect solution, it may be
> >>> necessary given available maintainer cycles, and the substantive
> >>> issues I described in the opening.
> >>>
> >>> If there is sufficient interest and effort to propose and build a
> >>> different solution, that would be a great outcome of this discussion.
> >>> I've thought at different points about reworking the Script Engine
> >>> integration to provide a cleaner path to external extension, so there
> >>> are definitely opportunities in that direction.
> >>>
> >>> With that, my primary concerns with the current implementation remain,
> >>> but glad to consider future alternatives.
> >>>
> >>> Regards,
> >>> David Handermann
> >>>
> >>>
> >>> On Wed, Aug 12, 2026 at 7:52 AM Matt Burgess <ma...@apache.org> wrote:
> >>>>
> >>>> I agree with Marc, the concerns of the implementation of the capability
> >>>> should not warrant deprecation and future removal of the capability, but
> >>>> certainly we want to ensure that anything we implement for NiFi doesn't
> >>>> negatively the reputation of Apache NiFi as a secure, robust, reliable
> >>>> system.
> >>>>
> >>>> I think we should instead consider a new architecture, one
> >>>> fully extensible so if the community decides to deprecate the capability,
> >>>> it could still be restored using some extension point. This kind of generic
> >>>> approach could allow other extensions not sanctioned by the Apache NiFi
> >>>> community to coexist in the "NiFi+" deployment. Not to get off-topic but it
> >>>> is somewhat analogous to exposing the Spring context such that alternate
> >>>> implementations of the beans could be provided.
> >>>>
> >>>> There are options to consider, I'm sure each with its own set of pros and
> >>>> cons. Off the top of my head:
> >>>>
> >>>> - Sidecar server responsible for executing Python scripts with FlowFile
> >>>> input(s) and returning FlowFile output(s) directly from/to processors, like
> >>>> Marc's suggestion but managed by NiFi
> >>>> - Jep [1] and it's OS-independent counterpart jep-distro [2]
> >>>> - Require Java bindings calling out to an embedded interpreter, such as
> >>>> WrapPy4J (which uses Jep under the hood) [3]
> >>>>
> >>>> I am far from a Python expert and I know there are several places in the
> >>>> framework where the Python stuff has landed, so I don't expect any other
> >>>> implementation to be any easier or take less time than the original, which
> >>>> I'm sure took quite a lot of time and effort which is very much
> >>>> appreciated. It seems that Py4J is the long pole in the tent for the
> >>>> concerns, not the capability itself.
> >>>>
> >>>> At the end of the day I think a discussion like this clearly shows the
> >>>> commitment of the whole community to ensure Apache NiFi is all that we want
> >>>> it to be and that we can introspect, improve, and collaborate to keep NiFi
> >>>> awesome :)
> >>>>
> >>>> Regards,
> >>>> Matt
> >>>>
> >>>> [1] https://urldefense.com/v3/__https://github.com/ninia/JEP?tab=readme-ov-file__;!!FtR4BK4x7WL3xYs!6mIniDSbJ62B5zII3b9g972tiJiM1K8M1E-1R8rvwlwl__KEfyNqtqWjM3xwIPbVawsv_VEsYzoNRToUxhft8Q3Rnnw$
> >>>> [2] https://urldefense.com/v3/__https://github.com/icemachined/jep-distro?tab=readme-ov-file__;!!FtR4BK4x7WL3xYs!6mIniDSbJ62B5zII3b9g972tiJiM1K8M1E-1R8rvwlwl__KEfyNqtqWjM3xwIPbVawsv_VEsYzoNRToUxhft703BJKY$
> >>>> [3] https://urldefense.com/v3/__https://github.com/313hemant313/wrappyj__;!!FtR4BK4x7WL3xYs!6mIniDSbJ62B5zII3b9g972tiJiM1K8M1E-1R8rvwlwl__KEfyNqtqWjM3xwIPbVawsv_VEsYzoNRToUxhftYE_fqvo$
> >>>>
> >>>> On Wed, Aug 12, 2026 at 8:19 AM Marc P. <ma...@gmail.com> wrote:
> >>>>
> >>>>> This is a feature I find personally valuable. I agree with several
> >>>>> underlying issues, particularly resource management, Py4J, dependency
> >>>>> handling, and the difficulty of maintaining a framework that spans both the
> >>>>> JVM and Python ecosystems.
> >>>>>
> >>>>> Where I disagree is with the conclusion that these issues necessarily point
> >>>>> toward deprecating Python Processor support itself.
> >>>>>
> >>>>> I think there is an important distinction between the value of Python as an
> >>>>> extension model and the architecture currently used to implement it.
> >>>>>
> >>>>> To be solution oriented one way to look at this is to consider what users
> >>>>> would do in the absence of Python Processor support. In many cases, the
> >>>>> alternative is not to rewrite the functionality in Java. It is likely to
> >>>>> become something like:
> >>>>>
> >>>>> Apache NiFi -> HTTP/gRPC -> Python microservice -> NiFi
> >>>>>
> >>>>> That architecture still has independent Python processes, Python dependency
> >>>>> management, memory and CPU consumption outside the JVM, and a Python
> >>>>> software supply chain. We have not eliminated any of those concerns; we
> >>>>> have moved them outside NiFi and made every operator build and manage that
> >>>>> integration themselves.
> >>>>>
> >>>>> There are also costs to that approach: additional network boundaries,
> >>>>> deployment artifacts, authentication, service discovery, observability,
> >>>>> retry semantics, and operational complexity. From a governance perspective,
> >>>>> NiFi may actually have less visibility and control over that external
> >>>>> Python workload than it would if Python execution remained part of a
> >>>>> defined NiFi extension model.
> >>>>>
> >>>>> That makes me wonder whether the better direction is to reconsider the
> >>>>> Python runtime architecture rather than Python support itself.
> >>>>>
> >>>>> For example, instead of treating the current Py4J-based implementation as
> >>>>> something that must be preserved indefinitely, we could define a stronger
> >>>>> process boundary around Python execution. A future implementation could use
> >>>>> a well-defined IPC or RPC contract, potentially based on gRPC/protobuf,
> >>>>> Unix domain sockets, shared memory, Arrow, or another mechanism designed
> >>>>> around coarse-grained operations.
> >>>>>
> >>>>> NiFi does not need to become a Python integrated application, but I think
> >>>>> there is substantial value in allowing those workloads to participate
> >>>>> directly in a governed NiFi dataflow without requiring users to construct a
> >>>>> separate microservice architecture for every Python integration.
> >>>>>
> >>>>> For that reason, I would favor separating two questions:
> >>>>>
> >>>>>   1. Is the current Python Processor implementation the architecture we
> >>>>>   want to support long term?
> >>>>>   2. Should NiFi continue to provide a first-class Python extension
> >>>>>   capability?
> >>>>>
> >>>>> I could see a strong argument that the answer to the first question is no,
> >>>>> particularly if Py4J is becoming a maintenance risk. I am much less
> >>>>> convinced that the answer to the second question should also be no.
> >>>>>
> >>>>> With that said I distinctly understand that time is limited and we would
> >>>>> all develop the coolest of things with infinite resources, so I yield to
> >>>>> those with more influence, but would happy to contribute to a better
> >>>>> architecture if others find it valuable.
> >>>>>
> >>>>> Thanks,
> >>>>>
> >>>>> Marc
> >>>>>
> >>>>> On Wed, Aug 12, 2026 at 12:10 AM Joe Witt <jo...@gmail.com> wrote:
> >>>>>
> >>>>>> You are totally fine to engage on this thread Sean. We appreciate it
> >>>>>>
> >>>>>>
> >>>>>> On Tue, Aug 11, 2026 at 8:17 PM Sean Hunter via dev <dev@nifi.apache.org
> >>>>>>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Hi David,
> >>>>>>>
> >>>>>>> As a user, rather than contributor, I think my primary concern is one
> >>>>> of
> >>>>>>> operations and implementation. What alternative would be available? We
> >>>>>> used
> >>>>>>> the 1.x version and were quite excited to move to 2.0 with its more
> >>>>>> native
> >>>>>>> integration that was easier to troubleshoot. We also do build a NAR as
> >>>>>> part
> >>>>>>> of our pipeline process, and I suspect since we used a blog as guidance
> >>>>>> to
> >>>>>>> develop that process, that others likely do as well. Our use case is a
> >>>>>>> particularly nasty data transform (Windows logs of some sort,
> >>>>> naturally)
> >>>>>>> that’s reasonable to write and execute in Python. It’s readable, while
> >>>>>>> still being performant enough for our needs, and seemingly not causing
> >>>>>>> system stability issues for us.
> >>>>>>>
> >>>>>>> I do understand and take your points. What I’d like to better
> >>>>> understand
> >>>>>>> are the options users have. Do you have any thoughts?
> >>>>>>>
> >>>>>>> Thank you and have a great day!
> >>>>>>> Sean
> >>>>>>>
> >>>>>>> PS - Please let me know if this is too far off topic for the dev
> >>>>> mailing
> >>>>>>> list and I’ll take it elsewhere.
> >>>>>>>
> >>>>>>> From: David Handermann <ex...@apache.org>
> >>>>>>> Date: Tuesday, August 11, 2026 at 8:38 PM
> >>>>>>> To: dev@nifi.apache.org <de...@nifi.apache.org>
> >>>>>>> Subject: [DISCUSS] Future Support for Python Processors?
> >>>>>>>
> >>>>>>> Be Careful: Email is from an External Source
> >>>>>>> Use caution when opening emails and do not click on links or
> >>>>> attachments
> >>>>>>> from unknown senders. For more resources, visit
> >>>>>> http://security.cat.com/phishing .
> >>>>>>>
> >>>>>>>
> >>>>>>> Team,
> >>>>>>>
> >>>>>>> The introduction of support for Python Processors was one of the major
> >>>>>>> new features of Apache NiFi 2. In light of experience, however, I
> >>>>>>> believe it is time to reconsider future support for this extension
> >>>>>>> strategy. The idea of removing this feature may raise some concerns,
> >>>>>>> but it is important to consider the options based on scalability,
> >>>>>>> security, and supportability.
> >>>>>>>
> >>>>>>> One of the greatest benefits of Python-based Processors is rapid
> >>>>>>> iteration. Although this is a strength during the development phase,
> >>>>>>> it can become a serious weakness when moving to production.
> >>>>>>>
> >>>>>>> NiFi is fundamentally a Java-based application, thus tuning resource
> >>>>>>> allocation is a key part of maintaining a production system. Running a
> >>>>>>> Python Processor requires one or more independent Python processes,
> >>>>>>> each with their own memory consumption and CPU usage patterns. What
> >>>>>>> might work at a small scale on a development machine breaks down when
> >>>>>>> attempting to run at production scale, without careful tuning. This is
> >>>>>>> challenging enough with Java-based components, but global settings
> >>>>>>> like heap and non-heap memory provide clear boundaries. Python
> >>>>>>> Processors, as currently implemented, do not support this level of
> >>>>>>> control. Beyond process control, operations that require communicating
> >>>>>>> between the NiFi JVM and Python processes involve socket
> >>>>>>> communication, which can be expensive for frequent calls. Thus an
> >>>>>>> apparent initial advantage of Python Processors becomes a major
> >>>>>>> problem when moving beyond limited types of operations.
> >>>>>>>
> >>>>>>> When it comes to security, Python Processors can involve more serious
> >>>>>>> risks than bundled Java extensions.
> >>>>>>>
> >>>>>>> With NiFi support for downloading dependencies at runtime, it is
> >>>>>>> essential for an administrator to provide a secure supply chain
> >>>>>>> strategy for retrieving these dependencies. Although the NiFi
> >>>>>>> framework also supports packaging Python Processors as NAR bundles,
> >>>>>>> this packaging capability is not currently part of the Apache NiFi
> >>>>>>> project. The project could build on third party work, but as Python
> >>>>>>> packaging is less opinionated, this is an additional maintenance
> >>>>>>> question that quickly becomes more involved. This combination of
> >>>>>>> supply chain security and lack of packaging support raises security
> >>>>>>> challenges.
> >>>>>>>
> >>>>>>> Supportability is always a question for any feature in an open source
> >>>>>>> project, but Python Processors are a particular point of concern.
> >>>>>>> Providing NiFi framework support requires a strong knowledge of the
> >>>>>>> NiFi project itself, and a strong knowledge of Python. Project
> >>>>>>> maintainers and contributors usually have expertise in particular
> >>>>>>> areas, but it is rare to find both. More to the point, however, the
> >>>>>>> Py4J project, which NiFi depends on for integration, has seen limited
> >>>>>>> maintenance, with the last tagged version dating to January 2025.
> >>>>>>> Without robust architecture support, the NiFi project cannot provide a
> >>>>>>> strong framework for future implementation.
> >>>>>>>
> >>>>>>> Taking these issues together, I believe the project should consider
> >>>>>>> deprecating support for Python Processors. Similar to other
> >>>>>>> deprecations, actual removal would not occur until a major release
> >>>>>>> version, and that would be a separate conversation.
> >>>>>>>
> >>>>>>> Rather than continuing to extend the life of Python Processor support,
> >>>>>>> it seems better to invest limited project resources into maintaining
> >>>>>>> and improving core functionality.
> >>>>>>>
> >>>>>>> Regards,
> >>>>>>> David Handermann
> >>>>>>> Apache NiFi PMC Chair
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>
>

Re: [DISCUSS] Future Support for Python Processors?

[Mark Payne <ma...@hotmail.com>]

Kevin,

Py4J is the glue that bridges the NiFi JVM to the python process that runs python processors. So there’s a py4j server that is run in a separate python process. Then a py4j client that runs in the JVM. Py4J handles the RPC to make it possible to call the Python processors as if they were Java objects.

Thank
-Makr


> On Aug 13, 2026, at 3:37 PM, Kevin Doran <kd...@apache.org> wrote:
> 
> Can someone clarify where Py4J is used today vs where subprocesses
> that invoke a configurable python executable are used? My
> understanding was that the Py4J was mostly restricted to the scripting
> processors, but if that is not the case I would be interested in
> hearing more.
> 
> On Thu, Aug 13, 2026 at 1:42 PM David Handermann
> <ex...@apache.org> wrote:
>> 
>> Dan,
>> 
>> Thanks for the substantive reply and consideration of the various
>> issues surrounding support for native Python Processors.
>> 
>> As you highlighted, the scalability issue is foundational. Although
>> Processor authors can work with self-imposed constraints, the NiFi
>> framework does not provide the same level of safeguards. Any future
>> solution would need to account for these types of production-level
>> concerns.
>> 
>> Regarding supply chain security, you are correct that there are
>> potential paths forward on the packaging strategy. The framework could
>> remove support for runtime loading as a safety measure. Right now,
>> lack of pip or uv effectively disables runtime loading. Adjusting the
>> default project container configuration would strengthen the standard
>> position, and thus require Python Processors to be packaged using a
>> NAR. The project could provide tooling, similar to the NiFi NAR Maven
>> Plugin, but this broadens the maintenance scope in a meaningful way.
>> Thus before going in this direction, the future support strategy needs
>> to be considered.
>> 
>> The Py4J architecture remains a critical concern and high risk for
>> future maintainability. Considering an alternative architecture would
>> require a substantial redesign and rewrite of Python Processor
>> support, abandoning a majority of the current implementation. Current
>> tooling makes this more plausible in theory, but is a significant
>> undertaking that is difficult to justify without serious commitment on
>> the part of multiple project maintainers. To be clear, deprecating the
>> current implementation for removal does not preclude future redesign.
>> These two questions should be considered independently.
>> 
>> Deprecation and removal without replacement will have a different
>> impact depending on the current scope of usage. Simpler Python
>> Processors should be easier to convert to Java, but more complex
>> implementations would likely require something outside of NiFi, as you
>> mentioned. Although these are difficult questions, it ultimately comes
>> down to project focus and available maintenance cycles. Apache NiFi
>> has historically done many things. NiFi 2 served as a focusing
>> function. Going forward, maintaining project health and stability is
>> paramount, which may at times involve removing certain capabilities.
>> 
>> Regards,
>> David Handermann
>> 
>> On Wed, Aug 12, 2026 at 3:27 PM Daniel Weidman via dev
>> <de...@nifi.apache.org> wrote:
>>> 
>>> David,
>>> 
>>> Thanks for raising this directly rather than letting it drift, and for laying
>>> out the reasoning rather than just the conclusion.
>>> 
>>> On scalability, your point lands. In my experience the memory behavior of a
>>> Python Processor is largely a function of how it is written, and disciplined
>>> implementations can be well behaved. But I recognize that is an argument about
>>> good citizenship, not about enforcement, and it does not give an operator the
>>> equivalent of a heap ceiling. That is a real gap. It does seem addressable at
>>> the process supervision layer, through cgroup or rlimit bounds on spawned
>>> interpreters, rather than being inherent to Python as an extension language.
>>> 
>>> On security, I would offer a friendly amendment. The meaningful difference does
>>> not seem to be Python versus Java, but when and where third party dependencies
>>> land. A NAR resolves its tree at build time and ships self contained, so it can
>>> be scanned, signed, and deployed into an air gapped environment, and unpacking
>>> it executes nothing. The current Python path resolves dependencies at runtime
>>> via pip on the NiFi host, and pip installation does execute packaging code.
>>> That is a real governance gap, and I will concede PyPI carries risks Maven
>>> Central does not.
>>> 
>>> But as you noted, the framework already supports loading Python Processors
>>> packaged as NARs. What is missing is project-provided tooling to build them.
>>> Sean's note about relying on a blog post to produce one seems telling. Closing
>>> that gap looks like a smaller undertaking than removing the extension point,
>>> and it would address the supply chain concern directly rather than by
>>> subtraction.
>>> 
>>> I will admit I had not realized the current implementation still rests on Py4J.
>>> That reframes a good deal of this for me, and it tracks with Matt's read that
>>> Py4J is the long pole rather than the capability itself.
>>> 
>>> For what it is worth as an operator, the practical value of Python Processors
>>> is not just iteration speed. It is that the people writing our transforms are
>>> Python native. Removal without a replacement does not convert that work to
>>> Java, it pushes it out of NiFi entirely, to sidecars or external services,
>>> where, as Marc noted, we keep every one of the concerns you listed but lose
>>> NiFi's visibility into them.
>>> 
>>> I would rather see the runtime architecture reconsidered than the capability
>>> retired. If the deciding constraint is maintainer cycles, it would be useful to
>>> know what a replacement would need in terms of effort and expertise. That is a
>>> more actionable ask of the community than a deprecation vote.
>>> 
>>> Regards,
>>> Dan
>>> 
>>> 
>>> From: David Handermann <ex...@apache.org>
>>> Date: Wednesday, August 12, 2026 at 9:10 AM
>>> To: dev@nifi.apache.org <de...@nifi.apache.org>
>>> Subject: Re: [DISCUSS] Future Support for Python Processors?
>>> 
>>> Thanks to everyone for the replies thus far!
>>> 
>>> The question about alternatives gets at the core issue, and the ideas
>>> around a more loosely coupled architecture provide some helpful
>>> possibilities.
>>> 
>>> At a basic level, simple deprecation and removal would leave the
>>> current set of options, including standard Java, Groovy scripting, or
>>> externalized Python scripts executed as a stream. None of these
>>> provide the same level of features for Python itself, which prompted
>>> the original native Python Processor implementation. Without a
>>> replacement, these alternatives would remain. With the rise of
>>> AI-enabled development, it is now much easier to go from Python to
>>> Java, but work is still required.
>>> 
>>> As to future strategies for Python integration, the current discussion
>>> around deprecation is more narrowly focused on the existing
>>> framework-level extension. Removal of the current implementation is
>>> not intended to halt future innovation. Although removal without an
>>> initial replacement might not be the perfect solution, it may be
>>> necessary given available maintainer cycles, and the substantive
>>> issues I described in the opening.
>>> 
>>> If there is sufficient interest and effort to propose and build a
>>> different solution, that would be a great outcome of this discussion.
>>> I've thought at different points about reworking the Script Engine
>>> integration to provide a cleaner path to external extension, so there
>>> are definitely opportunities in that direction.
>>> 
>>> With that, my primary concerns with the current implementation remain,
>>> but glad to consider future alternatives.
>>> 
>>> Regards,
>>> David Handermann
>>> 
>>> 
>>> On Wed, Aug 12, 2026 at 7:52 AM Matt Burgess <ma...@apache.org> wrote:
>>>> 
>>>> I agree with Marc, the concerns of the implementation of the capability
>>>> should not warrant deprecation and future removal of the capability, but
>>>> certainly we want to ensure that anything we implement for NiFi doesn't
>>>> negatively the reputation of Apache NiFi as a secure, robust, reliable
>>>> system.
>>>> 
>>>> I think we should instead consider a new architecture, one
>>>> fully extensible so if the community decides to deprecate the capability,
>>>> it could still be restored using some extension point. This kind of generic
>>>> approach could allow other extensions not sanctioned by the Apache NiFi
>>>> community to coexist in the "NiFi+" deployment. Not to get off-topic but it
>>>> is somewhat analogous to exposing the Spring context such that alternate
>>>> implementations of the beans could be provided.
>>>> 
>>>> There are options to consider, I'm sure each with its own set of pros and
>>>> cons. Off the top of my head:
>>>> 
>>>> - Sidecar server responsible for executing Python scripts with FlowFile
>>>> input(s) and returning FlowFile output(s) directly from/to processors, like
>>>> Marc's suggestion but managed by NiFi
>>>> - Jep [1] and it's OS-independent counterpart jep-distro [2]
>>>> - Require Java bindings calling out to an embedded interpreter, such as
>>>> WrapPy4J (which uses Jep under the hood) [3]
>>>> 
>>>> I am far from a Python expert and I know there are several places in the
>>>> framework where the Python stuff has landed, so I don't expect any other
>>>> implementation to be any easier or take less time than the original, which
>>>> I'm sure took quite a lot of time and effort which is very much
>>>> appreciated. It seems that Py4J is the long pole in the tent for the
>>>> concerns, not the capability itself.
>>>> 
>>>> At the end of the day I think a discussion like this clearly shows the
>>>> commitment of the whole community to ensure Apache NiFi is all that we want
>>>> it to be and that we can introspect, improve, and collaborate to keep NiFi
>>>> awesome :)
>>>> 
>>>> Regards,
>>>> Matt
>>>> 
>>>> [1] https://urldefense.com/v3/__https://github.com/ninia/JEP?tab=readme-ov-file__;!!FtR4BK4x7WL3xYs!6mIniDSbJ62B5zII3b9g972tiJiM1K8M1E-1R8rvwlwl__KEfyNqtqWjM3xwIPbVawsv_VEsYzoNRToUxhft8Q3Rnnw$
>>>> [2] https://urldefense.com/v3/__https://github.com/icemachined/jep-distro?tab=readme-ov-file__;!!FtR4BK4x7WL3xYs!6mIniDSbJ62B5zII3b9g972tiJiM1K8M1E-1R8rvwlwl__KEfyNqtqWjM3xwIPbVawsv_VEsYzoNRToUxhft703BJKY$
>>>> [3] https://urldefense.com/v3/__https://github.com/313hemant313/wrappyj__;!!FtR4BK4x7WL3xYs!6mIniDSbJ62B5zII3b9g972tiJiM1K8M1E-1R8rvwlwl__KEfyNqtqWjM3xwIPbVawsv_VEsYzoNRToUxhftYE_fqvo$
>>>> 
>>>> On Wed, Aug 12, 2026 at 8:19 AM Marc P. <ma...@gmail.com> wrote:
>>>> 
>>>>> This is a feature I find personally valuable. I agree with several
>>>>> underlying issues, particularly resource management, Py4J, dependency
>>>>> handling, and the difficulty of maintaining a framework that spans both the
>>>>> JVM and Python ecosystems.
>>>>> 
>>>>> Where I disagree is with the conclusion that these issues necessarily point
>>>>> toward deprecating Python Processor support itself.
>>>>> 
>>>>> I think there is an important distinction between the value of Python as an
>>>>> extension model and the architecture currently used to implement it.
>>>>> 
>>>>> To be solution oriented one way to look at this is to consider what users
>>>>> would do in the absence of Python Processor support. In many cases, the
>>>>> alternative is not to rewrite the functionality in Java. It is likely to
>>>>> become something like:
>>>>> 
>>>>> Apache NiFi -> HTTP/gRPC -> Python microservice -> NiFi
>>>>> 
>>>>> That architecture still has independent Python processes, Python dependency
>>>>> management, memory and CPU consumption outside the JVM, and a Python
>>>>> software supply chain. We have not eliminated any of those concerns; we
>>>>> have moved them outside NiFi and made every operator build and manage that
>>>>> integration themselves.
>>>>> 
>>>>> There are also costs to that approach: additional network boundaries,
>>>>> deployment artifacts, authentication, service discovery, observability,
>>>>> retry semantics, and operational complexity. From a governance perspective,
>>>>> NiFi may actually have less visibility and control over that external
>>>>> Python workload than it would if Python execution remained part of a
>>>>> defined NiFi extension model.
>>>>> 
>>>>> That makes me wonder whether the better direction is to reconsider the
>>>>> Python runtime architecture rather than Python support itself.
>>>>> 
>>>>> For example, instead of treating the current Py4J-based implementation as
>>>>> something that must be preserved indefinitely, we could define a stronger
>>>>> process boundary around Python execution. A future implementation could use
>>>>> a well-defined IPC or RPC contract, potentially based on gRPC/protobuf,
>>>>> Unix domain sockets, shared memory, Arrow, or another mechanism designed
>>>>> around coarse-grained operations.
>>>>> 
>>>>> NiFi does not need to become a Python integrated application, but I think
>>>>> there is substantial value in allowing those workloads to participate
>>>>> directly in a governed NiFi dataflow without requiring users to construct a
>>>>> separate microservice architecture for every Python integration.
>>>>> 
>>>>> For that reason, I would favor separating two questions:
>>>>> 
>>>>>   1. Is the current Python Processor implementation the architecture we
>>>>>   want to support long term?
>>>>>   2. Should NiFi continue to provide a first-class Python extension
>>>>>   capability?
>>>>> 
>>>>> I could see a strong argument that the answer to the first question is no,
>>>>> particularly if Py4J is becoming a maintenance risk. I am much less
>>>>> convinced that the answer to the second question should also be no.
>>>>> 
>>>>> With that said I distinctly understand that time is limited and we would
>>>>> all develop the coolest of things with infinite resources, so I yield to
>>>>> those with more influence, but would happy to contribute to a better
>>>>> architecture if others find it valuable.
>>>>> 
>>>>> Thanks,
>>>>> 
>>>>> Marc
>>>>> 
>>>>> On Wed, Aug 12, 2026 at 12:10 AM Joe Witt <jo...@gmail.com> wrote:
>>>>> 
>>>>>> You are totally fine to engage on this thread Sean. We appreciate it
>>>>>> 
>>>>>> 
>>>>>> On Tue, Aug 11, 2026 at 8:17 PM Sean Hunter via dev <dev@nifi.apache.org
>>>>>> 
>>>>>> wrote:
>>>>>> 
>>>>>>> Hi David,
>>>>>>> 
>>>>>>> As a user, rather than contributor, I think my primary concern is one
>>>>> of
>>>>>>> operations and implementation. What alternative would be available? We
>>>>>> used
>>>>>>> the 1.x version and were quite excited to move to 2.0 with its more
>>>>>> native
>>>>>>> integration that was easier to troubleshoot. We also do build a NAR as
>>>>>> part
>>>>>>> of our pipeline process, and I suspect since we used a blog as guidance
>>>>>> to
>>>>>>> develop that process, that others likely do as well. Our use case is a
>>>>>>> particularly nasty data transform (Windows logs of some sort,
>>>>> naturally)
>>>>>>> that’s reasonable to write and execute in Python. It’s readable, while
>>>>>>> still being performant enough for our needs, and seemingly not causing
>>>>>>> system stability issues for us.
>>>>>>> 
>>>>>>> I do understand and take your points. What I’d like to better
>>>>> understand
>>>>>>> are the options users have. Do you have any thoughts?
>>>>>>> 
>>>>>>> Thank you and have a great day!
>>>>>>> Sean
>>>>>>> 
>>>>>>> PS - Please let me know if this is too far off topic for the dev
>>>>> mailing
>>>>>>> list and I’ll take it elsewhere.
>>>>>>> 
>>>>>>> From: David Handermann <ex...@apache.org>
>>>>>>> Date: Tuesday, August 11, 2026 at 8:38 PM
>>>>>>> To: dev@nifi.apache.org <de...@nifi.apache.org>
>>>>>>> Subject: [DISCUSS] Future Support for Python Processors?
>>>>>>> 
>>>>>>> Be Careful: Email is from an External Source
>>>>>>> Use caution when opening emails and do not click on links or
>>>>> attachments
>>>>>>> from unknown senders. For more resources, visit
>>>>>> http://security.cat.com/phishing .
>>>>>>> 
>>>>>>> 
>>>>>>> Team,
>>>>>>> 
>>>>>>> The introduction of support for Python Processors was one of the major
>>>>>>> new features of Apache NiFi 2. In light of experience, however, I
>>>>>>> believe it is time to reconsider future support for this extension
>>>>>>> strategy. The idea of removing this feature may raise some concerns,
>>>>>>> but it is important to consider the options based on scalability,
>>>>>>> security, and supportability.
>>>>>>> 
>>>>>>> One of the greatest benefits of Python-based Processors is rapid
>>>>>>> iteration. Although this is a strength during the development phase,
>>>>>>> it can become a serious weakness when moving to production.
>>>>>>> 
>>>>>>> NiFi is fundamentally a Java-based application, thus tuning resource
>>>>>>> allocation is a key part of maintaining a production system. Running a
>>>>>>> Python Processor requires one or more independent Python processes,
>>>>>>> each with their own memory consumption and CPU usage patterns. What
>>>>>>> might work at a small scale on a development machine breaks down when
>>>>>>> attempting to run at production scale, without careful tuning. This is
>>>>>>> challenging enough with Java-based components, but global settings
>>>>>>> like heap and non-heap memory provide clear boundaries. Python
>>>>>>> Processors, as currently implemented, do not support this level of
>>>>>>> control. Beyond process control, operations that require communicating
>>>>>>> between the NiFi JVM and Python processes involve socket
>>>>>>> communication, which can be expensive for frequent calls. Thus an
>>>>>>> apparent initial advantage of Python Processors becomes a major
>>>>>>> problem when moving beyond limited types of operations.
>>>>>>> 
>>>>>>> When it comes to security, Python Processors can involve more serious
>>>>>>> risks than bundled Java extensions.
>>>>>>> 
>>>>>>> With NiFi support for downloading dependencies at runtime, it is
>>>>>>> essential for an administrator to provide a secure supply chain
>>>>>>> strategy for retrieving these dependencies. Although the NiFi
>>>>>>> framework also supports packaging Python Processors as NAR bundles,
>>>>>>> this packaging capability is not currently part of the Apache NiFi
>>>>>>> project. The project could build on third party work, but as Python
>>>>>>> packaging is less opinionated, this is an additional maintenance
>>>>>>> question that quickly becomes more involved. This combination of
>>>>>>> supply chain security and lack of packaging support raises security
>>>>>>> challenges.
>>>>>>> 
>>>>>>> Supportability is always a question for any feature in an open source
>>>>>>> project, but Python Processors are a particular point of concern.
>>>>>>> Providing NiFi framework support requires a strong knowledge of the
>>>>>>> NiFi project itself, and a strong knowledge of Python. Project
>>>>>>> maintainers and contributors usually have expertise in particular
>>>>>>> areas, but it is rare to find both. More to the point, however, the
>>>>>>> Py4J project, which NiFi depends on for integration, has seen limited
>>>>>>> maintenance, with the last tagged version dating to January 2025.
>>>>>>> Without robust architecture support, the NiFi project cannot provide a
>>>>>>> strong framework for future implementation.
>>>>>>> 
>>>>>>> Taking these issues together, I believe the project should consider
>>>>>>> deprecating support for Python Processors. Similar to other
>>>>>>> deprecations, actual removal would not occur until a major release
>>>>>>> version, and that would be a separate conversation.
>>>>>>> 
>>>>>>> Rather than continuing to extend the life of Python Processor support,
>>>>>>> it seems better to invest limited project resources into maintaining
>>>>>>> and improving core functionality.
>>>>>>> 
>>>>>>> Regards,
>>>>>>> David Handermann
>>>>>>> Apache NiFi PMC Chair
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>> 

Re: [DISCUSS] Future Support for Python Processors?

[David Handermann <ex...@apache.org>]

Kevin,

The PythonProcess class is one of the primary integration points,
which spawns a native Python process and initiates socket
communication:

https://github.com/apache/nifi/blob/main/nifi-extension-bundles/nifi-py4j-extension-bundle/nifi-py4j-bridge/src/main/java/org/apache/nifi/py4j/PythonProcess.java

On the Python side, the Controller class handles initializing the Py4J gateway:

https://github.com/apache/nifi/blob/main/nifi-framework-bundle/nifi-framework-extensions/nifi-py4j-framework-bundle/nifi-python-framework/src/main/python/framework/Controller.py

Sibling components in each of these packages provide supporting features.

Regards,
David Handermann

On Thu, Aug 13, 2026 at 2:37 PM Kevin Doran <kd...@apache.org> wrote:
>
> Can someone clarify where Py4J is used today vs where subprocesses
> that invoke a configurable python executable are used? My
> understanding was that the Py4J was mostly restricted to the scripting
> processors, but if that is not the case I would be interested in
> hearing more.
>
> On Thu, Aug 13, 2026 at 1:42 PM David Handermann
> <ex...@apache.org> wrote:
> >
> > Dan,
> >
> > Thanks for the substantive reply and consideration of the various
> > issues surrounding support for native Python Processors.
> >
> > As you highlighted, the scalability issue is foundational. Although
> > Processor authors can work with self-imposed constraints, the NiFi
> > framework does not provide the same level of safeguards. Any future
> > solution would need to account for these types of production-level
> > concerns.
> >
> > Regarding supply chain security, you are correct that there are
> > potential paths forward on the packaging strategy. The framework could
> > remove support for runtime loading as a safety measure. Right now,
> > lack of pip or uv effectively disables runtime loading. Adjusting the
> > default project container configuration would strengthen the standard
> > position, and thus require Python Processors to be packaged using a
> > NAR. The project could provide tooling, similar to the NiFi NAR Maven
> > Plugin, but this broadens the maintenance scope in a meaningful way.
> > Thus before going in this direction, the future support strategy needs
> > to be considered.
> >
> > The Py4J architecture remains a critical concern and high risk for
> > future maintainability. Considering an alternative architecture would
> > require a substantial redesign and rewrite of Python Processor
> > support, abandoning a majority of the current implementation. Current
> > tooling makes this more plausible in theory, but is a significant
> > undertaking that is difficult to justify without serious commitment on
> > the part of multiple project maintainers. To be clear, deprecating the
> > current implementation for removal does not preclude future redesign.
> > These two questions should be considered independently.
> >
> > Deprecation and removal without replacement will have a different
> > impact depending on the current scope of usage. Simpler Python
> > Processors should be easier to convert to Java, but more complex
> > implementations would likely require something outside of NiFi, as you
> > mentioned. Although these are difficult questions, it ultimately comes
> > down to project focus and available maintenance cycles. Apache NiFi
> > has historically done many things. NiFi 2 served as a focusing
> > function. Going forward, maintaining project health and stability is
> > paramount, which may at times involve removing certain capabilities.
> >
> > Regards,
> > David Handermann
> >
> > On Wed, Aug 12, 2026 at 3:27 PM Daniel Weidman via dev
> > <de...@nifi.apache.org> wrote:
> > >
> > > David,
> > >
> > > Thanks for raising this directly rather than letting it drift, and for laying
> > > out the reasoning rather than just the conclusion.
> > >
> > > On scalability, your point lands. In my experience the memory behavior of a
> > > Python Processor is largely a function of how it is written, and disciplined
> > > implementations can be well behaved. But I recognize that is an argument about
> > > good citizenship, not about enforcement, and it does not give an operator the
> > > equivalent of a heap ceiling. That is a real gap. It does seem addressable at
> > > the process supervision layer, through cgroup or rlimit bounds on spawned
> > > interpreters, rather than being inherent to Python as an extension language.
> > >
> > > On security, I would offer a friendly amendment. The meaningful difference does
> > > not seem to be Python versus Java, but when and where third party dependencies
> > > land. A NAR resolves its tree at build time and ships self contained, so it can
> > > be scanned, signed, and deployed into an air gapped environment, and unpacking
> > > it executes nothing. The current Python path resolves dependencies at runtime
> > > via pip on the NiFi host, and pip installation does execute packaging code.
> > > That is a real governance gap, and I will concede PyPI carries risks Maven
> > > Central does not.
> > >
> > > But as you noted, the framework already supports loading Python Processors
> > > packaged as NARs. What is missing is project-provided tooling to build them.
> > > Sean's note about relying on a blog post to produce one seems telling. Closing
> > > that gap looks like a smaller undertaking than removing the extension point,
> > > and it would address the supply chain concern directly rather than by
> > > subtraction.
> > >
> > > I will admit I had not realized the current implementation still rests on Py4J.
> > > That reframes a good deal of this for me, and it tracks with Matt's read that
> > > Py4J is the long pole rather than the capability itself.
> > >
> > > For what it is worth as an operator, the practical value of Python Processors
> > > is not just iteration speed. It is that the people writing our transforms are
> > > Python native. Removal without a replacement does not convert that work to
> > > Java, it pushes it out of NiFi entirely, to sidecars or external services,
> > > where, as Marc noted, we keep every one of the concerns you listed but lose
> > > NiFi's visibility into them.
> > >
> > > I would rather see the runtime architecture reconsidered than the capability
> > > retired. If the deciding constraint is maintainer cycles, it would be useful to
> > > know what a replacement would need in terms of effort and expertise. That is a
> > > more actionable ask of the community than a deprecation vote.
> > >
> > > Regards,
> > > Dan
> > >
> > >
> > > From: David Handermann <ex...@apache.org>
> > > Date: Wednesday, August 12, 2026 at 9:10 AM
> > > To: dev@nifi.apache.org <de...@nifi.apache.org>
> > > Subject: Re: [DISCUSS] Future Support for Python Processors?
> > >
> > > Thanks to everyone for the replies thus far!
> > >
> > > The question about alternatives gets at the core issue, and the ideas
> > > around a more loosely coupled architecture provide some helpful
> > > possibilities.
> > >
> > > At a basic level, simple deprecation and removal would leave the
> > > current set of options, including standard Java, Groovy scripting, or
> > > externalized Python scripts executed as a stream. None of these
> > > provide the same level of features for Python itself, which prompted
> > > the original native Python Processor implementation. Without a
> > > replacement, these alternatives would remain. With the rise of
> > > AI-enabled development, it is now much easier to go from Python to
> > > Java, but work is still required.
> > >
> > > As to future strategies for Python integration, the current discussion
> > > around deprecation is more narrowly focused on the existing
> > > framework-level extension. Removal of the current implementation is
> > > not intended to halt future innovation. Although removal without an
> > > initial replacement might not be the perfect solution, it may be
> > > necessary given available maintainer cycles, and the substantive
> > > issues I described in the opening.
> > >
> > > If there is sufficient interest and effort to propose and build a
> > > different solution, that would be a great outcome of this discussion.
> > > I've thought at different points about reworking the Script Engine
> > > integration to provide a cleaner path to external extension, so there
> > > are definitely opportunities in that direction.
> > >
> > > With that, my primary concerns with the current implementation remain,
> > > but glad to consider future alternatives.
> > >
> > > Regards,
> > > David Handermann
> > >
> > >
> > > On Wed, Aug 12, 2026 at 7:52 AM Matt Burgess <ma...@apache.org> wrote:
> > > >
> > > > I agree with Marc, the concerns of the implementation of the capability
> > > > should not warrant deprecation and future removal of the capability, but
> > > > certainly we want to ensure that anything we implement for NiFi doesn't
> > > > negatively the reputation of Apache NiFi as a secure, robust, reliable
> > > > system.
> > > >
> > > >  I think we should instead consider a new architecture, one
> > > > fully extensible so if the community decides to deprecate the capability,
> > > > it could still be restored using some extension point. This kind of generic
> > > > approach could allow other extensions not sanctioned by the Apache NiFi
> > > > community to coexist in the "NiFi+" deployment. Not to get off-topic but it
> > > > is somewhat analogous to exposing the Spring context such that alternate
> > > > implementations of the beans could be provided.
> > > >
> > > > There are options to consider, I'm sure each with its own set of pros and
> > > > cons. Off the top of my head:
> > > >
> > > > - Sidecar server responsible for executing Python scripts with FlowFile
> > > > input(s) and returning FlowFile output(s) directly from/to processors, like
> > > > Marc's suggestion but managed by NiFi
> > > > - Jep [1] and it's OS-independent counterpart jep-distro [2]
> > > > - Require Java bindings calling out to an embedded interpreter, such as
> > > > WrapPy4J (which uses Jep under the hood) [3]
> > > >
> > > > I am far from a Python expert and I know there are several places in the
> > > > framework where the Python stuff has landed, so I don't expect any other
> > > > implementation to be any easier or take less time than the original, which
> > > > I'm sure took quite a lot of time and effort which is very much
> > > > appreciated. It seems that Py4J is the long pole in the tent for the
> > > > concerns, not the capability itself.
> > > >
> > > > At the end of the day I think a discussion like this clearly shows the
> > > > commitment of the whole community to ensure Apache NiFi is all that we want
> > > > it to be and that we can introspect, improve, and collaborate to keep NiFi
> > > > awesome :)
> > > >
> > > > Regards,
> > > > Matt
> > > >
> > > > [1] https://urldefense.com/v3/__https://github.com/ninia/JEP?tab=readme-ov-file__;!!FtR4BK4x7WL3xYs!6mIniDSbJ62B5zII3b9g972tiJiM1K8M1E-1R8rvwlwl__KEfyNqtqWjM3xwIPbVawsv_VEsYzoNRToUxhft8Q3Rnnw$
> > > > [2] https://urldefense.com/v3/__https://github.com/icemachined/jep-distro?tab=readme-ov-file__;!!FtR4BK4x7WL3xYs!6mIniDSbJ62B5zII3b9g972tiJiM1K8M1E-1R8rvwlwl__KEfyNqtqWjM3xwIPbVawsv_VEsYzoNRToUxhft703BJKY$
> > > > [3] https://urldefense.com/v3/__https://github.com/313hemant313/wrappyj__;!!FtR4BK4x7WL3xYs!6mIniDSbJ62B5zII3b9g972tiJiM1K8M1E-1R8rvwlwl__KEfyNqtqWjM3xwIPbVawsv_VEsYzoNRToUxhftYE_fqvo$
> > > >
> > > > On Wed, Aug 12, 2026 at 8:19 AM Marc P. <ma...@gmail.com> wrote:
> > > >
> > > > > This is a feature I find personally valuable. I agree with several
> > > > > underlying issues, particularly resource management, Py4J, dependency
> > > > > handling, and the difficulty of maintaining a framework that spans both the
> > > > > JVM and Python ecosystems.
> > > > >
> > > > > Where I disagree is with the conclusion that these issues necessarily point
> > > > > toward deprecating Python Processor support itself.
> > > > >
> > > > > I think there is an important distinction between the value of Python as an
> > > > > extension model and the architecture currently used to implement it.
> > > > >
> > > > > To be solution oriented one way to look at this is to consider what users
> > > > > would do in the absence of Python Processor support. In many cases, the
> > > > > alternative is not to rewrite the functionality in Java. It is likely to
> > > > > become something like:
> > > > >
> > > > > Apache NiFi -> HTTP/gRPC -> Python microservice -> NiFi
> > > > >
> > > > > That architecture still has independent Python processes, Python dependency
> > > > > management, memory and CPU consumption outside the JVM, and a Python
> > > > > software supply chain. We have not eliminated any of those concerns; we
> > > > > have moved them outside NiFi and made every operator build and manage that
> > > > > integration themselves.
> > > > >
> > > > > There are also costs to that approach: additional network boundaries,
> > > > > deployment artifacts, authentication, service discovery, observability,
> > > > > retry semantics, and operational complexity. From a governance perspective,
> > > > > NiFi may actually have less visibility and control over that external
> > > > > Python workload than it would if Python execution remained part of a
> > > > > defined NiFi extension model.
> > > > >
> > > > > That makes me wonder whether the better direction is to reconsider the
> > > > > Python runtime architecture rather than Python support itself.
> > > > >
> > > > > For example, instead of treating the current Py4J-based implementation as
> > > > > something that must be preserved indefinitely, we could define a stronger
> > > > > process boundary around Python execution. A future implementation could use
> > > > > a well-defined IPC or RPC contract, potentially based on gRPC/protobuf,
> > > > > Unix domain sockets, shared memory, Arrow, or another mechanism designed
> > > > > around coarse-grained operations.
> > > > >
> > > > > NiFi does not need to become a Python integrated application, but I think
> > > > > there is substantial value in allowing those workloads to participate
> > > > > directly in a governed NiFi dataflow without requiring users to construct a
> > > > > separate microservice architecture for every Python integration.
> > > > >
> > > > > For that reason, I would favor separating two questions:
> > > > >
> > > > >    1. Is the current Python Processor implementation the architecture we
> > > > >    want to support long term?
> > > > >    2. Should NiFi continue to provide a first-class Python extension
> > > > >    capability?
> > > > >
> > > > > I could see a strong argument that the answer to the first question is no,
> > > > > particularly if Py4J is becoming a maintenance risk. I am much less
> > > > > convinced that the answer to the second question should also be no.
> > > > >
> > > > > With that said I distinctly understand that time is limited and we would
> > > > > all develop the coolest of things with infinite resources, so I yield to
> > > > > those with more influence, but would happy to contribute to a better
> > > > > architecture if others find it valuable.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Marc
> > > > >
> > > > > On Wed, Aug 12, 2026 at 12:10 AM Joe Witt <jo...@gmail.com> wrote:
> > > > >
> > > > > > You are totally fine to engage on this thread Sean. We appreciate it
> > > > > >
> > > > > >
> > > > > > On Tue, Aug 11, 2026 at 8:17 PM Sean Hunter via dev <dev@nifi.apache.org
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi David,
> > > > > > >
> > > > > > > As a user, rather than contributor, I think my primary concern is one
> > > > > of
> > > > > > > operations and implementation. What alternative would be available? We
> > > > > > used
> > > > > > > the 1.x version and were quite excited to move to 2.0 with its more
> > > > > > native
> > > > > > > integration that was easier to troubleshoot. We also do build a NAR as
> > > > > > part
> > > > > > > of our pipeline process, and I suspect since we used a blog as guidance
> > > > > > to
> > > > > > > develop that process, that others likely do as well. Our use case is a
> > > > > > > particularly nasty data transform (Windows logs of some sort,
> > > > > naturally)
> > > > > > > that’s reasonable to write and execute in Python. It’s readable, while
> > > > > > > still being performant enough for our needs, and seemingly not causing
> > > > > > > system stability issues for us.
> > > > > > >
> > > > > > > I do understand and take your points. What I’d like to better
> > > > > understand
> > > > > > > are the options users have. Do you have any thoughts?
> > > > > > >
> > > > > > > Thank you and have a great day!
> > > > > > > Sean
> > > > > > >
> > > > > > > PS - Please let me know if this is too far off topic for the dev
> > > > > mailing
> > > > > > > list and I’ll take it elsewhere.
> > > > > > >
> > > > > > > From: David Handermann <ex...@apache.org>
> > > > > > > Date: Tuesday, August 11, 2026 at 8:38 PM
> > > > > > > To: dev@nifi.apache.org <de...@nifi.apache.org>
> > > > > > > Subject: [DISCUSS] Future Support for Python Processors?
> > > > > > >
> > > > > > > Be Careful: Email is from an External Source
> > > > > > > Use caution when opening emails and do not click on links or
> > > > > attachments
> > > > > > > from unknown senders. For more resources, visit
> > > > > > http://security.cat.com/phishing .
> > > > > > >
> > > > > > >
> > > > > > > Team,
> > > > > > >
> > > > > > > The introduction of support for Python Processors was one of the major
> > > > > > > new features of Apache NiFi 2. In light of experience, however, I
> > > > > > > believe it is time to reconsider future support for this extension
> > > > > > > strategy. The idea of removing this feature may raise some concerns,
> > > > > > > but it is important to consider the options based on scalability,
> > > > > > > security, and supportability.
> > > > > > >
> > > > > > > One of the greatest benefits of Python-based Processors is rapid
> > > > > > > iteration. Although this is a strength during the development phase,
> > > > > > > it can become a serious weakness when moving to production.
> > > > > > >
> > > > > > > NiFi is fundamentally a Java-based application, thus tuning resource
> > > > > > > allocation is a key part of maintaining a production system. Running a
> > > > > > > Python Processor requires one or more independent Python processes,
> > > > > > > each with their own memory consumption and CPU usage patterns. What
> > > > > > > might work at a small scale on a development machine breaks down when
> > > > > > > attempting to run at production scale, without careful tuning. This is
> > > > > > > challenging enough with Java-based components, but global settings
> > > > > > > like heap and non-heap memory provide clear boundaries. Python
> > > > > > > Processors, as currently implemented, do not support this level of
> > > > > > > control. Beyond process control, operations that require communicating
> > > > > > > between the NiFi JVM and Python processes involve socket
> > > > > > > communication, which can be expensive for frequent calls. Thus an
> > > > > > > apparent initial advantage of Python Processors becomes a major
> > > > > > > problem when moving beyond limited types of operations.
> > > > > > >
> > > > > > > When it comes to security, Python Processors can involve more serious
> > > > > > > risks than bundled Java extensions.
> > > > > > >
> > > > > > > With NiFi support for downloading dependencies at runtime, it is
> > > > > > > essential for an administrator to provide a secure supply chain
> > > > > > > strategy for retrieving these dependencies. Although the NiFi
> > > > > > > framework also supports packaging Python Processors as NAR bundles,
> > > > > > > this packaging capability is not currently part of the Apache NiFi
> > > > > > > project. The project could build on third party work, but as Python
> > > > > > > packaging is less opinionated, this is an additional maintenance
> > > > > > > question that quickly becomes more involved. This combination of
> > > > > > > supply chain security and lack of packaging support raises security
> > > > > > > challenges.
> > > > > > >
> > > > > > > Supportability is always a question for any feature in an open source
> > > > > > > project, but Python Processors are a particular point of concern.
> > > > > > > Providing NiFi framework support requires a strong knowledge of the
> > > > > > > NiFi project itself, and a strong knowledge of Python. Project
> > > > > > > maintainers and contributors usually have expertise in particular
> > > > > > > areas, but it is rare to find both. More to the point, however, the
> > > > > > > Py4J project, which NiFi depends on for integration, has seen limited
> > > > > > > maintenance, with the last tagged version dating to January 2025.
> > > > > > > Without robust architecture support, the NiFi project cannot provide a
> > > > > > > strong framework for future implementation.
> > > > > > >
> > > > > > > Taking these issues together, I believe the project should consider
> > > > > > > deprecating support for Python Processors. Similar to other
> > > > > > > deprecations, actual removal would not occur until a major release
> > > > > > > version, and that would be a separate conversation.
> > > > > > >
> > > > > > > Rather than continuing to extend the life of Python Processor support,
> > > > > > > it seems better to invest limited project resources into maintaining
> > > > > > > and improving core functionality.
> > > > > > >
> > > > > > > Regards,
> > > > > > > David Handermann
> > > > > > > Apache NiFi PMC Chair
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >

Re: [DISCUSS] Future Support for Python Processors?

[Kevin Doran <kd...@apache.org>]

Can someone clarify where Py4J is used today vs where subprocesses
that invoke a configurable python executable are used? My
understanding was that the Py4J was mostly restricted to the scripting
processors, but if that is not the case I would be interested in
hearing more.

On Thu, Aug 13, 2026 at 1:42 PM David Handermann
<ex...@apache.org> wrote:
>
> Dan,
>
> Thanks for the substantive reply and consideration of the various
> issues surrounding support for native Python Processors.
>
> As you highlighted, the scalability issue is foundational. Although
> Processor authors can work with self-imposed constraints, the NiFi
> framework does not provide the same level of safeguards. Any future
> solution would need to account for these types of production-level
> concerns.
>
> Regarding supply chain security, you are correct that there are
> potential paths forward on the packaging strategy. The framework could
> remove support for runtime loading as a safety measure. Right now,
> lack of pip or uv effectively disables runtime loading. Adjusting the
> default project container configuration would strengthen the standard
> position, and thus require Python Processors to be packaged using a
> NAR. The project could provide tooling, similar to the NiFi NAR Maven
> Plugin, but this broadens the maintenance scope in a meaningful way.
> Thus before going in this direction, the future support strategy needs
> to be considered.
>
> The Py4J architecture remains a critical concern and high risk for
> future maintainability. Considering an alternative architecture would
> require a substantial redesign and rewrite of Python Processor
> support, abandoning a majority of the current implementation. Current
> tooling makes this more plausible in theory, but is a significant
> undertaking that is difficult to justify without serious commitment on
> the part of multiple project maintainers. To be clear, deprecating the
> current implementation for removal does not preclude future redesign.
> These two questions should be considered independently.
>
> Deprecation and removal without replacement will have a different
> impact depending on the current scope of usage. Simpler Python
> Processors should be easier to convert to Java, but more complex
> implementations would likely require something outside of NiFi, as you
> mentioned. Although these are difficult questions, it ultimately comes
> down to project focus and available maintenance cycles. Apache NiFi
> has historically done many things. NiFi 2 served as a focusing
> function. Going forward, maintaining project health and stability is
> paramount, which may at times involve removing certain capabilities.
>
> Regards,
> David Handermann
>
> On Wed, Aug 12, 2026 at 3:27 PM Daniel Weidman via dev
> <de...@nifi.apache.org> wrote:
> >
> > David,
> >
> > Thanks for raising this directly rather than letting it drift, and for laying
> > out the reasoning rather than just the conclusion.
> >
> > On scalability, your point lands. In my experience the memory behavior of a
> > Python Processor is largely a function of how it is written, and disciplined
> > implementations can be well behaved. But I recognize that is an argument about
> > good citizenship, not about enforcement, and it does not give an operator the
> > equivalent of a heap ceiling. That is a real gap. It does seem addressable at
> > the process supervision layer, through cgroup or rlimit bounds on spawned
> > interpreters, rather than being inherent to Python as an extension language.
> >
> > On security, I would offer a friendly amendment. The meaningful difference does
> > not seem to be Python versus Java, but when and where third party dependencies
> > land. A NAR resolves its tree at build time and ships self contained, so it can
> > be scanned, signed, and deployed into an air gapped environment, and unpacking
> > it executes nothing. The current Python path resolves dependencies at runtime
> > via pip on the NiFi host, and pip installation does execute packaging code.
> > That is a real governance gap, and I will concede PyPI carries risks Maven
> > Central does not.
> >
> > But as you noted, the framework already supports loading Python Processors
> > packaged as NARs. What is missing is project-provided tooling to build them.
> > Sean's note about relying on a blog post to produce one seems telling. Closing
> > that gap looks like a smaller undertaking than removing the extension point,
> > and it would address the supply chain concern directly rather than by
> > subtraction.
> >
> > I will admit I had not realized the current implementation still rests on Py4J.
> > That reframes a good deal of this for me, and it tracks with Matt's read that
> > Py4J is the long pole rather than the capability itself.
> >
> > For what it is worth as an operator, the practical value of Python Processors
> > is not just iteration speed. It is that the people writing our transforms are
> > Python native. Removal without a replacement does not convert that work to
> > Java, it pushes it out of NiFi entirely, to sidecars or external services,
> > where, as Marc noted, we keep every one of the concerns you listed but lose
> > NiFi's visibility into them.
> >
> > I would rather see the runtime architecture reconsidered than the capability
> > retired. If the deciding constraint is maintainer cycles, it would be useful to
> > know what a replacement would need in terms of effort and expertise. That is a
> > more actionable ask of the community than a deprecation vote.
> >
> > Regards,
> > Dan
> >
> >
> > From: David Handermann <ex...@apache.org>
> > Date: Wednesday, August 12, 2026 at 9:10 AM
> > To: dev@nifi.apache.org <de...@nifi.apache.org>
> > Subject: Re: [DISCUSS] Future Support for Python Processors?
> >
> > Thanks to everyone for the replies thus far!
> >
> > The question about alternatives gets at the core issue, and the ideas
> > around a more loosely coupled architecture provide some helpful
> > possibilities.
> >
> > At a basic level, simple deprecation and removal would leave the
> > current set of options, including standard Java, Groovy scripting, or
> > externalized Python scripts executed as a stream. None of these
> > provide the same level of features for Python itself, which prompted
> > the original native Python Processor implementation. Without a
> > replacement, these alternatives would remain. With the rise of
> > AI-enabled development, it is now much easier to go from Python to
> > Java, but work is still required.
> >
> > As to future strategies for Python integration, the current discussion
> > around deprecation is more narrowly focused on the existing
> > framework-level extension. Removal of the current implementation is
> > not intended to halt future innovation. Although removal without an
> > initial replacement might not be the perfect solution, it may be
> > necessary given available maintainer cycles, and the substantive
> > issues I described in the opening.
> >
> > If there is sufficient interest and effort to propose and build a
> > different solution, that would be a great outcome of this discussion.
> > I've thought at different points about reworking the Script Engine
> > integration to provide a cleaner path to external extension, so there
> > are definitely opportunities in that direction.
> >
> > With that, my primary concerns with the current implementation remain,
> > but glad to consider future alternatives.
> >
> > Regards,
> > David Handermann
> >
> >
> > On Wed, Aug 12, 2026 at 7:52 AM Matt Burgess <ma...@apache.org> wrote:
> > >
> > > I agree with Marc, the concerns of the implementation of the capability
> > > should not warrant deprecation and future removal of the capability, but
> > > certainly we want to ensure that anything we implement for NiFi doesn't
> > > negatively the reputation of Apache NiFi as a secure, robust, reliable
> > > system.
> > >
> > >  I think we should instead consider a new architecture, one
> > > fully extensible so if the community decides to deprecate the capability,
> > > it could still be restored using some extension point. This kind of generic
> > > approach could allow other extensions not sanctioned by the Apache NiFi
> > > community to coexist in the "NiFi+" deployment. Not to get off-topic but it
> > > is somewhat analogous to exposing the Spring context such that alternate
> > > implementations of the beans could be provided.
> > >
> > > There are options to consider, I'm sure each with its own set of pros and
> > > cons. Off the top of my head:
> > >
> > > - Sidecar server responsible for executing Python scripts with FlowFile
> > > input(s) and returning FlowFile output(s) directly from/to processors, like
> > > Marc's suggestion but managed by NiFi
> > > - Jep [1] and it's OS-independent counterpart jep-distro [2]
> > > - Require Java bindings calling out to an embedded interpreter, such as
> > > WrapPy4J (which uses Jep under the hood) [3]
> > >
> > > I am far from a Python expert and I know there are several places in the
> > > framework where the Python stuff has landed, so I don't expect any other
> > > implementation to be any easier or take less time than the original, which
> > > I'm sure took quite a lot of time and effort which is very much
> > > appreciated. It seems that Py4J is the long pole in the tent for the
> > > concerns, not the capability itself.
> > >
> > > At the end of the day I think a discussion like this clearly shows the
> > > commitment of the whole community to ensure Apache NiFi is all that we want
> > > it to be and that we can introspect, improve, and collaborate to keep NiFi
> > > awesome :)
> > >
> > > Regards,
> > > Matt
> > >
> > > [1] https://urldefense.com/v3/__https://github.com/ninia/JEP?tab=readme-ov-file__;!!FtR4BK4x7WL3xYs!6mIniDSbJ62B5zII3b9g972tiJiM1K8M1E-1R8rvwlwl__KEfyNqtqWjM3xwIPbVawsv_VEsYzoNRToUxhft8Q3Rnnw$
> > > [2] https://urldefense.com/v3/__https://github.com/icemachined/jep-distro?tab=readme-ov-file__;!!FtR4BK4x7WL3xYs!6mIniDSbJ62B5zII3b9g972tiJiM1K8M1E-1R8rvwlwl__KEfyNqtqWjM3xwIPbVawsv_VEsYzoNRToUxhft703BJKY$
> > > [3] https://urldefense.com/v3/__https://github.com/313hemant313/wrappyj__;!!FtR4BK4x7WL3xYs!6mIniDSbJ62B5zII3b9g972tiJiM1K8M1E-1R8rvwlwl__KEfyNqtqWjM3xwIPbVawsv_VEsYzoNRToUxhftYE_fqvo$
> > >
> > > On Wed, Aug 12, 2026 at 8:19 AM Marc P. <ma...@gmail.com> wrote:
> > >
> > > > This is a feature I find personally valuable. I agree with several
> > > > underlying issues, particularly resource management, Py4J, dependency
> > > > handling, and the difficulty of maintaining a framework that spans both the
> > > > JVM and Python ecosystems.
> > > >
> > > > Where I disagree is with the conclusion that these issues necessarily point
> > > > toward deprecating Python Processor support itself.
> > > >
> > > > I think there is an important distinction between the value of Python as an
> > > > extension model and the architecture currently used to implement it.
> > > >
> > > > To be solution oriented one way to look at this is to consider what users
> > > > would do in the absence of Python Processor support. In many cases, the
> > > > alternative is not to rewrite the functionality in Java. It is likely to
> > > > become something like:
> > > >
> > > > Apache NiFi -> HTTP/gRPC -> Python microservice -> NiFi
> > > >
> > > > That architecture still has independent Python processes, Python dependency
> > > > management, memory and CPU consumption outside the JVM, and a Python
> > > > software supply chain. We have not eliminated any of those concerns; we
> > > > have moved them outside NiFi and made every operator build and manage that
> > > > integration themselves.
> > > >
> > > > There are also costs to that approach: additional network boundaries,
> > > > deployment artifacts, authentication, service discovery, observability,
> > > > retry semantics, and operational complexity. From a governance perspective,
> > > > NiFi may actually have less visibility and control over that external
> > > > Python workload than it would if Python execution remained part of a
> > > > defined NiFi extension model.
> > > >
> > > > That makes me wonder whether the better direction is to reconsider the
> > > > Python runtime architecture rather than Python support itself.
> > > >
> > > > For example, instead of treating the current Py4J-based implementation as
> > > > something that must be preserved indefinitely, we could define a stronger
> > > > process boundary around Python execution. A future implementation could use
> > > > a well-defined IPC or RPC contract, potentially based on gRPC/protobuf,
> > > > Unix domain sockets, shared memory, Arrow, or another mechanism designed
> > > > around coarse-grained operations.
> > > >
> > > > NiFi does not need to become a Python integrated application, but I think
> > > > there is substantial value in allowing those workloads to participate
> > > > directly in a governed NiFi dataflow without requiring users to construct a
> > > > separate microservice architecture for every Python integration.
> > > >
> > > > For that reason, I would favor separating two questions:
> > > >
> > > >    1. Is the current Python Processor implementation the architecture we
> > > >    want to support long term?
> > > >    2. Should NiFi continue to provide a first-class Python extension
> > > >    capability?
> > > >
> > > > I could see a strong argument that the answer to the first question is no,
> > > > particularly if Py4J is becoming a maintenance risk. I am much less
> > > > convinced that the answer to the second question should also be no.
> > > >
> > > > With that said I distinctly understand that time is limited and we would
> > > > all develop the coolest of things with infinite resources, so I yield to
> > > > those with more influence, but would happy to contribute to a better
> > > > architecture if others find it valuable.
> > > >
> > > > Thanks,
> > > >
> > > > Marc
> > > >
> > > > On Wed, Aug 12, 2026 at 12:10 AM Joe Witt <jo...@gmail.com> wrote:
> > > >
> > > > > You are totally fine to engage on this thread Sean. We appreciate it
> > > > >
> > > > >
> > > > > On Tue, Aug 11, 2026 at 8:17 PM Sean Hunter via dev <dev@nifi.apache.org
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi David,
> > > > > >
> > > > > > As a user, rather than contributor, I think my primary concern is one
> > > > of
> > > > > > operations and implementation. What alternative would be available? We
> > > > > used
> > > > > > the 1.x version and were quite excited to move to 2.0 with its more
> > > > > native
> > > > > > integration that was easier to troubleshoot. We also do build a NAR as
> > > > > part
> > > > > > of our pipeline process, and I suspect since we used a blog as guidance
> > > > > to
> > > > > > develop that process, that others likely do as well. Our use case is a
> > > > > > particularly nasty data transform (Windows logs of some sort,
> > > > naturally)
> > > > > > that’s reasonable to write and execute in Python. It’s readable, while
> > > > > > still being performant enough for our needs, and seemingly not causing
> > > > > > system stability issues for us.
> > > > > >
> > > > > > I do understand and take your points. What I’d like to better
> > > > understand
> > > > > > are the options users have. Do you have any thoughts?
> > > > > >
> > > > > > Thank you and have a great day!
> > > > > > Sean
> > > > > >
> > > > > > PS - Please let me know if this is too far off topic for the dev
> > > > mailing
> > > > > > list and I’ll take it elsewhere.
> > > > > >
> > > > > > From: David Handermann <ex...@apache.org>
> > > > > > Date: Tuesday, August 11, 2026 at 8:38 PM
> > > > > > To: dev@nifi.apache.org <de...@nifi.apache.org>
> > > > > > Subject: [DISCUSS] Future Support for Python Processors?
> > > > > >
> > > > > > Be Careful: Email is from an External Source
> > > > > > Use caution when opening emails and do not click on links or
> > > > attachments
> > > > > > from unknown senders. For more resources, visit
> > > > > http://security.cat.com/phishing .
> > > > > >
> > > > > >
> > > > > > Team,
> > > > > >
> > > > > > The introduction of support for Python Processors was one of the major
> > > > > > new features of Apache NiFi 2. In light of experience, however, I
> > > > > > believe it is time to reconsider future support for this extension
> > > > > > strategy. The idea of removing this feature may raise some concerns,
> > > > > > but it is important to consider the options based on scalability,
> > > > > > security, and supportability.
> > > > > >
> > > > > > One of the greatest benefits of Python-based Processors is rapid
> > > > > > iteration. Although this is a strength during the development phase,
> > > > > > it can become a serious weakness when moving to production.
> > > > > >
> > > > > > NiFi is fundamentally a Java-based application, thus tuning resource
> > > > > > allocation is a key part of maintaining a production system. Running a
> > > > > > Python Processor requires one or more independent Python processes,
> > > > > > each with their own memory consumption and CPU usage patterns. What
> > > > > > might work at a small scale on a development machine breaks down when
> > > > > > attempting to run at production scale, without careful tuning. This is
> > > > > > challenging enough with Java-based components, but global settings
> > > > > > like heap and non-heap memory provide clear boundaries. Python
> > > > > > Processors, as currently implemented, do not support this level of
> > > > > > control. Beyond process control, operations that require communicating
> > > > > > between the NiFi JVM and Python processes involve socket
> > > > > > communication, which can be expensive for frequent calls. Thus an
> > > > > > apparent initial advantage of Python Processors becomes a major
> > > > > > problem when moving beyond limited types of operations.
> > > > > >
> > > > > > When it comes to security, Python Processors can involve more serious
> > > > > > risks than bundled Java extensions.
> > > > > >
> > > > > > With NiFi support for downloading dependencies at runtime, it is
> > > > > > essential for an administrator to provide a secure supply chain
> > > > > > strategy for retrieving these dependencies. Although the NiFi
> > > > > > framework also supports packaging Python Processors as NAR bundles,
> > > > > > this packaging capability is not currently part of the Apache NiFi
> > > > > > project. The project could build on third party work, but as Python
> > > > > > packaging is less opinionated, this is an additional maintenance
> > > > > > question that quickly becomes more involved. This combination of
> > > > > > supply chain security and lack of packaging support raises security
> > > > > > challenges.
> > > > > >
> > > > > > Supportability is always a question for any feature in an open source
> > > > > > project, but Python Processors are a particular point of concern.
> > > > > > Providing NiFi framework support requires a strong knowledge of the
> > > > > > NiFi project itself, and a strong knowledge of Python. Project
> > > > > > maintainers and contributors usually have expertise in particular
> > > > > > areas, but it is rare to find both. More to the point, however, the
> > > > > > Py4J project, which NiFi depends on for integration, has seen limited
> > > > > > maintenance, with the last tagged version dating to January 2025.
> > > > > > Without robust architecture support, the NiFi project cannot provide a
> > > > > > strong framework for future implementation.
> > > > > >
> > > > > > Taking these issues together, I believe the project should consider
> > > > > > deprecating support for Python Processors. Similar to other
> > > > > > deprecations, actual removal would not occur until a major release
> > > > > > version, and that would be a separate conversation.
> > > > > >
> > > > > > Rather than continuing to extend the life of Python Processor support,
> > > > > > it seems better to invest limited project resources into maintaining
> > > > > > and improving core functionality.
> > > > > >
> > > > > > Regards,
> > > > > > David Handermann
> > > > > > Apache NiFi PMC Chair
> > > > > >
> > > > > >
> > > > >
> > > >
> >

Re: [DISCUSS] Future Support for Python Processors?

[David Handermann <ex...@apache.org>]

Dan,

Thanks for the substantive reply and consideration of the various
issues surrounding support for native Python Processors.

As you highlighted, the scalability issue is foundational. Although
Processor authors can work with self-imposed constraints, the NiFi
framework does not provide the same level of safeguards. Any future
solution would need to account for these types of production-level
concerns.

Regarding supply chain security, you are correct that there are
potential paths forward on the packaging strategy. The framework could
remove support for runtime loading as a safety measure. Right now,
lack of pip or uv effectively disables runtime loading. Adjusting the
default project container configuration would strengthen the standard
position, and thus require Python Processors to be packaged using a
NAR. The project could provide tooling, similar to the NiFi NAR Maven
Plugin, but this broadens the maintenance scope in a meaningful way.
Thus before going in this direction, the future support strategy needs
to be considered.

The Py4J architecture remains a critical concern and high risk for
future maintainability. Considering an alternative architecture would
require a substantial redesign and rewrite of Python Processor
support, abandoning a majority of the current implementation. Current
tooling makes this more plausible in theory, but is a significant
undertaking that is difficult to justify without serious commitment on
the part of multiple project maintainers. To be clear, deprecating the
current implementation for removal does not preclude future redesign.
These two questions should be considered independently.

Deprecation and removal without replacement will have a different
impact depending on the current scope of usage. Simpler Python
Processors should be easier to convert to Java, but more complex
implementations would likely require something outside of NiFi, as you
mentioned. Although these are difficult questions, it ultimately comes
down to project focus and available maintenance cycles. Apache NiFi
has historically done many things. NiFi 2 served as a focusing
function. Going forward, maintaining project health and stability is
paramount, which may at times involve removing certain capabilities.

Regards,
David Handermann

On Wed, Aug 12, 2026 at 3:27 PM Daniel Weidman via dev
<de...@nifi.apache.org> wrote:
>
> David,
>
> Thanks for raising this directly rather than letting it drift, and for laying
> out the reasoning rather than just the conclusion.
>
> On scalability, your point lands. In my experience the memory behavior of a
> Python Processor is largely a function of how it is written, and disciplined
> implementations can be well behaved. But I recognize that is an argument about
> good citizenship, not about enforcement, and it does not give an operator the
> equivalent of a heap ceiling. That is a real gap. It does seem addressable at
> the process supervision layer, through cgroup or rlimit bounds on spawned
> interpreters, rather than being inherent to Python as an extension language.
>
> On security, I would offer a friendly amendment. The meaningful difference does
> not seem to be Python versus Java, but when and where third party dependencies
> land. A NAR resolves its tree at build time and ships self contained, so it can
> be scanned, signed, and deployed into an air gapped environment, and unpacking
> it executes nothing. The current Python path resolves dependencies at runtime
> via pip on the NiFi host, and pip installation does execute packaging code.
> That is a real governance gap, and I will concede PyPI carries risks Maven
> Central does not.
>
> But as you noted, the framework already supports loading Python Processors
> packaged as NARs. What is missing is project-provided tooling to build them.
> Sean's note about relying on a blog post to produce one seems telling. Closing
> that gap looks like a smaller undertaking than removing the extension point,
> and it would address the supply chain concern directly rather than by
> subtraction.
>
> I will admit I had not realized the current implementation still rests on Py4J.
> That reframes a good deal of this for me, and it tracks with Matt's read that
> Py4J is the long pole rather than the capability itself.
>
> For what it is worth as an operator, the practical value of Python Processors
> is not just iteration speed. It is that the people writing our transforms are
> Python native. Removal without a replacement does not convert that work to
> Java, it pushes it out of NiFi entirely, to sidecars or external services,
> where, as Marc noted, we keep every one of the concerns you listed but lose
> NiFi's visibility into them.
>
> I would rather see the runtime architecture reconsidered than the capability
> retired. If the deciding constraint is maintainer cycles, it would be useful to
> know what a replacement would need in terms of effort and expertise. That is a
> more actionable ask of the community than a deprecation vote.
>
> Regards,
> Dan
>
>
> From: David Handermann <ex...@apache.org>
> Date: Wednesday, August 12, 2026 at 9:10 AM
> To: dev@nifi.apache.org <de...@nifi.apache.org>
> Subject: Re: [DISCUSS] Future Support for Python Processors?
>
> Thanks to everyone for the replies thus far!
>
> The question about alternatives gets at the core issue, and the ideas
> around a more loosely coupled architecture provide some helpful
> possibilities.
>
> At a basic level, simple deprecation and removal would leave the
> current set of options, including standard Java, Groovy scripting, or
> externalized Python scripts executed as a stream. None of these
> provide the same level of features for Python itself, which prompted
> the original native Python Processor implementation. Without a
> replacement, these alternatives would remain. With the rise of
> AI-enabled development, it is now much easier to go from Python to
> Java, but work is still required.
>
> As to future strategies for Python integration, the current discussion
> around deprecation is more narrowly focused on the existing
> framework-level extension. Removal of the current implementation is
> not intended to halt future innovation. Although removal without an
> initial replacement might not be the perfect solution, it may be
> necessary given available maintainer cycles, and the substantive
> issues I described in the opening.
>
> If there is sufficient interest and effort to propose and build a
> different solution, that would be a great outcome of this discussion.
> I've thought at different points about reworking the Script Engine
> integration to provide a cleaner path to external extension, so there
> are definitely opportunities in that direction.
>
> With that, my primary concerns with the current implementation remain,
> but glad to consider future alternatives.
>
> Regards,
> David Handermann
>
>
> On Wed, Aug 12, 2026 at 7:52 AM Matt Burgess <ma...@apache.org> wrote:
> >
> > I agree with Marc, the concerns of the implementation of the capability
> > should not warrant deprecation and future removal of the capability, but
> > certainly we want to ensure that anything we implement for NiFi doesn't
> > negatively the reputation of Apache NiFi as a secure, robust, reliable
> > system.
> >
> >  I think we should instead consider a new architecture, one
> > fully extensible so if the community decides to deprecate the capability,
> > it could still be restored using some extension point. This kind of generic
> > approach could allow other extensions not sanctioned by the Apache NiFi
> > community to coexist in the "NiFi+" deployment. Not to get off-topic but it
> > is somewhat analogous to exposing the Spring context such that alternate
> > implementations of the beans could be provided.
> >
> > There are options to consider, I'm sure each with its own set of pros and
> > cons. Off the top of my head:
> >
> > - Sidecar server responsible for executing Python scripts with FlowFile
> > input(s) and returning FlowFile output(s) directly from/to processors, like
> > Marc's suggestion but managed by NiFi
> > - Jep [1] and it's OS-independent counterpart jep-distro [2]
> > - Require Java bindings calling out to an embedded interpreter, such as
> > WrapPy4J (which uses Jep under the hood) [3]
> >
> > I am far from a Python expert and I know there are several places in the
> > framework where the Python stuff has landed, so I don't expect any other
> > implementation to be any easier or take less time than the original, which
> > I'm sure took quite a lot of time and effort which is very much
> > appreciated. It seems that Py4J is the long pole in the tent for the
> > concerns, not the capability itself.
> >
> > At the end of the day I think a discussion like this clearly shows the
> > commitment of the whole community to ensure Apache NiFi is all that we want
> > it to be and that we can introspect, improve, and collaborate to keep NiFi
> > awesome :)
> >
> > Regards,
> > Matt
> >
> > [1] https://urldefense.com/v3/__https://github.com/ninia/JEP?tab=readme-ov-file__;!!FtR4BK4x7WL3xYs!6mIniDSbJ62B5zII3b9g972tiJiM1K8M1E-1R8rvwlwl__KEfyNqtqWjM3xwIPbVawsv_VEsYzoNRToUxhft8Q3Rnnw$
> > [2] https://urldefense.com/v3/__https://github.com/icemachined/jep-distro?tab=readme-ov-file__;!!FtR4BK4x7WL3xYs!6mIniDSbJ62B5zII3b9g972tiJiM1K8M1E-1R8rvwlwl__KEfyNqtqWjM3xwIPbVawsv_VEsYzoNRToUxhft703BJKY$
> > [3] https://urldefense.com/v3/__https://github.com/313hemant313/wrappyj__;!!FtR4BK4x7WL3xYs!6mIniDSbJ62B5zII3b9g972tiJiM1K8M1E-1R8rvwlwl__KEfyNqtqWjM3xwIPbVawsv_VEsYzoNRToUxhftYE_fqvo$
> >
> > On Wed, Aug 12, 2026 at 8:19 AM Marc P. <ma...@gmail.com> wrote:
> >
> > > This is a feature I find personally valuable. I agree with several
> > > underlying issues, particularly resource management, Py4J, dependency
> > > handling, and the difficulty of maintaining a framework that spans both the
> > > JVM and Python ecosystems.
> > >
> > > Where I disagree is with the conclusion that these issues necessarily point
> > > toward deprecating Python Processor support itself.
> > >
> > > I think there is an important distinction between the value of Python as an
> > > extension model and the architecture currently used to implement it.
> > >
> > > To be solution oriented one way to look at this is to consider what users
> > > would do in the absence of Python Processor support. In many cases, the
> > > alternative is not to rewrite the functionality in Java. It is likely to
> > > become something like:
> > >
> > > Apache NiFi -> HTTP/gRPC -> Python microservice -> NiFi
> > >
> > > That architecture still has independent Python processes, Python dependency
> > > management, memory and CPU consumption outside the JVM, and a Python
> > > software supply chain. We have not eliminated any of those concerns; we
> > > have moved them outside NiFi and made every operator build and manage that
> > > integration themselves.
> > >
> > > There are also costs to that approach: additional network boundaries,
> > > deployment artifacts, authentication, service discovery, observability,
> > > retry semantics, and operational complexity. From a governance perspective,
> > > NiFi may actually have less visibility and control over that external
> > > Python workload than it would if Python execution remained part of a
> > > defined NiFi extension model.
> > >
> > > That makes me wonder whether the better direction is to reconsider the
> > > Python runtime architecture rather than Python support itself.
> > >
> > > For example, instead of treating the current Py4J-based implementation as
> > > something that must be preserved indefinitely, we could define a stronger
> > > process boundary around Python execution. A future implementation could use
> > > a well-defined IPC or RPC contract, potentially based on gRPC/protobuf,
> > > Unix domain sockets, shared memory, Arrow, or another mechanism designed
> > > around coarse-grained operations.
> > >
> > > NiFi does not need to become a Python integrated application, but I think
> > > there is substantial value in allowing those workloads to participate
> > > directly in a governed NiFi dataflow without requiring users to construct a
> > > separate microservice architecture for every Python integration.
> > >
> > > For that reason, I would favor separating two questions:
> > >
> > >    1. Is the current Python Processor implementation the architecture we
> > >    want to support long term?
> > >    2. Should NiFi continue to provide a first-class Python extension
> > >    capability?
> > >
> > > I could see a strong argument that the answer to the first question is no,
> > > particularly if Py4J is becoming a maintenance risk. I am much less
> > > convinced that the answer to the second question should also be no.
> > >
> > > With that said I distinctly understand that time is limited and we would
> > > all develop the coolest of things with infinite resources, so I yield to
> > > those with more influence, but would happy to contribute to a better
> > > architecture if others find it valuable.
> > >
> > > Thanks,
> > >
> > > Marc
> > >
> > > On Wed, Aug 12, 2026 at 12:10 AM Joe Witt <jo...@gmail.com> wrote:
> > >
> > > > You are totally fine to engage on this thread Sean. We appreciate it
> > > >
> > > >
> > > > On Tue, Aug 11, 2026 at 8:17 PM Sean Hunter via dev <dev@nifi.apache.org
> > > >
> > > > wrote:
> > > >
> > > > > Hi David,
> > > > >
> > > > > As a user, rather than contributor, I think my primary concern is one
> > > of
> > > > > operations and implementation. What alternative would be available? We
> > > > used
> > > > > the 1.x version and were quite excited to move to 2.0 with its more
> > > > native
> > > > > integration that was easier to troubleshoot. We also do build a NAR as
> > > > part
> > > > > of our pipeline process, and I suspect since we used a blog as guidance
> > > > to
> > > > > develop that process, that others likely do as well. Our use case is a
> > > > > particularly nasty data transform (Windows logs of some sort,
> > > naturally)
> > > > > that’s reasonable to write and execute in Python. It’s readable, while
> > > > > still being performant enough for our needs, and seemingly not causing
> > > > > system stability issues for us.
> > > > >
> > > > > I do understand and take your points. What I’d like to better
> > > understand
> > > > > are the options users have. Do you have any thoughts?
> > > > >
> > > > > Thank you and have a great day!
> > > > > Sean
> > > > >
> > > > > PS - Please let me know if this is too far off topic for the dev
> > > mailing
> > > > > list and I’ll take it elsewhere.
> > > > >
> > > > > From: David Handermann <ex...@apache.org>
> > > > > Date: Tuesday, August 11, 2026 at 8:38 PM
> > > > > To: dev@nifi.apache.org <de...@nifi.apache.org>
> > > > > Subject: [DISCUSS] Future Support for Python Processors?
> > > > >
> > > > > Be Careful: Email is from an External Source
> > > > > Use caution when opening emails and do not click on links or
> > > attachments
> > > > > from unknown senders. For more resources, visit
> > > > http://security.cat.com/phishing .
> > > > >
> > > > >
> > > > > Team,
> > > > >
> > > > > The introduction of support for Python Processors was one of the major
> > > > > new features of Apache NiFi 2. In light of experience, however, I
> > > > > believe it is time to reconsider future support for this extension
> > > > > strategy. The idea of removing this feature may raise some concerns,
> > > > > but it is important to consider the options based on scalability,
> > > > > security, and supportability.
> > > > >
> > > > > One of the greatest benefits of Python-based Processors is rapid
> > > > > iteration. Although this is a strength during the development phase,
> > > > > it can become a serious weakness when moving to production.
> > > > >
> > > > > NiFi is fundamentally a Java-based application, thus tuning resource
> > > > > allocation is a key part of maintaining a production system. Running a
> > > > > Python Processor requires one or more independent Python processes,
> > > > > each with their own memory consumption and CPU usage patterns. What
> > > > > might work at a small scale on a development machine breaks down when
> > > > > attempting to run at production scale, without careful tuning. This is
> > > > > challenging enough with Java-based components, but global settings
> > > > > like heap and non-heap memory provide clear boundaries. Python
> > > > > Processors, as currently implemented, do not support this level of
> > > > > control. Beyond process control, operations that require communicating
> > > > > between the NiFi JVM and Python processes involve socket
> > > > > communication, which can be expensive for frequent calls. Thus an
> > > > > apparent initial advantage of Python Processors becomes a major
> > > > > problem when moving beyond limited types of operations.
> > > > >
> > > > > When it comes to security, Python Processors can involve more serious
> > > > > risks than bundled Java extensions.
> > > > >
> > > > > With NiFi support for downloading dependencies at runtime, it is
> > > > > essential for an administrator to provide a secure supply chain
> > > > > strategy for retrieving these dependencies. Although the NiFi
> > > > > framework also supports packaging Python Processors as NAR bundles,
> > > > > this packaging capability is not currently part of the Apache NiFi
> > > > > project. The project could build on third party work, but as Python
> > > > > packaging is less opinionated, this is an additional maintenance
> > > > > question that quickly becomes more involved. This combination of
> > > > > supply chain security and lack of packaging support raises security
> > > > > challenges.
> > > > >
> > > > > Supportability is always a question for any feature in an open source
> > > > > project, but Python Processors are a particular point of concern.
> > > > > Providing NiFi framework support requires a strong knowledge of the
> > > > > NiFi project itself, and a strong knowledge of Python. Project
> > > > > maintainers and contributors usually have expertise in particular
> > > > > areas, but it is rare to find both. More to the point, however, the
> > > > > Py4J project, which NiFi depends on for integration, has seen limited
> > > > > maintenance, with the last tagged version dating to January 2025.
> > > > > Without robust architecture support, the NiFi project cannot provide a
> > > > > strong framework for future implementation.
> > > > >
> > > > > Taking these issues together, I believe the project should consider
> > > > > deprecating support for Python Processors. Similar to other
> > > > > deprecations, actual removal would not occur until a major release
> > > > > version, and that would be a separate conversation.
> > > > >
> > > > > Rather than continuing to extend the life of Python Processor support,
> > > > > it seems better to invest limited project resources into maintaining
> > > > > and improving core functionality.
> > > > >
> > > > > Regards,
> > > > > David Handermann
> > > > > Apache NiFi PMC Chair
> > > > >
> > > > >
> > > >
> > >
>

Re: [DISCUSS] Future Support for Python Processors?

[Daniel Weidman via dev <de...@nifi.apache.org>]

David,

Thanks for raising this directly rather than letting it drift, and for laying
out the reasoning rather than just the conclusion.

On scalability, your point lands. In my experience the memory behavior of a
Python Processor is largely a function of how it is written, and disciplined
implementations can be well behaved. But I recognize that is an argument about
good citizenship, not about enforcement, and it does not give an operator the
equivalent of a heap ceiling. That is a real gap. It does seem addressable at
the process supervision layer, through cgroup or rlimit bounds on spawned
interpreters, rather than being inherent to Python as an extension language.

On security, I would offer a friendly amendment. The meaningful difference does
not seem to be Python versus Java, but when and where third party dependencies
land. A NAR resolves its tree at build time and ships self contained, so it can
be scanned, signed, and deployed into an air gapped environment, and unpacking
it executes nothing. The current Python path resolves dependencies at runtime
via pip on the NiFi host, and pip installation does execute packaging code.
That is a real governance gap, and I will concede PyPI carries risks Maven
Central does not.

But as you noted, the framework already supports loading Python Processors
packaged as NARs. What is missing is project-provided tooling to build them.
Sean's note about relying on a blog post to produce one seems telling. Closing
that gap looks like a smaller undertaking than removing the extension point,
and it would address the supply chain concern directly rather than by
subtraction.

I will admit I had not realized the current implementation still rests on Py4J.
That reframes a good deal of this for me, and it tracks with Matt's read that
Py4J is the long pole rather than the capability itself.

For what it is worth as an operator, the practical value of Python Processors
is not just iteration speed. It is that the people writing our transforms are
Python native. Removal without a replacement does not convert that work to
Java, it pushes it out of NiFi entirely, to sidecars or external services,
where, as Marc noted, we keep every one of the concerns you listed but lose
NiFi's visibility into them.

I would rather see the runtime architecture reconsidered than the capability
retired. If the deciding constraint is maintainer cycles, it would be useful to
know what a replacement would need in terms of effort and expertise. That is a
more actionable ask of the community than a deprecation vote.

Regards,
Dan


From: David Handermann <ex...@apache.org>
Date: Wednesday, August 12, 2026 at 9:10 AM
To: dev@nifi.apache.org <de...@nifi.apache.org>
Subject: Re: [DISCUSS] Future Support for Python Processors?

Thanks to everyone for the replies thus far!

The question about alternatives gets at the core issue, and the ideas
around a more loosely coupled architecture provide some helpful
possibilities.

At a basic level, simple deprecation and removal would leave the
current set of options, including standard Java, Groovy scripting, or
externalized Python scripts executed as a stream. None of these
provide the same level of features for Python itself, which prompted
the original native Python Processor implementation. Without a
replacement, these alternatives would remain. With the rise of
AI-enabled development, it is now much easier to go from Python to
Java, but work is still required.

As to future strategies for Python integration, the current discussion
around deprecation is more narrowly focused on the existing
framework-level extension. Removal of the current implementation is
not intended to halt future innovation. Although removal without an
initial replacement might not be the perfect solution, it may be
necessary given available maintainer cycles, and the substantive
issues I described in the opening.

If there is sufficient interest and effort to propose and build a
different solution, that would be a great outcome of this discussion.
I've thought at different points about reworking the Script Engine
integration to provide a cleaner path to external extension, so there
are definitely opportunities in that direction.

With that, my primary concerns with the current implementation remain,
but glad to consider future alternatives.

Regards,
David Handermann


On Wed, Aug 12, 2026 at 7:52 AM Matt Burgess <ma...@apache.org> wrote:
>
> I agree with Marc, the concerns of the implementation of the capability
> should not warrant deprecation and future removal of the capability, but
> certainly we want to ensure that anything we implement for NiFi doesn't
> negatively the reputation of Apache NiFi as a secure, robust, reliable
> system.
>
>  I think we should instead consider a new architecture, one
> fully extensible so if the community decides to deprecate the capability,
> it could still be restored using some extension point. This kind of generic
> approach could allow other extensions not sanctioned by the Apache NiFi
> community to coexist in the "NiFi+" deployment. Not to get off-topic but it
> is somewhat analogous to exposing the Spring context such that alternate
> implementations of the beans could be provided.
>
> There are options to consider, I'm sure each with its own set of pros and
> cons. Off the top of my head:
>
> - Sidecar server responsible for executing Python scripts with FlowFile
> input(s) and returning FlowFile output(s) directly from/to processors, like
> Marc's suggestion but managed by NiFi
> - Jep [1] and it's OS-independent counterpart jep-distro [2]
> - Require Java bindings calling out to an embedded interpreter, such as
> WrapPy4J (which uses Jep under the hood) [3]
>
> I am far from a Python expert and I know there are several places in the
> framework where the Python stuff has landed, so I don't expect any other
> implementation to be any easier or take less time than the original, which
> I'm sure took quite a lot of time and effort which is very much
> appreciated. It seems that Py4J is the long pole in the tent for the
> concerns, not the capability itself.
>
> At the end of the day I think a discussion like this clearly shows the
> commitment of the whole community to ensure Apache NiFi is all that we want
> it to be and that we can introspect, improve, and collaborate to keep NiFi
> awesome :)
>
> Regards,
> Matt
>
> [1] https://urldefense.com/v3/__https://github.com/ninia/JEP?tab=readme-ov-file__;!!FtR4BK4x7WL3xYs!6mIniDSbJ62B5zII3b9g972tiJiM1K8M1E-1R8rvwlwl__KEfyNqtqWjM3xwIPbVawsv_VEsYzoNRToUxhft8Q3Rnnw$
> [2] https://urldefense.com/v3/__https://github.com/icemachined/jep-distro?tab=readme-ov-file__;!!FtR4BK4x7WL3xYs!6mIniDSbJ62B5zII3b9g972tiJiM1K8M1E-1R8rvwlwl__KEfyNqtqWjM3xwIPbVawsv_VEsYzoNRToUxhft703BJKY$
> [3] https://urldefense.com/v3/__https://github.com/313hemant313/wrappyj__;!!FtR4BK4x7WL3xYs!6mIniDSbJ62B5zII3b9g972tiJiM1K8M1E-1R8rvwlwl__KEfyNqtqWjM3xwIPbVawsv_VEsYzoNRToUxhftYE_fqvo$
>
> On Wed, Aug 12, 2026 at 8:19 AM Marc P. <ma...@gmail.com> wrote:
>
> > This is a feature I find personally valuable. I agree with several
> > underlying issues, particularly resource management, Py4J, dependency
> > handling, and the difficulty of maintaining a framework that spans both the
> > JVM and Python ecosystems.
> >
> > Where I disagree is with the conclusion that these issues necessarily point
> > toward deprecating Python Processor support itself.
> >
> > I think there is an important distinction between the value of Python as an
> > extension model and the architecture currently used to implement it.
> >
> > To be solution oriented one way to look at this is to consider what users
> > would do in the absence of Python Processor support. In many cases, the
> > alternative is not to rewrite the functionality in Java. It is likely to
> > become something like:
> >
> > Apache NiFi -> HTTP/gRPC -> Python microservice -> NiFi
> >
> > That architecture still has independent Python processes, Python dependency
> > management, memory and CPU consumption outside the JVM, and a Python
> > software supply chain. We have not eliminated any of those concerns; we
> > have moved them outside NiFi and made every operator build and manage that
> > integration themselves.
> >
> > There are also costs to that approach: additional network boundaries,
> > deployment artifacts, authentication, service discovery, observability,
> > retry semantics, and operational complexity. From a governance perspective,
> > NiFi may actually have less visibility and control over that external
> > Python workload than it would if Python execution remained part of a
> > defined NiFi extension model.
> >
> > That makes me wonder whether the better direction is to reconsider the
> > Python runtime architecture rather than Python support itself.
> >
> > For example, instead of treating the current Py4J-based implementation as
> > something that must be preserved indefinitely, we could define a stronger
> > process boundary around Python execution. A future implementation could use
> > a well-defined IPC or RPC contract, potentially based on gRPC/protobuf,
> > Unix domain sockets, shared memory, Arrow, or another mechanism designed
> > around coarse-grained operations.
> >
> > NiFi does not need to become a Python integrated application, but I think
> > there is substantial value in allowing those workloads to participate
> > directly in a governed NiFi dataflow without requiring users to construct a
> > separate microservice architecture for every Python integration.
> >
> > For that reason, I would favor separating two questions:
> >
> >    1. Is the current Python Processor implementation the architecture we
> >    want to support long term?
> >    2. Should NiFi continue to provide a first-class Python extension
> >    capability?
> >
> > I could see a strong argument that the answer to the first question is no,
> > particularly if Py4J is becoming a maintenance risk. I am much less
> > convinced that the answer to the second question should also be no.
> >
> > With that said I distinctly understand that time is limited and we would
> > all develop the coolest of things with infinite resources, so I yield to
> > those with more influence, but would happy to contribute to a better
> > architecture if others find it valuable.
> >
> > Thanks,
> >
> > Marc
> >
> > On Wed, Aug 12, 2026 at 12:10 AM Joe Witt <jo...@gmail.com> wrote:
> >
> > > You are totally fine to engage on this thread Sean. We appreciate it
> > >
> > >
> > > On Tue, Aug 11, 2026 at 8:17 PM Sean Hunter via dev <dev@nifi.apache.org
> > >
> > > wrote:
> > >
> > > > Hi David,
> > > >
> > > > As a user, rather than contributor, I think my primary concern is one
> > of
> > > > operations and implementation. What alternative would be available? We
> > > used
> > > > the 1.x version and were quite excited to move to 2.0 with its more
> > > native
> > > > integration that was easier to troubleshoot. We also do build a NAR as
> > > part
> > > > of our pipeline process, and I suspect since we used a blog as guidance
> > > to
> > > > develop that process, that others likely do as well. Our use case is a
> > > > particularly nasty data transform (Windows logs of some sort,
> > naturally)
> > > > that’s reasonable to write and execute in Python. It’s readable, while
> > > > still being performant enough for our needs, and seemingly not causing
> > > > system stability issues for us.
> > > >
> > > > I do understand and take your points. What I’d like to better
> > understand
> > > > are the options users have. Do you have any thoughts?
> > > >
> > > > Thank you and have a great day!
> > > > Sean
> > > >
> > > > PS - Please let me know if this is too far off topic for the dev
> > mailing
> > > > list and I’ll take it elsewhere.
> > > >
> > > > From: David Handermann <ex...@apache.org>
> > > > Date: Tuesday, August 11, 2026 at 8:38 PM
> > > > To: dev@nifi.apache.org <de...@nifi.apache.org>
> > > > Subject: [DISCUSS] Future Support for Python Processors?
> > > >
> > > > Be Careful: Email is from an External Source
> > > > Use caution when opening emails and do not click on links or
> > attachments
> > > > from unknown senders. For more resources, visit
> > > http://security.cat.com/phishing .
> > > >
> > > >
> > > > Team,
> > > >
> > > > The introduction of support for Python Processors was one of the major
> > > > new features of Apache NiFi 2. In light of experience, however, I
> > > > believe it is time to reconsider future support for this extension
> > > > strategy. The idea of removing this feature may raise some concerns,
> > > > but it is important to consider the options based on scalability,
> > > > security, and supportability.
> > > >
> > > > One of the greatest benefits of Python-based Processors is rapid
> > > > iteration. Although this is a strength during the development phase,
> > > > it can become a serious weakness when moving to production.
> > > >
> > > > NiFi is fundamentally a Java-based application, thus tuning resource
> > > > allocation is a key part of maintaining a production system. Running a
> > > > Python Processor requires one or more independent Python processes,
> > > > each with their own memory consumption and CPU usage patterns. What
> > > > might work at a small scale on a development machine breaks down when
> > > > attempting to run at production scale, without careful tuning. This is
> > > > challenging enough with Java-based components, but global settings
> > > > like heap and non-heap memory provide clear boundaries. Python
> > > > Processors, as currently implemented, do not support this level of
> > > > control. Beyond process control, operations that require communicating
> > > > between the NiFi JVM and Python processes involve socket
> > > > communication, which can be expensive for frequent calls. Thus an
> > > > apparent initial advantage of Python Processors becomes a major
> > > > problem when moving beyond limited types of operations.
> > > >
> > > > When it comes to security, Python Processors can involve more serious
> > > > risks than bundled Java extensions.
> > > >
> > > > With NiFi support for downloading dependencies at runtime, it is
> > > > essential for an administrator to provide a secure supply chain
> > > > strategy for retrieving these dependencies. Although the NiFi
> > > > framework also supports packaging Python Processors as NAR bundles,
> > > > this packaging capability is not currently part of the Apache NiFi
> > > > project. The project could build on third party work, but as Python
> > > > packaging is less opinionated, this is an additional maintenance
> > > > question that quickly becomes more involved. This combination of
> > > > supply chain security and lack of packaging support raises security
> > > > challenges.
> > > >
> > > > Supportability is always a question for any feature in an open source
> > > > project, but Python Processors are a particular point of concern.
> > > > Providing NiFi framework support requires a strong knowledge of the
> > > > NiFi project itself, and a strong knowledge of Python. Project
> > > > maintainers and contributors usually have expertise in particular
> > > > areas, but it is rare to find both. More to the point, however, the
> > > > Py4J project, which NiFi depends on for integration, has seen limited
> > > > maintenance, with the last tagged version dating to January 2025.
> > > > Without robust architecture support, the NiFi project cannot provide a
> > > > strong framework for future implementation.
> > > >
> > > > Taking these issues together, I believe the project should consider
> > > > deprecating support for Python Processors. Similar to other
> > > > deprecations, actual removal would not occur until a major release
> > > > version, and that would be a separate conversation.
> > > >
> > > > Rather than continuing to extend the life of Python Processor support,
> > > > it seems better to invest limited project resources into maintaining
> > > > and improving core functionality.
> > > >
> > > > Regards,
> > > > David Handermann
> > > > Apache NiFi PMC Chair
> > > >
> > > >
> > >
> >

Re: [DISCUSS] Future Support for Python Processors?

[David Handermann <ex...@apache.org>]

Thanks to everyone for the replies thus far!

The question about alternatives gets at the core issue, and the ideas
around a more loosely coupled architecture provide some helpful
possibilities.

At a basic level, simple deprecation and removal would leave the
current set of options, including standard Java, Groovy scripting, or
externalized Python scripts executed as a stream. None of these
provide the same level of features for Python itself, which prompted
the original native Python Processor implementation. Without a
replacement, these alternatives would remain. With the rise of
AI-enabled development, it is now much easier to go from Python to
Java, but work is still required.

As to future strategies for Python integration, the current discussion
around deprecation is more narrowly focused on the existing
framework-level extension. Removal of the current implementation is
not intended to halt future innovation. Although removal without an
initial replacement might not be the perfect solution, it may be
necessary given available maintainer cycles, and the substantive
issues I described in the opening.

If there is sufficient interest and effort to propose and build a
different solution, that would be a great outcome of this discussion.
I've thought at different points about reworking the Script Engine
integration to provide a cleaner path to external extension, so there
are definitely opportunities in that direction.

With that, my primary concerns with the current implementation remain,
but glad to consider future alternatives.

Regards,
David Handermann


On Wed, Aug 12, 2026 at 7:52 AM Matt Burgess <ma...@apache.org> wrote:
>
> I agree with Marc, the concerns of the implementation of the capability
> should not warrant deprecation and future removal of the capability, but
> certainly we want to ensure that anything we implement for NiFi doesn't
> negatively the reputation of Apache NiFi as a secure, robust, reliable
> system.
>
>  I think we should instead consider a new architecture, one
> fully extensible so if the community decides to deprecate the capability,
> it could still be restored using some extension point. This kind of generic
> approach could allow other extensions not sanctioned by the Apache NiFi
> community to coexist in the "NiFi+" deployment. Not to get off-topic but it
> is somewhat analogous to exposing the Spring context such that alternate
> implementations of the beans could be provided.
>
> There are options to consider, I'm sure each with its own set of pros and
> cons. Off the top of my head:
>
> - Sidecar server responsible for executing Python scripts with FlowFile
> input(s) and returning FlowFile output(s) directly from/to processors, like
> Marc's suggestion but managed by NiFi
> - Jep [1] and it's OS-independent counterpart jep-distro [2]
> - Require Java bindings calling out to an embedded interpreter, such as
> WrapPy4J (which uses Jep under the hood) [3]
>
> I am far from a Python expert and I know there are several places in the
> framework where the Python stuff has landed, so I don't expect any other
> implementation to be any easier or take less time than the original, which
> I'm sure took quite a lot of time and effort which is very much
> appreciated. It seems that Py4J is the long pole in the tent for the
> concerns, not the capability itself.
>
> At the end of the day I think a discussion like this clearly shows the
> commitment of the whole community to ensure Apache NiFi is all that we want
> it to be and that we can introspect, improve, and collaborate to keep NiFi
> awesome :)
>
> Regards,
> Matt
>
> [1] https://github.com/ninia/JEP?tab=readme-ov-file
> [2] https://github.com/icemachined/jep-distro?tab=readme-ov-file
> [3] https://github.com/313hemant313/wrappyj
>
> On Wed, Aug 12, 2026 at 8:19 AM Marc P. <ma...@gmail.com> wrote:
>
> > This is a feature I find personally valuable. I agree with several
> > underlying issues, particularly resource management, Py4J, dependency
> > handling, and the difficulty of maintaining a framework that spans both the
> > JVM and Python ecosystems.
> >
> > Where I disagree is with the conclusion that these issues necessarily point
> > toward deprecating Python Processor support itself.
> >
> > I think there is an important distinction between the value of Python as an
> > extension model and the architecture currently used to implement it.
> >
> > To be solution oriented one way to look at this is to consider what users
> > would do in the absence of Python Processor support. In many cases, the
> > alternative is not to rewrite the functionality in Java. It is likely to
> > become something like:
> >
> > Apache NiFi -> HTTP/gRPC -> Python microservice -> NiFi
> >
> > That architecture still has independent Python processes, Python dependency
> > management, memory and CPU consumption outside the JVM, and a Python
> > software supply chain. We have not eliminated any of those concerns; we
> > have moved them outside NiFi and made every operator build and manage that
> > integration themselves.
> >
> > There are also costs to that approach: additional network boundaries,
> > deployment artifacts, authentication, service discovery, observability,
> > retry semantics, and operational complexity. From a governance perspective,
> > NiFi may actually have less visibility and control over that external
> > Python workload than it would if Python execution remained part of a
> > defined NiFi extension model.
> >
> > That makes me wonder whether the better direction is to reconsider the
> > Python runtime architecture rather than Python support itself.
> >
> > For example, instead of treating the current Py4J-based implementation as
> > something that must be preserved indefinitely, we could define a stronger
> > process boundary around Python execution. A future implementation could use
> > a well-defined IPC or RPC contract, potentially based on gRPC/protobuf,
> > Unix domain sockets, shared memory, Arrow, or another mechanism designed
> > around coarse-grained operations.
> >
> > NiFi does not need to become a Python integrated application, but I think
> > there is substantial value in allowing those workloads to participate
> > directly in a governed NiFi dataflow without requiring users to construct a
> > separate microservice architecture for every Python integration.
> >
> > For that reason, I would favor separating two questions:
> >
> >    1. Is the current Python Processor implementation the architecture we
> >    want to support long term?
> >    2. Should NiFi continue to provide a first-class Python extension
> >    capability?
> >
> > I could see a strong argument that the answer to the first question is no,
> > particularly if Py4J is becoming a maintenance risk. I am much less
> > convinced that the answer to the second question should also be no.
> >
> > With that said I distinctly understand that time is limited and we would
> > all develop the coolest of things with infinite resources, so I yield to
> > those with more influence, but would happy to contribute to a better
> > architecture if others find it valuable.
> >
> > Thanks,
> >
> > Marc
> >
> > On Wed, Aug 12, 2026 at 12:10 AM Joe Witt <jo...@gmail.com> wrote:
> >
> > > You are totally fine to engage on this thread Sean. We appreciate it
> > >
> > >
> > > On Tue, Aug 11, 2026 at 8:17 PM Sean Hunter via dev <dev@nifi.apache.org
> > >
> > > wrote:
> > >
> > > > Hi David,
> > > >
> > > > As a user, rather than contributor, I think my primary concern is one
> > of
> > > > operations and implementation. What alternative would be available? We
> > > used
> > > > the 1.x version and were quite excited to move to 2.0 with its more
> > > native
> > > > integration that was easier to troubleshoot. We also do build a NAR as
> > > part
> > > > of our pipeline process, and I suspect since we used a blog as guidance
> > > to
> > > > develop that process, that others likely do as well. Our use case is a
> > > > particularly nasty data transform (Windows logs of some sort,
> > naturally)
> > > > that’s reasonable to write and execute in Python. It’s readable, while
> > > > still being performant enough for our needs, and seemingly not causing
> > > > system stability issues for us.
> > > >
> > > > I do understand and take your points. What I’d like to better
> > understand
> > > > are the options users have. Do you have any thoughts?
> > > >
> > > > Thank you and have a great day!
> > > > Sean
> > > >
> > > > PS - Please let me know if this is too far off topic for the dev
> > mailing
> > > > list and I’ll take it elsewhere.
> > > >
> > > > From: David Handermann <ex...@apache.org>
> > > > Date: Tuesday, August 11, 2026 at 8:38 PM
> > > > To: dev@nifi.apache.org <de...@nifi.apache.org>
> > > > Subject: [DISCUSS] Future Support for Python Processors?
> > > >
> > > > Be Careful: Email is from an External Source
> > > > Use caution when opening emails and do not click on links or
> > attachments
> > > > from unknown senders. For more resources, visit
> > > security.cat.com/phishing.
> > > >
> > > >
> > > > Team,
> > > >
> > > > The introduction of support for Python Processors was one of the major
> > > > new features of Apache NiFi 2. In light of experience, however, I
> > > > believe it is time to reconsider future support for this extension
> > > > strategy. The idea of removing this feature may raise some concerns,
> > > > but it is important to consider the options based on scalability,
> > > > security, and supportability.
> > > >
> > > > One of the greatest benefits of Python-based Processors is rapid
> > > > iteration. Although this is a strength during the development phase,
> > > > it can become a serious weakness when moving to production.
> > > >
> > > > NiFi is fundamentally a Java-based application, thus tuning resource
> > > > allocation is a key part of maintaining a production system. Running a
> > > > Python Processor requires one or more independent Python processes,
> > > > each with their own memory consumption and CPU usage patterns. What
> > > > might work at a small scale on a development machine breaks down when
> > > > attempting to run at production scale, without careful tuning. This is
> > > > challenging enough with Java-based components, but global settings
> > > > like heap and non-heap memory provide clear boundaries. Python
> > > > Processors, as currently implemented, do not support this level of
> > > > control. Beyond process control, operations that require communicating
> > > > between the NiFi JVM and Python processes involve socket
> > > > communication, which can be expensive for frequent calls. Thus an
> > > > apparent initial advantage of Python Processors becomes a major
> > > > problem when moving beyond limited types of operations.
> > > >
> > > > When it comes to security, Python Processors can involve more serious
> > > > risks than bundled Java extensions.
> > > >
> > > > With NiFi support for downloading dependencies at runtime, it is
> > > > essential for an administrator to provide a secure supply chain
> > > > strategy for retrieving these dependencies. Although the NiFi
> > > > framework also supports packaging Python Processors as NAR bundles,
> > > > this packaging capability is not currently part of the Apache NiFi
> > > > project. The project could build on third party work, but as Python
> > > > packaging is less opinionated, this is an additional maintenance
> > > > question that quickly becomes more involved. This combination of
> > > > supply chain security and lack of packaging support raises security
> > > > challenges.
> > > >
> > > > Supportability is always a question for any feature in an open source
> > > > project, but Python Processors are a particular point of concern.
> > > > Providing NiFi framework support requires a strong knowledge of the
> > > > NiFi project itself, and a strong knowledge of Python. Project
> > > > maintainers and contributors usually have expertise in particular
> > > > areas, but it is rare to find both. More to the point, however, the
> > > > Py4J project, which NiFi depends on for integration, has seen limited
> > > > maintenance, with the last tagged version dating to January 2025.
> > > > Without robust architecture support, the NiFi project cannot provide a
> > > > strong framework for future implementation.
> > > >
> > > > Taking these issues together, I believe the project should consider
> > > > deprecating support for Python Processors. Similar to other
> > > > deprecations, actual removal would not occur until a major release
> > > > version, and that would be a separate conversation.
> > > >
> > > > Rather than continuing to extend the life of Python Processor support,
> > > > it seems better to invest limited project resources into maintaining
> > > > and improving core functionality.
> > > >
> > > > Regards,
> > > > David Handermann
> > > > Apache NiFi PMC Chair
> > > >
> > > >
> > >
> >

Re: [DISCUSS] Future Support for Python Processors?

[Matt Burgess <ma...@apache.org>]

I agree with Marc, the concerns of the implementation of the capability
should not warrant deprecation and future removal of the capability, but
certainly we want to ensure that anything we implement for NiFi doesn't
negatively the reputation of Apache NiFi as a secure, robust, reliable
system.

 I think we should instead consider a new architecture, one
fully extensible so if the community decides to deprecate the capability,
it could still be restored using some extension point. This kind of generic
approach could allow other extensions not sanctioned by the Apache NiFi
community to coexist in the "NiFi+" deployment. Not to get off-topic but it
is somewhat analogous to exposing the Spring context such that alternate
implementations of the beans could be provided.

There are options to consider, I'm sure each with its own set of pros and
cons. Off the top of my head:

- Sidecar server responsible for executing Python scripts with FlowFile
input(s) and returning FlowFile output(s) directly from/to processors, like
Marc's suggestion but managed by NiFi
- Jep [1] and it's OS-independent counterpart jep-distro [2]
- Require Java bindings calling out to an embedded interpreter, such as
WrapPy4J (which uses Jep under the hood) [3]

I am far from a Python expert and I know there are several places in the
framework where the Python stuff has landed, so I don't expect any other
implementation to be any easier or take less time than the original, which
I'm sure took quite a lot of time and effort which is very much
appreciated. It seems that Py4J is the long pole in the tent for the
concerns, not the capability itself.

At the end of the day I think a discussion like this clearly shows the
commitment of the whole community to ensure Apache NiFi is all that we want
it to be and that we can introspect, improve, and collaborate to keep NiFi
awesome :)

Regards,
Matt

[1] https://github.com/ninia/JEP?tab=readme-ov-file
[2] https://github.com/icemachined/jep-distro?tab=readme-ov-file
[3] https://github.com/313hemant313/wrappyj

On Wed, Aug 12, 2026 at 8:19 AM Marc P. <ma...@gmail.com> wrote:

> This is a feature I find personally valuable. I agree with several
> underlying issues, particularly resource management, Py4J, dependency
> handling, and the difficulty of maintaining a framework that spans both the
> JVM and Python ecosystems.
>
> Where I disagree is with the conclusion that these issues necessarily point
> toward deprecating Python Processor support itself.
>
> I think there is an important distinction between the value of Python as an
> extension model and the architecture currently used to implement it.
>
> To be solution oriented one way to look at this is to consider what users
> would do in the absence of Python Processor support. In many cases, the
> alternative is not to rewrite the functionality in Java. It is likely to
> become something like:
>
> Apache NiFi -> HTTP/gRPC -> Python microservice -> NiFi
>
> That architecture still has independent Python processes, Python dependency
> management, memory and CPU consumption outside the JVM, and a Python
> software supply chain. We have not eliminated any of those concerns; we
> have moved them outside NiFi and made every operator build and manage that
> integration themselves.
>
> There are also costs to that approach: additional network boundaries,
> deployment artifacts, authentication, service discovery, observability,
> retry semantics, and operational complexity. From a governance perspective,
> NiFi may actually have less visibility and control over that external
> Python workload than it would if Python execution remained part of a
> defined NiFi extension model.
>
> That makes me wonder whether the better direction is to reconsider the
> Python runtime architecture rather than Python support itself.
>
> For example, instead of treating the current Py4J-based implementation as
> something that must be preserved indefinitely, we could define a stronger
> process boundary around Python execution. A future implementation could use
> a well-defined IPC or RPC contract, potentially based on gRPC/protobuf,
> Unix domain sockets, shared memory, Arrow, or another mechanism designed
> around coarse-grained operations.
>
> NiFi does not need to become a Python integrated application, but I think
> there is substantial value in allowing those workloads to participate
> directly in a governed NiFi dataflow without requiring users to construct a
> separate microservice architecture for every Python integration.
>
> For that reason, I would favor separating two questions:
>
>    1. Is the current Python Processor implementation the architecture we
>    want to support long term?
>    2. Should NiFi continue to provide a first-class Python extension
>    capability?
>
> I could see a strong argument that the answer to the first question is no,
> particularly if Py4J is becoming a maintenance risk. I am much less
> convinced that the answer to the second question should also be no.
>
> With that said I distinctly understand that time is limited and we would
> all develop the coolest of things with infinite resources, so I yield to
> those with more influence, but would happy to contribute to a better
> architecture if others find it valuable.
>
> Thanks,
>
> Marc
>
> On Wed, Aug 12, 2026 at 12:10 AM Joe Witt <jo...@gmail.com> wrote:
>
> > You are totally fine to engage on this thread Sean. We appreciate it
> >
> >
> > On Tue, Aug 11, 2026 at 8:17 PM Sean Hunter via dev <dev@nifi.apache.org
> >
> > wrote:
> >
> > > Hi David,
> > >
> > > As a user, rather than contributor, I think my primary concern is one
> of
> > > operations and implementation. What alternative would be available? We
> > used
> > > the 1.x version and were quite excited to move to 2.0 with its more
> > native
> > > integration that was easier to troubleshoot. We also do build a NAR as
> > part
> > > of our pipeline process, and I suspect since we used a blog as guidance
> > to
> > > develop that process, that others likely do as well. Our use case is a
> > > particularly nasty data transform (Windows logs of some sort,
> naturally)
> > > that’s reasonable to write and execute in Python. It’s readable, while
> > > still being performant enough for our needs, and seemingly not causing
> > > system stability issues for us.
> > >
> > > I do understand and take your points. What I’d like to better
> understand
> > > are the options users have. Do you have any thoughts?
> > >
> > > Thank you and have a great day!
> > > Sean
> > >
> > > PS - Please let me know if this is too far off topic for the dev
> mailing
> > > list and I’ll take it elsewhere.
> > >
> > > From: David Handermann <ex...@apache.org>
> > > Date: Tuesday, August 11, 2026 at 8:38 PM
> > > To: dev@nifi.apache.org <de...@nifi.apache.org>
> > > Subject: [DISCUSS] Future Support for Python Processors?
> > >
> > > Be Careful: Email is from an External Source
> > > Use caution when opening emails and do not click on links or
> attachments
> > > from unknown senders. For more resources, visit
> > security.cat.com/phishing.
> > >
> > >
> > > Team,
> > >
> > > The introduction of support for Python Processors was one of the major
> > > new features of Apache NiFi 2. In light of experience, however, I
> > > believe it is time to reconsider future support for this extension
> > > strategy. The idea of removing this feature may raise some concerns,
> > > but it is important to consider the options based on scalability,
> > > security, and supportability.
> > >
> > > One of the greatest benefits of Python-based Processors is rapid
> > > iteration. Although this is a strength during the development phase,
> > > it can become a serious weakness when moving to production.
> > >
> > > NiFi is fundamentally a Java-based application, thus tuning resource
> > > allocation is a key part of maintaining a production system. Running a
> > > Python Processor requires one or more independent Python processes,
> > > each with their own memory consumption and CPU usage patterns. What
> > > might work at a small scale on a development machine breaks down when
> > > attempting to run at production scale, without careful tuning. This is
> > > challenging enough with Java-based components, but global settings
> > > like heap and non-heap memory provide clear boundaries. Python
> > > Processors, as currently implemented, do not support this level of
> > > control. Beyond process control, operations that require communicating
> > > between the NiFi JVM and Python processes involve socket
> > > communication, which can be expensive for frequent calls. Thus an
> > > apparent initial advantage of Python Processors becomes a major
> > > problem when moving beyond limited types of operations.
> > >
> > > When it comes to security, Python Processors can involve more serious
> > > risks than bundled Java extensions.
> > >
> > > With NiFi support for downloading dependencies at runtime, it is
> > > essential for an administrator to provide a secure supply chain
> > > strategy for retrieving these dependencies. Although the NiFi
> > > framework also supports packaging Python Processors as NAR bundles,
> > > this packaging capability is not currently part of the Apache NiFi
> > > project. The project could build on third party work, but as Python
> > > packaging is less opinionated, this is an additional maintenance
> > > question that quickly becomes more involved. This combination of
> > > supply chain security and lack of packaging support raises security
> > > challenges.
> > >
> > > Supportability is always a question for any feature in an open source
> > > project, but Python Processors are a particular point of concern.
> > > Providing NiFi framework support requires a strong knowledge of the
> > > NiFi project itself, and a strong knowledge of Python. Project
> > > maintainers and contributors usually have expertise in particular
> > > areas, but it is rare to find both. More to the point, however, the
> > > Py4J project, which NiFi depends on for integration, has seen limited
> > > maintenance, with the last tagged version dating to January 2025.
> > > Without robust architecture support, the NiFi project cannot provide a
> > > strong framework for future implementation.
> > >
> > > Taking these issues together, I believe the project should consider
> > > deprecating support for Python Processors. Similar to other
> > > deprecations, actual removal would not occur until a major release
> > > version, and that would be a separate conversation.
> > >
> > > Rather than continuing to extend the life of Python Processor support,
> > > it seems better to invest limited project resources into maintaining
> > > and improving core functionality.
> > >
> > > Regards,
> > > David Handermann
> > > Apache NiFi PMC Chair
> > >
> > >
> >
>

Re: [DISCUSS] Future Support for Python Processors?

["Marc P." <ma...@gmail.com>]

This is a feature I find personally valuable. I agree with several
underlying issues, particularly resource management, Py4J, dependency
handling, and the difficulty of maintaining a framework that spans both the
JVM and Python ecosystems.

Where I disagree is with the conclusion that these issues necessarily point
toward deprecating Python Processor support itself.

I think there is an important distinction between the value of Python as an
extension model and the architecture currently used to implement it.

To be solution oriented one way to look at this is to consider what users
would do in the absence of Python Processor support. In many cases, the
alternative is not to rewrite the functionality in Java. It is likely to
become something like:

Apache NiFi -> HTTP/gRPC -> Python microservice -> NiFi

That architecture still has independent Python processes, Python dependency
management, memory and CPU consumption outside the JVM, and a Python
software supply chain. We have not eliminated any of those concerns; we
have moved them outside NiFi and made every operator build and manage that
integration themselves.

There are also costs to that approach: additional network boundaries,
deployment artifacts, authentication, service discovery, observability,
retry semantics, and operational complexity. From a governance perspective,
NiFi may actually have less visibility and control over that external
Python workload than it would if Python execution remained part of a
defined NiFi extension model.

That makes me wonder whether the better direction is to reconsider the
Python runtime architecture rather than Python support itself.

For example, instead of treating the current Py4J-based implementation as
something that must be preserved indefinitely, we could define a stronger
process boundary around Python execution. A future implementation could use
a well-defined IPC or RPC contract, potentially based on gRPC/protobuf,
Unix domain sockets, shared memory, Arrow, or another mechanism designed
around coarse-grained operations.

NiFi does not need to become a Python integrated application, but I think
there is substantial value in allowing those workloads to participate
directly in a governed NiFi dataflow without requiring users to construct a
separate microservice architecture for every Python integration.

For that reason, I would favor separating two questions:

   1. Is the current Python Processor implementation the architecture we
   want to support long term?
   2. Should NiFi continue to provide a first-class Python extension
   capability?

I could see a strong argument that the answer to the first question is no,
particularly if Py4J is becoming a maintenance risk. I am much less
convinced that the answer to the second question should also be no.

With that said I distinctly understand that time is limited and we would
all develop the coolest of things with infinite resources, so I yield to
those with more influence, but would happy to contribute to a better
architecture if others find it valuable.

Thanks,

Marc

On Wed, Aug 12, 2026 at 12:10 AM Joe Witt <jo...@gmail.com> wrote:

> You are totally fine to engage on this thread Sean. We appreciate it
>
>
> On Tue, Aug 11, 2026 at 8:17 PM Sean Hunter via dev <de...@nifi.apache.org>
> wrote:
>
> > Hi David,
> >
> > As a user, rather than contributor, I think my primary concern is one of
> > operations and implementation. What alternative would be available? We
> used
> > the 1.x version and were quite excited to move to 2.0 with its more
> native
> > integration that was easier to troubleshoot. We also do build a NAR as
> part
> > of our pipeline process, and I suspect since we used a blog as guidance
> to
> > develop that process, that others likely do as well. Our use case is a
> > particularly nasty data transform (Windows logs of some sort, naturally)
> > that’s reasonable to write and execute in Python. It’s readable, while
> > still being performant enough for our needs, and seemingly not causing
> > system stability issues for us.
> >
> > I do understand and take your points. What I’d like to better understand
> > are the options users have. Do you have any thoughts?
> >
> > Thank you and have a great day!
> > Sean
> >
> > PS - Please let me know if this is too far off topic for the dev mailing
> > list and I’ll take it elsewhere.
> >
> > From: David Handermann <ex...@apache.org>
> > Date: Tuesday, August 11, 2026 at 8:38 PM
> > To: dev@nifi.apache.org <de...@nifi.apache.org>
> > Subject: [DISCUSS] Future Support for Python Processors?
> >
> > Be Careful: Email is from an External Source
> > Use caution when opening emails and do not click on links or attachments
> > from unknown senders. For more resources, visit
> security.cat.com/phishing.
> >
> >
> > Team,
> >
> > The introduction of support for Python Processors was one of the major
> > new features of Apache NiFi 2. In light of experience, however, I
> > believe it is time to reconsider future support for this extension
> > strategy. The idea of removing this feature may raise some concerns,
> > but it is important to consider the options based on scalability,
> > security, and supportability.
> >
> > One of the greatest benefits of Python-based Processors is rapid
> > iteration. Although this is a strength during the development phase,
> > it can become a serious weakness when moving to production.
> >
> > NiFi is fundamentally a Java-based application, thus tuning resource
> > allocation is a key part of maintaining a production system. Running a
> > Python Processor requires one or more independent Python processes,
> > each with their own memory consumption and CPU usage patterns. What
> > might work at a small scale on a development machine breaks down when
> > attempting to run at production scale, without careful tuning. This is
> > challenging enough with Java-based components, but global settings
> > like heap and non-heap memory provide clear boundaries. Python
> > Processors, as currently implemented, do not support this level of
> > control. Beyond process control, operations that require communicating
> > between the NiFi JVM and Python processes involve socket
> > communication, which can be expensive for frequent calls. Thus an
> > apparent initial advantage of Python Processors becomes a major
> > problem when moving beyond limited types of operations.
> >
> > When it comes to security, Python Processors can involve more serious
> > risks than bundled Java extensions.
> >
> > With NiFi support for downloading dependencies at runtime, it is
> > essential for an administrator to provide a secure supply chain
> > strategy for retrieving these dependencies. Although the NiFi
> > framework also supports packaging Python Processors as NAR bundles,
> > this packaging capability is not currently part of the Apache NiFi
> > project. The project could build on third party work, but as Python
> > packaging is less opinionated, this is an additional maintenance
> > question that quickly becomes more involved. This combination of
> > supply chain security and lack of packaging support raises security
> > challenges.
> >
> > Supportability is always a question for any feature in an open source
> > project, but Python Processors are a particular point of concern.
> > Providing NiFi framework support requires a strong knowledge of the
> > NiFi project itself, and a strong knowledge of Python. Project
> > maintainers and contributors usually have expertise in particular
> > areas, but it is rare to find both. More to the point, however, the
> > Py4J project, which NiFi depends on for integration, has seen limited
> > maintenance, with the last tagged version dating to January 2025.
> > Without robust architecture support, the NiFi project cannot provide a
> > strong framework for future implementation.
> >
> > Taking these issues together, I believe the project should consider
> > deprecating support for Python Processors. Similar to other
> > deprecations, actual removal would not occur until a major release
> > version, and that would be a separate conversation.
> >
> > Rather than continuing to extend the life of Python Processor support,
> > it seems better to invest limited project resources into maintaining
> > and improving core functionality.
> >
> > Regards,
> > David Handermann
> > Apache NiFi PMC Chair
> >
> >
>

Re: [DISCUSS] Future Support for Python Processors?

[Joe Witt <jo...@gmail.com>]

You are totally fine to engage on this thread Sean. We appreciate it


On Tue, Aug 11, 2026 at 8:17 PM Sean Hunter via dev <de...@nifi.apache.org>
wrote:

> Hi David,
>
> As a user, rather than contributor, I think my primary concern is one of
> operations and implementation. What alternative would be available? We used
> the 1.x version and were quite excited to move to 2.0 with its more native
> integration that was easier to troubleshoot. We also do build a NAR as part
> of our pipeline process, and I suspect since we used a blog as guidance to
> develop that process, that others likely do as well. Our use case is a
> particularly nasty data transform (Windows logs of some sort, naturally)
> that’s reasonable to write and execute in Python. It’s readable, while
> still being performant enough for our needs, and seemingly not causing
> system stability issues for us.
>
> I do understand and take your points. What I’d like to better understand
> are the options users have. Do you have any thoughts?
>
> Thank you and have a great day!
> Sean
>
> PS - Please let me know if this is too far off topic for the dev mailing
> list and I’ll take it elsewhere.
>
> From: David Handermann <ex...@apache.org>
> Date: Tuesday, August 11, 2026 at 8:38 PM
> To: dev@nifi.apache.org <de...@nifi.apache.org>
> Subject: [DISCUSS] Future Support for Python Processors?
>
> Be Careful: Email is from an External Source
> Use caution when opening emails and do not click on links or attachments
> from unknown senders. For more resources, visit security.cat.com/phishing.
>
>
> Team,
>
> The introduction of support for Python Processors was one of the major
> new features of Apache NiFi 2. In light of experience, however, I
> believe it is time to reconsider future support for this extension
> strategy. The idea of removing this feature may raise some concerns,
> but it is important to consider the options based on scalability,
> security, and supportability.
>
> One of the greatest benefits of Python-based Processors is rapid
> iteration. Although this is a strength during the development phase,
> it can become a serious weakness when moving to production.
>
> NiFi is fundamentally a Java-based application, thus tuning resource
> allocation is a key part of maintaining a production system. Running a
> Python Processor requires one or more independent Python processes,
> each with their own memory consumption and CPU usage patterns. What
> might work at a small scale on a development machine breaks down when
> attempting to run at production scale, without careful tuning. This is
> challenging enough with Java-based components, but global settings
> like heap and non-heap memory provide clear boundaries. Python
> Processors, as currently implemented, do not support this level of
> control. Beyond process control, operations that require communicating
> between the NiFi JVM and Python processes involve socket
> communication, which can be expensive for frequent calls. Thus an
> apparent initial advantage of Python Processors becomes a major
> problem when moving beyond limited types of operations.
>
> When it comes to security, Python Processors can involve more serious
> risks than bundled Java extensions.
>
> With NiFi support for downloading dependencies at runtime, it is
> essential for an administrator to provide a secure supply chain
> strategy for retrieving these dependencies. Although the NiFi
> framework also supports packaging Python Processors as NAR bundles,
> this packaging capability is not currently part of the Apache NiFi
> project. The project could build on third party work, but as Python
> packaging is less opinionated, this is an additional maintenance
> question that quickly becomes more involved. This combination of
> supply chain security and lack of packaging support raises security
> challenges.
>
> Supportability is always a question for any feature in an open source
> project, but Python Processors are a particular point of concern.
> Providing NiFi framework support requires a strong knowledge of the
> NiFi project itself, and a strong knowledge of Python. Project
> maintainers and contributors usually have expertise in particular
> areas, but it is rare to find both. More to the point, however, the
> Py4J project, which NiFi depends on for integration, has seen limited
> maintenance, with the last tagged version dating to January 2025.
> Without robust architecture support, the NiFi project cannot provide a
> strong framework for future implementation.
>
> Taking these issues together, I believe the project should consider
> deprecating support for Python Processors. Similar to other
> deprecations, actual removal would not occur until a major release
> version, and that would be a separate conversation.
>
> Rather than continuing to extend the life of Python Processor support,
> it seems better to invest limited project resources into maintaining
> and improving core functionality.
>
> Regards,
> David Handermann
> Apache NiFi PMC Chair
>
>

Re: [DISCUSS] Future Support for Python Processors?

[Sean Hunter via dev <de...@nifi.apache.org>]

Hi David,

As a user, rather than contributor, I think my primary concern is one of operations and implementation. What alternative would be available? We used the 1.x version and were quite excited to move to 2.0 with its more native integration that was easier to troubleshoot. We also do build a NAR as part of our pipeline process, and I suspect since we used a blog as guidance to develop that process, that others likely do as well. Our use case is a particularly nasty data transform (Windows logs of some sort, naturally) that’s reasonable to write and execute in Python. It’s readable, while still being performant enough for our needs, and seemingly not causing system stability issues for us.

I do understand and take your points. What I’d like to better understand are the options users have. Do you have any thoughts?

Thank you and have a great day!
Sean

PS - Please let me know if this is too far off topic for the dev mailing list and I’ll take it elsewhere.

From: David Handermann <ex...@apache.org>
Date: Tuesday, August 11, 2026 at 8:38 PM
To: dev@nifi.apache.org <de...@nifi.apache.org>
Subject: [DISCUSS] Future Support for Python Processors?

Be Careful: Email is from an External Source
Use caution when opening emails and do not click on links or attachments from unknown senders. For more resources, visit security.cat.com/phishing.


Team,

The introduction of support for Python Processors was one of the major
new features of Apache NiFi 2. In light of experience, however, I
believe it is time to reconsider future support for this extension
strategy. The idea of removing this feature may raise some concerns,
but it is important to consider the options based on scalability,
security, and supportability.

One of the greatest benefits of Python-based Processors is rapid
iteration. Although this is a strength during the development phase,
it can become a serious weakness when moving to production.

NiFi is fundamentally a Java-based application, thus tuning resource
allocation is a key part of maintaining a production system. Running a
Python Processor requires one or more independent Python processes,
each with their own memory consumption and CPU usage patterns. What
might work at a small scale on a development machine breaks down when
attempting to run at production scale, without careful tuning. This is
challenging enough with Java-based components, but global settings
like heap and non-heap memory provide clear boundaries. Python
Processors, as currently implemented, do not support this level of
control. Beyond process control, operations that require communicating
between the NiFi JVM and Python processes involve socket
communication, which can be expensive for frequent calls. Thus an
apparent initial advantage of Python Processors becomes a major
problem when moving beyond limited types of operations.

When it comes to security, Python Processors can involve more serious
risks than bundled Java extensions.

With NiFi support for downloading dependencies at runtime, it is
essential for an administrator to provide a secure supply chain
strategy for retrieving these dependencies. Although the NiFi
framework also supports packaging Python Processors as NAR bundles,
this packaging capability is not currently part of the Apache NiFi
project. The project could build on third party work, but as Python
packaging is less opinionated, this is an additional maintenance
question that quickly becomes more involved. This combination of
supply chain security and lack of packaging support raises security
challenges.

Supportability is always a question for any feature in an open source
project, but Python Processors are a particular point of concern.
Providing NiFi framework support requires a strong knowledge of the
NiFi project itself, and a strong knowledge of Python. Project
maintainers and contributors usually have expertise in particular
areas, but it is rare to find both. More to the point, however, the
Py4J project, which NiFi depends on for integration, has seen limited
maintenance, with the last tagged version dating to January 2025.
Without robust architecture support, the NiFi project cannot provide a
strong framework for future implementation.

Taking these issues together, I believe the project should consider
deprecating support for Python Processors. Similar to other
deprecations, actual removal would not occur until a major release
version, and that would be a separate conversation.

Rather than continuing to extend the life of Python Processor support,
it seems better to invest limited project resources into maintaining
and improving core functionality.

Regards,
David Handermann
Apache NiFi PMC Chair