NiPyAPI vs direct Python requests

[David Early via users <us...@nifi.apache.org>]

Hi all,

Looking for opinion or experience on this one.

I developed some basic nipyapi functions that got sidelined for a while.
Currently being picked up again but getting some questions about why we
should use nipyapi vs doing "direct" api queries without actually using the
api.

It has been over a year since I did the initial work....i would like to
continue with the library vs direct, but was hoping for any experience or
opinions on why it would be good/bad to use nipyapi.  What does it buy me?

Thanks for any feedback, just looking to give some reasons to the young
guys coming in behind me.

Dave

-- 
David Early, Ph.D.
david.early@grokstream.com
720-470-7460 Cell

Re: NiPyAPI vs direct Python requests

[Daniel Chaffelson <ch...@gmail.com>]

I am one of the primary NiPyAPI maintainers, so my opinion is a bit biased.

NiPy, by SLOC, is mostly a big library verbosely procedurally generated
from the NiFi swagger spec, covering every possible command that it
accepts.
However, when measured by developer-hours it is primarily a pile of proven
convenience methods and a few demo scripts which resolve some of the more
complex interactions you are likely to encounter, like fetching and
stopping all Process groups in any version of NiFi, or handling promotion
of flow design changes between Dev and Prod.
It also has fairly well tested implementations for handling authentication
and authorization, and generally I would advise anyone to use a tested
community library for these areas over implementing their own.
The impact of this however is literally hundreds of methods that you have
absolutely no interest in within the procedurally generated base clients,
and a very 2016-era swagger codegen generated codebase, which is extremely
repetitive (we'll get around to updating this eventually I am sure).

Naturally, if you only even use a few simple API calls then you could
reverse engineer those from the developer tools in your browser, implement
them in Requests and worry little about maintenance.
You could even just crib them from NiPy (it is OSS, after all) and skip
having the dependency. If you are only needing Bearer token access,
likewise this is fairly simple and you don't need a whole library to handle
it for you.

I think generally libraries like NiPy are a good default if they allow you
to reliably skip developing in-house tools in layers of your stack where
you aren't trying to innovate. Get things done sooner, do less maintenance
over time, and hopefully with fewer obscure errors. If it isn't quite
right, fork it (and hopefully PR back). If it looks dodgy, don't use it.
Non-trivial code that you touch rarely requires time to refamiliarise
yourself; a library of high-level abstractions like NiPy allows that
process to be quick as you don't need to dive back in and, for example,
relearn exactly how NiFi implements version control integrations two years
after you first figured it out.
However if you are sure you will only need a few simple commands, using
Requests over familiarising yourself with a new library is time saved.
Obviously the challenge is knowing in advance your likely depth of usage.

For me, NiPy grew out of a desire to reliably demo NiFi at short notice
when it came into Hortonworks, which resulted in a need to treat a command
& control service with design & deploy principals and tie it into on-demand
and CICD deployment frameworks.
I mostly get questions about the higher-level methods in the canvas,
versioning, security & utils modules in CICD pipelines - I suspect if you
are running some multi-NiFi pipeline you will quickly end up reimplementing
a lot of NiPy in Requests and would be better off with the library.

Either way, thanks for taking the time to try the library - it is always
heartwarming to see the community adoption.

D

On Fri, Aug 5, 2022 at 11:10 PM Cannon Palms <cp...@influxdata.com> wrote:

> It buys you what any API wrapper buys you: slightly increased developer
> velocity at the expense of occasional bugs that aren't in your code. Are
> you using python? Is your context that of a business that is attempting to
> trade engineering spend for bottom-line beats (or bottom-line improvements
> of any sort)? What is the project's timeline?
>
> I don't think anyone can answer this for you, but if you'd like my
> personal opinion: use whatever delivers the most value in the least amount
> of time.
>
>
> On Fri, Aug 5, 2022, 1:19 PM David Early via users <us...@nifi.apache.org>
> wrote:
>
>> Hi all,
>>
>> Looking for opinion or experience on this one.
>>
>> I developed some basic nipyapi functions that got sidelined for a while.
>> Currently being picked up again but getting some questions about why we
>> should use nipyapi vs doing "direct" api queries without actually using the
>> api.
>>
>> It has been over a year since I did the initial work....i would like to
>> continue with the library vs direct, but was hoping for any experience or
>> opinions on why it would be good/bad to use nipyapi.  What does it buy me?
>>
>> Thanks for any feedback, just looking to give some reasons to the young
>> guys coming in behind me.
>>
>> Dave
>>
>> --
>> David Early, Ph.D.
>> david.early@grokstream.com
>> 720-470-7460 Cell
>>
>>

Re: NiPyAPI vs direct Python requests

[Cannon Palms <cp...@influxdata.com>]

It buys you what any API wrapper buys you: slightly increased developer
velocity at the expense of occasional bugs that aren't in your code. Are
you using python? Is your context that of a business that is attempting to
trade engineering spend for bottom-line beats (or bottom-line improvements
of any sort)? What is the project's timeline?

I don't think anyone can answer this for you, but if you'd like my personal
opinion: use whatever delivers the most value in the least amount of time.


On Fri, Aug 5, 2022, 1:19 PM David Early via users <us...@nifi.apache.org>
wrote:

> Hi all,
>
> Looking for opinion or experience on this one.
>
> I developed some basic nipyapi functions that got sidelined for a while.
> Currently being picked up again but getting some questions about why we
> should use nipyapi vs doing "direct" api queries without actually using the
> api.
>
> It has been over a year since I did the initial work....i would like to
> continue with the library vs direct, but was hoping for any experience or
> opinions on why it would be good/bad to use nipyapi.  What does it buy me?
>
> Thanks for any feedback, just looking to give some reasons to the young
> guys coming in behind me.
>
> Dave
>
> --
> David Early, Ph.D.
> david.early@grokstream.com
> 720-470-7460 Cell
>
>