You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by "Peter Wicks (pwicks)" <pw...@micron.com> on 2018/01/24 02:22:02 UTC
RE: [EXT] Re: Embedded Nifi
Vincent,
Embedded NiFi still has a long ways to go to be really useful, in my opinion; and I don't know if anyone is actively working on those improvements.
The PR Andy mentioned simply allows you to startup NiFi inside your process instead of running it directly from a startup script, but that doesn't mean you magically have access to all of NiFi's internals (someone can correct me if I'm wrong). If you want to actually interact with your new NiFi instance you will still need to use the REST API.
Thanks,
Peter
-----Original Message-----
From: Vincent Russell [mailto:vincent.russell@gmail.com]
Sent: Tuesday, January 23, 2018 03:07
To: dev@nifi.apache.org
Subject: [EXT] Re: Embedded Nifi
Thanks Andy,
This looks a great first step. It would be nice to have a builder pattern
and the ability to download the "executable" from a nexus or the local filesystem like embedded elastic, but perhaps that might be better in some third party library.
https://github.com/allegro/embedded-elasticsearch
-Vincent
On Mon, Jan 22, 2018 at 1:37 PM, Andy LoPresto <al...@apache.org> wrote:
> Vincent,
>
> I plan to merge this pull request [1] for NIFI-4424 [2] by Peter
> Horvath today. Does this satisfy your requirements?
>
> [1] https://github.com/apache/nifi/pull/2251
> [2] https://issues.apache.org/jira/browse/NIFI-4424
>
> Andy LoPresto
> alopresto@apache.org
> *alopresto.apache@gmail.com <al...@gmail.com>* PGP
> Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
> On Jan 21, 2018, at 7:35 AM, Vincent Russell
> <vi...@gmail.com>
> wrote:
>
> Devs,
>
> Does an embedded nifi exist that would start a nifi with a provided
> workflow? I am aware of the Mock framework, but I am looking for
> something for integration tests.
>
> Thanks,
> Vincent
>
>
>
Re: [EXT] Re: Embedded Nifi
Posted by Daniel Chaffelson <ch...@gmail.com>.
Charlie,
This was merged last week to provide an easy method to get around the host
header issue with running 1.5.0 docker.
Essentially it allows you to specify the host:port via environment
variables.
https://github.com/apache/nifi/pull/2439
On Tue, Feb 6, 2018 at 12:12 AM Charlie Meyer <
charlie.meyer@civitaslearning.com> wrote:
> Hi Vincent,
>
> We do this for our test cases using
> https://github.com/palantir/docker-compose-rule. It works fairly well, but
> is a bit heavy weight to get tests started up locally. Also, with 1.5.0,
> the host header issue surfaces here and makes running in docker a pain.
>
> Hope that helps a bit!
>
> On Mon, Feb 5, 2018 at 6:07 PM, Vincent Russell <vincent.russell@gmail.com
> >
> wrote:
>
> > Perhaps Dan, but integrating docker into an automated test might prove a
> > little challenging.
> >
> > I'll check it out though.
> >
> > Thanks,
> >
> > On Wed, Jan 31, 2018 at 3:00 AM, Daniel Chaffelson <chaffelson@gmail.com
> >
> > wrote:
> >
> > > Vincent,
> > > I do something along these lines in Python to test NiFi automation
> work.
> > > NiPyApi creates the requisite Docker container(s) for the test suite,
> > > procedurally creates the NiFi/Registry objects (Process Groups,
> > Processors,
> > > Buckets, Flows, etc), recursively tests itself, then tears itself down.
> > >
> > > I know this is not 'embedded', but could automated and containerised be
> > > close enough to meet your needs?
> > >
> > > Thanks,
> > > Dan
> > >
> > > On Wed, Jan 24, 2018 at 2:22 AM Peter Wicks (pwicks) <
> pwicks@micron.com>
> > > wrote:
> > >
> > > > Vincent,
> > > >
> > > > Embedded NiFi still has a long ways to go to be really useful, in my
> > > > opinion; and I don't know if anyone is actively working on those
> > > > improvements.
> > > >
> > > > The PR Andy mentioned simply allows you to startup NiFi inside your
> > > > process instead of running it directly from a startup script, but
> that
> > > > doesn't mean you magically have access to all of NiFi's internals
> > > (someone
> > > > can correct me if I'm wrong). If you want to actually interact with
> > your
> > > > new NiFi instance you will still need to use the REST API.
> > > >
> > > > Thanks,
> > > > Peter
> > > >
> > > > -----Original Message-----
> > > > From: Vincent Russell [mailto:vincent.russell@gmail.com]
> > > > Sent: Tuesday, January 23, 2018 03:07
> > > > To: dev@nifi.apache.org
> > > > Subject: [EXT] Re: Embedded Nifi
> > > >
> > > > Thanks Andy,
> > > >
> > > > This looks a great first step. It would be nice to have a builder
> > > pattern
> > > > and the ability to download the "executable" from a nexus or the
> local
> > > > filesystem like embedded elastic, but perhaps that might be better in
> > > some
> > > > third party library.
> > > >
> > > > https://github.com/allegro/embedded-elasticsearch
> > > >
> > > > -Vincent
> > > >
> > > > On Mon, Jan 22, 2018 at 1:37 PM, Andy LoPresto <alopresto@apache.org
> >
> > > > wrote:
> > > >
> > > > > Vincent,
> > > > >
> > > > > I plan to merge this pull request [1] for NIFI-4424 [2] by Peter
> > > > > Horvath today. Does this satisfy your requirements?
> > > > >
> > > > > [1] https://github.com/apache/nifi/pull/2251
> > > > > [2] https://issues.apache.org/jira/browse/NIFI-4424
> > > > >
> > > > > Andy LoPresto
> > > > > alopresto@apache.org
> > > > > *alopresto.apache@gmail.com <al...@gmail.com>* PGP
> > > > > Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> > > > >
> > > > > On Jan 21, 2018, at 7:35 AM, Vincent Russell
> > > > > <vi...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > Devs,
> > > > >
> > > > > Does an embedded nifi exist that would start a nifi with a provided
> > > > > workflow? I am aware of the Mock framework, but I am looking for
> > > > > something for integration tests.
> > > > >
> > > > > Thanks,
> > > > > Vincent
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: [EXT] Re: Embedded Nifi
Posted by Charlie Meyer <ch...@civitaslearning.com>.
Hi Vincent,
We do this for our test cases using
https://github.com/palantir/docker-compose-rule. It works fairly well, but
is a bit heavy weight to get tests started up locally. Also, with 1.5.0,
the host header issue surfaces here and makes running in docker a pain.
Hope that helps a bit!
On Mon, Feb 5, 2018 at 6:07 PM, Vincent Russell <vi...@gmail.com>
wrote:
> Perhaps Dan, but integrating docker into an automated test might prove a
> little challenging.
>
> I'll check it out though.
>
> Thanks,
>
> On Wed, Jan 31, 2018 at 3:00 AM, Daniel Chaffelson <ch...@gmail.com>
> wrote:
>
> > Vincent,
> > I do something along these lines in Python to test NiFi automation work.
> > NiPyApi creates the requisite Docker container(s) for the test suite,
> > procedurally creates the NiFi/Registry objects (Process Groups,
> Processors,
> > Buckets, Flows, etc), recursively tests itself, then tears itself down.
> >
> > I know this is not 'embedded', but could automated and containerised be
> > close enough to meet your needs?
> >
> > Thanks,
> > Dan
> >
> > On Wed, Jan 24, 2018 at 2:22 AM Peter Wicks (pwicks) <pw...@micron.com>
> > wrote:
> >
> > > Vincent,
> > >
> > > Embedded NiFi still has a long ways to go to be really useful, in my
> > > opinion; and I don't know if anyone is actively working on those
> > > improvements.
> > >
> > > The PR Andy mentioned simply allows you to startup NiFi inside your
> > > process instead of running it directly from a startup script, but that
> > > doesn't mean you magically have access to all of NiFi's internals
> > (someone
> > > can correct me if I'm wrong). If you want to actually interact with
> your
> > > new NiFi instance you will still need to use the REST API.
> > >
> > > Thanks,
> > > Peter
> > >
> > > -----Original Message-----
> > > From: Vincent Russell [mailto:vincent.russell@gmail.com]
> > > Sent: Tuesday, January 23, 2018 03:07
> > > To: dev@nifi.apache.org
> > > Subject: [EXT] Re: Embedded Nifi
> > >
> > > Thanks Andy,
> > >
> > > This looks a great first step. It would be nice to have a builder
> > pattern
> > > and the ability to download the "executable" from a nexus or the local
> > > filesystem like embedded elastic, but perhaps that might be better in
> > some
> > > third party library.
> > >
> > > https://github.com/allegro/embedded-elasticsearch
> > >
> > > -Vincent
> > >
> > > On Mon, Jan 22, 2018 at 1:37 PM, Andy LoPresto <al...@apache.org>
> > > wrote:
> > >
> > > > Vincent,
> > > >
> > > > I plan to merge this pull request [1] for NIFI-4424 [2] by Peter
> > > > Horvath today. Does this satisfy your requirements?
> > > >
> > > > [1] https://github.com/apache/nifi/pull/2251
> > > > [2] https://issues.apache.org/jira/browse/NIFI-4424
> > > >
> > > > Andy LoPresto
> > > > alopresto@apache.org
> > > > *alopresto.apache@gmail.com <al...@gmail.com>* PGP
> > > > Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> > > >
> > > > On Jan 21, 2018, at 7:35 AM, Vincent Russell
> > > > <vi...@gmail.com>
> > > > wrote:
> > > >
> > > > Devs,
> > > >
> > > > Does an embedded nifi exist that would start a nifi with a provided
> > > > workflow? I am aware of the Mock framework, but I am looking for
> > > > something for integration tests.
> > > >
> > > > Thanks,
> > > > Vincent
> > > >
> > > >
> > > >
> > >
> >
>
Re: [EXT] Re: Embedded Nifi
Posted by Vincent Russell <vi...@gmail.com>.
Perhaps Dan, but integrating docker into an automated test might prove a
little challenging.
I'll check it out though.
Thanks,
On Wed, Jan 31, 2018 at 3:00 AM, Daniel Chaffelson <ch...@gmail.com>
wrote:
> Vincent,
> I do something along these lines in Python to test NiFi automation work.
> NiPyApi creates the requisite Docker container(s) for the test suite,
> procedurally creates the NiFi/Registry objects (Process Groups, Processors,
> Buckets, Flows, etc), recursively tests itself, then tears itself down.
>
> I know this is not 'embedded', but could automated and containerised be
> close enough to meet your needs?
>
> Thanks,
> Dan
>
> On Wed, Jan 24, 2018 at 2:22 AM Peter Wicks (pwicks) <pw...@micron.com>
> wrote:
>
> > Vincent,
> >
> > Embedded NiFi still has a long ways to go to be really useful, in my
> > opinion; and I don't know if anyone is actively working on those
> > improvements.
> >
> > The PR Andy mentioned simply allows you to startup NiFi inside your
> > process instead of running it directly from a startup script, but that
> > doesn't mean you magically have access to all of NiFi's internals
> (someone
> > can correct me if I'm wrong). If you want to actually interact with your
> > new NiFi instance you will still need to use the REST API.
> >
> > Thanks,
> > Peter
> >
> > -----Original Message-----
> > From: Vincent Russell [mailto:vincent.russell@gmail.com]
> > Sent: Tuesday, January 23, 2018 03:07
> > To: dev@nifi.apache.org
> > Subject: [EXT] Re: Embedded Nifi
> >
> > Thanks Andy,
> >
> > This looks a great first step. It would be nice to have a builder
> pattern
> > and the ability to download the "executable" from a nexus or the local
> > filesystem like embedded elastic, but perhaps that might be better in
> some
> > third party library.
> >
> > https://github.com/allegro/embedded-elasticsearch
> >
> > -Vincent
> >
> > On Mon, Jan 22, 2018 at 1:37 PM, Andy LoPresto <al...@apache.org>
> > wrote:
> >
> > > Vincent,
> > >
> > > I plan to merge this pull request [1] for NIFI-4424 [2] by Peter
> > > Horvath today. Does this satisfy your requirements?
> > >
> > > [1] https://github.com/apache/nifi/pull/2251
> > > [2] https://issues.apache.org/jira/browse/NIFI-4424
> > >
> > > Andy LoPresto
> > > alopresto@apache.org
> > > *alopresto.apache@gmail.com <al...@gmail.com>* PGP
> > > Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> > >
> > > On Jan 21, 2018, at 7:35 AM, Vincent Russell
> > > <vi...@gmail.com>
> > > wrote:
> > >
> > > Devs,
> > >
> > > Does an embedded nifi exist that would start a nifi with a provided
> > > workflow? I am aware of the Mock framework, but I am looking for
> > > something for integration tests.
> > >
> > > Thanks,
> > > Vincent
> > >
> > >
> > >
> >
>
Re: [EXT] Re: Embedded Nifi
Posted by Daniel Chaffelson <ch...@gmail.com>.
Vincent,
I do something along these lines in Python to test NiFi automation work.
NiPyApi creates the requisite Docker container(s) for the test suite,
procedurally creates the NiFi/Registry objects (Process Groups, Processors,
Buckets, Flows, etc), recursively tests itself, then tears itself down.
I know this is not 'embedded', but could automated and containerised be
close enough to meet your needs?
Thanks,
Dan
On Wed, Jan 24, 2018 at 2:22 AM Peter Wicks (pwicks) <pw...@micron.com>
wrote:
> Vincent,
>
> Embedded NiFi still has a long ways to go to be really useful, in my
> opinion; and I don't know if anyone is actively working on those
> improvements.
>
> The PR Andy mentioned simply allows you to startup NiFi inside your
> process instead of running it directly from a startup script, but that
> doesn't mean you magically have access to all of NiFi's internals (someone
> can correct me if I'm wrong). If you want to actually interact with your
> new NiFi instance you will still need to use the REST API.
>
> Thanks,
> Peter
>
> -----Original Message-----
> From: Vincent Russell [mailto:vincent.russell@gmail.com]
> Sent: Tuesday, January 23, 2018 03:07
> To: dev@nifi.apache.org
> Subject: [EXT] Re: Embedded Nifi
>
> Thanks Andy,
>
> This looks a great first step. It would be nice to have a builder pattern
> and the ability to download the "executable" from a nexus or the local
> filesystem like embedded elastic, but perhaps that might be better in some
> third party library.
>
> https://github.com/allegro/embedded-elasticsearch
>
> -Vincent
>
> On Mon, Jan 22, 2018 at 1:37 PM, Andy LoPresto <al...@apache.org>
> wrote:
>
> > Vincent,
> >
> > I plan to merge this pull request [1] for NIFI-4424 [2] by Peter
> > Horvath today. Does this satisfy your requirements?
> >
> > [1] https://github.com/apache/nifi/pull/2251
> > [2] https://issues.apache.org/jira/browse/NIFI-4424
> >
> > Andy LoPresto
> > alopresto@apache.org
> > *alopresto.apache@gmail.com <al...@gmail.com>* PGP
> > Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> >
> > On Jan 21, 2018, at 7:35 AM, Vincent Russell
> > <vi...@gmail.com>
> > wrote:
> >
> > Devs,
> >
> > Does an embedded nifi exist that would start a nifi with a provided
> > workflow? I am aware of the Mock framework, but I am looking for
> > something for integration tests.
> >
> > Thanks,
> > Vincent
> >
> >
> >
>
***UNCHECKED*** Re: [EXT] Re: Embedded Nifi
Posted by Vincent Russell <vi...@gmail.com>.
Thank you Peter.
At this point I only wish to make sure that an existing flow that is
already created in a flow.xml file works within an integration test. I am
hoping that this can be achieved with starting Nifi programmatically this
way.
Thanks,
On Tue, Jan 23, 2018 at 9:22 PM, Peter Wicks (pwicks) <pw...@micron.com>
wrote:
> Vincent,
>
> Embedded NiFi still has a long ways to go to be really useful, in my
> opinion; and I don't know if anyone is actively working on those
> improvements.
>
> The PR Andy mentioned simply allows you to startup NiFi inside your
> process instead of running it directly from a startup script, but that
> doesn't mean you magically have access to all of NiFi's internals (someone
> can correct me if I'm wrong). If you want to actually interact with your
> new NiFi instance you will still need to use the REST API.
>
> Thanks,
> Peter
>
> -----Original Message-----
> From: Vincent Russell [mailto:vincent.russell@gmail.com]
> Sent: Tuesday, January 23, 2018 03:07
> To: dev@nifi.apache.org
> Subject: [EXT] Re: Embedded Nifi
>
> Thanks Andy,
>
> This looks a great first step. It would be nice to have a builder pattern
> and the ability to download the "executable" from a nexus or the local
> filesystem like embedded elastic, but perhaps that might be better in some
> third party library.
>
> https://github.com/allegro/embedded-elasticsearch
>
> -Vincent
>
> On Mon, Jan 22, 2018 at 1:37 PM, Andy LoPresto <al...@apache.org>
> wrote:
>
> > Vincent,
> >
> > I plan to merge this pull request [1] for NIFI-4424 [2] by Peter
> > Horvath today. Does this satisfy your requirements?
> >
> > [1] https://github.com/apache/nifi/pull/2251
> > [2] https://issues.apache.org/jira/browse/NIFI-4424
> >
> > Andy LoPresto
> > alopresto@apache.org
> > *alopresto.apache@gmail.com <al...@gmail.com>* PGP
> > Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> >
> > On Jan 21, 2018, at 7:35 AM, Vincent Russell
> > <vi...@gmail.com>
> > wrote:
> >
> > Devs,
> >
> > Does an embedded nifi exist that would start a nifi with a provided
> > workflow? I am aware of the Mock framework, but I am looking for
> > something for integration tests.
> >
> > Thanks,
> > Vincent
> >
> >
> >
>