You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Vincent Russell <vi...@gmail.com> on 2018/02/06 00:07:45 UTC
Re: [EXT] Re: Embedded Nifi
Perhaps Dan, but integrating docker into an automated test might prove a
little challenging.
I'll check it out though.
Thanks,
On Wed, Jan 31, 2018 at 3:00 AM, Daniel Chaffelson <ch...@gmail.com>
wrote:
> Vincent,
> I do something along these lines in Python to test NiFi automation work.
> NiPyApi creates the requisite Docker container(s) for the test suite,
> procedurally creates the NiFi/Registry objects (Process Groups, Processors,
> Buckets, Flows, etc), recursively tests itself, then tears itself down.
>
> I know this is not 'embedded', but could automated and containerised be
> close enough to meet your needs?
>
> Thanks,
> Dan
>
> On Wed, Jan 24, 2018 at 2:22 AM Peter Wicks (pwicks) <pw...@micron.com>
> wrote:
>
> > Vincent,
> >
> > Embedded NiFi still has a long ways to go to be really useful, in my
> > opinion; and I don't know if anyone is actively working on those
> > improvements.
> >
> > The PR Andy mentioned simply allows you to startup NiFi inside your
> > process instead of running it directly from a startup script, but that
> > doesn't mean you magically have access to all of NiFi's internals
> (someone
> > can correct me if I'm wrong). If you want to actually interact with your
> > new NiFi instance you will still need to use the REST API.
> >
> > Thanks,
> > Peter
> >
> > -----Original Message-----
> > From: Vincent Russell [mailto:vincent.russell@gmail.com]
> > Sent: Tuesday, January 23, 2018 03:07
> > To: dev@nifi.apache.org
> > Subject: [EXT] Re: Embedded Nifi
> >
> > Thanks Andy,
> >
> > This looks a great first step. It would be nice to have a builder
> pattern
> > and the ability to download the "executable" from a nexus or the local
> > filesystem like embedded elastic, but perhaps that might be better in
> some
> > third party library.
> >
> > https://github.com/allegro/embedded-elasticsearch
> >
> > -Vincent
> >
> > On Mon, Jan 22, 2018 at 1:37 PM, Andy LoPresto <al...@apache.org>
> > wrote:
> >
> > > Vincent,
> > >
> > > I plan to merge this pull request [1] for NIFI-4424 [2] by Peter
> > > Horvath today. Does this satisfy your requirements?
> > >
> > > [1] https://github.com/apache/nifi/pull/2251
> > > [2] https://issues.apache.org/jira/browse/NIFI-4424
> > >
> > > Andy LoPresto
> > > alopresto@apache.org
> > > *alopresto.apache@gmail.com <al...@gmail.com>* PGP
> > > Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> > >
> > > On Jan 21, 2018, at 7:35 AM, Vincent Russell
> > > <vi...@gmail.com>
> > > wrote:
> > >
> > > Devs,
> > >
> > > Does an embedded nifi exist that would start a nifi with a provided
> > > workflow? I am aware of the Mock framework, but I am looking for
> > > something for integration tests.
> > >
> > > Thanks,
> > > Vincent
> > >
> > >
> > >
> >
>
Re: [EXT] Re: Embedded Nifi
Posted by Daniel Chaffelson <ch...@gmail.com>.
Charlie,
This was merged last week to provide an easy method to get around the host
header issue with running 1.5.0 docker.
Essentially it allows you to specify the host:port via environment
variables.
https://github.com/apache/nifi/pull/2439
On Tue, Feb 6, 2018 at 12:12 AM Charlie Meyer <
charlie.meyer@civitaslearning.com> wrote:
> Hi Vincent,
>
> We do this for our test cases using
> https://github.com/palantir/docker-compose-rule. It works fairly well, but
> is a bit heavy weight to get tests started up locally. Also, with 1.5.0,
> the host header issue surfaces here and makes running in docker a pain.
>
> Hope that helps a bit!
>
> On Mon, Feb 5, 2018 at 6:07 PM, Vincent Russell <vincent.russell@gmail.com
> >
> wrote:
>
> > Perhaps Dan, but integrating docker into an automated test might prove a
> > little challenging.
> >
> > I'll check it out though.
> >
> > Thanks,
> >
> > On Wed, Jan 31, 2018 at 3:00 AM, Daniel Chaffelson <chaffelson@gmail.com
> >
> > wrote:
> >
> > > Vincent,
> > > I do something along these lines in Python to test NiFi automation
> work.
> > > NiPyApi creates the requisite Docker container(s) for the test suite,
> > > procedurally creates the NiFi/Registry objects (Process Groups,
> > Processors,
> > > Buckets, Flows, etc), recursively tests itself, then tears itself down.
> > >
> > > I know this is not 'embedded', but could automated and containerised be
> > > close enough to meet your needs?
> > >
> > > Thanks,
> > > Dan
> > >
> > > On Wed, Jan 24, 2018 at 2:22 AM Peter Wicks (pwicks) <
> pwicks@micron.com>
> > > wrote:
> > >
> > > > Vincent,
> > > >
> > > > Embedded NiFi still has a long ways to go to be really useful, in my
> > > > opinion; and I don't know if anyone is actively working on those
> > > > improvements.
> > > >
> > > > The PR Andy mentioned simply allows you to startup NiFi inside your
> > > > process instead of running it directly from a startup script, but
> that
> > > > doesn't mean you magically have access to all of NiFi's internals
> > > (someone
> > > > can correct me if I'm wrong). If you want to actually interact with
> > your
> > > > new NiFi instance you will still need to use the REST API.
> > > >
> > > > Thanks,
> > > > Peter
> > > >
> > > > -----Original Message-----
> > > > From: Vincent Russell [mailto:vincent.russell@gmail.com]
> > > > Sent: Tuesday, January 23, 2018 03:07
> > > > To: dev@nifi.apache.org
> > > > Subject: [EXT] Re: Embedded Nifi
> > > >
> > > > Thanks Andy,
> > > >
> > > > This looks a great first step. It would be nice to have a builder
> > > pattern
> > > > and the ability to download the "executable" from a nexus or the
> local
> > > > filesystem like embedded elastic, but perhaps that might be better in
> > > some
> > > > third party library.
> > > >
> > > > https://github.com/allegro/embedded-elasticsearch
> > > >
> > > > -Vincent
> > > >
> > > > On Mon, Jan 22, 2018 at 1:37 PM, Andy LoPresto <alopresto@apache.org
> >
> > > > wrote:
> > > >
> > > > > Vincent,
> > > > >
> > > > > I plan to merge this pull request [1] for NIFI-4424 [2] by Peter
> > > > > Horvath today. Does this satisfy your requirements?
> > > > >
> > > > > [1] https://github.com/apache/nifi/pull/2251
> > > > > [2] https://issues.apache.org/jira/browse/NIFI-4424
> > > > >
> > > > > Andy LoPresto
> > > > > alopresto@apache.org
> > > > > *alopresto.apache@gmail.com <al...@gmail.com>* PGP
> > > > > Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> > > > >
> > > > > On Jan 21, 2018, at 7:35 AM, Vincent Russell
> > > > > <vi...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > Devs,
> > > > >
> > > > > Does an embedded nifi exist that would start a nifi with a provided
> > > > > workflow? I am aware of the Mock framework, but I am looking for
> > > > > something for integration tests.
> > > > >
> > > > > Thanks,
> > > > > Vincent
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: [EXT] Re: Embedded Nifi
Posted by Charlie Meyer <ch...@civitaslearning.com>.
Hi Vincent,
We do this for our test cases using
https://github.com/palantir/docker-compose-rule. It works fairly well, but
is a bit heavy weight to get tests started up locally. Also, with 1.5.0,
the host header issue surfaces here and makes running in docker a pain.
Hope that helps a bit!
On Mon, Feb 5, 2018 at 6:07 PM, Vincent Russell <vi...@gmail.com>
wrote:
> Perhaps Dan, but integrating docker into an automated test might prove a
> little challenging.
>
> I'll check it out though.
>
> Thanks,
>
> On Wed, Jan 31, 2018 at 3:00 AM, Daniel Chaffelson <ch...@gmail.com>
> wrote:
>
> > Vincent,
> > I do something along these lines in Python to test NiFi automation work.
> > NiPyApi creates the requisite Docker container(s) for the test suite,
> > procedurally creates the NiFi/Registry objects (Process Groups,
> Processors,
> > Buckets, Flows, etc), recursively tests itself, then tears itself down.
> >
> > I know this is not 'embedded', but could automated and containerised be
> > close enough to meet your needs?
> >
> > Thanks,
> > Dan
> >
> > On Wed, Jan 24, 2018 at 2:22 AM Peter Wicks (pwicks) <pw...@micron.com>
> > wrote:
> >
> > > Vincent,
> > >
> > > Embedded NiFi still has a long ways to go to be really useful, in my
> > > opinion; and I don't know if anyone is actively working on those
> > > improvements.
> > >
> > > The PR Andy mentioned simply allows you to startup NiFi inside your
> > > process instead of running it directly from a startup script, but that
> > > doesn't mean you magically have access to all of NiFi's internals
> > (someone
> > > can correct me if I'm wrong). If you want to actually interact with
> your
> > > new NiFi instance you will still need to use the REST API.
> > >
> > > Thanks,
> > > Peter
> > >
> > > -----Original Message-----
> > > From: Vincent Russell [mailto:vincent.russell@gmail.com]
> > > Sent: Tuesday, January 23, 2018 03:07
> > > To: dev@nifi.apache.org
> > > Subject: [EXT] Re: Embedded Nifi
> > >
> > > Thanks Andy,
> > >
> > > This looks a great first step. It would be nice to have a builder
> > pattern
> > > and the ability to download the "executable" from a nexus or the local
> > > filesystem like embedded elastic, but perhaps that might be better in
> > some
> > > third party library.
> > >
> > > https://github.com/allegro/embedded-elasticsearch
> > >
> > > -Vincent
> > >
> > > On Mon, Jan 22, 2018 at 1:37 PM, Andy LoPresto <al...@apache.org>
> > > wrote:
> > >
> > > > Vincent,
> > > >
> > > > I plan to merge this pull request [1] for NIFI-4424 [2] by Peter
> > > > Horvath today. Does this satisfy your requirements?
> > > >
> > > > [1] https://github.com/apache/nifi/pull/2251
> > > > [2] https://issues.apache.org/jira/browse/NIFI-4424
> > > >
> > > > Andy LoPresto
> > > > alopresto@apache.org
> > > > *alopresto.apache@gmail.com <al...@gmail.com>* PGP
> > > > Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> > > >
> > > > On Jan 21, 2018, at 7:35 AM, Vincent Russell
> > > > <vi...@gmail.com>
> > > > wrote:
> > > >
> > > > Devs,
> > > >
> > > > Does an embedded nifi exist that would start a nifi with a provided
> > > > workflow? I am aware of the Mock framework, but I am looking for
> > > > something for integration tests.
> > > >
> > > > Thanks,
> > > > Vincent
> > > >
> > > >
> > > >
> > >
> >
>