You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by "Zenin, Alexei" <al...@scotiabank.com> on 2019/03/06 16:47:39 UTC
Automated Testing of Nifi Flow by injecting data into Processors
Hi,
Is there any way in Nifi to test out your whole flow without having the source systems online and running. I need this kind of capability to run automated integration tests which tests if the flow behaves as expected. None of the tools or libraries I have seen accomplish this and the Nifi API does not seem to allow to inject data directly into a processor (Nifi's TestRunner seems to be able to create FlowFiles but can we inject mock data to a processor through the Nifi REST API?). I think this would be a critical feature to make this system more production ready and abide by best practices for CICD.
Regards,
Alexei
To unsubscribe from receiving further commercial electronic messages from The Bank of Nova Scotia in Canada, please click here: https://unsubscribe.scotiabank.com?entid=BNS&buid=SBNK
To unsubscribe from receiving further commercial electronic messages from certain other senders set out in the attached list, please click here: https://www.unsubscribe.gwm.scotiabank.com?page=gwm
This email may contain confidential information the use of which by an unintended recipient is unauthorized. This email may also contain important disclosure information for the records of the intended recipient(s). For details please click here: http://www.scotiabank.com/email_disclaimer/email_english.html
Pour vous désabonner et cesser de recevoir des messages électroniques commerciaux de La Banque de Nouvelle-Écosse, veuillez cliquer ici : https://desabonnement.banquescotia.com?entid=BNS&buid=SBNK
Pour vous désabonner et cesser de recevoir des messages électroniques commerciaux de certains autres expéditeurs figurant dans la liste ci-jointe, veuillez cliquer ici : https://www.unsubscribe.gwm.scotiabank.com?page=gwm&lang=fr
Cette transmission peut contenir de l'information confidentielle et son utilisation par toute personne autre que la personne à laquelle cette transmission est destinée est interdite. Le présent courriel peut aussi contenir des renseignements importants pour les dossiers du ou des destinataires prévus. Pour plus de détails, veuillez vous diriger vers: http://www.scotiabank.com/email_disclaimer/email_francais.html
Para dejar de recibir mensajes electrónicos comerciales de The Bank of Nova Scotia en Canadá, haga clic aquí: https://cancelarsuscripcion.scotiabank.com?entid=BNS&buid=SBNK
Para dejar de recibir mensajes electrónicos comerciales de otras entidades incluidas en la lista adjunta, haga clic aquí: https://www.unsubscribe.gwm.scotiabank.com?page=gwm&lang=es
Este correo electrónico puede contener información confidencial cuyo uso por parte de personas distintas de los destinatarios del mismo está prohibido. El mensaje puede tener también datos importantes para los archivos del(de los) destinatario(s) a quien(es) está dirigido. Para más detalles, por favor dirigirse a: http://www.scotiabank.com/email_disclaimer/email_espanol.html
Re: Automated Testing of Nifi Flow by injecting data into Processors
Posted by Andrew Psaltis <ps...@gmail.com>.
Hi Alexei,
If you're talking about testing your flow with simulated data as if it's
deployed, basically not unit tests. I've seen this done in several
different ways:
1. Using the GenerateFlowFile processor and adding in custom text. At
first, that may seem obvious, however, if you think about it in that custom
text field you can add in a block of JSON data that then in a subsequent
processor you can manipulate to populate various values and then convert it
to the format you need. From here you can have the data continue on to the
rest of your flow, or if you want to test the reading of data from a type
of source, perhaps File then you can write the data out to test location
and have your regular flow read it in. There are various ways you can take
this to make it as elaborate as necessary.
2. Build a custom processor to do the data generation. Here is an example
for TimeSeries data:
https://medium.com/hashmapinc/its-here-an-apache-nifi-simulator-for-generating-random-and-realistic-time-series-data-d7e463aa5e78
Does this help with idea on how to do it?
On Wed, Mar 6, 2019 at 11:50 AM Zenin, Alexei <al...@scotiabank.com>
wrote:
> Hi,
>
> Is there any way in Nifi to test out your whole flow without having the
> source systems online and running. I need this kind of capability to run
> automated integration tests which tests if the flow behaves as expected.
> None of the tools or libraries I have seen accomplish this and the Nifi API
> does not seem to allow to inject data directly into a processor (Nifi's
> TestRunner seems to be able to create FlowFiles but can we inject mock data
> to a processor through the Nifi REST API?). I think this would be a
> critical feature to make this system more production ready and abide by
> best practices for CICD.
>
> Regards,
> Alexei
> To unsubscribe from receiving further commercial electronic messages from
> The Bank of Nova Scotia in Canada, please click here:
> https://unsubscribe.scotiabank.com?entid=BNS&buid=SBNK
>
> To unsubscribe from receiving further commercial electronic messages from
> certain other senders set out in the attached list, please click here:
> https://www.unsubscribe.gwm.scotiabank.com?page=gwm
>
> This email may contain confidential information the use of which by an
> unintended recipient is unauthorized. This email may also contain important
> disclosure information for the records of the intended recipient(s). For
> details please click here:
> http://www.scotiabank.com/email_disclaimer/email_english.html
>
> Pour vous désabonner et cesser de recevoir des messages électroniques
> commerciaux de La Banque de Nouvelle-Écosse, veuillez cliquer ici :
> https://desabonnement.banquescotia.com?entid=BNS&buid=SBNK
>
> Pour vous désabonner et cesser de recevoir des messages électroniques
> commerciaux de certains autres expéditeurs figurant dans la liste
> ci-jointe, veuillez cliquer ici :
> https://www.unsubscribe.gwm.scotiabank.com?page=gwm&lang=fr
>
> Cette transmission peut contenir de l'information confidentielle et son
> utilisation par toute personne autre que la personne à laquelle cette
> transmission est destinée est interdite. Le présent courriel peut aussi
> contenir des renseignements importants pour les dossiers du ou des
> destinataires prévus. Pour plus de détails, veuillez vous diriger vers:
> http://www.scotiabank.com/email_disclaimer/email_francais.html
>
> Para dejar de recibir mensajes electrónicos comerciales de The Bank of
> Nova Scotia en Canadá, haga clic aquí:
> https://cancelarsuscripcion.scotiabank.com?entid=BNS&buid=SBNK
>
> Para dejar de recibir mensajes electrónicos comerciales de otras entidades
> incluidas en la lista adjunta, haga clic aquí:
> https://www.unsubscribe.gwm.scotiabank.com?page=gwm&lang=es
>
> Este correo electrónico puede contener información confidencial cuyo uso
> por parte de personas distintas de los destinatarios del mismo está
> prohibido. El mensaje puede tener también datos importantes para los
> archivos del(de los) destinatario(s) a quien(es) está dirigido. Para más
> detalles, por favor dirigirse a:
> http://www.scotiabank.com/email_disclaimer/email_espanol.html
Re: Automated Testing of Nifi Flow by injecting data into Processors
Posted by Bronislav Jitnikov <sh...@gmail.com>.
I think, you can use https://pypi.org/project/nipyapi/ to create test
scenario for integration testing of complete flow without access data in
source system. I work on template for similar task, but only in start of my
way.
As an idea Create Python script that
1. deploy flow from Bucket to testing enviroment
2. load test data to quey after source system access processor (source)
3. Run flow except source step by step
4. Check result in queue after each processor.
May be create some module that can smiplify this task.
ср, 6 мар. 2019 г. в 19:50, Zenin, Alexei <al...@scotiabank.com>:
> Hi,
>
> Is there any way in Nifi to test out your whole flow without having the
> source systems online and running. I need this kind of capability to run
> automated integration tests which tests if the flow behaves as expected.
> None of the tools or libraries I have seen accomplish this and the Nifi API
> does not seem to allow to inject data directly into a processor (Nifi's
> TestRunner seems to be able to create FlowFiles but can we inject mock data
> to a processor through the Nifi REST API?). I think this would be a
> critical feature to make this system more production ready and abide by
> best practices for CICD.
>
> Regards,
> Alexei
> To unsubscribe from receiving further commercial electronic messages from
> The Bank of Nova Scotia in Canada, please click here:
> https://unsubscribe.scotiabank.com?entid=BNS&buid=SBNK
>
> To unsubscribe from receiving further commercial electronic messages from
> certain other senders set out in the attached list, please click here:
> https://www.unsubscribe.gwm.scotiabank.com?page=gwm
>
> This email may contain confidential information the use of which by an
> unintended recipient is unauthorized. This email may also contain important
> disclosure information for the records of the intended recipient(s). For
> details please click here:
> http://www.scotiabank.com/email_disclaimer/email_english.html
>
> Pour vous désabonner et cesser de recevoir des messages électroniques
> commerciaux de La Banque de Nouvelle-Écosse, veuillez cliquer ici :
> https://desabonnement.banquescotia.com?entid=BNS&buid=SBNK
>
> Pour vous désabonner et cesser de recevoir des messages électroniques
> commerciaux de certains autres expéditeurs figurant dans la liste
> ci-jointe, veuillez cliquer ici :
> https://www.unsubscribe.gwm.scotiabank.com?page=gwm&lang=fr
>
> Cette transmission peut contenir de l'information confidentielle et son
> utilisation par toute personne autre que la personne à laquelle cette
> transmission est destinée est interdite. Le présent courriel peut aussi
> contenir des renseignements importants pour les dossiers du ou des
> destinataires prévus. Pour plus de détails, veuillez vous diriger vers:
> http://www.scotiabank.com/email_disclaimer/email_francais.html
>
> Para dejar de recibir mensajes electrónicos comerciales de The Bank of
> Nova Scotia en Canadá, haga clic aquí:
> https://cancelarsuscripcion.scotiabank.com?entid=BNS&buid=SBNK
>
> Para dejar de recibir mensajes electrónicos comerciales de otras entidades
> incluidas en la lista adjunta, haga clic aquí:
> https://www.unsubscribe.gwm.scotiabank.com?page=gwm&lang=es
>
> Este correo electrónico puede contener información confidencial cuyo uso
> por parte de personas distintas de los destinatarios del mismo está
> prohibido. El mensaje puede tener también datos importantes para los
> archivos del(de los) destinatario(s) a quien(es) está dirigido. Para más
> detalles, por favor dirigirse a:
> http://www.scotiabank.com/email_disclaimer/email_espanol.html