You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Vibhath Ileperuma <vi...@gmail.com> on 2021/05/19 11:31:50 UTC
Getting a notification once a request is finished
Hi all,
I'm developing a NIFI flow to extract data in a database and write into
files.This flow starts with a ListenHttp processor. A python script sends a
list of table names required to be extracted to the port ListenHttp
processor is listening to. NIFI flow has been developed to iterate through
that list and extract the data in given tables.
I would like to know whether there is a way to get a notification to the
python script once all the tables are extracted.
If it is not possible, is it possible to log a message informing all the
tables have been extracted.
Thank You.
Best Regards,
Vibhath.
Re: Getting a notification once a request is finished
Posted by Joseph Gresock <jg...@cloudera.com>.
Hi Vibhath,
In your case, I'd consider using HandleHttpRequest
<https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.HandleHttpRequest/index.html>
/ HandleHttpResponse
<https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.HandleHttpResponse/index.html>
instead
of ListenHTTP. This will allow you to send an HTTP call to NiFi and wait
for the processing to complete, before sending the HTTP response back to
your script via HandleHttpResponse. This will be simple if your flow
generates only one flowfile, since you can simply route this to
HandleHttpResponse. If you have multiple flowfiles as output, you may have
to use MergeContent after all the data has been extracted, so that you can
send a single response. Or even better, if there's some way for you to
know in the flow which is the last result, you could route only that
flowfile to HandleHttpResponse.
Joe Gresock
On Wed, May 19, 2021 at 7:32 AM Vibhath Ileperuma <
vibhatharunapriya@gmail.com> wrote:
> Hi all,
>
> I'm developing a NIFI flow to extract data in a database and write into
> files.This flow starts with a ListenHttp processor. A python script sends a
> list of table names required to be extracted to the port ListenHttp
> processor is listening to. NIFI flow has been developed to iterate through
> that list and extract the data in given tables.
>
> I would like to know whether there is a way to get a notification to the
> python script once all the tables are extracted.
> If it is not possible, is it possible to log a message informing all the
> tables have been extracted.
>
> Thank You.
> Best Regards,
> Vibhath.
>
Re: Getting a notification once a request is finished
Posted by Otto Fowler <ot...@gmail.com>.
Maybe
SCRIPT:
get list of tables
send list of tables
run http server ( built into python )
listen for tables complete
when all complete ????
profit
NIFI
HandleRequest
Iterate tables and write to file
after each write, HttpInvoke the table name to the script http server
> On May 19, 2021, at 07:31, Vibhath Ileperuma <vi...@gmail.com> wrote:
>
> Hi all,
>
> I'm developing a NIFI flow to extract data in a database and write into files.This flow starts with a ListenHttp processor. A python script sends a list of table names required to be extracted to the port ListenHttp processor is listening to. NIFI flow has been developed to iterate through that list and extract the data in given tables.
>
> I would like to know whether there is a way to get a notification to the python script once all the tables are extracted.
> If it is not possible, is it possible to log a message informing all the tables have been extracted.
>
> Thank You.
> Best Regards,
> Vibhath.
Re: Getting a notification once a request is finished
Posted by Vibhath Ileperuma <vi...@gmail.com>.
Hi Asmath,
Could you please explain a little bit more. Are you writing the status of
the extraction in another database and read that database from your script.
Thanks,
Vibhath
On Wed, May 19, 2021 at 5:07 PM KhajaAsmath Mohammed <
mdkhajaasmath@gmail.com> wrote:
> Hi Vibhath,
>
> After a lot of research, I did this way.
>
> Let’s say, you have 50 tables that needs to be processed. I will have
> status as loading, success or fail foe each file processing. If anyone of
> the flow file is failed, then process is considered as failed . I will keep
> checking the status using db query for these tables and Mark as complete
> based on db query status
>
> Thanks,
> Asmath
>
> > On May 19, 2021, at 6:32 AM, Vibhath Ileperuma <
> vibhatharunapriya@gmail.com> wrote:
> >
> >
> > Hi all,
> >
> > I'm developing a NIFI flow to extract data in a database and write into
> files.This flow starts with a ListenHttp processor. A python script sends a
> list of table names required to be extracted to the port ListenHttp
> processor is listening to. NIFI flow has been developed to iterate through
> that list and extract the data in given tables.
> >
> > I would like to know whether there is a way to get a notification to the
> python script once all the tables are extracted.
> > If it is not possible, is it possible to log a message informing all the
> tables have been extracted.
> >
> > Thank You.
> > Best Regards,
> > Vibhath.
>
Re: Getting a notification once a request is finished
Posted by KhajaAsmath Mohammed <md...@gmail.com>.
Hi Vibhath,
After a lot of research, I did this way.
Let’s say, you have 50 tables that needs to be processed. I will have status as loading, success or fail foe each file processing. If anyone of the flow file is failed, then process is considered as failed . I will keep checking the status using db query for these tables and Mark as complete based on db query status
Thanks,
Asmath
> On May 19, 2021, at 6:32 AM, Vibhath Ileperuma <vi...@gmail.com> wrote:
>
>
> Hi all,
>
> I'm developing a NIFI flow to extract data in a database and write into files.This flow starts with a ListenHttp processor. A python script sends a list of table names required to be extracted to the port ListenHttp processor is listening to. NIFI flow has been developed to iterate through that list and extract the data in given tables.
>
> I would like to know whether there is a way to get a notification to the python script once all the tables are extracted.
> If it is not possible, is it possible to log a message informing all the tables have been extracted.
>
> Thank You.
> Best Regards,
> Vibhath.