Fwd: Is there any way to make Nifi processor to wait until a number of Flow files arrives.

[Yuri Nikonovich <ut...@gmail.com>]

---------- Forwarded message ----------
From: Yuri Nikonovich <ut...@gmail.com>
Date: 2016-05-26 19:57 GMT+03:00
Subject: Is there any way to make Nifi processor to wait until a number of
Flow files arrives.
To: users@nifi.apache.org


Hi
I'm evaluating Nifi for our ETL process.
I wan't to build the following flow:
Fetch a lot of data from SQL database -> Split into chunks 1000 records
each -> Count error records in each chunk -> Count total number of error
records -> If it exceeds  a threshold  Fail process -> else save each chunk
to the database.

The problem I can't resolve is how to wait until all chunks are validated.
If for example I have 5 validation tasks working concurrently, I need some
kind of barrier to wait until all chunks are processed and only after that
run error count processor because I don't want to save invalid data and
delete it if the threshold is reached.

The other question I have is if there is any possibility to run this
validation processor on multiple nodes in parallel and still have the
possibility to wait until they all are completed.

Thanks,
Yury