NiFi - How to Wait until all Inserts into Table are done, then Create/Analyze Indexes on the Table afterwards

["matthewmichala@northwesternmutual.com" <ma...@northwesternmutual.com>]

NiFi Developers,

I tried reaching out to Stack Overflow Apache-Nifi Tag, but am getting no replies - so normally I wouldn't ask this email list this type of question, but am not sure where else to turn?

So to keep simple, my 1st Process Group (PG) Truncates then drops indexes in table. Then that output port routes to input port in 2nd process group the does the 500K Inserts into the table. After successfully inserting the 500K rows, I want to create the indexes on the table and analyze it via 3rd process group. This is typical Data Warehouse methodology. Can anyone please give advice on how to do this?  I'd like to be able to not even start the 3rd process group, until first 2 complete successfully, and the Insert Count matches the Select Count.

I've tried setting counters, then comparing total inserted count to match initial select count, but it appears that I cannot reference Counters in the Expression Language because this "${InsertCounter}"  syntax always returns nulls??   So maybe instead I should be using the Wait & Notify Processors, so could Wait until all the inserts are done, then Notify the next process (the create & analyze analyze indexes process) to start after all the inserts are done?  Here's example code on that from Stack Overflow:

In the wait processor set the Target Signal Count to ${fragment.count}.

Set the Release Signal Identifier in both the notify and wait processor to ${fragment.identifier}
Thanks for any help anyone can provide!

Thanks & Regards,

Matt Michala
Senior Engineer
Digital Workplace & Corp. Solutions
Northwestern Mutual (T07-150)
414-661-4668



This e-mail and any attachments may contain confidential information of Northwestern Mutual. If you are not the intended recipient of this message, be aware that any disclosure, copying, distribution or use of this e-mail and any attachments is prohibited. If you have received this e-mail in error, please notify Northwestern Mutual immediately by returning it to the sender and delete all copies from your system. Please be advised that communications with {SECURE MESSAGE} in the subject line have been sent using a secure messaging system. Communications that do not have this tag may not be secure and could be observed by a third party. Thank you for your cooperation. 

Re: NiFi - How to Wait until all Inserts into Table are done, then Create/Analyze Indexes on the Table afterwards

[Koji Kawamura <ij...@gmail.com>]

Hi Matt,

I posted my answer to your Stackoverflow question.
https://stackoverflow.com/questions/55483317/nifi-create-indexes-after-inserting-records-into-table/55486259#55486259

Thanks,
Koji

On Wed, Apr 3, 2019 at 8:04 AM matthewmichala@northwesternmutual.com
<ma...@northwesternmutual.com> wrote:
>
> NiFi Developers,
>
> I tried reaching out to Stack Overflow Apache-Nifi Tag, but am getting no replies - so normally I wouldn't ask this email list this type of question, but am not sure where else to turn?
>
> So to keep simple, my 1st Process Group (PG) Truncates then drops indexes in table. Then that output port routes to input port in 2nd process group the does the 500K Inserts into the table. After successfully inserting the 500K rows, I want to create the indexes on the table and analyze it via 3rd process group. This is typical Data Warehouse methodology. Can anyone please give advice on how to do this?  I'd like to be able to not even start the 3rd process group, until first 2 complete successfully, and the Insert Count matches the Select Count.
>
> I've tried setting counters, then comparing total inserted count to match initial select count, but it appears that I cannot reference Counters in the Expression Language because this "${InsertCounter}"  syntax always returns nulls??   So maybe instead I should be using the Wait & Notify Processors, so could Wait until all the inserts are done, then Notify the next process (the create & analyze analyze indexes process) to start after all the inserts are done?  Here's example code on that from Stack Overflow:
>
> In the wait processor set the Target Signal Count to ${fragment.count}.
>
> Set the Release Signal Identifier in both the notify and wait processor to ${fragment.identifier}
> Thanks for any help anyone can provide!
>
> Thanks & Regards,
>
> Matt Michala
> Senior Engineer
> Digital Workplace & Corp. Solutions
> Northwestern Mutual (T07-150)
> 414-661-4668
>
>
>
> This e-mail and any attachments may contain confidential information of Northwestern Mutual. If you are not the intended recipient of this message, be aware that any disclosure, copying, distribution or use of this e-mail and any attachments is prohibited. If you have received this e-mail in error, please notify Northwestern Mutual immediately by returning it to the sender and delete all copies from your system. Please be advised that communications with {SECURE MESSAGE} in the subject line have been sent using a secure messaging system. Communications that do not have this tag may not be secure and could be observed by a third party. Thank you for your cooperation.
>
>