You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Deepanshu Lodhi <De...@inventia.in> on 2024/10/23 10:49:38 UTC

Delay in Data Insertion Using PutDatabaseRecord Processor with CRON Scheduling

Hello Everyone,

I hope you're doing well!

I'm encountering a delay with the PutDatabaseRecord processor when inserting data at a scheduled time using CRON-driven scheduling. While the QueryDatabaseTable processor fetches the data on time, the insertion through PutDatabaseRecord is significantly delayed. However, when using Time-driven scheduling, the processor works as expected without delays.
I suspect the issue might be related to the DBCPConnectionPool configuration. Could anyone suggest the best possible configuration or other optimizations to ensure timely data insertion?

Thank you in advance for your support!

Best regards,
Deepanshu

Re: Delay in Data Insertion Using PutDatabaseRecord Processor with CRON Scheduling

Posted by Deepanshu Lodhi <De...@inventia.in>.
Hii Mike,

Thanks for the update...
Yes, it's working fine now 😇

- Deepanshu

________________________________
From: Michael Moser <mo...@gmail.com>
Sent: Wednesday, October 23, 2024 10:16:38 pm
To: users@nifi.apache.org <us...@nifi.apache.org>
Subject: Re: Delay in Data Insertion Using PutDatabaseRecord Processor with CRON Scheduling

Hi Deepanshu,

Assuming your flow is a QueryDatabaseTable followed later in the flow by a PutDatabaseRecord, then it is recommended to use CRON-driven scheduling for QueryDatabaseTable and Timer-driven scheduling for PutDatabaeRecord.

If you set both processors to the same CRON schedule, then PutDatabaseRecord will trigger and have nothing to do, because QueryDatabaseTable has triggered at the exact same time but takes a bit of time to complete its job and create a flowfile in NiFi.

Regards,
-- Mike


On Wed, Oct 23, 2024 at 6:50 AM Deepanshu Lodhi <De...@inventia.in>> wrote:
Hello Everyone,

I hope you're doing well!

I'm encountering a delay with the PutDatabaseRecord processor when inserting data at a scheduled time using CRON-driven scheduling. While the QueryDatabaseTable processor fetches the data on time, the insertion through PutDatabaseRecord is significantly delayed. However, when using Time-driven scheduling, the processor works as expected without delays.
I suspect the issue might be related to the DBCPConnectionPool configuration. Could anyone suggest the best possible configuration or other optimizations to ensure timely data insertion?

Thank you in advance for your support!

Best regards,
Deepanshu

Re: Delay in Data Insertion Using PutDatabaseRecord Processor with CRON Scheduling

Posted by Michael Moser <mo...@gmail.com>.
Hi Deepanshu,

Assuming your flow is a QueryDatabaseTable followed later in the flow by a
PutDatabaseRecord, then it is recommended to use CRON-driven scheduling for
QueryDatabaseTable and Timer-driven scheduling for PutDatabaeRecord.

If you set both processors to the same CRON schedule, then
PutDatabaseRecord will trigger and have nothing to do, because
QueryDatabaseTable has triggered at the exact same time but takes a bit of
time to complete its job and create a flowfile in NiFi.

Regards,
-- Mike


On Wed, Oct 23, 2024 at 6:50 AM Deepanshu Lodhi <De...@inventia.in>
wrote:

> Hello Everyone,
>
>
>
> *I hope you're doing well!*
>
>
>
> I'm encountering a delay with the *PutDatabaseRecord* processor when
> inserting data at a scheduled time using CRON-driven scheduling. While the
> *QueryDatabaseTable* processor fetches the data on time, the insertion
> through *PutDatabaseRecord* is significantly delayed. However, when using
> Time-driven scheduling, the processor works as expected without delays.
>
> I suspect the issue might be related to the *DBCPConnectionPool*
> configuration. Could anyone suggest the best possible configuration or
> other optimizations to ensure timely data insertion?
>
>
>
> Thank you in advance for your support!
>
>
>
> Best regards,
> Deepanshu
>
>
>