NiFi 1.1.1 & 1.2.0 with PostgreSql 9.5

[Raymond Rogers <ra...@gmail.com>]

We are having a performance issue where we are inserting a 2000+ rows into
a PostreSql database.

The database server is basically idle, disk writes 200-300 KB per second.
We are using the postgresql-42.1.1.jar JDBC driver.
The NiFi server has over 10,000 flow files waiting in queue for the PutSQL
processor.
It appears that the PutSQL has completely stopped processing flow-files.
The two servers are sitting in the same rack and network traffic is less
the 300 KB.
I can't find where anything is logging any errors.

Can anyone make any suggestions?

Re: NiFi 1.1.1 & 1.2.0 with PostgreSql 9.5

[Koji Kawamura <ij...@gmail.com>]

Hello Raymond,

Does the PutSQL has some number (probably 1) shown on its top right
corner in NiFi UI? If so, the execution thread is stuck at some state,
then looking at a thread dump may help to investigate where it stops.
Could you share "$NIFI_HOME/bin/nifi.sh dump" result? The output will
be written into logs/nifi-bootstrap.log.

Or if PutSQL has downstream flow such as 'success' or 'failure' and if
those relationships have 10,000 FlowFiles queued in it, then 'Back
Pressure' is enabled for those. In that case, NiFi won't schedule
PutSQL until those queued FlowFiles move forward or removed and number
of queued FlowFile and total size become less than configured
back-pressure threshold, 10,000 FlowFiles or 1GB in size by default.

Hope this helps.

Thanks,
Koji

On Sat, Jun 10, 2017 at 5:36 AM, Raymond Rogers
<ra...@gmail.com> wrote:
> We are having a performance issue where we are inserting a 2000+ rows into a
> PostreSql database.
>
> The database server is basically idle, disk writes 200-300 KB per second.
> We are using the postgresql-42.1.1.jar JDBC driver.
> The NiFi server has over 10,000 flow files waiting in queue for the PutSQL
> processor.
> It appears that the PutSQL has completely stopped processing flow-files.
> The two servers are sitting in the same rack and network traffic is less the
> 300 KB.
> I can't find where anything is logging any errors.
>
> Can anyone make any suggestions?