[jira] [Created] (NIFI-12027) PutDatabaseRecord should invalidate schema cache entries

["Joe Witt (Jira)" <ji...@apache.org>]

Joe Witt created NIFI-12027:
-------------------------------

             Summary: PutDatabaseRecord should invalidate schema cache entries
                 Key: NIFI-12027
                 URL: https://issues.apache.org/jira/browse/NIFI-12027
             Project: Apache NiFi
          Issue Type: Improvement
            Reporter: Joe Witt


On nifi main/2.0 line but confirmed also by a user on 1.15.x line....

If you have a flow of records, such as CSV records, feeding into a PutDatabaseRecord and you add columns to the source data things flow normally if you have 'ignore new columns/fields' properties set as those new column values are just ignored.  However, when you add the new columns present in the data to also be in the database you're writing to the values are not sent and end up as nulls (if the database allows them).  If you stop/start the PutDatabaseRecord processor though then the values start getting set.

This is with the default table schema cache value of 100.  If you set that value to 0 then it appears to work fine without restarting.  This suggests that our caching default is likely too simplistic.  We should have some mechanism whereby schema changes in the database are detected and invalidate any schemas we have cached.  Or we do it if we detect a difference in the incoming schema of the data.  But the current behavior at least for defaults leaves the user, I think, having to choose from the default which likely is 'faster' or a slower but more dynamic path.

Perhaps the other question here is how valuable is that schema cache in the first place?



--
This message was sent by Atlassian Jira
(v8.20.10#820010)