[jira] [Commented] (NIFI-8101) Improvement PutDatabaseRecord for refresh table schema cache

["Matt Burgess (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-8101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17260889#comment-17260889 ] 

Matt Burgess commented on NIFI-8101:
------------------------------------

This looks related to NIFI-7862, except here the use case is that when data comes in that doesn't match the table, the new property can be set to refresh the table in the expectation that the table has been updated and should accept the new data. This feels like a sort of more manual "data drift" capability; NIFI-7862 would add (and possibly delete) columns in the table based on the incoming data, so the table doesn't have to be externally updated to accept the new data.

Having said that, if I understand the intent of this Jira correctly, we wouldn't want fields/columns to be ignored or throw errors when the table has been externally altered. Could we instead just re-fetch the table schema if something doesn't match, and then check again (regardless of whether we ignore, warn, or fail)? Seems like the worst that would happen is the same that would happen if we had stopped and restarted the processor after the table had been altered. If, after updating the schema, the fields don't match the current columns, the processor would do what it is configured to do. That way we don't need a new property, we just use a possible error condition to bring things up-to-date and check again. Or do you see use cases when we'd only want to fetch the table schema if fields didn't match columns, or vice versa? IMO if something doesn't match, we'd just make sure we have the latest information and proceed on that basis. What are your thoughts?


> Improvement  PutDatabaseRecord for refresh table schema cache
> -------------------------------------------------------------
>
>                 Key: NIFI-8101
>                 URL: https://issues.apache.org/jira/browse/NIFI-8101
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Extensions
>            Reporter: ZhangCheng
>            Assignee: ZhangCheng
>            Priority: Major
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> * Sometimes,  the target table has changed and the `PutDatabaseRecord` cached outdated  table schema informatiion.  Maybe we need a new property to tell the `PutDatabaseRecord` to refresh the table schema cache under certain conditions. 
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)