[jira] [Commented] (NIFI-4004) Refactor RecordReaderFactory and SchemaAccessStrategy to be used without incoming FlowFile

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-4004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16130698#comment-16130698 ] 

ASF GitHub Bot commented on NIFI-4004:
--------------------------------------

Github user markap14 commented on the issue:

    https://github.com/apache/nifi/pull/1877
  
    @ijokarumawak it does look good now. Unfortunately, though, it looks like it needs to be rebased. It looks like this is a rather large PR, though, that touches a lot of different points. And it looks like there are several new processors, controller services, etc. that have been merged to master since the PR was created. I wonder if it may make sense, instead of changing RecordReaderFactory to take `Map<String, String>` variables instead of `FlowFIle flowFile`, I wonder if it makes sense to add both methods to the interface, and then have a default method in the interface, such as:
    
    ```
    default RecordReader createRecordReader(FlowFile flowFile, InputStream in, ComponentLog logger) throws MalformedRecordException, IOException, SchemaNotFoundException {
            return createRecordReader(flowFile == null ? Collections.emptyMap() : flowFile.getAttributes(), in, logger);
        }
    
        RecordReader createRecordReader(Map<String, String> variables, InputStream in, ComponentLog logger) throws MalformedRecordException, IOException, SchemaNotFoundException;
    
    ```
    
    Thoughts?


> Refactor RecordReaderFactory and SchemaAccessStrategy to be used without incoming FlowFile
> ------------------------------------------------------------------------------------------
>
>                 Key: NIFI-4004
>                 URL: https://issues.apache.org/jira/browse/NIFI-4004
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Extensions
>    Affects Versions: 1.2.0
>            Reporter: Koji Kawamura
>            Assignee: Koji Kawamura
>
> Current RecordReaderFactory and SchemaAccessStrategy implementation assumes there's always an incoming FlowFile available, and use it to resolve Record Schema.
> That is fine for components those convert or update incoming FlowFiles, however there are other components those does not have any incoming FlowFiles, for example, ConsumeKafkaRecord_0_10. Typically, ones fetches data from external system do not have incoming FlowFile. And current API doesn't fit well with these as it requires a FlowFile.
> In fact, [ConsumeKafkaRecord creates a temporal FlowFile|https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-0-10-processors/src/main/java/org/apache/nifi/processors/kafka/pubsub/ConsumerLease.java#L426] only to get RecordSchema. This should be avoided as we expect more components start using Record reader mechanism.
> This JIRA proposes refactoring current API to allow accessing RecordReaders without needing an incoming FlowFile.
> Additionally, since there's Schema Access Strategy that requires incoming FlowFile containing attribute values to access schema registry, it'd be useful if we could tell user when such RecordReader is specified that it can't be used.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)