[GitHub] nifi issue #2003: NIFI-4181 CSVReader and CSVRecordSetWriter can be used by ...

[markap14 <gi...@git.apache.org>]

Github user markap14 commented on the issue:

    https://github.com/apache/nifi/pull/2003
  
    @Wesley-Lawrence I definitely agree that having to build up an entire Avro schema can be a pain -- especially if you're not already familiar with Avro schemas. And I absolutely love that you saw something that can be improved and jumped in to improve the user experience! However, I am a bit concerned with the approach taken here because while it does scratch an itch, it does so only for CSV data and as a result isn't really consistent.
    
    I would like to start a discussion on how we could perhaps handle this in a more generic way. My first thought is to actually provide an alternative implementation of the Schema Registry. Call it ExplicitStringFieldSchemaRegistry for lack of a better name (i'm suggesting calling it this for the sake of the discussion, not necessarily creating an implementation with this name). The idea here, though, is that there would be a new implementation of SchemaRegistry. In this new implementation you would add properties just like AvroSchemaRegistry. The name of the property would be the name of a schema. But the value, instead of an Avro Schema, would be a comma-separated list of column names, and all would be assumed to be Strings. This approach gives us a few different benefits.
    
    First, it allows this same approach to be taken with other data formats (flat JSON, for example). Also, it keeps things consistent in terms of how we access the schema (we can still use the Schema Name property along with the Schema Registry). It also makes the CSV Reader more re-usable because we can use Expression Language to access the name of the schema, so a single CSV Reader can be used by many different processors spread throughout the flow.
    
    What I would love to see at some point is a more powerful SchemaRegistry service that provides a Custom/Advanced UI for actually building schemas. I think this would be extremely powerful and useful and far easier to use. But I promise that I am the last person that you (or any user) wants building a UI :)
    
    Any thoughts here?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---