[GitHub] [druid] clintropolis commented on pull request #13653: allow using nested column indexer for schema discovery

[GitBox <gi...@apache.org>]

clintropolis commented on PR #13653:
URL: https://github.com/apache/druid/pull/13653#issuecomment-1378003784

   > Is there a way to do this without a new configuration setting? It would be nice if this could just be the default without having anyone think about configuration.
   
   I think my intention is for this to be temporary while I make improvements to the feature to ensure it is production ready. After the storage optimizations are done, I think we can switch this to be the default, or could just remove the setting entirely once comfortable that it fully replaces the existing string only discovery behavior.
   
   >I wasn't super sure reading the description, but for SQL-based ingestion, is Select * the only command needed?
   
   There is still a bit of work to be done to support this with SQL based ingestion which will be in a follow-up PR, but that is my goal - that we only have to do a select *.
   
   > Finally, does this work on auto-type detection apply to evolving schemas?
   
   I do think this is where this feature will truly find its best use, particularly with streams that have some variety and/or change over time. This PR enable all columns from some input to be discovered and ingested as the correct type, which is certainly part of the schema evolution story. However, this PR doesn't make any changes in cases when the schema varies between segments, so it will use druid's existing best effort mechanisms to execute the query as asked, potentially casting values whenever necessary and the like. I think there is potentially some room for further improvement in this area as well, such as adding support for coercing values to a certain type to match existing schemas, etc, but that will be future enhancements.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org