[I] Make the configuration for `StreamTable` more generic to support more stream sources [datafusion]

["matthewmturner (via GitHub)" <gi...@apache.org>]

matthewmturner opened a new issue, #10599:
URL: https://github.com/apache/datafusion/issues/10599

   ### Is your feature request related to a problem or challenge?
   
   I am working on a websocket `TableProvider` and initially I went about creating my own `TableProvider` but then after reviewing the `StreamTable` and `StreamConfig` it seems it was made with the potential to be generic at some point.
   
   Ref: "Currently only reading from / appending to a single file in-place is supported, but other stream sources and sinks may be added in future."
   
   So I decided to try making a more generic interface for this.  As a first step I created the new interface and ported the existing implementation to use this (I will have a PR up shortly).  I havent finished cleaning up yet but i think the API is in good enough shape to start getting feedback.
   
   The TLDR of the new interface is that there is a new trait `StreamSource` which provides the `Schema` and `RecordBatchReader` and `RecordBatchWriter`.  Then there are implementations like `FileStreamSource` or `WebsocketStreamSource`.
   
   Assuming this is going in the right direction I can cleanup more, get this merged, and then follow on with an implementation of `WebsocketStreamSource`.
   
   ### Describe the solution you'd like
   
   _No response_
   
   ### Describe alternatives you've considered
   
   _No response_
   
   ### Additional context
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] Make the configuration for `StreamTable` more generic to support more stream sources [datafusion]

["alamb (via GitHub)" <gi...@apache.org>]

alamb closed issue #10599: Make the configuration for `StreamTable` more generic to support more stream sources
URL: https://github.com/apache/datafusion/issues/10599


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org