[GitHub] [arrow-datafusion] alamb commented on issue #1544: Streaming support for DataFusion

[GitBox <gi...@apache.org>]

alamb commented on issue #1544:
URL: https://github.com/apache/arrow-datafusion/issues/1544#issuecomment-1009919116


   @hntd187  -- I think the proposal sounds good.
   
   I think you were asking about mechanics here:
   
   > we will at least have to develop the API inside datafusion for now it would seem unless we can do it all via contrib. I'm not totally aware of all the package scoping that might prevent it from existing in contrib
   
   Here is what I suggest:
   1. Create your contrib repo
   2. in `Crates.toml` link to a `datafusion` git repo rather than the version published on crates.io . For example:
   
   ```toml
   datafusion = { git = "https://github.com/apache/arrow-datafusion.git", rev="401271377cd84dc1546827f66bda1b242860a6a8", default-features = false, package = "datafusion" }
   ```
   
   Protip: while developing you can change this to point at your own local checkout like this:
   
   ```toml
   [patch.crates-io]
   datafusion = { path='/path/to/you/local/arrow-datafusion/checkout' }
   ```
   
   Then as you work on the streaming implementation, add whatever new DataFusion API / changes you need to a branch (perhaps against your own datafusion fork)
   
   Once you have settled on an API, then open a PR to the main DataFusion repo to incorporate your new API
   
   This will let you iterate on the new DataFusion API without having to worry about the test suite, reviews, delays in merging, etc.
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org