[GitHub] [arrow-datafusion] alamb commented on issue #7573: Should parallel collecting statistics like infer schema?

["alamb (via GitHub)" <gi...@apache.org>]

alamb commented on issue #7573:
URL: https://github.com/apache/arrow-datafusion/issues/7573#issuecomment-1724504600

   Hi @hengfeiyang  -- this sound like a great idea to me
   
   Rather than hard coding the concurrency, perhaps you can add a config parameter, perhaps like `ExecutionOptions::meta_fetch_concurrency`, following the model of https://docs.rs/datafusion/latest/datafusion/config/struct.ExecutionOptions.html#structfield.planning_concurrency
   
   The rationale for a configuration setting is that the optimal value will likely depend on the network configuration of the system, so there is no good constant that would work in all cases. 
   
   Perhaps it can default to 10 or 32 ?
   
   cc @Ted-Jiang  who I think was working on some other settings to cache statistics for multiple queries in the same session. See https://github.com/apache/arrow-datafusion/pull/7570
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org