[GitHub] [arrow-ballista] andygrove opened a new issue, #20: Ballista: Fix hacks around concurrency=2 to force hash-partitioned joins

[GitBox <gi...@apache.org>]

andygrove opened a new issue, #20:
URL: https://github.com/apache/arrow-ballista/issues/20

   **Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
   
   By default, DataFusion uses hash-partitioned joins if concurrency > 1 which led to me adding this hacky code in a couple of places in Ballista.
   
   ```rust
   let config = ExecutionConfig::new().with_concurrency(2); // TODO: this is hack to enable partitioned joins
   let mut ctx = ExecutionContext::with_config(config);
   ```
   
   **Describe the solution you'd like**
   I'm actually not sure what the solution should be, but I would like to be able to tell the context to use hash-partitioned joins, separately from specifying concurrency.
   
   **Describe alternatives you've considered**
   None
   
   **Additional context**
   This code is running in the scheduler, not in the executor where the query actually executes. The scheduler concurrency should not impact how the query is planned.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org