You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "demetribu (via GitHub)" <gi...@apache.org> on 2024/12/06 20:02:19 UTC

Re: [I] Create memory table with target partitions [datafusion]

demetribu commented on issue #12905:
URL: https://github.com/apache/datafusion/issues/12905#issuecomment-2524060879

   I tried to implement this in a naive way by just adding the following code:
   
   ```
   let target_partitions = self.state().config_options().execution.target_partitions;
   
   let physical = DataFrame::new(self.state(), input);
   let batches: Vec<_> = physical
       .repartition(Partitioning::RoundRobinBatch(target_partitions))?
       .collect_partitioned().await?;
   ```
   
   to https://github.com/apache/datafusion/blob/fc703238b1d7794bd132a7fb6b97cad9ba4c7446/datafusion/core/src/execution/context/mod.rs#L795
   
   But I found that doing Repartition on top of a simple Projection gets overridden by physical optimization: https://github.com/apache/datafusion/blob/fc703238b1d7794bd132a7fb6b97cad9ba4c7446/datafusion/core/src/physical_optimizer/enforce_distribution.rs#L975
   
   PS: When I skip remove_dist_changing_operators, the behavior still looks weird. The number of partitions matches target_partitions, but the data isn’t distributed properly. There’s one partition with all the data and others are empty.
   
   
   @jayzhan211 Would appreciate any ideas. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org