You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "Remi Dettai (Jira)" <ji...@apache.org> on 2020/10/23 08:43:00 UTC
[jira] [Updated] (ARROW-10368) [Rust][DataFusion] Refactor scan
nodes to allow extensions
[ https://issues.apache.org/jira/browse/ARROW-10368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Remi Dettai updated ARROW-10368:
--------------------------------
Summary: [Rust][DataFusion] Refactor scan nodes to allow extensions (was: [Rust][DataFusion] Make InMemoryScan work on iterators of RecordBatch)
> [Rust][DataFusion] Refactor scan nodes to allow extensions
> ----------------------------------------------------------
>
> Key: ARROW-10368
> URL: https://issues.apache.org/jira/browse/ARROW-10368
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Rust, Rust - DataFusion
> Reporter: Remi Dettai
> Priority: Major
>
> Currently, InMemoryScan takes a Vec<Vec<RecordBatch>> as data.
> - the outer Vec separates the partitions
> - the inner Vec contains all the RecordBatch for one partition
> The inner Vec is then converted into an iterator when the LogicalPlan is turned into a PhysicalPlan.
> I suggest that InMemoryScan should take Vec<Iter<RecordBatch>>. This would make it possible to plug custom Scan implementations into datafusion without the need to read them entirely into memory. It would still work pretty seamlessly with Vec<Vec<RecordBatch>> that would just need a to be converted with data.map(|x| x.iter()) first.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)