You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "Wes McKinney (JIRA)" <ji...@apache.org> on 2019/02/08 05:17:00 UTC
[jira] [Commented] (ARROW-4333) [C++] Sketch out design for kernels
and "query" execution in compute layer
[ https://issues.apache.org/jira/browse/ARROW-4333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16763321#comment-16763321 ]
Wes McKinney commented on ARROW-4333:
-------------------------------------
Yes, I agree with coming up with a plan for all these concerns. I think the volcano batch model is the way to go. This is what Impala and many other systems use quite successfully
> [C++] Sketch out design for kernels and "query" execution in compute layer
> --------------------------------------------------------------------------
>
> Key: ARROW-4333
> URL: https://issues.apache.org/jira/browse/ARROW-4333
> Project: Apache Arrow
> Issue Type: New Feature
> Components: C++
> Reporter: Micah Kornfield
> Priority: Major
> Labels: analytics
> Fix For: 0.13.0
>
>
> It would be good to formalize the design of kernels and the controlling query execution layer (e.g. volcano batch model?) to understand the following:
> Contracts for kernels:
> * Thread safety of kernels?
> * When Kernels should allocate memory vs expect preallocated memory? How to communicate requirements for a kernels memory allocaiton?
> * How to communicate the whether a kernels execution is parallelizable across a ChunkedArray? How to determine if the order to execution across a ChunkedArray is important?
> * How to communicate when it is safe to re-use the same buffers and input and output to the same kernel?
> What does the threading model look like for the higher level of control? Where should synchronization happen?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)