[jira] [Comment Edited] (CALCITE-3582) Make RelStructuredTypeFlattener configurable in PlannerImpl

["Rui Wang (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CALCITE-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16992807#comment-16992807 ] 

Rui Wang edited comment on CALCITE-3582 at 12/10/19 6:21 PM:
-------------------------------------------------------------

[~jinxing6042@126.com] I remember that it was more like a historic decision. Quotes from the relevant thread "Flattening was introduced mainly because the original engine used flat column-oriented storage".


[~zabetak] Indeed the use case is to maintain the final structure in the top SELECT. In most cases struct does not matter in the middle of the plan. Sounds like it maybe just because flattener fails to reconstruct the struct. I will double check it.


was (Author: amaliujia):
[~jinxing6042@126.com] I remember that it was more like a historic decision. Quotes from the relevant thread "Flattening was introduced mainly because the original engine used flat column-oriented storage".


[~zabetak] Indeed the use case is to maintain the final structure in the top SELECT. So sounds like it maybe just because flattener fails to reconstruct the struct. I will double check it.

> Make RelStructuredTypeFlattener configurable in PlannerImpl
> -----------------------------------------------------------
>
>                 Key: CALCITE-3582
>                 URL: https://issues.apache.org/jira/browse/CALCITE-3582
>             Project: Calcite
>          Issue Type: Task
>            Reporter: Rui Wang
>            Assignee: Rui Wang
>            Priority: Major
>
> There is a use case where users want to mix Beam programming model with Beam SQL together to process a dataset. The following is an example of the use case:
> dataset.apply(something user defined)
>             .apply(SELECT row_a, row_b ...)
>             .apply(something user defined)
> As you can see, after the SQL statement is applied, the data structure should be preserved for further processing(e.g. users expect to see there are two rows, than flattened fields of rows)
> So far as RelStructuredTypeFlattener is enabled by default in PlannerImpl, Row/Struct will be flattened. But users are very likely defined an output scheme that include row/struct, as that's what they processed in SQL statement. 
> Make calling RelStructuredTypeFlattener configurable in PlannerImpl to allow disabling it when not needed can be a solution.  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)