[jira] [Commented] (DRILL-4505) Can't group by or sort across files with different schema

["Terence Namusonge Sifuna (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/DRILL-4505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16795157#comment-16795157 ] 

Terence Namusonge Sifuna commented on DRILL-4505:
-------------------------------------------------

Run into this issue on ver 1.15 with json storage format as well. I run `alter session set `exec.enable_union_type` = true`

and it made no difference.

`select created,id,data from dfs.tmp.alerts a order by a.created;`

{color:#FF0000}Error: UNSUPPORTED_OPERATION ERROR: Unexpected schema change - likely code error.{color}

The column data is a nested json which has changed slightly along the way

 

> Can't group by or sort across files with different schema
> ---------------------------------------------------------
>
>                 Key: DRILL-4505
>                 URL: https://issues.apache.org/jira/browse/DRILL-4505
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Storage - Parquet
>    Affects Versions: 1.5.0
>         Environment: Java 1.8
>            Reporter: Tobias
>            Priority: Major
>         Attachments: Screenshot from 2019-03-18 19-12-32.png
>
>
> We are currently trying out the support for querying across parquet files with different schemas.
> Simple selects work well but when we wan't to do sort or group by Drill returns "UNSUPPORTED_OPERATION ERROR: Sort doesn't currently support sorts with changing schemas Fragment 0:0 [Error Id: ff490670-64c1-4fb8-990e-a02aa44ac010 on zookeeper-1:31010]"
> This is despite not even including the new columns in the query.
> Expected result would be to treat the non existing columns in certain files as either null or default value and allow them to be grouped and sorted
> Example
> SELECT APPLICATION_ID ,dir0 AS year_ FROM dfs.`/PRO/UTC/1` WHERE dir2 >='2016-01-01' AND dir2<'2016-04-02' work with changing schema
> but SELECT max(APPLICATION_ID ),dir0 AS year_ FROM dfs.`/PRO/UTC/1` WHERE dir2 >='2016-01-01' AND dir2<'2016-04-02'  group by dir0 does not work
> For us this hampers any possibility to have an evolving schema with moderatly complex queries



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)