[jira] [Commented] (KYLIN-1111) Ignore unsupported hive column types when sync hive table

["Wu Bin (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KYLIN-1111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16701796#comment-16701796 ] 

Wu Bin commented on KYLIN-1111:
-------------------------------

May I have a try?

Based on my understanding, there is already a method to fit some gaps between Hive's columnTypes & Kylin's : 

HiveMetadataExplorer#extractColumnFromMeta:
{code:java}
private ColumnDesc[] extractColumnFromMeta(HiveTableMeta hiveTableMeta) {
    int columnNumber = hiveTableMeta.allColumns.size();
    List<ColumnDesc> columns = new ArrayList<ColumnDesc>(columnNumber);

    for (int i = 0; i < columnNumber; i++) {
        HiveTableMeta.HiveTableColumnMeta field = hiveTableMeta.allColumns.get(i);
        ColumnDesc cdesc = new ColumnDesc();
        cdesc.setName(field.name.toUpperCase(Locale.ROOT));

        // use "double" in kylin for "float"
        if ("float".equalsIgnoreCase(field.dataType)) {
            cdesc.setDatatype("double");
        } else {
            cdesc.setDatatype(field.dataType);
        }

        cdesc.setId(String.valueOf(i + 1));
        cdesc.setComment(field.comment);
        columns.add(cdesc);
    }

    return columns.toArray(new ColumnDesc[columnNumber]);
}
{code}
Can we filter the unsupported columns during the loop?

> Ignore unsupported hive column types when sync hive table
> ---------------------------------------------------------
>
>                 Key: KYLIN-1111
>                 URL: https://issues.apache.org/jira/browse/KYLIN-1111
>             Project: Kylin
>          Issue Type: Improvement
>            Reporter: liyang
>            Priority: Major
>              Labels: newbie
>
> Currently unsupported hive column type like map, array will cause failing to load hive table. Instead, we could just ignore the unsupported column, and still load the table with remaining columns.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)