You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Zoltán Borók-Nagy (JIRA)" <ji...@apache.org> on 2019/03/19 10:59:00 UTC
[jira] [Assigned] (IMPALA-3722) Avro codegen can be unnecessarily
disabled
[ https://issues.apache.org/jira/browse/IMPALA-3722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zoltán Borók-Nagy reassigned IMPALA-3722:
-----------------------------------------
Assignee: (was: Zoltán Borók-Nagy)
> Avro codegen can be unnecessarily disabled
> ------------------------------------------
>
> Key: IMPALA-3722
> URL: https://issues.apache.org/jira/browse/IMPALA-3722
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Affects Versions: Impala 2.6.0
> Reporter: Skye Wanderman-Milne
> Priority: Major
> Labels: avro, codegen, ramp-up
>
> We use avro_schema_equal() from the Avro C library to determine if a file's schema matches the table schema, and if they don't match we disable codegen for that file (https://github.com/cloudera/Impala/blob/cdh5-trunk/be/src/exec/hdfs-avro-scanner.cc#L153). However, avro_schema_equal() is unnecessarily restrictive, because it compares the records' names and namespaces, which don't have to be the same to enable codegen. There are probably other checks we don't need as well, e.g. default values. We should write our own schema comparison function that is tailored to what must match for codegen specifically.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org