You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Tim Armstrong (JIRA)" <ji...@apache.org> on 2019/03/18 20:54:00 UTC
[jira] [Commented] (IMPALA-3722) Avro codegen can be unnecessarily
disabled
[ https://issues.apache.org/jira/browse/IMPALA-3722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16795384#comment-16795384 ]
Tim Armstrong commented on IMPALA-3722:
---------------------------------------
[~boroknagyz] are you planning to do this? It probably isn't that critical, we could add it back to the pool in case someone else wants to pick it up.
> Avro codegen can be unnecessarily disabled
> ------------------------------------------
>
> Key: IMPALA-3722
> URL: https://issues.apache.org/jira/browse/IMPALA-3722
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Affects Versions: Impala 2.6.0
> Reporter: Skye Wanderman-Milne
> Assignee: Zoltán Borók-Nagy
> Priority: Major
> Labels: avro, codegen, ramp-up
>
> We use avro_schema_equal() from the Avro C library to determine if a file's schema matches the table schema, and if they don't match we disable codegen for that file (https://github.com/cloudera/Impala/blob/cdh5-trunk/be/src/exec/hdfs-avro-scanner.cc#L153). However, avro_schema_equal() is unnecessarily restrictive, because it compares the records' names and namespaces, which don't have to be the same to enable codegen. There are probably other checks we don't need as well, e.g. default values. We should write our own schema comparison function that is tailored to what must match for codegen specifically.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org