You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "pengdou1990 (Jira)" <ji...@apache.org> on 2021/07/21 06:58:00 UTC
[jira] [Comment Edited] (IMPALA-10785) when union kudu table and
hdfs table, union passthrough does not take effect
[ https://issues.apache.org/jira/browse/IMPALA-10785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17384646#comment-17384646 ]
pengdou1990 edited comment on IMPALA-10785 at 7/21/21, 6:57 AM:
----------------------------------------------------------------
union output tuple layout depence on it's first child node：
[https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/analysis/SetOperationStmt.java#L599]
each slot's isNullable_ value and null indicator depence on all childnode's corresponding slot：
[https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/analysis/SetOperationStmt.java#L644]
in kudu table tuple, the primary key slot‘s isNullable_ = false
in hdfs table tuple, all slot's isNullable_ = true
in kudu table union with hdfs table situation, tuple memory layout may as hdfs table’s tuple memory layout (string type length is 12 byte， without padding), and isNullable_ of slots in tuple as kudu table’s
neither hdfs table nor kudu table can pass isChildPassthrough check, so pass through does not take effect
was (Author: pengdou1990):
union output tuple layout depence on it's first child node：
[https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/analysis/SetOperationStmt.java#L599]
each slot's isNullable_ value and null indicator depence on all childnode's corresponding slot：
[https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/analysis/SetOperationStmt.java#L644]
in kudu table tuple, the primary key slot‘s isNullable_ = false
in hdfs table tuple, all slot's isNullable_ = true
in kudu table union with hdfs table situation, tuple memory layout may as hdfs table’s tuple memory layout (string type length is 12 byte， without padding), and isNullable_ of slots in tuple as kudu table’s
neither hdfs table nor kudu table can't pass isChildPassthrough check, so pass through does not take effect
> when union kudu table and hdfs table, union passthrough does not take effect
> ----------------------------------------------------------------------------
>
> Key: IMPALA-10785
> URL: https://issues.apache.org/jira/browse/IMPALA-10785
> Project: IMPALA
> Issue Type: Improvement
> Reporter: pengdou1990
> Priority: Major
>
> IMPALA-3586 already supports union passthrough, and brings great performance improvements in union， but there is still some problems when union between hdfs table and kudu table ，several points cause the problem:
> # in kudu scanner node output TupleDescriptor, string slot is 16B,while in hdfs scanner node output TupleDescriptor, string slot is 12B,cause tuple memory layout mismatch
> # in kudu scanner node output TupleDescriptor, string slot is 16B, while in Union output TupleDescriptor, string slot is 12B,cause tuple memory layout mismatch
> # in Kudu Scannode, row key slot is not null, while in hdfs node, not null slot can't get from the metadata, cause tuple memory layout mismatch
> I hive resolved the 1st and 2nd points, how should I do with the 3rd point?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org