impala, kudu and hive interaction

[Рябков Алексей Николаевич <a....@ntc-vulkan.ru>]

1.       When I use DDL (create table etc) how interact all components? I think impalad send request to impala-catalog , then catalog update hive metastore, then catalog  send request to kudu-master, and kudu master send request to kudu-tablet. Is it correct?



2.       When I use DML (select, update) how interact components? I think impalad query metadata from impala-catalog, then impalad send DML to  other impalad (on node), and impalad(on node) get/set data from/to kudu-tablet. Is it correct?



3.       Why  impala use hive metastore in such case? May be impala-catalog just interact with kudu-master.

Re: impala, kudu and hive interaction

[Matthew Jacobs <mj...@cloudera.com>]

Good questions.

On Thu, Jan 19, 2017 at 1:12 AM, Рябков Алексей Николаевич
<a....@ntc-vulkan.ru> wrote:
> 1.       When I use DDL (create table etc) how interact all components? I
> think impalad send request to impala-catalog , then catalog update hive
> metastore, then catalog  send request to kudu-master, and kudu master send
> request to kudu-tablet. Is it correct?
>

Yes

>
>
> 2.       When I use DML (select, update) how interact components? I think
> impalad query metadata from impala-catalog, then impalad send DML to  other
> impalad (on node), and impalad(on node) get/set data from/to kudu-tablet. Is
> it correct?
>

Yes


>
>
> 3.       Why  impala use hive metastore in such case? May be impala-catalog
> just interact with kudu-master.
>

It's complicated, but ideally we wouldn't use HMS in addition to
Kudu's metadata -- there is duplication. We aim to improve this, but
there are some problems:
* how to have a single set of databases/tables that work across storage layers
* how to create/store metadata that isn't kept by kudu, e.g. table/col
statistics and views
* how would the authorization system work across both?
and more...

>
>
>