Next gen metastore

[Edward Capriolo <ed...@gmail.com>]

While not active in the development community as much I have been using
hive in the field as well as spark and impala for some time.

My ancmecdotal opinion is that the current metastore needs a significant re
write to deal with "next generation" workloads. By next generation I
actually mean last generation.

Currently cloudera's impala advice is . No more then 1k rows in table. And
tables with lots of partitions are problematic.

Thus really "wont get it done" at the "new" web scale. Hive server can have
memory problems with tables with 2k columns and 5k partitions.

It feels like design ideas like "surely we can fetch all the columns of a
table in one go' dont make sense universally.

Amazon has glue which can scale to amazon scale. Hive metastore cant even
really scale to q single organization. So what are the next steps,  I dont
think its simple as "move it to nosql" I think it has to be reworked from
ground up.


-- 
Sorry this was sent from mobile. Will do less grammar and spell check than
usual.

Re: Next gen metastore

[Peter Vary <pv...@cloudera.com>]

Hi Edward,

We are currently working on integrating Apache Iceberg tables to Hive.
In the latest released Hive 4.0.0-alpha-1 it is possible to create tables backed by Iceberg tables, and those could be queried by Hive. You can define the partitioning using Iceberg specification like this:

CREATE EXTERNAL TABLE ice_table (id bigint, year_field date) PARTITIONED BY SPEC (year(year_field)) STORED BY ICEBERG;

These partitions will be handled by Iceberg, and in the HMS no partitions are stored.
This will remove a serious part of the load from the HMS and will allow higher number of partitions for a single table.

Also there is an ongoing work at the Impala project to read/write these Hive-Iceberg tables.

https://blog.cloudera.com/introducing-apache-iceberg-in-cloudera-data-platform/ <https://blog.cloudera.com/introducing-apache-iceberg-in-cloudera-data-platform/>

I hope this helps,
Peter


> On 2022. Apr 2., at 23:33, Edward Capriolo <ed...@gmail.com> wrote:
> 
> While not active in the development community as much I have been using hive in the field as well as spark and impala for some time.
> 
> My ancmecdotal opinion is that the current metastore needs a significant re write to deal with "next generation" workloads. By next generation I actually mean last generation. 
> 
> Currently cloudera's impala advice is . No more then 1k rows in table. And tables with lots of partitions are problematic.
> 
> Thus really "wont get it done" at the "new" web scale. Hive server can have memory problems with tables with 2k columns and 5k partitions. 
> 
> It feels like design ideas like "surely we can fetch all the columns of a table in one go' dont make sense universally.
> 
> Amazon has glue which can scale to amazon scale. Hive metastore cant even really scale to q single organization. So what are the next steps,  I dont think its simple as "move it to nosql" I think it has to be reworked from ground up.
> 
> 
> -- 
> Sorry this was sent from mobile. Will do less grammar and spell check than usual.