[jira] [Created] (HIVE-26313) Aggregate all column statistics into a single field

["Alessandro Solimando (Jira)" <ji...@apache.org>]

Alessandro Solimando created HIVE-26313:
-------------------------------------------

             Summary: Aggregate all column statistics into a single field
                 Key: HIVE-26313
                 URL: https://issues.apache.org/jira/browse/HIVE-26313
             Project: Hive
          Issue Type: Improvement
          Components: Standalone Metastore, Statistics
    Affects Versions: 4.0.0-alpha-2
            Reporter: Alessandro Solimando


At the moment, column statistics tables in the metastore schema look like this (it's similar for _PART_COL_STATS_):

{noformat}
CREATE TABLE "APP"."TAB_COL_STATS"(
    "CAT_NAME" VARCHAR(256) NOT NULL,
    "DB_NAME" VARCHAR(128) NOT NULL,
    "TABLE_NAME" VARCHAR(256) NOT NULL,
    "COLUMN_NAME" VARCHAR(767) NOT NULL,
    "COLUMN_TYPE" VARCHAR(128) NOT NULL,
    "LONG_LOW_VALUE" BIGINT,
    "LONG_HIGH_VALUE" BIGINT,
    "DOUBLE_LOW_VALUE" DOUBLE,
    "DOUBLE_HIGH_VALUE" DOUBLE,
    "BIG_DECIMAL_LOW_VALUE" VARCHAR(4000),
    "BIG_DECIMAL_HIGH_VALUE" VARCHAR(4000),
    "NUM_DISTINCTS" BIGINT,
    "NUM_NULLS" BIGINT NOT NULL,
    "AVG_COL_LEN" DOUBLE,
    "MAX_COL_LEN" BIGINT,
    "NUM_TRUES" BIGINT,
    "NUM_FALSES" BIGINT,
    "LAST_ANALYZED" BIGINT,
    "CS_ID" BIGINT NOT NULL,
    "TBL_ID" BIGINT NOT NULL,
    "BIT_VECTOR" BLOB,
    "ENGINE" VARCHAR(128) NOT NULL
);
{noformat}

The idea is to have a single blob named _STATISTICS_ to replace them, as follows:

{noformat}
CREATE TABLE "APP"."TAB_COL_STATS"(
    "CAT_NAME" VARCHAR(256) NOT NULL,
    "DB_NAME" VARCHAR(128) NOT NULL,
    "TABLE_NAME" VARCHAR(256) NOT NULL,
    "COLUMN_NAME" VARCHAR(767) NOT NULL,
    "COLUMN_TYPE" VARCHAR(128) NOT NULL,
    "STATISTICS" BLOB,
    "LAST_ANALYZED" BIGINT,
    "CS_ID" BIGINT NOT NULL,
    "TBL_ID" BIGINT NOT NULL,
    "ENGINE" VARCHAR(128) NOT NULL
);
{noformat}

The _STATISTICS_ column could be the serialization of a Json-encoded string, which will be consumed in a "schema-on-read" fashion.

At first at least the removed column statistics will be encoded in the _STATISTICS_ column, but since each "consumer" will read the portion of the schema it is interested into, multiple engines (see the _ENGINE_ column) can read and write statistics as they deem fit.

Another advantage is that, if we plan to add more statistics in the future, we won't need to change the thrift interface for the metastore again.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)