[jira] [Created] (IMPALA-6536) CREATE TABLE on S3 takes a very long time

["Alexander Behm (JIRA)" <ji...@apache.org>]

Alexander Behm created IMPALA-6536:
--------------------------------------

             Summary: CREATE TABLE on S3 takes a very long time
                 Key: IMPALA-6536
                 URL: https://issues.apache.org/jira/browse/IMPALA-6536
             Project: IMPALA
          Issue Type: Improvement
          Components: Frontend
    Affects Versions: Impala 2.11.0, Impala 2.10.0, Impala 2.9.0
            Reporter: Alexander Behm
            Assignee: Alexander Behm


*Summary*
Creating a table that points to existing data in S3 can take an excessive amount of time.

*Reason*
If the Hive Metastore is configured with "hive.stats.autogather=true" then Hive lists the files of newly created tables to populate basic statistics like file count and file byte sizes. Unfortunately, this listing operation can take an excessive amount of time particularly on S3.

*Workarounds*
* Add TBLPROPERTIES("DO_NOT_UPDATE_STATS"="true") to your CREATE TABLE
* Or reconfigure the Hive Metastore with "hive.stats.autogather=false"

*Example*
{code}
CREATE EXTERNAL TABLE tpch_lineitem_s3 (
  l_orderkey BIGINT,
  l_partkey BIGINT,
  l_suppkey BIGINT,
  l_linenumber BIGINT,
  l_quantity DECIMAL(12,2),
  l_extendedprice DECIMAL(12,2),
  l_discount DECIMAL(12,2),
  l_tax DECIMAL(12,2),
  l_returnflag STRING,
  l_linestatus STRING,
  l_shipdate STRING,
  l_commitdate STRING,
  l_receiptdate STRING,
  l_shipinstruct STRING,
  l_shipmode STRING,
  l_comment STRING
)
STORED AS PARQUET
LOCATION "s3a://some_location/my_existing_data"
TBLPROPERTIES("DO_NOT_UPDATE_STATS"="true"); <--- Add this as a workaround
{code}

Impala should probably add the workaround automatically in CREATE TABLE since Impala does not even use those basic statistics populated by the Hive Metastore.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)