You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sujith Chacko (JIRA)" <ji...@apache.org> on 2019/04/07 18:20:00 UTC

[jira] [Created] (SPARK-27403) Failed to update the table size automatically even though spark.sql.statistics.size.autoUpdate.enabled is set as rue

Sujith Chacko created SPARK-27403:
-------------------------------------

             Summary: Failed to update the table size automatically even though spark.sql.statistics.size.autoUpdate.enabled is set as rue
                 Key: SPARK-27403
                 URL: https://issues.apache.org/jira/browse/SPARK-27403
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 2.4.1
         Environment: system shall able to update the table stats automatically if user sets  spark.sql.statistics.size.autoUpdate.enabled as true  , Currently this property doesnt have any significance even if it is enabled  or disabled.

Please follow the below steps to reproduce the issue

scala> spark.sql("create table x222 (name string,age int) stored as parquet")

19/04/07 23:41:56 WARN HiveMetaStore: Location: file:/D:/spark-2.4.1-bin-hadoop2.7/bin/spark-warehouse/table1 specified for non-external table:table1
-chgrp: 'HTIPL-23270\None' does not match expected pattern for group
Usage: hadoop fs [generic options] -chgrp [-R] GROUP PATH...
res0: org.apache.spark.sql.DataFrame = []

scala> spark.sql("insert into table1 select 'a',29")
res2: org.apache.spark.sql.DataFrame = []

scala> spark.sql("desc extended table1").show(false)
+----------------------------+--------------------------------------------------------------+-------+
|col_name |data_type |comment|
+----------------------------+--------------------------------------------------------------+-------+
|name |string |null |
|age |int |null |
| | | |
|# Detailed Table Information| | |
|Database |default | |
|Table |table1 | |
|Owner |Administrator | |
|Created Time |Sun Apr 07 23:41:56 IST 2019 | |
|Last Access |Thu Jan 01 05:30:00 IST 1970 | |
|Created By |Spark 2.4.1 | |
|Type |MANAGED | |
|Provider |hive | |
|Table Properties |[transient_lastDdlTime=1554660716] | |
|Location |file:/D:/spark-2.4.1-bin-hadoop2.7/bin/spark-warehouse/table1 | |
|Serde Library |org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe | |
|InputFormat |org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat | |
|OutputFormat |org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat| |
|Storage Properties |[serialization.format=1] | |
|Partition Provider |Catalog | |
+----------------------------+--------------------------------------------------------------+-------+
            Reporter: Sujith Chacko






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org