You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Daniel (Jira)" <ji...@apache.org> on 2020/11/03 12:04:00 UTC
[jira] [Updated] (SPARK-33326) Partition Parameters are not updated
even after ANALYZE TABLE command
[ https://issues.apache.org/jira/browse/SPARK-33326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel updated SPARK-33326:
---------------------------
Description:
Here are the reproduction steps:
{code:java}
scala> spark.sql("CREATE TABLE t (a string,b string) PARTITIONED BY (p string) STORED AS PARQUET")
Hive Session ID = d44e21ee-2d5c-48ab-91bf-26cb25775486
res0: org.apache.spark.sql.DataFrame = []
scala> spark.sql("INSERT INTO t PARTITION(p='p1') VALUES ('aaa', 'bbb')")
res1: org.apache.spark.sql.DataFrame = []
scala> spark.sql("INSERT INTO t PARTITION(p='p1') VALUES ('ccc', 'ddd')")
res2: org.apache.spark.sql.DataFrame = []
scala> spark.sql("ANALYZE TABLE t PARTITION(p='p1') COMPUTE STATISTICS")
res3: org.apache.spark.sql.DataFrame = []
scala> spark.sql("DESCRIBE FORMATTED t PARTITION (p='p1')").show(50, false)
...
|Partition Parameters |{rawDataSize=0, numFiles=1, numFilesErasureCoded=0, transient_lastDdlTime=1604404640, totalSize=532, COLUMN_STATS_ACCURATE={"BASIC_STATS":"true","COLUMN_STATS":{"a":"true","b":"true"}}, numRows=0}| |
...
|Partition Statistics |1064 bytes, 2 rows | |
...
{code}
My expectation would be that the Partition Parameters should be updated after ANALYZE TABLE.
was:
Here are the reproduction steps:
{code:java}
scala> spark.sql("CREATE TABLE t (a string,b string) PARTITIONED BY (p string) STORED AS PARQUET")
Hive Session ID = d44e21ee-2d5c-48ab-91bf-26cb25775486
res0: org.apache.spark.sql.DataFrame = []
scala> spark.sql("INSERT INTO t PARTITION(p='p1') VALUES ('aaa', 'bbb')")
res1: org.apache.spark.sql.DataFrame = []
scala> spark.sql("INSERT INTO t PARTITION(p='p1') VALUES ('ccc', 'ddd')")
res2: org.apache.spark.sql.DataFrame = []
scala> spark.sql("ANALYZE TABLE t PARTITION(p='p1') COMPUTE STATISTICS")
res3: org.apache.spark.sql.DataFrame = []
scala> spark.sql("DESCRIBE FORMATTED t PARTITION (p='p1')").show(50, false)
...
|Partition Parameters |{rawDataSize=0, numFiles=1, numFilesErasureCoded=0, transient_lastDdlTime=1604404640, totalSize=532, COLUMN_STATS_ACCURATE={"BASIC_STATS":"true","COLUMN_STATS":{"a":"true","b":"true"}}, numRows=0}| |
...
|Partition Statistics |1064 bytes, 2 rows | |
...
{code}
My expectation would be that the Partition Parameters should be updated after ANALYZE TABLE.
> Partition Parameters are not updated even after ANALYZE TABLE command
> ---------------------------------------------------------------------
>
> Key: SPARK-33326
> URL: https://issues.apache.org/jira/browse/SPARK-33326
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.0.1
> Reporter: Daniel
> Priority: Major
>
> Here are the reproduction steps:
> {code:java}
> scala> spark.sql("CREATE TABLE t (a string,b string) PARTITIONED BY (p string) STORED AS PARQUET")
> Hive Session ID = d44e21ee-2d5c-48ab-91bf-26cb25775486
> res0: org.apache.spark.sql.DataFrame = []
> scala> spark.sql("INSERT INTO t PARTITION(p='p1') VALUES ('aaa', 'bbb')")
> res1: org.apache.spark.sql.DataFrame = []
> scala> spark.sql("INSERT INTO t PARTITION(p='p1') VALUES ('ccc', 'ddd')")
> res2: org.apache.spark.sql.DataFrame = []
> scala> spark.sql("ANALYZE TABLE t PARTITION(p='p1') COMPUTE STATISTICS")
> res3: org.apache.spark.sql.DataFrame = []
> scala> spark.sql("DESCRIBE FORMATTED t PARTITION (p='p1')").show(50, false)
> ...
> |Partition Parameters |{rawDataSize=0, numFiles=1, numFilesErasureCoded=0, transient_lastDdlTime=1604404640, totalSize=532, COLUMN_STATS_ACCURATE={"BASIC_STATS":"true","COLUMN_STATS":{"a":"true","b":"true"}}, numRows=0}| |
> ...
> |Partition Statistics |1064 bytes, 2 rows | |
> ...
> {code}
> My expectation would be that the Partition Parameters should be updated after ANALYZE TABLE.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org