You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (Jira)" <ji...@apache.org> on 2022/07/19 11:46:00 UTC
[jira] [Commented] (SPARK-33326) Partition Parameters are not updated even after ANALYZE TABLE command
[ https://issues.apache.org/jira/browse/SPARK-33326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17568495#comment-17568495 ]
Apache Spark commented on SPARK-33326:
--------------------------------------
User 'RS131419' has created a pull request for this issue:
https://github.com/apache/spark/pull/37227
> Partition Parameters are not updated even after ANALYZE TABLE command
> ---------------------------------------------------------------------
>
> Key: SPARK-33326
> URL: https://issues.apache.org/jira/browse/SPARK-33326
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.0.1
> Reporter: Daniel Bondor
> Priority: Major
>
> Here are the reproduction steps:
> {code:java}
> scala> spark.sql("CREATE TABLE t (a string,b string) PARTITIONED BY (p string) STORED AS PARQUET")
> Hive Session ID = d44e21ee-2d5c-48ab-91bf-26cb25775486
> res0: org.apache.spark.sql.DataFrame = []
> scala> spark.sql("INSERT INTO t PARTITION(p='p1') VALUES ('aaa', 'bbb')")
> res1: org.apache.spark.sql.DataFrame = []
> scala> spark.sql("INSERT INTO t PARTITION(p='p1') VALUES ('ccc', 'ddd')")
> res2: org.apache.spark.sql.DataFrame = []
> scala> spark.sql("ANALYZE TABLE t PARTITION(p='p1') COMPUTE STATISTICS")
> res3: org.apache.spark.sql.DataFrame = []
> scala> spark.sql("DESCRIBE FORMATTED t PARTITION (p='p1')").show(50, false)
> ...
> |Partition Parameters |{rawDataSize=0, numFiles=1, numFilesErasureCoded=0, transient_lastDdlTime=1604404640, totalSize=532, COLUMN_STATS_ACCURATE={"BASIC_STATS":"true","COLUMN_STATS":{"a":"true","b":"true"}}, numRows=0}| |
> ...
> |Partition Statistics |1064 bytes, 2 rows | |
> ...
> {code}
> My expectation would be that the Partition Parameters should be updated after ANALYZE TABLE.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org