You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/01/09 18:11:38 UTC

[GitHub] [spark] sunchao commented on a change in pull request #31101: [SPARK-34055][SQL] Refresh cache in `ALTER TABLE .. ADD PARTITION`

sunchao commented on a change in pull request #31101:
URL: https://github.com/apache/spark/pull/31101#discussion_r554455100



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/AlterTableAddPartitionSuite.scala
##########
@@ -41,6 +45,33 @@ trait AlterTableAddPartitionSuiteBase extends command.AlterTableAddPartitionSuit
         "The spec ([p1=]) contains an empty partition column value"))
     }
   }
+
+  test("SPARK-34055: refresh cache in partition adding") {
+    withTable("t") {
+      sql(s"CREATE TABLE t (id int, part int) USING parquet PARTITIONED BY (part)")
+      sql("INSERT INTO t PARTITION (part=0) SELECT 0")
+      assert(!spark.catalog.isCached("t"))
+      sql("CACHE TABLE t")
+      assert(spark.catalog.isCached("t"))
+      checkAnswer(sql("SELECT * FROM t"), Seq(Row(0, 0)))
+
+      // Create new partition (part = 1) in the filesystem
+      val information = sql("SHOW TABLE EXTENDED LIKE 't' PARTITION (part = 0)")

Review comment:
       nit: it may worth to pull this into an util method since it's useful in multiple places seems, e.g., the other PR on "ALTER TABLE .. RECOVER PARTITIONS"

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala
##########
@@ -485,6 +485,7 @@ case class AlterTableAddPartitionCommand(
       catalog.createPartitions(table.identifier, batch, ignoreIfExists = ifNotExists)
     }
 
+    sparkSession.catalog.refreshTable(table.identifier.quotedString)

Review comment:
       just curious, does it matter whether we refresh cache before or after the stats are updated?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org