[spark] branch master updated: [SPARK-34074][SQL][TESTS][FOLLOWUP] Fix table size parsing from statistics

[gu...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 514172a  [SPARK-34074][SQL][TESTS][FOLLOWUP] Fix table size parsing from statistics
514172a is described below

commit 514172aae79e6e7ed542870323193d4458f9b01a
Author: Max Gekk <ma...@gmail.com>
AuthorDate: Tue Jan 19 09:51:20 2021 +0900

    [SPARK-34074][SQL][TESTS][FOLLOWUP] Fix table size parsing from statistics
    
    ### What changes were proposed in this pull request?
    Fix table size parsing from the `Statistics` field which is formed at: https://github.com/apache/spark/blob/c3d81fbe79014f693cf93c02e31af401727761d7/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala#L573 . Before the fix, `getTableSize()` returns only the last digit. This works for Hive table in the tests because its size < 10 bytes, and accidentally works for V1 In-Memory catalog table in the tests.
    
    ### Why are the changes needed?
    This makes tests more reliable. For example, the parsing can not work in `AlterTableDropPartitionSuite` when table size before partition dropping:
    ```
    +---------+
    |data_type|
    +---------+
    |878 bytes|
    +---------+
    ```
    
    After:
    ```
    +---------+
    |data_type|
    +---------+
    |439 bytes|
    +---------+
    ```
    at:
    ```scala
            val onePartSize = getTableSize(t)
            assert(0 < onePartSize && onePartSize < twoPartSize)
    ```
    
    ### Does this PR introduce _any_ user-facing change?
    No
    
    ### How was this patch tested?
    By existing test suites:
    ```
    $ build/sbt -Phive-2.3 -Phive-thriftserver "test:testOnly *.AlterTableAddPartitionSuite"
    $ build/sbt -Phive-2.3 -Phive-thriftserver "test:testOnly *.AlterTableDropPartitionSuite"
    ```
    
    Closes #31237 from MaxGekk/optimize-updateTableStats-followup.
    
    Authored-by: Max Gekk <ma...@gmail.com>
    Signed-off-by: HyukjinKwon <gu...@apache.org>
---
 .../org/apache/spark/sql/execution/command/DDLCommandTestUtils.scala    | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLCommandTestUtils.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLCommandTestUtils.scala
index 573c450..15f9305 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLCommandTestUtils.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLCommandTestUtils.scala
@@ -101,7 +101,7 @@ trait DDLCommandTestUtils extends SQLTestUtils {
     if (stats.isEmpty) {
       throw new IllegalArgumentException(s"The table $tableName does not have stats")
     }
-    val tableSizeInStats = ".*(\\d) bytes.*".r
+    val tableSizeInStats = "^(\\d+) bytes.*$".r
     val size = stats.first().getString(0) match {
       case tableSizeInStats(s) => s.toInt
       case _ => throw new IllegalArgumentException("Not found table size in stats")


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org