You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by aokolnychyi <gi...@git.apache.org> on 2017/09/16 12:18:08 UTC

[GitHub] spark pull request #19252: [SPARK-21969][SQL] CommandUtils.updateTableStats ...

GitHub user aokolnychyi opened a pull request:

    https://github.com/apache/spark/pull/19252

    [SPARK-21969][SQL] CommandUtils.updateTableStats should call refreshTable

    ## What changes were proposed in this pull request?
    
    Tables in the catalog cache are not invalidated once their statistics are updated. As a consequence, existing sessions will use the cached information even though it is not valid anymore. Consider and an example below. 
    
    ```
    // step 1
    spark.range(100).write.saveAsTable("tab1")
    // step 2
    spark.sql("analyze table tab1 compute statistics")
    // step 3
    spark.sql("explain cost select distinct * from tab1").show(false)
    // step 4
    spark.range(100).write.mode("append").saveAsTable("tab1")
    // step 5
    spark.sql("explain cost select distinct * from tab1").show(false)
    ```
    
    After step 3, the table will be present in the catalog relation cache. Step 4 will correctly update the metadata inside the catalog but will NOT invalidate the cache.
    
    By the way, ``spark.sql("analyze table tab1 compute statistics")`` between step 3 and step 4 would also solve the problem.
    
    ## How was this patch tested?
    
    Current and additional unit tests.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/aokolnychyi/spark spark-21969

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/19252.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #19252
    
----
commit ba963b46cd2917315bc2bd0cf237c7d9f79e9d65
Author: aokolnychyi <an...@sap.com>
Date:   2017-09-16T11:57:52Z

    [SPARK-21969][SQL] CommandUtils.updateTableStats should call refreshTable

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19252: [SPARK-21969][SQL] CommandUtils.updateTableStats should ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19252
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81897/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19252: [SPARK-21969][SQL] CommandUtils.updateTableStats should ...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:

    https://github.com/apache/spark/pull/19252
  
    Actually, the right fix should add `refreshTable(identifier)` to the SessionCatalog's [alterTableStats API](https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala#L379).
    



---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19252: [SPARK-21969][SQL] CommandUtils.updateTableStats ...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19252#discussion_r139569666
  
    --- Diff: sql/core/src/test/scala/org/apache/spark/sql/StatisticsCollectionSuite.scala ---
    @@ -261,6 +261,11 @@ class StatisticsCollectionSuite extends StatisticsCollectionTestBase with Shared
               assert(fetched1.get.sizeInBytes == 0)
               assert(fetched1.get.colStats.size == 2)
     
    +          // compute stats based on the catalog table metadata and
    +          // put the relation into the catalog cache
    +          sql(s"EXPLAIN COST SELECT DISTINCT * FROM $table")
    --- End diff --
    
    Could you replace the usage of `EXPLAIN COST` by
    ```Scala
    // Table lookup will make the table cached.
    spark.table(table)
    ```


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19252: [SPARK-21969][SQL] CommandUtils.updateTableStats should ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19252
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19252: [SPARK-21969][SQL] CommandUtils.updateTableStats should ...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:

    https://github.com/apache/spark/pull/19252
  
    Thanks! Merged to master.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19252: [SPARK-21969][SQL] CommandUtils.updateTableStats ...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/19252


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19252: [SPARK-21969][SQL] CommandUtils.updateTableStats should ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19252
  
    **[Test build #81897 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81897/testReport)** for PR 19252 at commit [`a5cb16d`](https://github.com/apache/spark/commit/a5cb16d12c6338a3a0d88804f8fb9a6904fca734).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19252: [SPARK-21969][SQL] CommandUtils.updateTableStats should ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19252
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19252: [SPARK-21969][SQL] CommandUtils.updateTableStats should ...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:

    https://github.com/apache/spark/pull/19252
  
    LGTM


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19252: [SPARK-21969][SQL] CommandUtils.updateTableStats should ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19252
  
    **[Test build #81842 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81842/testReport)** for PR 19252 at commit [`ba963b4`](https://github.com/apache/spark/commit/ba963b46cd2917315bc2bd0cf237c7d9f79e9d65).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19252: [SPARK-21969][SQL] CommandUtils.updateTableStats should ...

Posted by aokolnychyi <gi...@git.apache.org>.
Github user aokolnychyi commented on the issue:

    https://github.com/apache/spark/pull/19252
  
    @gatorsmile thanks for the feedback. I also covered ``TruncateTableCommand`` with additional tests. However, I see a bit strange behavior while creating a test for ``AlterTableAddPartitionCommand ``.
    
    ```
    sql(s"CREATE TABLE t1 (col1 int, col2 int) USING PARQUET")
    sql(s"INSERT INTO TABLE t1 SELECT 1, 2")
    sql(s"INSERT INTO TABLE t1 SELECT 2, 4")
    sql("SELECT * FROM t1").show()
    +----+----+
    |col1|col2|
    +----+----+
    |   1|   2|
    |   2|   4|
    +----+----+
    
    sql(s"CREATE TABLE t2 (col1 int, col2 int) USING PARQUET PARTITIONED BY (col1)")
    sql(s"INSERT INTO TABLE t2 SELECT 1, 2")
    sql(s"INSERT INTO TABLE t2 SELECT 2, 4")
    sql("SELECT * FROM t2").show()
    +----+----+
    |col2|col1|
    +----+----+
    |   2|   4|
    |   1|   2|
    +----+----+
    ```
    
    Why are the results different? Is it a bug?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19252: [SPARK-21969][SQL] CommandUtils.updateTableStats should ...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:

    https://github.com/apache/spark/pull/19252
  
    This is not a bug. We just follow [the behavior of Hive's dynamic partition insert](https://docs.databricks.com/spark/latest/spark-sql/language-manual/insert.html#dynamic-partition-inserts). 
    
    > The dynamic partition columns must be specified last in both part_spec and the input result set (of the row value lists or the select query). They are resolved by position, instead of by names. Thus, the orders must be exactly matched.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19252: [SPARK-21969][SQL] CommandUtils.updateTableStats should ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19252
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19252: [SPARK-21969][SQL] CommandUtils.updateTableStats should ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19252
  
    **[Test build #81842 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81842/testReport)** for PR 19252 at commit [`ba963b4`](https://github.com/apache/spark/commit/ba963b46cd2917315bc2bd0cf237c7d9f79e9d65).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19252: [SPARK-21969][SQL] CommandUtils.updateTableStats should ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19252
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81842/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19252: [SPARK-21969][SQL] CommandUtils.updateTableStats ...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19252#discussion_r139317292
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/CommandUtils.scala ---
    @@ -44,6 +44,7 @@ object CommandUtils extends Logging {
           } else {
             catalog.alterTableStats(table.identifier, None)
           }
    +      catalog.refreshTable(table.identifier)
    --- End diff --
    
    Add a comment above this line:
    > Invalidate the table relation cache


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19252: [SPARK-21969][SQL] CommandUtils.updateTableStats ...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19252#discussion_r139569908
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala ---
    @@ -377,6 +377,8 @@ class SessionCatalog(
         requireDbExists(db)
         requireTableExists(tableIdentifier)
         externalCatalog.alterTableStats(db, table, newStats)
    +    // Invalidate the table relation cache
    +    refreshTable(identifier)
    --- End diff --
    
    Could you remove the unneeded `refreshTable` calls in `AnalyzeTableCommand` and `AnalyzeColumnCommand`?
    



---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19252: [SPARK-21969][SQL] CommandUtils.updateTableStats should ...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:

    https://github.com/apache/spark/pull/19252
  
    `TruncateTableCommand` also has similar issues. Could you also fix it in this PR?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19252: [SPARK-21969][SQL] CommandUtils.updateTableStats should ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19252
  
    **[Test build #81897 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81897/testReport)** for PR 19252 at commit [`a5cb16d`](https://github.com/apache/spark/commit/a5cb16d12c6338a3a0d88804f8fb9a6904fca734).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19252: [SPARK-21969][SQL] CommandUtils.updateTableStats should ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19252
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81896/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19252: [SPARK-21969][SQL] CommandUtils.updateTableStats should ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19252
  
    **[Test build #81896 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81896/testReport)** for PR 19252 at commit [`ca09962`](https://github.com/apache/spark/commit/ca09962e04f0ebc9f103af59796bf1a5ba4189f3).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19252: [SPARK-21969][SQL] CommandUtils.updateTableStats should ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19252
  
    **[Test build #81941 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81941/testReport)** for PR 19252 at commit [`63f9dc2`](https://github.com/apache/spark/commit/63f9dc23f781bdf6e8c891e1a502b91ccdbc0281).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19252: [SPARK-21969][SQL] CommandUtils.updateTableStats should ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19252
  
    **[Test build #81896 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81896/testReport)** for PR 19252 at commit [`ca09962`](https://github.com/apache/spark/commit/ca09962e04f0ebc9f103af59796bf1a5ba4189f3).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19252: [SPARK-21969][SQL] CommandUtils.updateTableStats should ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19252
  
    **[Test build #81941 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81941/testReport)** for PR 19252 at commit [`63f9dc2`](https://github.com/apache/spark/commit/63f9dc23f781bdf6e8c891e1a502b91ccdbc0281).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19252: [SPARK-21969][SQL] CommandUtils.updateTableStats should ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19252
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org