You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/12/07 13:46:35 UTC

[GitHub] [spark] wangyum opened a new pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

wangyum opened a new pull request #30648:
URL: https://github.com/apache/spark/pull/30648


   ### What changes were proposed in this pull request?
   
   This pr add support analyze all tables in a specific database:
   ```g4
    ANALYZE TABLES ((FROM | IN) multipartIdentifier)? COMPUTE STATISTICS (identifier)?
   ```
   
   ### Why are the changes needed?
   
   1. Make it easy to analyze all tables in a specific database.
   2. PostgreSQL has a similar implementation: https://www.postgresql.org/docs/12/sql-analyze.html.
   
   
   ### Does this PR introduce _any_ user-facing change?
   
   No.
   
   
   ### How was this patch tested?
   
   Unit test.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-741623534


   **[Test build #132477 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132477/testReport)** for PR 30648 at commit [`aafa658`](https://github.com/apache/spark/commit/aafa65812c0d70e615a052dcf93b972c0fd41452).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] maropu commented on a change in pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #30648:
URL: https://github.com/apache/spark/pull/30648#discussion_r537916908



##########
File path: sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4
##########
@@ -134,6 +134,8 @@ statement
         (AS? query)?                                                   #replaceTable
     | ANALYZE TABLE multipartIdentifier partitionSpec? COMPUTE STATISTICS
         (identifier | FOR COLUMNS identifierSeq | FOR ALL COLUMNS)?    #analyze
+    | ANALYZE TABLES ((FROM | IN) multipartIdentifier)? COMPUTE STATISTICS
+        (identifier)?                                                  #analyzeTables

Review comment:
       If` identifier` is only for `NOSCAN`, how about defining a ANTLR token for that?

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
##########
@@ -3547,6 +3547,25 @@ class AstBuilder extends SqlBaseBaseVisitor[AnyRef] with SQLConfHelper with Logg
     }
   }
 
+  /**
+   * Create an [[AnalyzeTables]].
+   * Example SQL for analyzing all tables in default database:
+   * {{{
+   *   ANALYZE TABLES IN default COMPUTE STATISTICS;

Review comment:
       `ANALYZE TABLES IN default COMPUTE STATISTICS;` -> `ANALYZE TABLES IN multi_part_name COMPUTE STATISTICS [NOSCAN];`?

##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DDLParserSuite.scala
##########
@@ -1898,6 +1898,15 @@ class DDLParserSuite extends AnalysisTest {
       "Expected `NOSCAN` instead of `xxxx`")
   }
 
+  test("analyze tables statistics") {
+    comparePlans(parsePlan("analyze tables in a.b.c compute statistics"),

Review comment:
       Please use uppercases for the SQL keywords where possible.

##########
File path: sql/core/src/test/scala/org/apache/spark/sql/StatisticsCollectionSuite.scala
##########
@@ -671,4 +671,24 @@ class StatisticsCollectionSuite extends StatisticsCollectionTestBase with Shared
       }
     }
   }
+
+  test("analyze all tables in a specific database") {
+    withTempDir { dir =>
+      withTable("t1", "t2") {
+        spark.range(10).write.saveAsTable("t1")
+        sql(s"CREATE EXTERNAL TABLE t2 USING parquet LOCATION '${dir.toURI}' " +
+          "AS SELECT * FROM range(20)")
+        withView("v1") {
+          sql(s"CREATE VIEW v1 AS SELECT * FROM t1")
+          sql(s"ANALYZE TABLES IN default COMPUTE STATISTICS NOSCAN")
+          checkTableStats("t1", hasSizeInBytes = true, expectedRowCounts = None)
+          checkTableStats("t2", hasSizeInBytes = true, expectedRowCounts = None)
+
+          sql(s"ANALYZE TABLES COMPUTE STATISTICS")

Review comment:
       nit: remove `s`.

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/AnalyzeTablesCommand.scala
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import org.apache.spark.sql.{Row, SparkSession}
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.CatalogTableType
+
+
+/**
+ * Analyzes all tables in the given database to generate statistics.
+ */
+case class AnalyzeTablesCommand(
+    databaseName: Option[String],
+    noScan: Boolean) extends RunnableCommand {
+
+  override def run(sparkSession: SparkSession): Seq[Row] = {
+
+    val catalog = sparkSession.sessionState.catalog
+    val db = databaseName.getOrElse(catalog.getCurrentDatabase)
+    catalog.listTables(db).foreach { tbl =>
+      try {
+        val tableMeta = catalog.getTableMetadata(tbl)
+        if (tableMeta.tableType == CatalogTableType.MANAGED ||

Review comment:
       How about cached views? `ANALYZE TABLE` supports cached case for views:
   ```
   scala> sql("create temporary view v as select 1")
   scala> sql("analyze table v COMPUTE STATISTICS FOR ALL COLUMNS")
   org.apache.spark.sql.AnalysisException: Temporary view `v` is not cached for analyzing columns.;
     at org.apache.spark.sql.execution.command.AnalyzeColumnCommand.analyzeColumnInTempView(AnalyzeColumnCommand.scala:75)
     ...
   
   scala> spark.table("v").cache()
   scala> sql("analyze table v COMPUTE STATISTICS FOR ALL COLUMNS")
   res3: org.apache.spark.sql.DataFrame = []  
   ```

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
##########
@@ -3547,6 +3547,25 @@ class AstBuilder extends SqlBaseBaseVisitor[AnyRef] with SQLConfHelper with Logg
     }
   }
 
+  /**
+   * Create an [[AnalyzeTables]].
+   * Example SQL for analyzing all tables in default database:
+   * {{{
+   *   ANALYZE TABLES IN default COMPUTE STATISTICS;
+   * }}}
+   */
+  override def visitAnalyzeTables(ctx: AnalyzeTablesContext): LogicalPlan = withOrigin(ctx) {
+    if (ctx.identifier != null &&
+      ctx.identifier.getText.toLowerCase(Locale.ROOT) != "noscan") {
+      throw new ParseException(s"Expected `NOSCAN` instead of `${ctx.identifier.getText}`",
+        ctx.identifier())
+    }
+    val multiPart = Option(ctx.multipartIdentifier).map(visitMultipartIdentifier)
+    AnalyzeTables(
+      UnresolvedNamespace(multiPart.getOrElse(Seq.empty[String])),

Review comment:
       nit: `Seq.empty[String]` -> `Nil`?

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala
##########
@@ -401,6 +401,13 @@ class ResolveSessionCatalog(
         AnalyzePartitionCommand(ident.asTableIdentifier, partitionSpec, noScan)
       }
 
+    case AnalyzeTables(SessionCatalogAndNamespace(_, ns), noScan) =>
+      if (ns.length > 1) {
+        throw new AnalysisException(
+          s"The database name is not valid: ${ns.quoted}")

Review comment:
       nit: we can put this in a single line: `throw new AnalysisException(s"The database name is not valid: ${ns.quoted}")`

##########
File path: sql/core/src/test/scala/org/apache/spark/sql/StatisticsCollectionSuite.scala
##########
@@ -671,4 +671,24 @@ class StatisticsCollectionSuite extends StatisticsCollectionTestBase with Shared
       }
     }
   }
+
+  test("analyze all tables in a specific database") {
+    withTempDir { dir =>
+      withTable("t1", "t2") {
+        spark.range(10).write.saveAsTable("t1")
+        sql(s"CREATE EXTERNAL TABLE t2 USING parquet LOCATION '${dir.toURI}' " +
+          "AS SELECT * FROM range(20)")
+        withView("v1") {
+          sql(s"CREATE VIEW v1 AS SELECT * FROM t1")
+          sql(s"ANALYZE TABLES IN default COMPUTE STATISTICS NOSCAN")

Review comment:
       Also, could you add tests for the case where no database found?

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/AnalyzeTablesCommand.scala
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import org.apache.spark.sql.{Row, SparkSession}
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.CatalogTableType
+
+
+/**
+ * Analyzes all tables in the given database to generate statistics.
+ */
+case class AnalyzeTablesCommand(
+    databaseName: Option[String],
+    noScan: Boolean) extends RunnableCommand {
+
+  override def run(sparkSession: SparkSession): Seq[Row] = {
+
+    val catalog = sparkSession.sessionState.catalog
+    val db = databaseName.getOrElse(catalog.getCurrentDatabase)
+    catalog.listTables(db).foreach { tbl =>
+      try {
+        val tableMeta = catalog.getTableMetadata(tbl)
+        if (tableMeta.tableType == CatalogTableType.MANAGED ||
+          tableMeta.tableType == CatalogTableType.EXTERNAL) {
+          // Compute stats for the whole table
+          val newTotalSize = CommandUtils.calculateTotalSize(sparkSession, tableMeta)
+          val tableIdentWithDB = TableIdentifier(tbl.table, Some(db))
+          val newRowCount =
+            if (noScan) None else Some(BigInt(sparkSession.table(tableIdentWithDB).count()))
+
+          // Update the metastore if the above statistics of the table are different from those
+          // recorded in the metastore.
+          val newStats =
+            CommandUtils.compareAndGetNewStats(tableMeta.stats, newTotalSize, newRowCount)
+          if (newStats.isDefined) {
+            catalog.alterTableStats(tableIdentWithDB, newStats)
+          }
+        }
+      } catch {
+        case e: Exception =>
+          logError(s"Failed to analyze table: ${tbl.identifier}.", e)

Review comment:
       `AnalysisException` instead of `logError`, I think. And, we need tests for this code path.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #30648:
URL: https://github.com/apache/spark/pull/30648#discussion_r537908153



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/StatisticsCollectionSuite.scala
##########
@@ -671,4 +671,24 @@ class StatisticsCollectionSuite extends StatisticsCollectionTestBase with Shared
       }
     }
   }
+
+  test("analyze all tables in a specific database") {

Review comment:
       Shall we add `SPARK-33687: ` prefix?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] maropu commented on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-787551927


   Thanks! Merged to master.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-785694624


   **[Test build #135449 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135449/testReport)** for PR 30648 at commit [`81efb9d`](https://github.com/apache/spark/commit/81efb9d1f0dc43b3053d03d3ea183e436a972576).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-741614172


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37075/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-785622026


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40029/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-741655570


   Looks fine to me


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-741687368


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132477/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-785550053


   **[Test build #135449 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135449/testReport)** for PR 30648 at commit [`81efb9d`](https://github.com/apache/spark/commit/81efb9d1f0dc43b3053d03d3ea183e436a972576).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-741599929


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37069/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] wangyum commented on a change in pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
wangyum commented on a change in pull request #30648:
URL: https://github.com/apache/spark/pull/30648#discussion_r539055992



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/StatisticsCollectionSuite.scala
##########
@@ -671,4 +671,24 @@ class StatisticsCollectionSuite extends StatisticsCollectionTestBase with Shared
       }
     }
   }
+
+  test("analyze all tables in a specific database") {
+    withTempDir { dir =>
+      withTable("t1", "t2") {
+        spark.range(10).write.saveAsTable("t1")
+        sql(s"CREATE EXTERNAL TABLE t2 USING parquet LOCATION '${dir.toURI}' " +
+          "AS SELECT * FROM range(20)")
+        withView("v1") {
+          sql(s"CREATE VIEW v1 AS SELECT * FROM t1")
+          sql(s"ANALYZE TABLES IN default COMPUTE STATISTICS NOSCAN")

Review comment:
       +1




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-741599914


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37069/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-741596373


   **[Test build #132473 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132473/testReport)** for PR 30648 at commit [`1a16397`](https://github.com/apache/spark/commit/1a163972331ad4d173e193b48096edf85f5f7759).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] wangyum commented on a change in pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
wangyum commented on a change in pull request #30648:
URL: https://github.com/apache/spark/pull/30648#discussion_r582531982



##########
File path: docs/sql-ref-syntax-aux-analyze-tables.md
##########
@@ -0,0 +1,149 @@
+---
+layout: global
+title: ANALYZE TABLES
+displayTitle: ANALYZE TABLES
+license: |
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+ 
+     http://www.apache.org/licenses/LICENSE-2.0
+ 
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+---
+
+### Description
+
+The `ANALYZE TABLES` statement collects statistics about all the tables in a database to be used by the query optimizer to find a better query execution plan.
+
+### Syntax
+
+```sql
+ANALYZE TABLES [ { FROM | IN } database_name ] COMPUTE STATISTICS [ NOSCAN ]
+```
+
+### Parameters
+
+* **{ FROM `|` IN } database_name**
+
+    Specifies the name of the database to be analyzed. Without a database name, `ANALYZE` collects all tables in the current database that the current user has permission to analyze.
+
+* **[ NOSCAN ]**
+
+    Collects only the table's size in bytes ( which does not require scanning the entire table ).

Review comment:
       fixed




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] wangyum commented on a change in pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
wangyum commented on a change in pull request #30648:
URL: https://github.com/apache/spark/pull/30648#discussion_r582448513



##########
File path: docs/sql-ref-syntax-aux-analyze-tables.md
##########
@@ -0,0 +1,149 @@
+---
+layout: global
+title: ANALYZE TABLES
+displayTitle: ANALYZE TABLES
+license: |
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+ 
+     http://www.apache.org/licenses/LICENSE-2.0
+ 
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+---
+
+### Description
+
+The `ANALYZE TABLES` statement collects statistics about all the tables in a database to be used by the query optimizer to find a better query execution plan.
+
+### Syntax
+
+```sql
+ANALYZE TABLES [ { FROM | IN } database_name ] COMPUTE STATISTICS [ NOSCAN ]
+```
+
+### Parameters
+
+* **{ FROM `|` IN } database_name**
+
+    Specifies the name of the database to be analyzed. Without a database name, `ANALYZE` collects all tables in the current database that the current user has permission to analyze.
+
+* **[ NOSCAN ]**
+
+    Collects only the table's size in bytes ( which does not require scanning the entire table ).

Review comment:
       It also has spaces:
   https://github.com/apache/spark/blob/aafa65812c0d70e615a052dcf93b972c0fd41452/docs/sql-ref-syntax-aux-analyze-table.md#L51-L53




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-785615136


   I am okay with this. cc @peter-toth too if you're interested in this.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-785239925


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40008/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] maropu commented on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-785028758


   @wangyum Could you resolve the conflict? This feature looks useful, so I wanna finish and merge it.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-787557188


   Thank you, @wangyum and all!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-740030345


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132370/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-741569030


   **[Test build #132467 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132467/testReport)** for PR 30648 at commit [`198babc`](https://github.com/apache/spark/commit/198babc120e942ed085b50be118779e37c35c3ae).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-741666853


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37079/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #30648:
URL: https://github.com/apache/spark/pull/30648#discussion_r537910759



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/AnalyzeTablesCommand.scala
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import org.apache.spark.sql.{Row, SparkSession}
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.CatalogTableType
+
+
+/**
+ * Analyzes all tables in the given database to generate statistics.
+ */
+case class AnalyzeTablesCommand(
+    databaseName: Option[String],
+    noScan: Boolean) extends RunnableCommand {
+
+  override def run(sparkSession: SparkSession): Seq[Row] = {
+
+    val catalog = sparkSession.sessionState.catalog
+    val db = databaseName.getOrElse(catalog.getCurrentDatabase)
+    catalog.listTables(db).foreach { tbl =>
+      try {
+        val tableMeta = catalog.getTableMetadata(tbl)
+        if (tableMeta.tableType == CatalogTableType.MANAGED ||
+          tableMeta.tableType == CatalogTableType.EXTERNAL) {
+          // Compute stats for the whole table
+          val newTotalSize = CommandUtils.calculateTotalSize(sparkSession, tableMeta)
+          val tableIdentWithDB = TableIdentifier(tbl.table, Some(db))
+          val newRowCount =
+            if (noScan) None else Some(BigInt(sparkSession.table(tableIdentWithDB).count()))
+
+          // Update the metastore if the above statistics of the table are different from those
+          // recorded in the metastore.
+          val newStats =
+            CommandUtils.compareAndGetNewStats(tableMeta.stats, newTotalSize, newRowCount)
+          if (newStats.isDefined) {
+            catalog.alterTableStats(tableIdentWithDB, newStats)
+          }
+        }
+      } catch {
+        case e: Exception =>

Review comment:
       This looks like too general. Can we use more specific one instead of `Exception`?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-741599929


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37069/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-739992708


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36970/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] maropu commented on a change in pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #30648:
URL: https://github.com/apache/spark/pull/30648#discussion_r539073924



##########
File path: docs/sql-ref-syntax-aux-analyze-tables.md
##########
@@ -0,0 +1,90 @@
+---
+layout: global
+title: ANALYZE TABLES
+displayTitle: ANALYZE TABLES
+license: |
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+ 
+     http://www.apache.org/licenses/LICENSE-2.0
+ 
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+---
+
+### Description
+
+The `ANALYZE TABLES` statement collects statistics about all the tables in a database to be used by the query optimizer to find a better query execution plan.
+
+### Syntax
+
+```sql
+ANALYZE TABLES [ { FROM | IN } database_name ] COMPUTE STATISTICS [ NOSCAN ]
+```
+
+### Parameters
+
+* **{ FROM `|` IN } database_name**
+
+    Specifies the name of the database to be analyzed. Without a database name, `ANALYZE` collects all tables in the current database that the current user has permission to analyze.
+
+* **[ NOSCAN ]**
+
+    Collects only the table's size in bytes ( which does not require scanning the entire table ).
+
+### Examples
+
+```sql
+CREATE DATABASE school_db;
+USE school_db;
+
+CREATE TABLE teachers (name STRING, teacher_id INT);
+INSERT INTO teachers VALUES ('Tom', 1), ('Jerry', 2);
+
+CREATE TABLE students (name STRING, student_id INT);
+INSERT INTO students VALUES ('Mark', 111111), ('John', 222222);
+
+ANALYZE TABLES IN school_db COMPUTE STATISTICS NOSCAN;
+
+SHOW TABLE EXTENDED IN school_db LIKE '*';
++------------+------------+--------------+----------------------------------------------------+
+|  database  | tableName  | isTemporary  |                    information                     |
++------------+------------+--------------+----------------------------------------------------+
+| school_db  | students   | false        | Database: school_db
+Table: students
+...
+Statistics: 24 bytes
+...
+
+| school_db  | teachers   | false        | Database: school_db
+Table: teachers
+...
+Statistics: 14 bytes
+...
++------------+------------+--------------+----------------------------------------------------+
+
+ANALYZE TABLES COMPUTE STATISTICS;
+
+SHOW TABLE EXTENDED IN school_db LIKE '*';
++------------+------------+--------------+----------------------------------------------------+
+|  database  | tableName  | isTemporary  |                    information                     |
++------------+------------+--------------+----------------------------------------------------+
+| school_db  | students   | false        | Database: school_db
+Table: students
+...
+Statistics: 24 bytes, 2 rows
+...
+| school_db  | teachers   | false        | Database: school_db
+Table: teachers
+...
+Statistics: 14 bytes, 2 rows
+...
++------------+------------+--------------+----------------------------------------------------+
+```

Review comment:
       Could you add a link to the `ANALYZE TABLE` page in the `Related Statements` section? Also, I think its better to add a link to this doc page in the `ANALYZE TABLE` page, too.
   

##########
File path: docs/sql-ref-syntax-aux-analyze-tables.md
##########
@@ -0,0 +1,90 @@
+---
+layout: global
+title: ANALYZE TABLES
+displayTitle: ANALYZE TABLES
+license: |
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+ 
+     http://www.apache.org/licenses/LICENSE-2.0
+ 
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+---
+
+### Description
+
+The `ANALYZE TABLES` statement collects statistics about all the tables in a database to be used by the query optimizer to find a better query execution plan.

Review comment:
       cc: @huaxingao 

##########
File path: docs/sql-ref-syntax-aux-analyze-tables.md
##########
@@ -0,0 +1,90 @@
+---
+layout: global
+title: ANALYZE TABLES
+displayTitle: ANALYZE TABLES
+license: |
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+ 
+     http://www.apache.org/licenses/LICENSE-2.0
+ 
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+---
+
+### Description
+
+The `ANALYZE TABLES` statement collects statistics about all the tables in a database to be used by the query optimizer to find a better query execution plan.
+
+### Syntax
+
+```sql
+ANALYZE TABLES [ { FROM | IN } database_name ] COMPUTE STATISTICS [ NOSCAN ]
+```
+
+### Parameters
+
+* **{ FROM `|` IN } database_name**
+
+    Specifies the name of the database to be analyzed. Without a database name, `ANALYZE` collects all tables in the current database that the current user has permission to analyze.
+
+* **[ NOSCAN ]**
+
+    Collects only the table's size in bytes ( which does not require scanning the entire table ).
+
+### Examples
+
+```sql
+CREATE DATABASE school_db;
+USE school_db;
+
+CREATE TABLE teachers (name STRING, teacher_id INT);
+INSERT INTO teachers VALUES ('Tom', 1), ('Jerry', 2);
+
+CREATE TABLE students (name STRING, student_id INT);
+INSERT INTO students VALUES ('Mark', 111111), ('John', 222222);
+
+ANALYZE TABLES IN school_db COMPUTE STATISTICS NOSCAN;
+
+SHOW TABLE EXTENDED IN school_db LIKE '*';
++------------+------------+--------------+----------------------------------------------------+
+|  database  | tableName  | isTemporary  |                    information                     |
++------------+------------+--------------+----------------------------------------------------+
+| school_db  | students   | false        | Database: school_db
+Table: students

Review comment:
       Formatting this like the `SHOW TABLE EXTENDED` page looks better:
   https://spark.apache.org/docs/latest/sql-ref-syntax-aux-show-table.html#examples

##########
File path: sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4
##########
@@ -134,6 +134,8 @@ statement
         (AS? query)?                                                   #replaceTable
     | ANALYZE TABLE multipartIdentifier partitionSpec? COMPUTE STATISTICS
         (identifier | FOR COLUMNS identifierSeq | FOR ALL COLUMNS)?    #analyze
+    | ANALYZE TABLES ((FROM | IN) multipartIdentifier)? COMPUTE STATISTICS
+        (identifier)?                                                  #analyzeTables

Review comment:
       sgtm




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-741666831


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37079/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-741655168


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132467/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-741666853


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37079/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] maropu edited a comment on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
maropu edited a comment on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-740260400


   Also, I think we need a SQL doc page for this new feature.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-741628010


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37075/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] wangyum commented on a change in pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
wangyum commented on a change in pull request #30648:
URL: https://github.com/apache/spark/pull/30648#discussion_r537908641



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/StatisticsCollectionSuite.scala
##########
@@ -671,4 +671,24 @@ class StatisticsCollectionSuite extends StatisticsCollectionTestBase with Shared
       }
     }
   }
+
+  test("analyze all tables in a specific database") {

Review comment:
       ok.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-785173487


   **[Test build #135428 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135428/testReport)** for PR 30648 at commit [`a75a30d`](https://github.com/apache/spark/commit/a75a30d58ebfff478d2a147bb7f3615757fa2235).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-740011971


   **[Test build #132370 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132370/testReport)** for PR 30648 at commit [`78b9ffc`](https://github.com/apache/spark/commit/78b9ffc48d1aa917300da1579e6254b9a5083677).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds the following public classes _(experimental)_:
     * `case class AnalyzeTables(`
     * `case class AnalyzeTablesCommand(`


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #30648:
URL: https://github.com/apache/spark/pull/30648#discussion_r537908780



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/StatisticsCollectionSuite.scala
##########
@@ -671,4 +671,24 @@ class StatisticsCollectionSuite extends StatisticsCollectionTestBase with Shared
       }
     }
   }
+
+  test("analyze all tables in a specific database") {
+    withTempDir { dir =>
+      withTable("t1", "t2") {
+        spark.range(10).write.saveAsTable("t1")
+        sql(s"CREATE EXTERNAL TABLE t2 USING parquet LOCATION '${dir.toURI}' " +
+          "AS SELECT * FROM range(20)")
+        withView("v1") {
+          sql(s"CREATE VIEW v1 AS SELECT * FROM t1")
+          sql(s"ANALYZE TABLES IN default COMPUTE STATISTICS NOSCAN")

Review comment:
       Could you create a new database and use it here instead of `default`?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-785351209


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135428/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-741628010


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37075/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] wangyum commented on a change in pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
wangyum commented on a change in pull request #30648:
URL: https://github.com/apache/spark/pull/30648#discussion_r582441658



##########
File path: docs/sql-ref-syntax-aux-analyze-tables.md
##########
@@ -0,0 +1,149 @@
+---
+layout: global
+title: ANALYZE TABLES
+displayTitle: ANALYZE TABLES
+license: |
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+ 
+     http://www.apache.org/licenses/LICENSE-2.0
+ 
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+---
+
+### Description
+
+The `ANALYZE TABLES` statement collects statistics about all the tables in a database to be used by the query optimizer to find a better query execution plan.
+
+### Syntax
+
+```sql
+ANALYZE TABLES [ { FROM | IN } database_name ] COMPUTE STATISTICS [ NOSCAN ]
+```
+
+### Parameters
+
+* **{ FROM `|` IN } database_name**
+
+    Specifies the name of the database to be analyzed. Without a database name, `ANALYZE` collects all tables in the current database that the current user has permission to analyze.

Review comment:
       I think this is needed, because it will ignore instead of throw exception if user do not have permission to analyze.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] wangyum commented on a change in pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
wangyum commented on a change in pull request #30648:
URL: https://github.com/apache/spark/pull/30648#discussion_r538974583



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/AnalyzeTablesCommand.scala
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import org.apache.spark.sql.{Row, SparkSession}
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.CatalogTableType
+
+
+/**
+ * Analyzes all tables in the given database to generate statistics.
+ */
+case class AnalyzeTablesCommand(
+    databaseName: Option[String],
+    noScan: Boolean) extends RunnableCommand {
+
+  override def run(sparkSession: SparkSession): Seq[Row] = {
+
+    val catalog = sparkSession.sessionState.catalog
+    val db = databaseName.getOrElse(catalog.getCurrentDatabase)
+    catalog.listTables(db).foreach { tbl =>
+      try {
+        val tableMeta = catalog.getTableMetadata(tbl)
+        if (tableMeta.tableType == CatalogTableType.MANAGED ||
+          tableMeta.tableType == CatalogTableType.EXTERNAL) {
+          // Compute stats for the whole table
+          val newTotalSize = CommandUtils.calculateTotalSize(sparkSession, tableMeta)
+          val tableIdentWithDB = TableIdentifier(tbl.table, Some(db))
+          val newRowCount =
+            if (noScan) None else Some(BigInt(sparkSession.table(tableIdentWithDB).count()))
+
+          // Update the metastore if the above statistics of the table are different from those
+          // recorded in the metastore.
+          val newStats =
+            CommandUtils.compareAndGetNewStats(tableMeta.stats, newTotalSize, newRowCount)
+          if (newStats.isDefined) {
+            catalog.alterTableStats(tableIdentWithDB, newStats)
+          }
+        }
+      } catch {
+        case e: Exception =>
+          logError(s"Failed to analyze table: ${tbl.identifier}.", e)

Review comment:
       This is because the current user may not have permission for some tables.  
   
   This is PostgreSQL doc:
   ```
   ANALYZE processes every table and materialized view in the current database that the current user has permission to analyze.
   ```
   
   https://www.postgresql.org/docs/12/sql-analyze.html




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-741623534


   **[Test build #132477 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132477/testReport)** for PR 30648 at commit [`aafa658`](https://github.com/apache/spark/commit/aafa65812c0d70e615a052dcf93b972c0fd41452).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] wangyum commented on a change in pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
wangyum commented on a change in pull request #30648:
URL: https://github.com/apache/spark/pull/30648#discussion_r539050414



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/StatisticsCollectionSuite.scala
##########
@@ -671,4 +671,24 @@ class StatisticsCollectionSuite extends StatisticsCollectionTestBase with Shared
       }
     }
   }
+
+  test("analyze all tables in a specific database") {
+    withTempDir { dir =>
+      withTable("t1", "t2") {
+        spark.range(10).write.saveAsTable("t1")
+        sql(s"CREATE EXTERNAL TABLE t2 USING parquet LOCATION '${dir.toURI}' " +
+          "AS SELECT * FROM range(20)")
+        withView("v1") {
+          sql(s"CREATE VIEW v1 AS SELECT * FROM t1")
+          sql(s"ANALYZE TABLES IN default COMPUTE STATISTICS NOSCAN")
+          checkTableStats("t1", hasSizeInBytes = true, expectedRowCounts = None)
+          checkTableStats("t2", hasSizeInBytes = true, expectedRowCounts = None)
+
+          sql(s"ANALYZE TABLES COMPUTE STATISTICS")

Review comment:
       +1




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-741648956


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37079/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-739971416


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36970/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-785714301


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135449/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] wangyum commented on a change in pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
wangyum commented on a change in pull request #30648:
URL: https://github.com/apache/spark/pull/30648#discussion_r539050836



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/AnalyzeTablesCommand.scala
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import org.apache.spark.sql.{Row, SparkSession}
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.CatalogTableType
+
+
+/**
+ * Analyzes all tables in the given database to generate statistics.
+ */
+case class AnalyzeTablesCommand(
+    databaseName: Option[String],
+    noScan: Boolean) extends RunnableCommand {
+
+  override def run(sparkSession: SparkSession): Seq[Row] = {
+
+    val catalog = sparkSession.sessionState.catalog
+    val db = databaseName.getOrElse(catalog.getCurrentDatabase)
+    catalog.listTables(db).foreach { tbl =>
+      try {
+        val tableMeta = catalog.getTableMetadata(tbl)
+        if (tableMeta.tableType == CatalogTableType.MANAGED ||

Review comment:
       Yes. add support analyze cached views.

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala
##########
@@ -401,6 +401,13 @@ class ResolveSessionCatalog(
         AnalyzePartitionCommand(ident.asTableIdentifier, partitionSpec, noScan)
       }
 
+    case AnalyzeTables(SessionCatalogAndNamespace(_, ns), noScan) =>
+      if (ns.length > 1) {
+        throw new AnalysisException(
+          s"The database name is not valid: ${ns.quoted}")

Review comment:
       +1

##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DDLParserSuite.scala
##########
@@ -1898,6 +1898,15 @@ class DDLParserSuite extends AnalysisTest {
       "Expected `NOSCAN` instead of `xxxx`")
   }
 
+  test("analyze tables statistics") {
+    comparePlans(parsePlan("analyze tables in a.b.c compute statistics"),

Review comment:
       +1




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-785604082


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40029/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-741687368


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132477/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-741634682


   **[Test build #132467 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132467/testReport)** for PR 30648 at commit [`198babc`](https://github.com/apache/spark/commit/198babc120e942ed085b50be118779e37c35c3ae).
    * This patch **fails Spark unit tests**.
    * This patch **does not merge cleanly**.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] maropu commented on a change in pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #30648:
URL: https://github.com/apache/spark/pull/30648#discussion_r537916908



##########
File path: sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4
##########
@@ -134,6 +134,8 @@ statement
         (AS? query)?                                                   #replaceTable
     | ANALYZE TABLE multipartIdentifier partitionSpec? COMPUTE STATISTICS
         (identifier | FOR COLUMNS identifierSeq | FOR ALL COLUMNS)?    #analyze
+    | ANALYZE TABLES ((FROM | IN) multipartIdentifier)? COMPUTE STATISTICS
+        (identifier)?                                                  #analyzeTables

Review comment:
       If` identifier` is only for `NOSCAN`, how about defining a new ANTLR token for that?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-741569030


   **[Test build #132467 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132467/testReport)** for PR 30648 at commit [`198babc`](https://github.com/apache/spark/commit/198babc120e942ed085b50be118779e37c35c3ae).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] maropu commented on a change in pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #30648:
URL: https://github.com/apache/spark/pull/30648#discussion_r582452728



##########
File path: docs/sql-ref-syntax-aux-analyze-tables.md
##########
@@ -0,0 +1,149 @@
+---
+layout: global
+title: ANALYZE TABLES
+displayTitle: ANALYZE TABLES
+license: |
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+ 
+     http://www.apache.org/licenses/LICENSE-2.0
+ 
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+---
+
+### Description
+
+The `ANALYZE TABLES` statement collects statistics about all the tables in a database to be used by the query optimizer to find a better query execution plan.
+
+### Syntax
+
+```sql
+ANALYZE TABLES [ { FROM | IN } database_name ] COMPUTE STATISTICS [ NOSCAN ]
+```
+
+### Parameters
+
+* **{ FROM `|` IN } database_name**
+
+    Specifies the name of the database to be analyzed. Without a database name, `ANALYZE` collects all tables in the current database that the current user has permission to analyze.
+
+* **[ NOSCAN ]**
+
+    Collects only the table's size in bytes ( which does not require scanning the entire table ).

Review comment:
       yea, can you fix them, too?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-739929100


   **[Test build #132370 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132370/testReport)** for PR 30648 at commit [`78b9ffc`](https://github.com/apache/spark/commit/78b9ffc48d1aa917300da1579e6254b9a5083677).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-785550053


   **[Test build #135449 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135449/testReport)** for PR 30648 at commit [`81efb9d`](https://github.com/apache/spark/commit/81efb9d1f0dc43b3053d03d3ea183e436a972576).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-739955172


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36970/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] wangyum commented on a change in pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
wangyum commented on a change in pull request #30648:
URL: https://github.com/apache/spark/pull/30648#discussion_r539050973



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
##########
@@ -3547,6 +3547,25 @@ class AstBuilder extends SqlBaseBaseVisitor[AnyRef] with SQLConfHelper with Logg
     }
   }
 
+  /**
+   * Create an [[AnalyzeTables]].
+   * Example SQL for analyzing all tables in default database:
+   * {{{
+   *   ANALYZE TABLES IN default COMPUTE STATISTICS;
+   * }}}
+   */
+  override def visitAnalyzeTables(ctx: AnalyzeTablesContext): LogicalPlan = withOrigin(ctx) {
+    if (ctx.identifier != null &&
+      ctx.identifier.getText.toLowerCase(Locale.ROOT) != "noscan") {
+      throw new ParseException(s"Expected `NOSCAN` instead of `${ctx.identifier.getText}`",
+        ctx.identifier())
+    }
+    val multiPart = Option(ctx.multipartIdentifier).map(visitMultipartIdentifier)
+    AnalyzeTables(
+      UnresolvedNamespace(multiPart.getOrElse(Seq.empty[String])),

Review comment:
       +1




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-785222042


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40008/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-785615453


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40029/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #30648:
URL: https://github.com/apache/spark/pull/30648#discussion_r537911714



##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DDLParserSuite.scala
##########
@@ -1898,6 +1898,15 @@ class DDLParserSuite extends AnalysisTest {
       "Expected `NOSCAN` instead of `xxxx`")
   }
 
+  test("analyze tables statistics") {

Review comment:
       Could you add `SPARK-33687: ` prefix here?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-741655168


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132467/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-741678712


   **[Test build #132477 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132477/testReport)** for PR 30648 at commit [`aafa658`](https://github.com/apache/spark/commit/aafa65812c0d70e615a052dcf93b972c0fd41452).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] wangyum commented on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
wangyum commented on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-741580560


   > Looks a nice feature. Btw, (just a question) Any system you referred to for defining this syntax?
   
   PostgreSQL syntax is:
   ```
   ANALYZE [ ( option [, ...] ) ] [ table_and_columns [, ...] ]
   ANALYZE [ VERBOSE ] [ table_and_columns [, ...] ]
   
   where option can be one of:
   
       VERBOSE [ boolean ]
       SKIP_LOCKED [ boolean ]
   
   and table_and_columns is:
   
       table_name [ ( column_name [, ...] ) ]
   ```
   
   This syntax (` ANALYZE TABLES ((FROM | IN) multipartIdentifier)? COMPUTE STATISTICS (identifier)?`) is from
   https://github.com/apache/spark/blob/1a163972331ad4d173e193b48096edf85f5f7759/sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4#L135-L136
   and 
   https://github.com/apache/spark/blob/1a163972331ad4d173e193b48096edf85f5f7759/sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4#L200-L201


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] wangyum commented on a change in pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
wangyum commented on a change in pull request #30648:
URL: https://github.com/apache/spark/pull/30648#discussion_r539055309



##########
File path: sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4
##########
@@ -134,6 +134,8 @@ statement
         (AS? query)?                                                   #replaceTable
     | ANALYZE TABLE multipartIdentifier partitionSpec? COMPUTE STATISTICS
         (identifier | FOR COLUMNS identifierSeq | FOR ALL COLUMNS)?    #analyze
+    | ANALYZE TABLES ((FROM | IN) multipartIdentifier)? COMPUTE STATISTICS
+        (identifier)?                                                  #analyzeTables

Review comment:
       analyze also uses this `identifier `, how about do it in a separate pr?

##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DDLParserSuite.scala
##########
@@ -1898,6 +1898,15 @@ class DDLParserSuite extends AnalysisTest {
       "Expected `NOSCAN` instead of `xxxx`")
   }
 
+  test("analyze tables statistics") {

Review comment:
       +1




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-739992708


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36970/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #30648:
URL: https://github.com/apache/spark/pull/30648#discussion_r537909175



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/AnalyzeTablesCommand.scala
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import org.apache.spark.sql.{Row, SparkSession}
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.CatalogTableType
+
+
+/**
+ * Analyzes all tables in the given database to generate statistics.
+ */
+case class AnalyzeTablesCommand(
+    databaseName: Option[String],
+    noScan: Boolean) extends RunnableCommand {
+
+  override def run(sparkSession: SparkSession): Seq[Row] = {
+
+    val catalog = sparkSession.sessionState.catalog
+    val db = databaseName.getOrElse(catalog.getCurrentDatabase)
+    catalog.listTables(db).foreach { tbl =>
+      try {
+        val tableMeta = catalog.getTableMetadata(tbl)
+        if (tableMeta.tableType == CatalogTableType.MANAGED ||
+          tableMeta.tableType == CatalogTableType.EXTERNAL) {

Review comment:
       indentation?

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/AnalyzeTablesCommand.scala
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import org.apache.spark.sql.{Row, SparkSession}
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.CatalogTableType
+
+
+/**
+ * Analyzes all tables in the given database to generate statistics.
+ */
+case class AnalyzeTablesCommand(
+    databaseName: Option[String],
+    noScan: Boolean) extends RunnableCommand {
+
+  override def run(sparkSession: SparkSession): Seq[Row] = {
+

Review comment:
       Redundant empty line?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-785239925


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40008/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-785714301


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135449/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-741627975


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37075/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-741596373


   **[Test build #132473 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132473/testReport)** for PR 30648 at commit [`1a16397`](https://github.com/apache/spark/commit/1a163972331ad4d173e193b48096edf85f5f7759).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-741717796


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132473/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-741587697


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37069/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-741712594


   **[Test build #132473 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132473/testReport)** for PR 30648 at commit [`1a16397`](https://github.com/apache/spark/commit/1a163972331ad4d173e193b48096edf85f5f7759).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] maropu commented on a change in pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #30648:
URL: https://github.com/apache/spark/pull/30648#discussion_r582406908



##########
File path: docs/sql-ref-syntax-aux-analyze.md
##########
@@ -20,3 +20,4 @@ license: |
 ---
 
  * [ANALYZE TABLE statement](sql-ref-syntax-aux-analyze-table.html)
+ * [ANALYZE TABLES statement](sql-ref-syntax-aux-analyze-tables.html)

Review comment:
       plz update the index page, too.

##########
File path: docs/sql-ref-syntax-aux-analyze-tables.md
##########
@@ -0,0 +1,149 @@
+---
+layout: global
+title: ANALYZE TABLES
+displayTitle: ANALYZE TABLES
+license: |
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+ 
+     http://www.apache.org/licenses/LICENSE-2.0
+ 
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+---
+
+### Description
+
+The `ANALYZE TABLES` statement collects statistics about all the tables in a database to be used by the query optimizer to find a better query execution plan.

Review comment:
       nit: `in a database` -> `in a specified database`?

##########
File path: docs/sql-ref-syntax-aux-analyze-tables.md
##########
@@ -0,0 +1,149 @@
+---
+layout: global
+title: ANALYZE TABLES
+displayTitle: ANALYZE TABLES
+license: |
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+ 
+     http://www.apache.org/licenses/LICENSE-2.0
+ 
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+---
+
+### Description
+
+The `ANALYZE TABLES` statement collects statistics about all the tables in a database to be used by the query optimizer to find a better query execution plan.
+
+### Syntax
+
+```sql
+ANALYZE TABLES [ { FROM | IN } database_name ] COMPUTE STATISTICS [ NOSCAN ]
+```
+
+### Parameters
+
+* **{ FROM `|` IN } database_name**
+
+    Specifies the name of the database to be analyzed. Without a database name, `ANALYZE` collects all tables in the current database that the current user has permission to analyze.
+
+* **[ NOSCAN ]**
+
+    Collects only the table's size in bytes ( which does not require scanning the entire table ).
+
+### Examples
+
+```sql
+CREATE DATABASE school_db;
+USE school_db;
+
+CREATE TABLE teachers (name STRING, teacher_id INT);
+INSERT INTO teachers VALUES ('Tom', 1), ('Jerry', 2);
+
+CREATE TABLE students (name STRING, student_id INT);
+INSERT INTO students VALUES ('Mark', 111111), ('John', 222222);
+
+ANALYZE TABLES IN school_db COMPUTE STATISTICS NOSCAN;
+
+SHOW TABLE EXTENDED IN school_db LIKE '*';
++------------+------------+--------------+----------------------------------------------------+
+|  database  | tableName  | isTemporary  |                    information                     |
++------------+------------+--------------+----------------------------------------------------+
+|school_db   |students    |false         |Database: school_db
+                                          Table: students
+                                          Owner: root
+                                          Created Time: Wed Dec 09 14:23:25 CST 2020
+                                          Last Access: UNKNOWN
+                                          Created By: Spark 3.2.0-SNAPSHOT
+                                          Type: MANAGED
+                                          Provider: hive
+                                          Table Properties: [transient_lastDdlTime=1607495032]
+                                          Statistics: 24 bytes
+                                          Location: file:/opt/spark1/spark/spark-warehouse/school_db.db/students
+                                          Serde Library: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+                                          InputFormat: org.apache.hadoop.mapred.TextInputFormat
+                                          OutputFormat: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+                                          Storage Properties: [serialization.format=1]
+                                          Partition Provider: Catalog
+                                          Schema: root
+                                           |-- name: string (nullable = true)
+                                           |-- student_id: integer (nullable = true)          |
+|school_db   |teachers    |false         |Database: school_db
+                                          Table: teachers
+                                          Owner: root
+                                          Created Time: Wed Dec 09 14:24:15 CST 2020
+                                          Last Access: UNKNOWN
+                                          Created By: Spark 3.2.0-SNAPSHOT
+                                          Type: MANAGED
+                                          Provider: hive
+                                          Table Properties: [transient_lastDdlTime=1607495059]
+                                          Statistics: 14 bytes
+                                          Location: file:/opt/spark1/spark/spark-warehouse/school_db.db/teachers
+                                          Serde Library: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+                                          InputFormat: org.apache.hadoop.mapred.TextInputFormat
+                                          OutputFormat: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+                                          Storage Properties: [serialization.format=1]
+                                          Partition Provider: Catalog
+                                          Schema: root
+                                           |-- name: string (nullable = true)
+                                           |-- teacher_id: integer (nullable = true)          |
++------------+------------+--------------+----------------------------------------------------+
+
+ANALYZE TABLES COMPUTE STATISTICS;
+
+SHOW TABLE EXTENDED IN school_db LIKE '*';
++------------+------------+--------------+----------------------------------------------------+
+|  database  | tableName  | isTemporary  |                    information                     |
++------------+------------+--------------+----------------------------------------------------+
+|school_db   |students    |false         |Database: school_db
+                                          Table: students
+                                          Owner: root
+                                          Created Time: Wed Dec 09 14:23:25 CST 2020
+                                          Last Access: UNKNOWN
+                                          Created By: Spark 3.2.0-SNAPSHOT
+                                          Type: MANAGED
+                                          Provider: hive
+                                          Table Properties: [transient_lastDdlTime=1607495311]
+                                          Statistics: 24 bytes, 2 rows
+                                          Location: file:/opt/spark1/spark/spark-warehouse/school_db.db/students

Review comment:
       Could you use `DESC EXTENDED` to follow the `ANALYZE TABLE` example? https://spark.apache.org/docs/3.0.2/sql-ref-syntax-aux-analyze-table.html I think the current example has many unnecessary information. The simpler is the better.

##########
File path: docs/sql-ref-syntax-aux-analyze-tables.md
##########
@@ -0,0 +1,149 @@
+---
+layout: global
+title: ANALYZE TABLES
+displayTitle: ANALYZE TABLES
+license: |
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+ 
+     http://www.apache.org/licenses/LICENSE-2.0
+ 
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+---
+
+### Description
+
+The `ANALYZE TABLES` statement collects statistics about all the tables in a database to be used by the query optimizer to find a better query execution plan.
+
+### Syntax
+
+```sql
+ANALYZE TABLES [ { FROM | IN } database_name ] COMPUTE STATISTICS [ NOSCAN ]
+```
+
+### Parameters
+
+* **{ FROM `|` IN } database_name**
+
+    Specifies the name of the database to be analyzed. Without a database name, `ANALYZE` collects all tables in the current database that the current user has permission to analyze.

Review comment:
       We need t say `...that the current user has permission to analyze.` here?
   https://github.com/apache/spark/blame/master/docs/sql-ref-syntax-aux-show-tables.md#L39

##########
File path: docs/sql-ref-syntax-aux-analyze-tables.md
##########
@@ -0,0 +1,149 @@
+---
+layout: global
+title: ANALYZE TABLES
+displayTitle: ANALYZE TABLES
+license: |
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+ 
+     http://www.apache.org/licenses/LICENSE-2.0
+ 
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+---
+
+### Description
+
+The `ANALYZE TABLES` statement collects statistics about all the tables in a database to be used by the query optimizer to find a better query execution plan.
+
+### Syntax
+
+```sql
+ANALYZE TABLES [ { FROM | IN } database_name ] COMPUTE STATISTICS [ NOSCAN ]
+```
+
+### Parameters
+
+* **{ FROM `|` IN } database_name**
+
+    Specifies the name of the database to be analyzed. Without a database name, `ANALYZE` collects all tables in the current database that the current user has permission to analyze.
+
+* **[ NOSCAN ]**
+
+    Collects only the table's size in bytes ( which does not require scanning the entire table ).

Review comment:
       nit: it seems unnecessary spaces found: `( which` and `table )`




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-785173487


   **[Test build #135428 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135428/testReport)** for PR 30648 at commit [`a75a30d`](https://github.com/apache/spark/commit/a75a30d58ebfff478d2a147bb7f3615757fa2235).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] wangyum commented on a change in pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
wangyum commented on a change in pull request #30648:
URL: https://github.com/apache/spark/pull/30648#discussion_r539053292



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
##########
@@ -3547,6 +3547,25 @@ class AstBuilder extends SqlBaseBaseVisitor[AnyRef] with SQLConfHelper with Logg
     }
   }
 
+  /**
+   * Create an [[AnalyzeTables]].
+   * Example SQL for analyzing all tables in default database:
+   * {{{
+   *   ANALYZE TABLES IN default COMPUTE STATISTICS;

Review comment:
       This is example, similar to https://github.com/apache/spark/blob/29fed23ba16d580e6247b6e70e9c9eef0698aa95/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala#L3494-L3508




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-741717796


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132473/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] maropu commented on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-740260400


   Also, we need a SQL doc page for this new feature.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] maropu closed pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
maropu closed pull request #30648:
URL: https://github.com/apache/spark/pull/30648


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] maropu commented on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-786635200


   If no one have more comments, I'll merge this in a few days.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] wangyum commented on a change in pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
wangyum commented on a change in pull request #30648:
URL: https://github.com/apache/spark/pull/30648#discussion_r539055697



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/AnalyzeTablesCommand.scala
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import org.apache.spark.sql.{Row, SparkSession}
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.CatalogTableType
+
+
+/**
+ * Analyzes all tables in the given database to generate statistics.
+ */
+case class AnalyzeTablesCommand(
+    databaseName: Option[String],
+    noScan: Boolean) extends RunnableCommand {
+
+  override def run(sparkSession: SparkSession): Seq[Row] = {
+
+    val catalog = sparkSession.sessionState.catalog
+    val db = databaseName.getOrElse(catalog.getCurrentDatabase)
+    catalog.listTables(db).foreach { tbl =>
+      try {
+        val tableMeta = catalog.getTableMetadata(tbl)
+        if (tableMeta.tableType == CatalogTableType.MANAGED ||
+          tableMeta.tableType == CatalogTableType.EXTERNAL) {
+          // Compute stats for the whole table
+          val newTotalSize = CommandUtils.calculateTotalSize(sparkSession, tableMeta)
+          val tableIdentWithDB = TableIdentifier(tbl.table, Some(db))
+          val newRowCount =
+            if (noScan) None else Some(BigInt(sparkSession.table(tableIdentWithDB).count()))
+
+          // Update the metastore if the above statistics of the table are different from those
+          // recorded in the metastore.
+          val newStats =
+            CommandUtils.compareAndGetNewStats(tableMeta.stats, newTotalSize, newRowCount)
+          if (newStats.isDefined) {
+            catalog.alterTableStats(tableIdentWithDB, newStats)
+          }
+        }
+      } catch {
+        case e: Exception =>

Review comment:
       Change it to:
   ```scala
           case NonFatal(e) =>
             logWarning(s"Failed to analyze table ${tbl.table} in the " +
               s"database $db because of ${e.toString}", e)
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-739929100


   **[Test build #132370 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132370/testReport)** for PR 30648 at commit [`78b9ffc`](https://github.com/apache/spark/commit/78b9ffc48d1aa917300da1579e6254b9a5083677).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-785201198


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40008/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-785351209


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135428/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-785339561


   **[Test build #135428 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135428/testReport)** for PR 30648 at commit [`a75a30d`](https://github.com/apache/spark/commit/a75a30d58ebfff478d2a147bb7f3615757fa2235).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-785622026


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40029/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] maropu commented on a change in pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #30648:
URL: https://github.com/apache/spark/pull/30648#discussion_r537923418



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/AnalyzeTablesCommand.scala
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import org.apache.spark.sql.{Row, SparkSession}
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.CatalogTableType
+
+
+/**
+ * Analyzes all tables in the given database to generate statistics.
+ */
+case class AnalyzeTablesCommand(
+    databaseName: Option[String],
+    noScan: Boolean) extends RunnableCommand {
+
+  override def run(sparkSession: SparkSession): Seq[Row] = {
+
+    val catalog = sparkSession.sessionState.catalog
+    val db = databaseName.getOrElse(catalog.getCurrentDatabase)
+    catalog.listTables(db).foreach { tbl =>
+      try {
+        val tableMeta = catalog.getTableMetadata(tbl)
+        if (tableMeta.tableType == CatalogTableType.MANAGED ||

Review comment:
       How about cached views? `ANALYZE TABLE` supports cached case for views:
   ```
   scala> sql("create view v as select 1")
   scala> sql("analyze table v COMPUTE STATISTICS FOR ALL COLUMNS")
   20/12/08 09:06:15 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
     ...
   
   scala> spark.table("v").cache()
   scala> sql("analyze table v COMPUTE STATISTICS FOR ALL COLUMNS")
   res3: org.apache.spark.sql.DataFrame = []  
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-740030345


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132370/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #30648: [SPARK-33687][SQL] Support analyze all tables in a specific database

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30648:
URL: https://github.com/apache/spark/pull/30648#issuecomment-740240463


   Thank you for making this contribution, @wangyum .


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org