You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/06/29 16:48:34 UTC

[GitHub] [spark] MaxGekk opened a new pull request #33137: [SPARK-35935][SQL] Prevent failure of `MSCK REPAIR TABLE` on table refreshing

MaxGekk opened a new pull request #33137:
URL: https://github.com/apache/spark/pull/33137


   ### What changes were proposed in this pull request?
   In the PR, I propose to catch all non-fatal exceptions coming `refreshTable()` at the final stage of table repairing, and output an error message instead of failing with an exception.
   
   ### Why are the changes needed?
   1. The uncaught exceptions from table refreshing might be considered as regression comparing to previous Spark versions. Table refreshing was introduced by https://github.com/apache/spark/pull/31066.
   2. This should improve user experience with Spark SQL. For instance, when the `MSCK REPAIR TABLE` is performed in a chain of command in SQL where catching exception is difficult or even impossible.
   
   ### Does this PR introduce _any_ user-facing change?
   Yes. Before the changes the `MSCK REPAIR TABLE` command can fail with the exception portrayed in SPARK-35935. After the changes, the same command outputs error message, and completes successfully.
   
   ### How was this patch tested?
   By existing test suites.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #33137: [SPARK-35935][SQL] Prevent failure of `MSCK REPAIR TABLE` on table refreshing

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33137:
URL: https://github.com/apache/spark/pull/33137#issuecomment-871166923


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44943/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] MaxGekk commented on pull request #33137: [SPARK-35935][SQL] Prevent failure of `MSCK REPAIR TABLE` on table refreshing

Posted by GitBox <gi...@apache.org>.
MaxGekk commented on pull request #33137:
URL: https://github.com/apache/spark/pull/33137#issuecomment-871158461


   Here are backports to 3.1/3.0: https://github.com/apache/spark/pull/33152


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33137: [SPARK-35935][SQL] Prevent failure of `MSCK REPAIR TABLE` on table refreshing

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33137:
URL: https://github.com/apache/spark/pull/33137#issuecomment-871274106


   **[Test build #140428 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140428/testReport)** for PR 33137 at commit [`85954ae`](https://github.com/apache/spark/commit/85954aef15ec30a14c4b4ec762d32b85d69bd133).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33137: [SPARK-35935][SQL] Prevent failure of `MSCK REPAIR TABLE` on table refreshing

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33137:
URL: https://github.com/apache/spark/pull/33137#issuecomment-871145684


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44943/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] MaxGekk commented on pull request #33137: [SPARK-35935][SQL] Prevent failure of `MSCK REPAIR TABLE` on table refreshing

Posted by GitBox <gi...@apache.org>.
MaxGekk commented on pull request #33137:
URL: https://github.com/apache/spark/pull/33137#issuecomment-871139935


   GA passed. Merging to master.
   Thank you, @cloud-fan for review.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33137: [SPARK-35935][SQL] Prevent failure of `MSCK REPAIR TABLE` on table refreshing

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33137:
URL: https://github.com/apache/spark/pull/33137#issuecomment-870824055


   **[Test build #140355 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140355/testReport)** for PR 33137 at commit [`2543fb4`](https://github.com/apache/spark/commit/2543fb4e943185177cb803be83fa0e4517eb8ab4).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33137: [SPARK-35935][SQL] Prevent failure of `MSCK REPAIR TABLE` on table refreshing

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33137:
URL: https://github.com/apache/spark/pull/33137#issuecomment-871110470


   **[Test build #140428 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140428/testReport)** for PR 33137 at commit [`85954ae`](https://github.com/apache/spark/commit/85954aef15ec30a14c4b4ec762d32b85d69bd133).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] MaxGekk closed pull request #33137: [SPARK-35935][SQL] Prevent failure of `MSCK REPAIR TABLE` on table refreshing

Posted by GitBox <gi...@apache.org>.
MaxGekk closed pull request #33137:
URL: https://github.com/apache/spark/pull/33137


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33137: [SPARK-35935][SQL] Prevent failure of `MSCK REPAIR TABLE` on table refreshing

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33137:
URL: https://github.com/apache/spark/pull/33137#issuecomment-871130009


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44943/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #33137: [SPARK-35935][SQL] Prevent failure of `MSCK REPAIR TABLE` on table refreshing

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33137:
URL: https://github.com/apache/spark/pull/33137#issuecomment-871287525


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140428/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #33137: [SPARK-35935][SQL] Prevent failure of `MSCK REPAIR TABLE` on table refreshing

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33137:
URL: https://github.com/apache/spark/pull/33137#issuecomment-871010059


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140355/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33137: [SPARK-35935][SQL] Prevent failure of `MSCK REPAIR TABLE` on table refreshing

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33137:
URL: https://github.com/apache/spark/pull/33137#issuecomment-871003899


   **[Test build #140355 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140355/testReport)** for PR 33137 at commit [`2543fb4`](https://github.com/apache/spark/commit/2543fb4e943185177cb803be83fa0e4517eb8ab4).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #33137: [SPARK-35935][SQL] Prevent failure of `MSCK REPAIR TABLE` on table refreshing

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33137:
URL: https://github.com/apache/spark/pull/33137#issuecomment-871010059


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140355/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #33137: [SPARK-35935][SQL] Prevent failure of `MSCK REPAIR TABLE` on table refreshing

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33137:
URL: https://github.com/apache/spark/pull/33137#issuecomment-870824055


   **[Test build #140355 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140355/testReport)** for PR 33137 at commit [`2543fb4`](https://github.com/apache/spark/commit/2543fb4e943185177cb803be83fa0e4517eb8ab4).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #33137: [SPARK-35935][SQL] Prevent failure of `MSCK REPAIR TABLE` on table refreshing

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33137:
URL: https://github.com/apache/spark/pull/33137#issuecomment-871166923


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44943/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #33137: [SPARK-35935][SQL] Prevent failure of `MSCK REPAIR TABLE` on table refreshing

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #33137:
URL: https://github.com/apache/spark/pull/33137#discussion_r660909444



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala
##########
@@ -675,7 +675,15 @@ case class RepairTableCommand(
     // This is always the case for Hive format tables, but is not true for Datasource tables created
     // before Spark 2.1 unless they are converted via `msck repair table`.
     spark.sessionState.catalog.alterTable(table.copy(tracksPartitionsInCatalog = true))
-    spark.catalog.refreshTable(tableIdentWithDB)
+    try {
+      spark.catalog.refreshTable(tableIdentWithDB)
+    } catch {
+      case NonFatal(_) =>
+        logError(s"Cannot refresh the table '$tableIdentWithDB'. A query of the table " +
+          "might return wrong result if the table was cached. To avoid such issue, you should " +
+          "uncache the table manually via the UNCACHE TABLE command after table recovering will " +
+          "complete fully.")

Review comment:
       shall we also log the exception itself?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #33137: [SPARK-35935][SQL] Prevent failure of `MSCK REPAIR TABLE` on table refreshing

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33137:
URL: https://github.com/apache/spark/pull/33137#issuecomment-871110470


   **[Test build #140428 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140428/testReport)** for PR 33137 at commit [`85954ae`](https://github.com/apache/spark/commit/85954aef15ec30a14c4b4ec762d32b85d69bd133).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #33137: [SPARK-35935][SQL] Prevent failure of `MSCK REPAIR TABLE` on table refreshing

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33137:
URL: https://github.com/apache/spark/pull/33137#issuecomment-871287525


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140428/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org