You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2021/04/19 19:26:44 UTC
[spark-website] branch asf-site updated: Add error message
guidelines (#332)
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/spark-website.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 4115e72 Add error message guidelines (#332)
4115e72 is described below
commit 4115e720870cbbd700e7f9da7d2af5c3e64e832d
Author: Karen Feng <ka...@gmail.com>
AuthorDate: Mon Apr 19 12:26:10 2021 -0700
Add error message guidelines (#332)
Adds error message guidelines to the contributing guidelines, as [discussed in the developers list](http://apache-spark-developers-list.1001551.n3.nabble.com/DISCUSS-Build-error-message-guideline-td31076.html).
For brevity, I included a condensed version of the error message guideline on the main contributing page, and linked to the full guideline for more detail.
Main contributing page:
Full guideline:
---
contributing.md | 23 ++
error-message-guidelines.md | 385 +++++++++++++++++++++++
site/contributing.html | 27 ++
site/error-message-guidelines.html | 618 +++++++++++++++++++++++++++++++++++++
site/sitemap.xml | 4 +
5 files changed, 1057 insertions(+)
diff --git a/contributing.md b/contributing.md
index 43d18d7..9e8d67d 100644
--- a/contributing.md
+++ b/contributing.md
@@ -186,6 +186,29 @@ that maintainability, consistency, and code quality come first. New algorithms s
- Come with a reasonable expectation of developer support.
- Have `@Since` annotation on public classes, methods, and variables.
+<h3>Error Message Guidelines</h3>
+
+Exceptions thrown in Spark should be associated with standardized and actionable
+error messages.
+
+Error messages should answer the following questions:
+
+- **What** was the problem?
+- **Why** did the problem happen?
+- **How** can the problem be solved?
+
+When writing error messages, you should:
+
+- Use active voice
+- Avoid time-based statements, such as promises of future support
+- Use the present tense to describe the error and provide suggestions
+- Provide concrete examples if the resolution is unclear
+- Avoid sounding accusatory, judgmental, or insulting
+- Be direct
+- Do not use programming jargon in user-facing errors
+
+See the <a href="{{site.baseurl}}/error-message-guidelines.html">error message guidelines</a> for more details.
+
<h3>Code Review Criteria</h3>
Before considering how to contribute code, it's useful to understand how code is reviewed,
diff --git a/error-message-guidelines.md b/error-message-guidelines.md
new file mode 100644
index 0000000..069f39a
--- /dev/null
+++ b/error-message-guidelines.md
@@ -0,0 +1,385 @@
+---
+layout: global
+title: Error message guidelines
+type: "page singular"
+navigation:
+ weight: 5
+ show: true
+---
+
+## Error Message Guidelines
+
+This guide is a reference for composing standardized and actionable
+error messages in Apache Spark.
+
+### Include What, Why, and How
+
+Exceptions thrown from Spark should answer the Five W's and How:
+
+- **Who** encountered the problem?
+- **What** was the problem?
+- **When** did the problem happen?
+- **Where** did the problem happen?
+- **Why** did the problem happen?
+- **How** can the problem be solved?
+
+The context provided by exceptions can help answer **who** (usually the
+user), **when** (usually included in the log via <code>log4j</code>), and **where**
+(usually included in the stack trace). However, these answers alone are
+often insufficient for the user to solve the problem. An error message
+that answers the remaining questions
+--- **what**, **why**, and **how** --- minimizes user frustration.
+
+#### Explicitly answer What, Why and How
+
+In many cases, the error message should explicitly answer **what**,
+**why**, and **how**.
+
+##### Example 1
+
+<code style="white-space:normal">
+ <a href="https://github.com/apache/spark/blob/569fb133d09e24e4ed56ed7efff641512d98b01b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala#L160">
+ Unable to generate an encoder for inner class {} without access to the
+ scope that this class was defined in. Try moving this class out of its
+ parent class.
+ </a>
+</code>
+
+- **What:** Unable to generate encoder inner class.
+- **Why:** Did not have access to the scope that the class was defined in.
+- **How:** Try moving this class out of its parent class.
+
+##### Example 2
+
+If the proposed fix (**how**) feels arbitrary, providing an explanation
+for **why** the error occurred can reduce user frustration.
+
+**Before**
+
+<code style="white-space:normal">
+ <a href="https://github.com/apache/spark/blob/03dd33cc982ebb3de4354274ac49da31521b8195/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala#L498">
+ <span style="background-color: #ffcccc">Unsupported</span> function name {}.
+ </a>
+</code>
+
+- **What:** Unsupported function name.
+- **Why:** Unclear.
+- **How:** Unclear.
+
+**After**
+
+*Function name {} <span style="background-color: #ccffcc">is invalid. Temporary functions cannot belong to a
+catalog. Specify a function name with one or two parts.</span>*
+
+- **What:** Invalid function name.
+- **Why:** Temporary functions cannot belong to a catalog.
+- **How:** Specify a function name with one or two parts.
+
+#### Implicitly answer How
+
+Not all error messages should be this verbose. Sometimes, explicitly
+explaining **how** to resolve the problem would be redundant;
+you may skip an explicit explanation in this case.
+
+##### Example 1
+
+<code style="white-space:normal">
+ <a href="https://github.com/apache/spark/blob/e5d972e84e973d9a2e62312dc471df30c35269bc/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala#L63">
+ Invalid pivot column {}. Pivot columns must be comparable.
+ </a>
+</code>
+
+- **What:** Invalid pivot column.
+- **Why:** Pivot columns must be comparable.
+- **How (*****implied by Why*****):** Use comparable pivot columns.
+
+##### Example 2
+
+**Before**
+
+<code style="white-space:normal">
+ <a href="https://github.com/apache/spark/blob/9809a2f1c5187205c81542dbdc84b71db535f6e1/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala#L325">
+ Cannot specify window frame for {} function
+ </a>
+</code>
+
+- **What:** Cannot specify window frame for the function.
+- **Why**: Unclear.
+- **How:** Unclear.
+
+**After**
+
+<code style="white-space:normal">
+ Cannot specify frame for window expression {}. <span style="background-color: #ccffcc">Window expression
+ contains mismatch between function frame {} and specification frame {}.</span>
+</code>
+
+- **What:** Cannot specify the frame for the window expression.
+- **Why:** Window expression contains mismatch between function frame
+ and specification frame.
+- **How (*****implied by Why*****):** Match the function frame and
+ specification frame.
+
+##### Example 3
+
+**Before**
+
+<code style="white-space:normal">
+ <a href="https://github.com/apache/spark/blob/aff6c0febb40d9713895ba00d8c77ba00f04bd16/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala#L93">
+ Cannot parse any decimal.
+ </a>
+</code>
+
+- **What:** Cannot parse decimal.
+- **Why**: Unclear.
+- **How:** Unclear.
+
+**After**
+
+<code style="white-space:normal">
+ Invalid decimal {}; <span style="background-color: #ccffcc">encountered error while parsing at position {}.</span>
+</code>
+
+- **What:** Invalid decimal.
+- **Why**: The decimal parser encountered an error at the specified position.
+- **How (*****implied by Why*****):** Fix the error at the specified position.
+
+#### Implicitly answer Why and How
+
+Sometimes, even explicitly explaining **why** the problem happened would
+be redundant; you may skip an explicit explanation in this case.
+
+<code style="white-space:normal">
+ <a href="https://github.com/apache/spark/blob/569fb133d09e24e4ed56ed7efff641512d98b01b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala#L770">
+ Path does not exist: {}
+ </a>
+</code>
+
+- **What:** Path does not exist.
+- **Why (*****implied by What*****):** User specified an invalid path.
+- **How (*****implied by What*****):** Use a different path.
+
+### Use clear language
+
+#### Diction guide
+
+<table class="table table-hover">
+ <thead>
+ <tr>
+ <th scope="col">Phrases</th>
+ <th scope="col">When to use</th>
+ <th scope="col">Example</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <th scope="row">Unsupported</th>
+ <td>
+ The user may reasonably assume that the operation is supported, but it is not. This error may go away in the future if developers add support for the operation.
+ </td>
+ <td>
+ <code style="white-space:normal">
+ Data type {} is <span style="background-color: #ccffff">unsupported</span>.
+ </code>
+ </td>
+ </tr>
+ <tr>
+ <th rowspan="3" scope="rowgroup">
+ Invalid / Not allowed / Unexpected
+ </th>
+ <td rowspan="3">
+ The user made a mistake when specifying an operation. The message should inform the user of how to resolve the error.
+ </td>
+ <td>
+ <code style="white-space:normal">
+ Array has size {}, index {} is <span style="background-color: #ccffff">invalid</span>.
+ </code>
+ </td>
+ </tr>
+ <tr>
+ <td>
+ <code style="white-space:normal">
+ Found {} generators for the clause {}. Only one generator is <span style="background-color: #ccffff">allowed</span>.
+ </code>
+ </td>
+ </tr>
+ <tr>
+ <td>
+ <code style="white-space:normal">
+ Found an <span style="background-color: #ccffff">unexpected</span> state format version {}. Expected versions 1 or 2.
+ </code>
+ </td>
+ </tr>
+ <tr>
+ <th scope="row">Failed to</th>
+ <td>
+ The system encountered an unexpected error that cannot be reasonably attributed to user error.
+ </td>
+ <td>
+ <code style="white-space:normal">
+ <span style="background-color: #ccffff">Failed to</span> compile {}.
+ </code>
+ </td>
+ </tr>
+ <tr>
+ <th scope="row">Cannot</th>
+ <td>
+ Any time, preferably only if one of the above alternatives does not apply.
+ </td>
+ <td>
+ <code style="white-space:normal">
+ <span style="background-color: #ccffff">Cannot</span> generate code for unsupported type {}.
+ </code>
+ </td>
+ </tr>
+ </tbody>
+</table>
+
+#### Wording guide
+
+<table class="table table-hover">
+ <thead>
+ <tr>
+ <th scope="col">Best practice</th>
+ <th scope="col">Before</th>
+ <th scope="col">After</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <th scope="row">
+ Use active voice
+ </th>
+ <td>
+ <code style="white-space:normal">
+ <a href="https://github.com/apache/spark/blob/73857cdd87757d2888bd92f6b7c2fad709701484/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala#L704">
+ DataType {} is <span style="background-color: #ffcccc">not supported by</span> {}.
+ </a>
+ </code>
+ </td>
+ <td>
+ <code style="white-space:normal">
+ {} <span style="background-color: #ccffcc">does not</span> support datatype {}.
+ </code>
+ </td>
+ </tr>
+ <tr>
+ <th rowspan="2" scope="rowgroup">
+ Avoid time-based statements, such as promises of future support
+ </th>
+ <td>
+ <code style="white-space:normal">
+ <a href="https://github.com/apache/spark/blob/27bec91bc971b393bd91f2ec8c6483b33f844f12/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala#L185">
+ Pandas UDF aggregate expressions are <span style="background-color: #ffcccc">currently</span> not supported in pivot.
+ </a>
+ </code>
+ </td>
+ <td>
+ <code style="white-space:normal">
+ Pivot <span style="background-color: #ccffcc">does not</span> support Pandas UDF aggregate expressions.
+ </code>
+ </td>
+ </tr>
+ <tr>
+ <td>
+ <code style="white-space:normal">
+ <a href="https://github.com/apache/spark/blob/569fb133d09e24e4ed56ed7efff641512d98b01b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala#L1076">
+ Parquet type not <span style="background-color: #ffcccc">yet</span> supported: {}.
+ </a>
+ </code>
+ </td>
+ <td>
+ <code style="white-space:normal">
+ {} <span style="background-color: #ccffcc">does not</span> support Parquet type.
+ </code>
+ </td>
+ </tr>
+ <tr>
+ <th rowspan="2" scope="rowgroup">Use the present tense to describe the error and provide suggestions</th>
+ <td>
+ <code style="white-space:normal">
+ <a href="https://github.com/apache/spark/blob/9809a2f1c5187205c81542dbdc84b71db535f6e1/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala#L166">
+ <span style="background-color: #ffcccc">Couldn't</span> find the reference column for {} at {}.
+ </a>
+ </code>
+ </td>
+ <td>
+ <code style="white-space:normal"><span style="background-color: #ccffcc">Cannot</span> find the reference column for {} at {}.</code>
+ </td>
+ </tr>
+ <tr>
+ <td>
+ <code style="white-space:normal">
+ <a href="https://github.com/apache/spark/blob/9809a2f1c5187205c81542dbdc84b71db535f6e1/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala#L409">
+ Join strategy hint parameter <span style="background-color: #ffcccc">should be</span> an identifier or string but was {}.
+ </a>
+ </code>
+ </td>
+ <td>
+ <code style="white-space:normal">
+ Cannot use join strategy hint parameter {}. <span style="background-color: #ccffcc">Use</span> a table name or identifier to specify the parameter.
+ </code>
+ </td>
+ </tr>
+ <tr>
+ <th scope="row">Provide concrete examples if the resolution is unclear</th>
+ <td>
+ <code style="white-space:normal">
+ <a href="https://github.com/apache/spark/blob/569fb133d09e24e4ed56ed7efff641512d98b01b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala#L422">
+ {} Hint expects a partition number as a parameter.
+ </a>
+ </code>
+ </td>
+ <td>
+ <code style="white-space:normal">
+ {} Hint expects a partition number as a parameter. <span style="background-color: #ccffcc">For example, specify 3 partitions with {}(3)</span>.
+ </code>
+ </td>
+ </tr>
+ <tr>
+ <th scope="row">Avoid sounding accusatory, judgmental, or insulting</th>
+ <td>
+ <code style="white-space:normal">
+ <a href="https://github.com/apache/spark/blob/569fb133d09e24e4ed56ed7efff641512d98b01b/core/src/main/scala/org/apache/spark/resource/ResourceUtils.scala#L143">
+ <span style="background-color: #ffcccc">You must</span> specify an amount for {}.
+ </a>
+ </code>
+ </td>
+ <td>
+ <code style="white-space:normal">
+ {} cannot be empty. Specify an amount for {}.
+ </code>
+ </td>
+ </tr>
+ <tr>
+ <th scope="row">Be direct</th>
+ <td>
+ <code style="white-space:normal">
+ <a href="https://github.com/apache/spark/blob/4b5fc1da752ec008468ef80a5717c8beab468387/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala#L119">
+ LEGACY store assignment policy is disallowed in Spark data source V2. <span style="background-color: #ffcccc">Please</span> set the configuration spark.sql.storeAssignmentPolicy to <span style="background-color: #ffcccc">other values</span>.
+ </a>
+ </code>
+ </td>
+ <td>
+ <code style="white-space:normal">
+ Spark data source V2 does not allow the LEGACY store assignment policy. Set the configuration spark.sql.storeAssignment to ANSI or STRICT.
+ </code>
+ </td>
+ </tr>
+ <tr>
+ <th scope="row">Do not use programming jargon in user-facing errors</th>
+ <td>
+ <code style="white-space:normal">
+ <a href="https://github.com/apache/spark/blob/4b5fc1da752ec008468ef80a5717c8beab468387/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala#L583">
+ RENAME TABLE source and destination databases do not match: '{}' <span style="background-color: #ffcccc">!=</span> '{}'.
+ </a>
+ </code>
+ </td>
+ <td>
+ <code style="white-space:normal">
+ RENAME TABLE source and destination databases do not match. The source database is {}, but the destination database is {}.
+ </code>
+ </td>
+ </tr>
+ </tbody>
+</table>
diff --git a/site/contributing.html b/site/contributing.html
index 4bc7ca3..cde06ad 100644
--- a/site/contributing.html
+++ b/site/contributing.html
@@ -394,6 +394,33 @@ that maintainability, consistency, and code quality come first. New algorithms s
<li>Have <code class="language-plaintext highlighter-rouge">@Since</code> annotation on public classes, methods, and variables.</li>
</ul>
+<h3>Error Message Guidelines</h3>
+
+<p>Exceptions thrown in Spark should be associated with standardized and actionable
+error messages.</p>
+
+<p>Error messages should answer the following questions:</p>
+
+<ul>
+ <li><strong>What</strong> was the problem?</li>
+ <li><strong>Why</strong> did the problem happen?</li>
+ <li><strong>How</strong> can the problem be solved?</li>
+</ul>
+
+<p>When writing error messages, you should:</p>
+
+<ul>
+ <li>Use active voice</li>
+ <li>Avoid time-based statements, such as promises of future support</li>
+ <li>Use the present tense to describe the error and provide suggestions</li>
+ <li>Provide concrete examples if the resolution is unclear</li>
+ <li>Avoid sounding accusatory, judgmental, or insulting</li>
+ <li>Be direct</li>
+ <li>Do not use programming jargon in user-facing errors</li>
+</ul>
+
+<p>See the <a href="/error-message-guidelines.html">error message guidelines</a> for more details.</p>
+
<h3>Code Review Criteria</h3>
<p>Before considering how to contribute code, it’s useful to understand how code is reviewed,
diff --git a/site/error-message-guidelines.html b/site/error-message-guidelines.html
new file mode 100644
index 0000000..5eb54db
--- /dev/null
+++ b/site/error-message-guidelines.html
@@ -0,0 +1,618 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>
+ Error message guidelines | Apache Spark
+
+ </title>
+
+
+
+
+
+ <!-- Bootstrap core CSS -->
+ <link href="/css/cerulean.min.css" rel="stylesheet">
+ <link href="/css/custom.css" rel="stylesheet">
+
+ <!-- Code highlighter CSS -->
+ <link href="/css/pygments-default.css" rel="stylesheet">
+
+ <script type="text/javascript">
+ <!-- Google Analytics initialization -->
+ var _gaq = _gaq || [];
+ _gaq.push(['_setAccount', 'UA-32518208-2']);
+ _gaq.push(['_trackPageview']);
+ (function() {
+ var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
+ ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
+ var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
+ })();
+
+ <!-- Adds slight delay to links to allow async reporting -->
+ function trackOutboundLink(link, category, action) {
+ try {
+ _gaq.push(['_trackEvent', category , action]);
+ } catch(err){}
+
+ setTimeout(function() {
+ document.location.href = link.href;
+ }, 100);
+ }
+ </script>
+
+ <!-- HTML5 shim and Respond.js IE8 support of HTML5 elements and media queries -->
+ <!--[if lt IE 9]>
+ <script src="https://oss.maxcdn.com/libs/html5shiv/3.7.0/html5shiv.js"></script>
+ <script src="https://oss.maxcdn.com/libs/respond.js/1.3.0/respond.min.js"></script>
+ <![endif]-->
+</head>
+
+<body>
+
+<script src="https://code.jquery.com/jquery.js"></script>
+<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.3/js/bootstrap.min.js"></script>
+<script src="/js/lang-tabs.js"></script>
+<script src="/js/downloads.js"></script>
+
+<div class="container" style="max-width: 1200px;">
+
+<div class="masthead">
+
+ <p class="lead">
+ <a href="/">
+ <img src="/images/spark-logo-trademark.png"
+ style="height:100px; width:auto; vertical-align: bottom; margin-top: 20px;"></a><span class="tagline">
+ Lightning-fast unified analytics engine
+ </span>
+ </p>
+
+</div>
+
+<nav class="navbar navbar-default" role="navigation">
+ <!-- Brand and toggle get grouped for better mobile display -->
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle" data-toggle="collapse"
+ data-target="#navbar-collapse-1">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ </div>
+
+ <!-- Collect the nav links, forms, and other content for toggling -->
+ <div class="collapse navbar-collapse" id="navbar-collapse-1">
+ <ul class="nav navbar-nav">
+ <li><a href="/downloads.html">Download</a></li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">
+ Libraries <b class="caret"></b>
+ </a>
+ <ul class="dropdown-menu">
+ <li><a href="/sql/">SQL and DataFrames</a></li>
+ <li><a href="/streaming/">Spark Streaming</a></li>
+ <li><a href="/mllib/">MLlib (machine learning)</a></li>
+ <li><a href="/graphx/">GraphX (graph)</a></li>
+ <li class="divider"></li>
+ <li><a href="/third-party-projects.html">Third-Party Projects</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">
+ Documentation <b class="caret"></b>
+ </a>
+ <ul class="dropdown-menu">
+ <li><a href="/docs/latest/">Latest Release (Spark 3.1.1)</a></li>
+ <li><a href="/documentation.html">Older Versions and Other Resources</a></li>
+ <li><a href="/faq.html">Frequently Asked Questions</a></li>
+ </ul>
+ </li>
+ <li><a href="/examples.html">Examples</a></li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">
+ Community <b class="caret"></b>
+ </a>
+ <ul class="dropdown-menu">
+ <li><a href="/community.html">Mailing Lists & Resources</a></li>
+ <li><a href="/contributing.html">Contributing to Spark</a></li>
+ <li><a href="/improvement-proposals.html">Improvement Proposals (SPIP)</a></li>
+ <li><a href="https://issues.apache.org/jira/browse/SPARK">Issue Tracker</a></li>
+ <li><a href="/powered-by.html">Powered By</a></li>
+ <li><a href="/committers.html">Project Committers</a></li>
+ <li><a href="/history.html">Project History</a></li>
+ </ul>
+ </li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">
+ Developers <b class="caret"></b>
+ </a>
+ <ul class="dropdown-menu">
+ <li><a href="/developer-tools.html">Useful Developer Tools</a></li>
+ <li><a href="/versioning-policy.html">Versioning Policy</a></li>
+ <li><a href="/release-process.html">Release Process</a></li>
+ <li><a href="/security.html">Security</a></li>
+ </ul>
+ </li>
+ </ul>
+ <ul class="nav navbar-nav navbar-right">
+ <li class="dropdown">
+ <a href="https://www.apache.org/" class="dropdown-toggle" data-toggle="dropdown">
+ Apache Software Foundation <b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="https://www.apache.org/">Apache Homepage</a></li>
+ <li><a href="https://www.apache.org/licenses/">License</a></li>
+ <li><a href="https://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
+ <li><a href="https://www.apache.org/foundation/thanks.html">Thanks</a></li>
+ <li><a href="https://www.apache.org/security/">Security</a></li>
+ </ul>
+ </li>
+ </ul>
+ </div>
+ <!-- /.navbar-collapse -->
+</nav>
+
+
+<div class="row">
+ <div class="col-md-3 col-md-push-9">
+ <div class="news" style="margin-bottom: 20px;">
+ <h5>Latest News</h5>
+ <ul class="list-unstyled">
+
+ <li><a href="/news/spark-3-1-1-released.html">Spark 3.1.1 released</a>
+ <span class="small">(Mar 02, 2021)</span></li>
+
+ <li><a href="/news/spark-3-0-2-released.html">Spark 3.0.2 released</a>
+ <span class="small">(Feb 19, 2021)</span></li>
+
+ <li><a href="/news/next-official-release-spark-3.1.1.html">Next official release: Spark 3.1.1</a>
+ <span class="small">(Jan 07, 2021)</span></li>
+
+ <li><a href="/news/spark-2-4-7-released.html">Spark 2.4.7 released</a>
+ <span class="small">(Sep 12, 2020)</span></li>
+
+ </ul>
+ <p class="small" style="text-align: right;"><a href="/news/index.html">Archive</a></p>
+ </div>
+ <div style="text-align:center; margin-bottom: 20px;">
+ <a href="https://www.apache.org/events/current-event.html">
+ <img src="https://www.apache.org/events/current-event-234x60.png"/>
+ </a>
+ </div>
+ <div class="hidden-xs hidden-sm">
+ <a href="/downloads.html" class="btn btn-success btn-lg btn-block" style="margin-bottom: 30px;">
+ Download Spark
+ </a>
+ <p style="font-size: 16px; font-weight: 500; color: #555;">
+ Built-in Libraries:
+ </p>
+ <ul class="list-none">
+ <li><a href="/sql/">SQL and DataFrames</a></li>
+ <li><a href="/streaming/">Spark Streaming</a></li>
+ <li><a href="/mllib/">MLlib (machine learning)</a></li>
+ <li><a href="/graphx/">GraphX (graph)</a></li>
+ </ul>
+ <a href="/third-party-projects.html">Third-Party Projects</a>
+ </div>
+ </div>
+
+ <div class="col-md-9 col-md-pull-3">
+ <h2 id="error-message-guidelines">Error Message Guidelines</h2>
+
+<p>This guide is a reference for composing standardized and actionable
+error messages in Apache Spark.</p>
+
+<h3 id="include-what-why-and-how">Include What, Why, and How</h3>
+
+<p>Exceptions thrown from Spark should answer the Five W’s and How:</p>
+
+<ul>
+ <li><strong>Who</strong> encountered the problem?</li>
+ <li><strong>What</strong> was the problem?</li>
+ <li><strong>When</strong> did the problem happen?</li>
+ <li><strong>Where</strong> did the problem happen?</li>
+ <li><strong>Why</strong> did the problem happen?</li>
+ <li><strong>How</strong> can the problem be solved?</li>
+</ul>
+
+<p>The context provided by exceptions can help answer <strong>who</strong> (usually the
+user), <strong>when</strong> (usually included in the log via <code>log4j</code>), and <strong>where</strong>
+(usually included in the stack trace). However, these answers alone are
+often insufficient for the user to solve the problem. An error message
+that answers the remaining questions
+— <strong>what</strong>, <strong>why</strong>, and <strong>how</strong> — minimizes user frustration.</p>
+
+<h4 id="explicitly-answer-what-why-and-how">Explicitly answer What, Why and How</h4>
+
+<p>In many cases, the error message should explicitly answer <strong>what</strong>,
+<strong>why</strong>, and <strong>how</strong>.</p>
+
+<h5 id="example-1">Example 1</h5>
+
+<p><code style="white-space:normal">
+ <a href="https://github.com/apache/spark/blob/569fb133d09e24e4ed56ed7efff641512d98b01b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala#L160">
+ Unable to generate an encoder for inner class {} without access to the
+ scope that this class was defined in. Try moving this class out of its
+ parent class.
+ </a>
+</code></p>
+
+<ul>
+ <li><strong>What:</strong> Unable to generate encoder inner class.</li>
+ <li><strong>Why:</strong> Did not have access to the scope that the class was defined in.</li>
+ <li><strong>How:</strong> Try moving this class out of its parent class.</li>
+</ul>
+
+<h5 id="example-2">Example 2</h5>
+
+<p>If the proposed fix (<strong>how</strong>) feels arbitrary, providing an explanation
+for <strong>why</strong> the error occurred can reduce user frustration.</p>
+
+<p><strong>Before</strong></p>
+
+<p><code style="white-space:normal">
+ <a href="https://github.com/apache/spark/blob/03dd33cc982ebb3de4354274ac49da31521b8195/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala#L498">
+ <span style="background-color: #ffcccc">Unsupported</span> function name {}.
+ </a>
+</code></p>
+
+<ul>
+ <li><strong>What:</strong> Unsupported function name.</li>
+ <li><strong>Why:</strong> Unclear.</li>
+ <li><strong>How:</strong> Unclear.</li>
+</ul>
+
+<p><strong>After</strong></p>
+
+<p><em>Function name {} <span style="background-color: #ccffcc">is invalid. Temporary functions cannot belong to a
+catalog. Specify a function name with one or two parts.</span></em></p>
+
+<ul>
+ <li><strong>What:</strong> Invalid function name.</li>
+ <li><strong>Why:</strong> Temporary functions cannot belong to a catalog.</li>
+ <li><strong>How:</strong> Specify a function name with one or two parts.</li>
+</ul>
+
+<h4 id="implicitly-answer-how">Implicitly answer How</h4>
+
+<p>Not all error messages should be this verbose. Sometimes, explicitly
+explaining <strong>how</strong> to resolve the problem would be redundant;
+you may skip an explicit explanation in this case.</p>
+
+<h5 id="example-1-1">Example 1</h5>
+
+<p><code style="white-space:normal">
+ <a href="https://github.com/apache/spark/blob/e5d972e84e973d9a2e62312dc471df30c35269bc/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala#L63">
+ Invalid pivot column {}. Pivot columns must be comparable.
+ </a>
+</code></p>
+
+<ul>
+ <li><strong>What:</strong> Invalid pivot column.</li>
+ <li><strong>Why:</strong> Pivot columns must be comparable.</li>
+ <li><strong>How (</strong><strong><em>implied by Why</em></strong><strong>):</strong> Use comparable pivot columns.</li>
+</ul>
+
+<h5 id="example-2-1">Example 2</h5>
+
+<p><strong>Before</strong></p>
+
+<p><code style="white-space:normal">
+ <a href="https://github.com/apache/spark/blob/9809a2f1c5187205c81542dbdc84b71db535f6e1/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala#L325">
+ Cannot specify window frame for {} function
+ </a>
+</code></p>
+
+<ul>
+ <li><strong>What:</strong> Cannot specify window frame for the function.</li>
+ <li><strong>Why</strong>: Unclear.</li>
+ <li><strong>How:</strong> Unclear.</li>
+</ul>
+
+<p><strong>After</strong></p>
+
+<p><code style="white-space:normal">
+ Cannot specify frame for window expression {}. <span style="background-color: #ccffcc">Window expression
+ contains mismatch between function frame {} and specification frame {}.</span>
+</code></p>
+
+<ul>
+ <li><strong>What:</strong> Cannot specify the frame for the window expression.</li>
+ <li><strong>Why:</strong> Window expression contains mismatch between function frame
+and specification frame.</li>
+ <li><strong>How (</strong><strong><em>implied by Why</em></strong><strong>):</strong> Match the function frame and
+specification frame.</li>
+</ul>
+
+<h5 id="example-3">Example 3</h5>
+
+<p><strong>Before</strong></p>
+
+<p><code style="white-space:normal">
+ <a href="https://github.com/apache/spark/blob/aff6c0febb40d9713895ba00d8c77ba00f04bd16/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala#L93">
+ Cannot parse any decimal.
+ </a>
+</code></p>
+
+<ul>
+ <li><strong>What:</strong> Cannot parse decimal.</li>
+ <li><strong>Why</strong>: Unclear.</li>
+ <li><strong>How:</strong> Unclear.</li>
+</ul>
+
+<p><strong>After</strong></p>
+
+<p><code style="white-space:normal">
+ Invalid decimal {}; <span style="background-color: #ccffcc">encountered error while parsing at position {}.</span>
+</code></p>
+
+<ul>
+ <li><strong>What:</strong> Invalid decimal.</li>
+ <li><strong>Why</strong>: The decimal parser encountered an error at the specified position.</li>
+ <li><strong>How (</strong><strong><em>implied by Why</em></strong><strong>):</strong> Fix the error at the specified position.</li>
+</ul>
+
+<h4 id="implicitly-answer-why-and-how">Implicitly answer Why and How</h4>
+
+<p>Sometimes, even explicitly explaining <strong>why</strong> the problem happened would
+be redundant; you may skip an explicit explanation in this case.</p>
+
+<p><code style="white-space:normal">
+ <a href="https://github.com/apache/spark/blob/569fb133d09e24e4ed56ed7efff641512d98b01b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala#L770">
+ Path does not exist: {}
+ </a>
+</code></p>
+
+<ul>
+ <li><strong>What:</strong> Path does not exist.</li>
+ <li><strong>Why (</strong><strong><em>implied by What</em></strong><strong>):</strong> User specified an invalid path.</li>
+ <li><strong>How (</strong><strong><em>implied by What</em></strong><strong>):</strong> Use a different path.</li>
+</ul>
+
+<h3 id="use-clear-language">Use clear language</h3>
+
+<h4 id="diction-guide">Diction guide</h4>
+
+<table class="table table-hover">
+ <thead>
+ <tr>
+ <th scope="col">Phrases</th>
+ <th scope="col">When to use</th>
+ <th scope="col">Example</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <th scope="row">Unsupported</th>
+ <td>
+ The user may reasonably assume that the operation is supported, but it is not. This error may go away in the future if developers add support for the operation.
+ </td>
+ <td>
+ <code style="white-space:normal">
+ Data type {} is <span style="background-color: #ccffff">unsupported</span>.
+ </code>
+ </td>
+ </tr>
+ <tr>
+ <th rowspan="3" scope="rowgroup">
+ Invalid / Not allowed / Unexpected
+ </th>
+ <td rowspan="3">
+ The user made a mistake when specifying an operation. The message should inform the user of how to resolve the error.
+ </td>
+ <td>
+ <code style="white-space:normal">
+ Array has size {}, index {} is <span style="background-color: #ccffff">invalid</span>.
+ </code>
+ </td>
+ </tr>
+ <tr>
+ <td>
+ <code style="white-space:normal">
+ Found {} generators for the clause {}. Only one generator is <span style="background-color: #ccffff">allowed</span>.
+ </code>
+ </td>
+ </tr>
+ <tr>
+ <td>
+ <code style="white-space:normal">
+ Found an <span style="background-color: #ccffff">unexpected</span> state format version {}. Expected versions 1 or 2.
+ </code>
+ </td>
+ </tr>
+ <tr>
+ <th scope="row">Failed to</th>
+ <td>
+ The system encountered an unexpected error that cannot be reasonably attributed to user error.
+ </td>
+ <td>
+ <code style="white-space:normal">
+ <span style="background-color: #ccffff">Failed to</span> compile {}.
+ </code>
+ </td>
+ </tr>
+ <tr>
+ <th scope="row">Cannot</th>
+ <td>
+ Any time, preferably only if one of the above alternatives does not apply.
+ </td>
+ <td>
+ <code style="white-space:normal">
+ <span style="background-color: #ccffff">Cannot</span> generate code for unsupported type {}.
+ </code>
+ </td>
+ </tr>
+ </tbody>
+</table>
+
+<h4 id="wording-guide">Wording guide</h4>
+
+<table class="table table-hover">
+ <thead>
+ <tr>
+ <th scope="col">Best practice</th>
+ <th scope="col">Before</th>
+ <th scope="col">After</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <th scope="row">
+ Use active voice
+ </th>
+ <td>
+ <code style="white-space:normal">
+ <a href="https://github.com/apache/spark/blob/73857cdd87757d2888bd92f6b7c2fad709701484/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala#L704">
+ DataType {} is <span style="background-color: #ffcccc">not supported by</span> {}.
+ </a>
+ </code>
+ </td>
+ <td>
+ <code style="white-space:normal">
+ {} <span style="background-color: #ccffcc">does not</span> support datatype {}.
+ </code>
+ </td>
+ </tr>
+ <tr>
+ <th rowspan="2" scope="rowgroup">
+ Avoid time-based statements, such as promises of future support
+ </th>
+ <td>
+ <code style="white-space:normal">
+ <a href="https://github.com/apache/spark/blob/27bec91bc971b393bd91f2ec8c6483b33f844f12/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala#L185">
+ Pandas UDF aggregate expressions are <span style="background-color: #ffcccc">currently</span> not supported in pivot.
+ </a>
+ </code>
+ </td>
+ <td>
+ <code style="white-space:normal">
+ Pivot <span style="background-color: #ccffcc">does not</span> support Pandas UDF aggregate expressions.
+ </code>
+ </td>
+ </tr>
+ <tr>
+ <td>
+ <code style="white-space:normal">
+ <a href="https://github.com/apache/spark/blob/569fb133d09e24e4ed56ed7efff641512d98b01b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala#L1076">
+ Parquet type not <span style="background-color: #ffcccc">yet</span> supported: {}.
+ </a>
+ </code>
+ </td>
+ <td>
+ <code style="white-space:normal">
+ {} <span style="background-color: #ccffcc">does not</span> support Parquet type.
+ </code>
+ </td>
+ </tr>
+ <tr>
+ <th rowspan="2" scope="rowgroup">Use the present tense to describe the error and provide suggestions</th>
+ <td>
+ <code style="white-space:normal">
+ <a href="https://github.com/apache/spark/blob/9809a2f1c5187205c81542dbdc84b71db535f6e1/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala#L166">
+ <span style="background-color: #ffcccc">Couldn't</span> find the reference column for {} at {}.
+ </a>
+ </code>
+ </td>
+ <td>
+ <code style="white-space:normal"><span style="background-color: #ccffcc">Cannot</span> find the reference column for {} at {}.</code>
+ </td>
+ </tr>
+ <tr>
+ <td>
+ <code style="white-space:normal">
+ <a href="https://github.com/apache/spark/blob/9809a2f1c5187205c81542dbdc84b71db535f6e1/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala#L409">
+ Join strategy hint parameter <span style="background-color: #ffcccc">should be</span> an identifier or string but was {}.
+ </a>
+ </code>
+ </td>
+ <td>
+ <code style="white-space:normal">
+ Cannot use join strategy hint parameter {}. <span style="background-color: #ccffcc">Use</span> a table name or identifier to specify the parameter.
+ </code>
+ </td>
+ </tr>
+ <tr>
+ <th scope="row">Provide concrete examples if the resolution is unclear</th>
+ <td>
+ <code style="white-space:normal">
+ <a href="https://github.com/apache/spark/blob/569fb133d09e24e4ed56ed7efff641512d98b01b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala#L422">
+ {} Hint expects a partition number as a parameter.
+ </a>
+ </code>
+ </td>
+ <td>
+ <code style="white-space:normal">
+ {} Hint expects a partition number as a parameter. <span style="background-color: #ccffcc">For example, specify 3 partitions with {}(3)</span>.
+ </code>
+ </td>
+ </tr>
+ <tr>
+ <th scope="row">Avoid sounding accusatory, judgmental, or insulting</th>
+ <td>
+ <code style="white-space:normal">
+ <a href="https://github.com/apache/spark/blob/569fb133d09e24e4ed56ed7efff641512d98b01b/core/src/main/scala/org/apache/spark/resource/ResourceUtils.scala#L143">
+ <span style="background-color: #ffcccc">You must</span> specify an amount for {}.
+ </a>
+ </code>
+ </td>
+ <td>
+ <code style="white-space:normal">
+ {} cannot be empty. Specify an amount for {}.
+ </code>
+ </td>
+ </tr>
+ <tr>
+ <th scope="row">Be direct</th>
+ <td>
+ <code style="white-space:normal">
+ <a href="https://github.com/apache/spark/blob/4b5fc1da752ec008468ef80a5717c8beab468387/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala#L119">
+ LEGACY store assignment policy is disallowed in Spark data source V2. <span style="background-color: #ffcccc">Please</span> set the configuration spark.sql.storeAssignmentPolicy to <span style="background-color: #ffcccc">other values</span>.
+ </a>
+ </code>
+ </td>
+ <td>
+ <code style="white-space:normal">
+ Spark data source V2 does not allow the LEGACY store assignment policy. Set the configuration spark.sql.storeAssignment to ANSI or STRICT.
+ </code>
+ </td>
+ </tr>
+ <tr>
+ <th scope="row">Do not use programming jargon in user-facing errors</th>
+ <td>
+ <code style="white-space:normal">
+ <a href="https://github.com/apache/spark/blob/4b5fc1da752ec008468ef80a5717c8beab468387/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala#L583">
+ RENAME TABLE source and destination databases do not match: '{}' <span style="background-color: #ffcccc">!=</span> '{}'.
+ </a>
+ </code>
+ </td>
+ <td>
+ <code style="white-space:normal">
+ RENAME TABLE source and destination databases do not match. The source database is {}, but the destination database is {}.
+ </code>
+ </td>
+ </tr>
+ </tbody>
+</table>
+
+ </div>
+</div>
+
+
+
+<footer class="small">
+ <hr>
+ Apache Spark, Spark, Apache, the Apache feather logo, and the Apache Spark project logo are either registered
+ trademarks or trademarks of The Apache Software Foundation in the United States and other countries.
+ See guidance on use of Apache Spark <a href="/trademarks.html">trademarks</a>.
+ All other marks mentioned may be trademarks or registered trademarks of their respective owners.
+ Copyright © 2018 The Apache Software Foundation, Licensed under the
+ <a href="https://www.apache.org/licenses/">Apache License, Version 2.0</a>.
+</footer>
+
+</div>
+
+</body>
+</html>
diff --git a/site/sitemap.xml b/site/sitemap.xml
index dc08efc..cdd113b 100644
--- a/site/sitemap.xml
+++ b/site/sitemap.xml
@@ -852,6 +852,10 @@
<changefreq>weekly</changefreq>
</url>
<url>
+ <loc>https://spark.apache.org/error-message-guidelines.html</loc>
+ <changefreq>weekly</changefreq>
+</url>
+<url>
<loc>https://spark.apache.org/examples.html</loc>
<changefreq>weekly</changefreq>
</url>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org