You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2020/06/12 15:23:40 UTC
[GitHub] [kafka] rhauch commented on a change in pull request #8858: KAFKA-10153: Error Reporting in Connect Documentation
rhauch commented on a change in pull request #8858:
URL: https://github.com/apache/kafka/pull/8858#discussion_r439467273
##########
File path: docs/connect.html
##########
@@ -258,6 +258,29 @@ <h4><a id="connect_rest" href="#connect_rest">REST API</a></h4>
<li><code>GET /</code>- return basic information about the Kafka Connect cluster such as the version of the Connect worker that serves the REST request (including git commit ID of the source code) and the Kafka cluster ID that is connected to.
</ul>
+ <h4><a id="connect_errorreporting" href="#connect_errorreporting">Error Reporting in Connect</a></h4>
+
+ <p>Kafka Connect provides error reporting for errors encountered along various stages of processing in Kafka Connect. A connector can be configured to enable log reporting, which logs the error context along with the standard application logs, as well as to enable dead letter queue reporting (within sink connectors only), which will write the original record (from the Kafka topic the sink connector is consuming from) into a configurable Kafka topic. Configuring these reporters will enable for automatic error reporting along both the conversion and transformation stages of Connect. For error reporting for records after they reach the connector, the <code>Errant Record Reporter</code> must be implemented within the specific connector along with the configuration.</p>
+
+ <h5><a id="connect_errantrecordreporting" href="#connect-errantrecordreporting">Errant Record Reporter</a></h5>
+
+ <p>The <code>ErrantRecordReporter</code> can be initialized within the <code>SinkTask</code> while preserving backwards compatibility with the following example:</p>
+
+ <pre class="brush: java;">
+ private ErrantRecordReporter reporter;
+
+ @Override
+ public void start(Map<String, String> props) {
+ ...
+ try {
+ reporter = context.failedRecordReporter(); // may be null if DLQ not enabled
+ } catch (NoSuchMethodException | NoClassDefFoundError e) {
+ // Will occur in Connect runtimes earlier than 2.6
+ reporter = null;
+ }
+ }
+ </pre>
+
Review comment:
Let's add a new section just before "Resuming from Previous Offsets" that talks about how sink tasks can use the error reporter, similar to the ["Example Usage" in the KIP](https://cwiki.apache.org/confluence/display/KAFKA/KIP-610%3A+Error+Reporting+in+Sink+Connectors#KIP610:ErrorReportinginSinkConnectors-ExampleUsage), but don't show using the returned future, since the guarantees provided by the framework about completing the writing of the errors are likely good enough for *most* connectors, and developers of any more sophisticated connector that needs stronger guarantees can read the JavaDoc.
##########
File path: docs/connect.html
##########
@@ -258,6 +258,29 @@ <h4><a id="connect_rest" href="#connect_rest">REST API</a></h4>
<li><code>GET /</code>- return basic information about the Kafka Connect cluster such as the version of the Connect worker that serves the REST request (including git commit ID of the source code) and the Kafka cluster ID that is connected to.
</ul>
+ <h4><a id="connect_errorreporting" href="#connect_errorreporting">Error Reporting in Connect</a></h4>
+
+ <p>Kafka Connect provides error reporting for errors encountered along various stages of processing in Kafka Connect. A connector can be configured to enable log reporting, which logs the error context along with the standard application logs, as well as to enable dead letter queue reporting (within sink connectors only), which will write the original record (from the Kafka topic the sink connector is consuming from) into a configurable Kafka topic. Configuring these reporters will enable for automatic error reporting along both the conversion and transformation stages of Connect. For error reporting for records after they reach the connector, the <code>Errant Record Reporter</code> must be implemented within the specific connector along with the configuration.</p>
Review comment:
A few suggestions for readability and to add some information that could have been added with [KIP-298](https://cwiki.apache.org/confluence/display/KAFKA/KIP-298%3A+Error+Handling+in+Connect):
```suggestion
<p>Kafka Connect provides error reporting to handle errors encountered along various stages of processing. By default, any error encountered during conversion or within transformations will cause the connector to fail. Each connector configuration can also enable tolerating such errors by skipping them, optionally writing each error and the details of the failed operation and problematic record (with various levels of detail) to the Connect application log. These mechanisms also capture errors when a sink connector is processing the messages consumed from its Kafka topics, and all of the errors can be written to a configurable "dead letter queue" (DLQ) Kafka topic.</p>
<p>To report errors to the log and/or DLQ, set <code>errors.tolerance=all</code> in the connector configuration. Likewise, set <code>errors.log.enable=true</code> in the connector configuration to log details of each error and problem record's topic, partition, and offset. For additional debugging purposes, set <code>errors.log.include.messages=true</code> to also log the problem record key, value, and headers to the log (note this may log sensitive information).
<p>To report errors within a sink connector's converter, transforms, or in the connector itself to a dead letter queue topic, set <code>errors.deadletterqueue.topic.name</code>, and optionally <code>errors.deadletterqueue.context.headers.enable=true</code>.</p>
<p>For example, a connector will fail immediately upon an error or exception. Although it is not necessary to add extra configuration properties for this behavior, adding the following properties to a sink connector configuration would achieve this "fail fast" behavior:</p>
<pre class="brush: text;">
# disable retries on failure
errors.retry.timeout=0
# do not log the error and their contexts
errors.log.enable=false
# do not record errors in a dead letter queue topic
errors.deadletterqueue.topic.name=
# Fail on first error
errors.tolerance=none
</pre>
<p>The following configuration shows how to setup error handling with multiple retries, logging both to the application logs and a Kafka topic with infinite tolerance:</p>
<pre class="brush: text;">
# retry for at most 10 minutes times waiting up to 30 seconds between consecutive failures
errors.retry.timeout=600000
errors.retry.delay.max.ms=30000
# log error context along with application logs, but do not include configs and messages
errors.log.enable=true
errors.log.include.messages=false
# produce error context into the Kafka topic
errors.deadletterqueue.topic.name=my-connector-errors
# Tolerate all errors.
errors.tolerance=all
</pre>
```
@aakashnshah, @wicknicks: what do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org