[jira] [Created] (FLINK-35989) Document and log error on partially failed requests for AWS sinks

["Hong Liang Teoh (Jira)" <ji...@apache.org>]

Hong Liang Teoh created FLINK-35989:
---------------------------------------

             Summary: Document and log error on partially failed requests for AWS sinks
                 Key: FLINK-35989
                 URL: https://issues.apache.org/jira/browse/FLINK-35989
             Project: Flink
          Issue Type: Improvement
          Components: Connectors / DynamoDB, Connectors / Firehose, Connectors / Kinesis
    Affects Versions: aws-connector-4.3.0
            Reporter: Hong Liang Teoh
             Fix For: aws-connector-4.4.0


h2. Problem

Currently, AWS sinks send batch write requests to endpoints.

On partial failure within the batch (only some records fail), we perform retries or fail the job, but we don't log the error returned.
{code:java}
    private void handlePartiallyFailedRequest(
            PutRecordsResponse response,
            List<PutRecordsRequestEntry> requestEntries,
            Consumer<List<PutRecordsRequestEntry>> requestResult) {
        LOG.warn(
                "KDS Sink failed to write and will retry {} entries to KDS",
                response.failedRecordCount());
        numRecordsOutErrorsCounter.inc(response.failedRecordCount());        if (failOnError) {
            getFatalExceptionCons()
                    .accept(new KinesisStreamsException.KinesisStreamsFailFastException());
            return;
        }
        List<PutRecordsRequestEntry> failedRequestEntries =
                new ArrayList<>(response.failedRecordCount());
        List<PutRecordsResultEntry> records = response.records();        for (int i = 0; i < records.size(); i++) {
            if (records.get(i).errorCode() != null) {
                failedRequestEntries.add(requestEntries.get(i));
            }
        }        requestResult.accept(failedRequestEntries);
    }{code}
Code link: [https://github.com/apache/flink-connector-aws/blob/main/flink-connector-aws/flink-connector-aws-kinesis-streams/src/main/java/org/apache/flink/connector/kinesis/sink/KinesisStreamsSinkWriter.java#L254-L274]
h2. What we want to do

We should log the reason for failure to assist user debugging.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)