[jira] [Created] (FLINK-29854) Make Record Size Flush Strategy Optional for Async Sink

["Danny Cranmer (Jira)" <ji...@apache.org>]

Danny Cranmer created FLINK-29854:
-------------------------------------

             Summary: Make Record Size Flush Strategy Optional for Async Sink
                 Key: FLINK-29854
                 URL: https://issues.apache.org/jira/browse/FLINK-29854
             Project: Flink
          Issue Type: Improvement
          Components: Connectors / Common
            Reporter: Danny Cranmer


h3. Background

Currently AsyncSinkWriter supports three mechanisms that trigger a flush to the destination:
 * TIme based 
 * Batch size in bytes
 * Number of records in the batch

For "batch size in bytes" one must implement [getSizeInBytes|https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/sink/writer/AsyncSinkWriter.java#L202] in order for the base to calculate the total batch size. In some cases computing the batch size within the AsyncSinkWriter is an expensive operation, or not possible. For example, the DynamoDB connector needs to determined the serialized size of {{DynamoDbWriteRequest}}.

h3. Scope

Add a feature to make "size in bytes" support optional, this includes:
- Connectors will not be required to implement {{getSizeInBytes}}
- Batches will not be validated for max size
- Records will not be validated size

The sink implementer can decide if it is appropriate to enable this feature.







--
This message was sent by Atlassian Jira
(v8.20.10#820010)