[jira] [Commented] (NIFI-6724) PutSQS does not check for individual status in batch message

["Pierre Villard (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-6724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16978627#comment-16978627 ] 

Pierre Villard commented on NIFI-6724:
--------------------------------------

Submitted a PR. Also updated DeleteSQS where the same remark applies. I am a bit surprised to see that the batch API is used as we are processing one flow file at a time. We could definitely go for the direct API but didn't want to change too much code. Could be done with another Jira to improve the SQS processor and really leverage the batch API.

> PutSQS does not check for individual status in batch message
> ------------------------------------------------------------
>
>                 Key: NIFI-6724
>                 URL: https://issues.apache.org/jira/browse/NIFI-6724
>             Project: Apache NiFi
>          Issue Type: Bug
>          Components: Extensions
>    Affects Versions: 1.9.2
>         Environment: Docker running in AWS
>            Reporter: Martin
>            Priority: Critical
>              Labels: Bug, PutSQS, SQS
>   Original Estimate: 2h
>          Time Spent: 10m
>  Remaining Estimate: 1h 50m
>
> PutSQS does not check for individual status in batch message. This could lead to PutSQS reporting success and dropping files even when individual files actually failed in SQS. 
> As specified in the AWSSimpleQueueService API Reference for [SendMessageBatch|https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_SendMessageBatch.html]:
> {quote}The result of sending each message is reported individually in the response. Because the batch request can result in a combination of successful and unsuccessful actions, *you should check for batch errors even when the call returns an HTTP status code of 200.*
> {quote}
>  
> We learned this as we did not Base64-encode compressed files (gzip) before sending them to PutSQS. All the compressed files failed to queue in SQS, but SQS acknowledged the batch message and reported status 200 (success). Thus PutSQS also reported success and dropped the files, when they actually failed. 
> I suggest a simple fix by implementing [SendMessageBatchResult|https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/sqs/model/SendMessageBatchResult.html] to check status on individual files by implementing something similar to this
> {code:java}
> SendMessageBatchResult sendMessageBatchResult = sqsClient.sendMessageBatch(request);
>                 if (!sendMessageBatchResult.getFailed().isEmpty()) {
>                     // Error
>                 }
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)