You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/04/28 10:42:17 UTC

[GitHub] [pulsar] RobertIndie opened a new pull request #10420: [Kinesis]Fix kinesis sink can not retry to send messages

RobertIndie opened a new pull request #10420:
URL: https://github.com/apache/pulsar/pull/10420


   
   ### Motivation
   
   Currently, when the kinesis sink connector fails to send a message, it will not retry. In this case, if `retainOrdering` is enabled, it will lead to subsequent messages can not be sent like the following:
   > 17:09:40.923 [crm/messaging-service/messaging-service-reply-0] WARN  org.apache.pulsar.io.kinesis.KinesisSink - Skip acking message to retain ordering with previous failed message prod_extapi.reply.message-Optional[26380226003034]
   
   
   ### Modifications
   
   * Add retry logic for the kinesis sink connector. When sending a message fails, it will retry to send.
   
   ### Verify
   
   * Disconnect the network
   * Send a message to the input topic of the kinesis sink connector. It will output logs:
   > 18:12:58.448 [public/default/pulsar-kinesis-sink-0] DEBUG org.apache.pulsar.io.kinesis.KinesisSink - Published message to kinesis stream test-io with size 3
   > 18:13:16.619 [kpl-daemon-0003] ERROR com.amazonaws.services.kinesis.producer.LogInputStreamReader - [2021-04-28 18:13:16.618959] [0x00007b57][0x0000700009bf1000] [error] [AWS Log: ERROR]\(CurlHttpClient)Curl returned error code 28 - Timeout was reached
   18:13:16.619 [kpl-daemon-0003] ERROR com.amazonaws.services.kinesis.producer.LogInputStreamReader - [2021-04-28 18:13:16.619002] [0x00007b57][0x0000700009bf1000] [error] [AWS Log: ERROR]\(AWSClient)HTTP response code: -1
   Exception name: 
   Error message: Unable to connect to endpoint
   0 response headers:
   > ...
   * When the internal retry of `KinesisProducer` fails, it will go into the `onFailure` method of `ProducerSendCallback` and start retry.
   > 18:13:28.742 [pool-2-thread-1] INFO  org.apache.pulsar.io.kinesis.KinesisSink - [test-io] Retry to publish message for replicator of Optional[persistent://public/default/test-kinesis-0]-Optional[2684354588] after 1000 ms.
   > ...
   > 18:14:07.141 [pool-2-thread-1] INFO  org.apache.pulsar.io.kinesis.KinesisSink - [test-io] Retry to publish message for replicator of Optional[persistent://public/default/test-kinesis-0]-Optional[2684354588] after 1914 ms.
   > ...
   > 18:14:39.385 [pool-2-thread-1] INFO  org.apache.pulsar.io.kinesis.KinesisSink - [test-io] Retry to publish message for replicator of Optional[persistent://public/default/test-kinesis-0]-Optional[2684354588] after 3713 ms.
   * Reconnect the network. It will send the message successfully.
   > 18:14:47.102 [kpl-callback-pool-0-thread-0] DEBUG org.apache.pulsar.io.kinesis.KinesisSink - Successfully published message for test-io-shardId-000000000000 with latency 108652
   * It can also successfully send subsequent messages
   > 18:14:52.166 [public/default/pulsar-kinesis-sink-0] DEBUG org.apache.pulsar.io.kinesis.KinesisSink - Published message to kinesis stream test-io with size 3
   18:14:52.805 [kpl-callback-pool-0-thread-0] DEBUG org.apache.pulsar.io.kinesis.KinesisSink - Successfully published message for test-io-shardId-000000000000 with latency 640
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] RobertIndie commented on pull request #10420: [Kinesis]Fix kinesis sink can not retry to send messages

Posted by GitBox <gi...@apache.org>.
RobertIndie commented on pull request #10420:
URL: https://github.com/apache/pulsar/pull/10420#issuecomment-829228354


   Here I did some stress tests:
   * Start the sink connector with `retainOrdering` enabled.
   ![image](https://user-images.githubusercontent.com/16974619/116555693-40c2d700-a92f-11eb-9c35-97213f4cddde.png)
   * Disconnect the network to let the kinesis producer fails to send messages.
   * Produce 1 million messages with rate 100000/s.
   ![image](https://user-images.githubusercontent.com/16974619/116555868-6fd94880-a92f-11eb-8252-b70c9a31e2c0.png)
   * The sink connector will skip all the subsequent messages to retain ordering.
   ![image](https://user-images.githubusercontent.com/16974619/116555963-87183600-a92f-11eb-8fd0-ff24f8e28033.png)
   * The memory of sink connector has not grown significantly.
   
   When `retainOrdering` is disabled, there will be no retry. The changes made by this PR will not affect this case.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] dlg99 commented on pull request #10420: [Kinesis]Fix kinesis sink can not retry to send messages

Posted by GitBox <gi...@apache.org>.
dlg99 commented on pull request #10420:
URL: https://github.com/apache/pulsar/pull/10420#issuecomment-828858706


   If I understood this approach correctly and to my best understanding of how the sink works, it will end up accumulating data in memory until the connection with kinesis is restored. If that takes too long, the PulsarSource will keep producing the data for the sink and eventually this will OOM. it needs a way (through the context?) to tell PulsarSource to pause. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] eolivelli commented on pull request #10420: [Kinesis]Fix kinesis sink can not retry to send messages

Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #10420:
URL: https://github.com/apache/pulsar/pull/10420#issuecomment-829443606


   Okay. Feel free to cherry pick. Otherwise I will do it tomorrow


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] eolivelli commented on pull request #10420: [Kinesis]Fix kinesis sink can not retry to send messages

Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #10420:
URL: https://github.com/apache/pulsar/pull/10420#issuecomment-828726239


   @dlg99 PTAL


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] RobertIndie commented on a change in pull request #10420: [Kinesis]Fix kinesis sink can not retry to send messages

Posted by GitBox <gi...@apache.org>.
RobertIndie commented on a change in pull request #10420:
URL: https://github.com/apache/pulsar/pull/10420#discussion_r622801176



##########
File path: pulsar-io/kinesis/src/main/java/org/apache/pulsar/io/kinesis/KinesisSink.java
##########
@@ -121,10 +130,8 @@ public void write(Record<byte[]> record) throws Exception {
                 ? partitionedKey.substring(0, maxPartitionedKeyLength - 1)
                 : partitionedKey; // partitionedKey Length must be at least one, and at most 256
         ByteBuffer data = createKinesisMessage(kinesisSinkConfig.getMessageFormat(), record);
-        ListenableFuture<UserRecordResult> addRecordResult = kinesisProducer.addUserRecord(this.streamName,
-                partitionedKey, data);
-        addCallback(addRecordResult,
-                ProducerSendCallback.create(this, record, System.nanoTime()), directExecutor());
+        Backoff backoff = new Backoff(1, TimeUnit.SECONDS, 1, TimeUnit.MINUTES, 0, TimeUnit.SECONDS);

Review comment:
       Thanks! I have fixed it.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] dlg99 edited a comment on pull request #10420: [Kinesis]Fix kinesis sink can not retry to send messages

Posted by GitBox <gi...@apache.org>.
dlg99 edited a comment on pull request #10420:
URL: https://github.com/apache/pulsar/pull/10420#issuecomment-828858706


   If I understood this approach correctly and to my best understanding of how the sink works, it will end up accumulating data in memory until the connection with kinesis is restored. If that takes too long, the PulsarSource will keep producing the data for the sink and eventually this will OOM. it needs a way (through the context?) to tell PulsarSource to pause. 
   
   I suggest trying your test with "Send a million messages to the input topic of the kinesis sink connector." and monitoring jvm's memory


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] codelipenghui merged pull request #10420: [Kinesis]Fix kinesis sink can not retry to send messages

Posted by GitBox <gi...@apache.org>.
codelipenghui merged pull request #10420:
URL: https://github.com/apache/pulsar/pull/10420


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] codelipenghui commented on pull request #10420: [Kinesis]Fix kinesis sink can not retry to send messages

Posted by GitBox <gi...@apache.org>.
codelipenghui commented on pull request #10420:
URL: https://github.com/apache/pulsar/pull/10420#issuecomment-829091168


   @dlg99 @RobertIndie We can leverage the max unack message limitation to limit the memory of the connector when using `retainOrdering=false`, it needs users to use the Shared subscription for the connector. For `retainedOrdering=true`, we only retry the current message until success, so this will not introduce OOM issues.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] sijie commented on pull request #10420: [Kinesis]Fix kinesis sink can not retry to send messages

Posted by GitBox <gi...@apache.org>.
sijie commented on pull request #10420:
URL: https://github.com/apache/pulsar/pull/10420#issuecomment-829439993


   @eolivelli @codelipenghui I marked this issue for 2.7.2


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] codelipenghui commented on pull request #10420: [Kinesis]Fix kinesis sink can not retry to send messages

Posted by GitBox <gi...@apache.org>.
codelipenghui commented on pull request #10420:
URL: https://github.com/apache/pulsar/pull/10420#issuecomment-829233360


   @dlg99 Please help take a look.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] codelipenghui commented on a change in pull request #10420: [Kinesis]Fix kinesis sink can not retry to send messages

Posted by GitBox <gi...@apache.org>.
codelipenghui commented on a change in pull request #10420:
URL: https://github.com/apache/pulsar/pull/10420#discussion_r622081653



##########
File path: pulsar-io/kinesis/src/main/java/org/apache/pulsar/io/kinesis/KinesisSink.java
##########
@@ -121,10 +130,8 @@ public void write(Record<byte[]> record) throws Exception {
                 ? partitionedKey.substring(0, maxPartitionedKeyLength - 1)
                 : partitionedKey; // partitionedKey Length must be at least one, and at most 256
         ByteBuffer data = createKinesisMessage(kinesisSinkConfig.getMessageFormat(), record);
-        ListenableFuture<UserRecordResult> addRecordResult = kinesisProducer.addUserRecord(this.streamName,
-                partitionedKey, data);
-        addCallback(addRecordResult,
-                ProducerSendCallback.create(this, record, System.nanoTime()), directExecutor());
+        Backoff backoff = new Backoff(1, TimeUnit.SECONDS, 1, TimeUnit.MINUTES, 0, TimeUnit.SECONDS);

Review comment:
       And it's better to make the initial timeout and max timeout configuable

##########
File path: pulsar-io/kinesis/src/main/java/org/apache/pulsar/io/kinesis/KinesisSink.java
##########
@@ -121,10 +130,8 @@ public void write(Record<byte[]> record) throws Exception {
                 ? partitionedKey.substring(0, maxPartitionedKeyLength - 1)
                 : partitionedKey; // partitionedKey Length must be at least one, and at most 256
         ByteBuffer data = createKinesisMessage(kinesisSinkConfig.getMessageFormat(), record);
-        ListenableFuture<UserRecordResult> addRecordResult = kinesisProducer.addUserRecord(this.streamName,
-                partitionedKey, data);
-        addCallback(addRecordResult,
-                ProducerSendCallback.create(this, record, System.nanoTime()), directExecutor());
+        Backoff backoff = new Backoff(1, TimeUnit.SECONDS, 1, TimeUnit.MINUTES, 0, TimeUnit.SECONDS);

Review comment:
       Do we need to create a Backoff instance for each data write process? We can just reuse the object by using the reset() method to reset the backoff?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org