[jira] [Updated] (SAMZA-1392) KafkaSystemProducer performance and correctness with concurrent sends and flushes

["Jake Maes (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SAMZA-1392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jake Maes updated SAMZA-1392:
-----------------------------
    Attachment: Producer Performance Tests for SAMZA-1392 - Sheet1.pdf

Attaching some performance test results for the patch. 

The results look pretty good across the board. As expected, the biggest improvement was with compression and multithreading enabled, but it's slightly faster all around. The fix to the flush logic (first bullet point in the list below) should have slowed us down a bit, so that must have been offset by the other improvements in the change. 

I’m a little snowblind from looking at the spreadsheet for the past few days, so if you spot any anomalies, let me know. :-)

Background:
In short, we wanted to remove the sendLock that we use to get the latestFuture because the KafkaProducer does compression and serialization in the user thread, which causes contention on that lock when multithreading is enabled. 

Recall that we were tracking the latestFuture and waiting on it for flush(). That code was written at a time before Kafka's flush() was available/working. Since the Kafka flush() operation is now available in all the versions supported by Samza, the plan was to switch to that API. With that model, we can flush without tracking the latestFuture which means we can remove the sendLock. 

Along the way, I found and fixed a couple data loss issues which mostly stem from our synchronous SystemProducer API trying to handle asynchronous error scenarios with multithreading enabled.

Summary of the improvements:
* Fixed a bug in flush() logic that waits on the latest Future per task rather than per partition, which could lead to data loss.
* Fixed a bug with exception handling when task.async.commit=true that could cause data loss.
* Significantly improved send() performance with multithreading by removing the sendLock in favor of the KafkaProducer flush() method. (see performance results in google sheet)
* Improved availability guarantees by introducing task.drop.producer.errors, which guarantees that a container will never fail due to async producer errors. This fixes the case where the application wants to swallow producer exceptions, but can't do so when they occur in flush(), so they still get occasional container failures. 
* Code is much more simple now and therefore easier to maintain. 
* More thorough unit tests added. 
* Potentially better batching in low throughput scenarios now because linger.ms=10 by default.


> KafkaSystemProducer performance and correctness with concurrent sends and flushes
> ---------------------------------------------------------------------------------
>
>                 Key: SAMZA-1392
>                 URL: https://issues.apache.org/jira/browse/SAMZA-1392
>             Project: Samza
>          Issue Type: Bug
>            Reporter: Jake Maes
>            Assignee: Jake Maes
>             Fix For: 0.14.0
>
>         Attachments: Producer Performance Tests for SAMZA-1392 - Sheet1.pdf
>
>
> There are 2 issues we need to fix in the KafkaSystemProducer when sends and flushes are called concurrently:
> 1. Concurrent sends contend for the sendlock, especially when producer compression is enabled. The fix is to use the producer.flush() API, which kafka has supported since at least version 0.9.x. This way we won't need to track the latest future, so we won't need the lock.
> 2. When task.async.commit is enabled, the threads calling send() could set the exceptionInCallback to null before the exception is handled in user code or flush(). This could allow us to checkpoint offsets for which the corresponding output was not successfully sent.
> The short term solution here is to only handle the callback exceptions from flush() and allow users to configure the exceptions as ignorable in case they don't want flush to fail.
> The long term solution is to support a fully asynchronous SystemProducer. Ticket SAMZA-1393.
> I found issue #2 while working on issue #1, so while they're separate issues, it's easier to fix them with one ticket/patch.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)