[jira] [Commented] (SAMZA-203) Bad performance in BrokerProxy when restoring changelogs

["Chris Riccomini (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SAMZA-203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13944646#comment-13944646 ] 

Chris Riccomini commented on SAMZA-203:
---------------------------------------

Commenting out the Thread.sleep line, my MacBook Air SSD can restore 1 million 200 byte rows in 7 seconds.

If we leave Thread.sleep as it was before, the restoration takes many minutes (I usually ctrl-c it).

> Bad performance in BrokerProxy when restoring changelogs
> --------------------------------------------------------
>
>                 Key: SAMZA-203
>                 URL: https://issues.apache.org/jira/browse/SAMZA-203
>             Project: Samza
>          Issue Type: Bug
>          Components: kv
>    Affects Versions: 0.6.0
>            Reporter: Chris Riccomini
>             Fix For: 0.7.0
>
>
> As part of SAMZA-126, we introduced a Thread.sleep call in BrokerProxy's fetchMessages method. The goal was to skip fetch requests on SimpleConsumer when the topicAndPartitionsToFetch variable was empty. Since we had no topic/partitions to fetch, we slowed the thread down by calling Thread.sleep(sleepMSWhileNoTopicPartitions), which defaults to 1000ms.
> We now see that we are only getting about 1mb/s when restoring changelogs. This is very slow. Upon investigation, it appears that the BrokerProxy thread is sleeping 90% of the time during restore, and the main SamzaContainer thread is polling for more messages about 60% of the time.
> The reason for the poor restore performance is that the BrokerProxy sleeps for 1 second every time the message queue for the restore topic is not empty. Effectively, the proxy starts throttling the reads. If I comment out the Thread.sleep line in the BrokerProxy, I get about 64mb/s network usage on my loopback (one broker running locally), 10mb/s disk read, and 70mb/s disk write on my MacBook Air SSD--the write appears to be the bottleneck (since we're writing all the values to the LevelDB store). This is much much faster than before.



--
This message was sent by Atlassian JIRA
(v6.2#6252)