[jira] [Resolved] (IMPALA-682) Improve statestore network performance with large topics

["Tim Armstrong (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IMPALA-682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tim Armstrong resolved IMPALA-682.
----------------------------------
    Resolution: Won't Fix

With the local catalog improvements, the metadata topic only contains invalidations. So no need to invest in this.

> Improve statestore network performance with large topics
> --------------------------------------------------------
>
>                 Key: IMPALA-682
>                 URL: https://issues.apache.org/jira/browse/IMPALA-682
>             Project: IMPALA
>          Issue Type: Improvement
>          Components: Distributed Exec
>    Affects Versions: Impala 1.2.2
>            Reporter: Henry Robinson
>            Assignee: Henry Robinson
>            Priority: Minor
>
> When the statestore has a large topic to transmit (e.g. over 100MB), a lot of network bandwidth will be used. This is particularly acute at startup, when many subscribers are competing for the complete version of a single large topic.
> A lot of the statestore's content is textual, and therefore likely to be very compressible. We can use Thrift's {{TZlibTransport}} to transparently compress large topics, but the problem then is that we'll be doing a lot of redundant work to compress the same topic many times. 
> Instead, maybe we will have to serialise the Topic's thrift structure to a byte string (much as we do for topic values), and then compress it in a single thread. We can do this repeatedly in the background as topics get updated. There'll be some double-serialisation cost to pay, since presumably serialising and deserialising Thrift structs in the application involves another copy, but it should be worth it.
> We can also mitigate the startup problem by having subscribers wait for a lot longer to get their first heartbeat after registration, since the first set of topic updates is going to be large.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org