[jira] [Created] (SOLR-18415) CrossDC Common: ExposedByteArrayOutputStream optimizes wrong aspect

["Andrzej Bialecki (Jira)" <ji...@apache.org>]

Andrzej Bialecki created SOLR-18415:
---------------------------------------

             Summary: CrossDC Common: ExposedByteArrayOutputStream optimizes wrong aspect
                 Key: SOLR-18415
                 URL: https://issues.apache.org/jira/browse/SOLR-18415
             Project: Solr
          Issue Type: Bug
          Components: module - crossDC
    Affects Versions: 9.10.1, 10.0
            Reporter: Andrzej Bialecki
            Assignee: Andrzej Bialecki


{{MirroringSolrRequestSerializer}} uses a subclass of {{ByteArrayOutputStream}} in an attempt to optimize data transfer by avoiding array copy - instead it passes the whole buffer directly as the output bytes from {{{}serialize(){}}}.

This sounds like a plausible optimization. However, the initial size of the buffer (32 bytes) is doubled on each overflow as the codec writes to the stream, which means that on average the actual serialized data occupies around half of the buffer, the rest are zeroes. Deserialization will handle these trailing zeros just fine, it's the cost of transmitting roughly twice as much data over the wire that is questionable - this buffer is then passed as-is to the output stream and eventually ends up as network traffic.

This "optimization" avoids one array copy cost (which reduces GC) but generates multiple costs on the way - not only the empty data has to be sent over the network but also it needs to be stored in source Kafka, in MirrorMaker and in target Kafka, to be finally retrieved by the Consumer (which has to allocate a buffer sufficiently large to fit also the zeroes), only to discard the zeroes during deserialization.

I propose to get rid of this optimization and use the plain {{ByteArrayOutputStream}} instead.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org