You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Maxim Valyanskiy (JIRA)" <ji...@apache.org> on 2011/07/28 13:07:09 UTC

[jira] [Created] (SOLR-2679) Optimize memory usage in JavaBinCodec.readStr()

Optimize memory usage in JavaBinCodec.readStr()
-----------------------------------------------

                 Key: SOLR-2679
                 URL: https://issues.apache.org/jira/browse/SOLR-2679
             Project: Solr
          Issue Type: Improvement
          Components: clients - java, Response Writers, update
    Affects Versions: 3.3
            Reporter: Maxim Valyanskiy
             Fix For: 3.4


In Solr 3.3, JavaBinCodec.readStr() reads complete string into byte[] and then converts it into String. FastInputStream is already buffered, so this it not necessary and may take a lot of memory if data string is large.

This patch replaces usage of big byte[] with usage of dis.readUnsignedByte() for getting characters directly from stream.

Patch reduces memory usage and may slightly improve performance of reading text data in javabin-codec.

JavaBinCodec.readStr() function already has unit test in TestJavaBinCodec, no additional unit test is required.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] [Updated] (SOLR-2679) Optimize memory usage in JavaBinCodec.readStr()

Posted by "Maxim Valyanskiy (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/SOLR-2679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Maxim Valyanskiy updated SOLR-2679:
-----------------------------------

    Attachment: SOLR-2679.patch

patch

> Optimize memory usage in JavaBinCodec.readStr()
> -----------------------------------------------
>
>                 Key: SOLR-2679
>                 URL: https://issues.apache.org/jira/browse/SOLR-2679
>             Project: Solr
>          Issue Type: Improvement
>          Components: clients - java, Response Writers, update
>    Affects Versions: 3.3
>            Reporter: Maxim Valyanskiy
>             Fix For: 3.4
>
>         Attachments: SOLR-2679.patch
>
>
> In Solr 3.3, JavaBinCodec.readStr() reads complete string into byte[] and then converts it into String. FastInputStream is already buffered, so this it not necessary and may take a lot of memory if data string is large.
> This patch replaces usage of big byte[] with usage of dis.readUnsignedByte() for getting characters directly from stream.
> Patch reduces memory usage and may slightly improve performance of reading text data in javabin-codec.
> JavaBinCodec.readStr() function already has unit test in TestJavaBinCodec, no additional unit test is required.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] [Commented] (SOLR-2679) Optimize memory usage in JavaBinCodec.readStr()

Posted by "Erick Erickson (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/SOLR-2679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13165575#comment-13165575 ] 

Erick Erickson commented on SOLR-2679:
--------------------------------------

Does anyone have an opinion on this? It seems to me that the tradeoff is that the current form of this code reads the bytes from a local array, where the proposed patch makes a method call for every byte that's read, which seems like it'd be slower since the dis.readFully method does an arrayCopy under the covers.

I'll volunteer to apply or close this, it seems like a time/space tradeoff. And the underlying byte[] grows to a max size and gets re-used.

Since the class that's used to read data is called FastInputStream, my straw-man proposal is that we do NOT apply this patch and I'll close the JIRA.
                
> Optimize memory usage in JavaBinCodec.readStr()
> -----------------------------------------------
>
>                 Key: SOLR-2679
>                 URL: https://issues.apache.org/jira/browse/SOLR-2679
>             Project: Solr
>          Issue Type: Improvement
>          Components: clients - java, Response Writers, update
>    Affects Versions: 3.3
>            Reporter: Maxim Valyanskiy
>             Fix For: 3.6, 4.0
>
>         Attachments: SOLR-2679.patch
>
>
> In Solr 3.3, JavaBinCodec.readStr() reads complete string into byte[] and then converts it into String. FastInputStream is already buffered, so this it not necessary and may take a lot of memory if data string is large.
> This patch replaces usage of big byte[] with usage of dis.readUnsignedByte() for getting characters directly from stream.
> Patch reduces memory usage and may slightly improve performance of reading text data in javabin-codec.
> JavaBinCodec.readStr() function already has unit test in TestJavaBinCodec, no additional unit test is required.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] [Commented] (SOLR-2679) Optimize memory usage in JavaBinCodec.readStr()

Posted by "Maxim Valyanskiy (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/SOLR-2679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169224#comment-13169224 ] 

Maxim Valyanskiy commented on SOLR-2679:
----------------------------------------

I think that there is no performance degradation after apply of my patch - I hope that JIT compiler is smart enough to optimize that method call when we read from stream buffer. I can make a benchmark if you think that proof is required.

My patch brings other change - it builds character array while reading data from (buffered) stream so it may be even faster when very large text field is read from network stream. Current version reads complete text field into byte array and then builds character array.
                
> Optimize memory usage in JavaBinCodec.readStr()
> -----------------------------------------------
>
>                 Key: SOLR-2679
>                 URL: https://issues.apache.org/jira/browse/SOLR-2679
>             Project: Solr
>          Issue Type: Improvement
>          Components: clients - java, Response Writers, update
>    Affects Versions: 3.3
>            Reporter: Maxim Valyanskiy
>             Fix For: 3.6, 4.0
>
>         Attachments: SOLR-2679.patch
>
>
> In Solr 3.3, JavaBinCodec.readStr() reads complete string into byte[] and then converts it into String. FastInputStream is already buffered, so this it not necessary and may take a lot of memory if data string is large.
> This patch replaces usage of big byte[] with usage of dis.readUnsignedByte() for getting characters directly from stream.
> Patch reduces memory usage and may slightly improve performance of reading text data in javabin-codec.
> JavaBinCodec.readStr() function already has unit test in TestJavaBinCodec, no additional unit test is required.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] [Commented] (SOLR-2679) Optimize memory usage in JavaBinCodec.readStr()

Posted by "Yonik Seeley (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/SOLR-2679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13165692#comment-13165692 ] 

Yonik Seeley commented on SOLR-2679:
------------------------------------

I think the right approach to increased efficiency here is to expose the byte[] of the FastInputStream.
It's not a super-simple approach since it needs to operate chunk-at-a-time and deal with multi-byte chars crossing chunks.
The goal of eliminating the double-buffering seems fine though - I think we can leave this issue open.
                
> Optimize memory usage in JavaBinCodec.readStr()
> -----------------------------------------------
>
>                 Key: SOLR-2679
>                 URL: https://issues.apache.org/jira/browse/SOLR-2679
>             Project: Solr
>          Issue Type: Improvement
>          Components: clients - java, Response Writers, update
>    Affects Versions: 3.3
>            Reporter: Maxim Valyanskiy
>             Fix For: 3.6, 4.0
>
>         Attachments: SOLR-2679.patch
>
>
> In Solr 3.3, JavaBinCodec.readStr() reads complete string into byte[] and then converts it into String. FastInputStream is already buffered, so this it not necessary and may take a lot of memory if data string is large.
> This patch replaces usage of big byte[] with usage of dis.readUnsignedByte() for getting characters directly from stream.
> Patch reduces memory usage and may slightly improve performance of reading text data in javabin-codec.
> JavaBinCodec.readStr() function already has unit test in TestJavaBinCodec, no additional unit test is required.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org