You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Shalin Shekhar Mangar (JIRA)" <ji...@apache.org> on 2014/02/06 08:20:09 UTC
[jira] [Updated] (SOLR-4227) StreamingUpdateSolrServer does not
buffer OutputStreamWriter with BufferedWriter, causing encoding explosion
[ https://issues.apache.org/jira/browse/SOLR-4227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shalin Shekhar Mangar updated SOLR-4227:
----------------------------------------
Attachment: SOLR-4227.patch
After SOLR-1565, the relevant code lies in RequestWriter.
I'll commit this shortly.
> StreamingUpdateSolrServer does not buffer OutputStreamWriter with BufferedWriter, causing encoding explosion
> ------------------------------------------------------------------------------------------------------------
>
> Key: SOLR-4227
> URL: https://issues.apache.org/jira/browse/SOLR-4227
> Project: Solr
> Issue Type: Improvement
> Affects Versions: 3.2
> Environment: Java 1.6, Linux. I am running SOLR 3.2, but the code doesn't seem different in 3.5.
> Reporter: Conrad Herrmann
> Fix For: 4.7
>
> Attachments: SOLR-4227.patch
>
>
> org.apache.solr.client.solrj.impl.StreamingUpdateSolrServer line 112 is:
> OutputStreamWriter writer = new OutputStreamWriter(out, "UTF-8");
> and then we call
> req.writeXML( writer );
> Because the writer is not buffered, this causes the XML writer to call the UTF-8 encoder for each atom being written, like in org.apache.solr.common.util.XML.writeXML:
> out.write('<');
> This causes the stream encoder to allocate a char array to hold it, and
> sun.nio.cs.StreamEncoder.implWrite allocates a CharBuffer to wrap it. All just for one character.
> This is particularly a problem when you have a lot of threads (100?) writing to the SOLR server, they rapidly eat up all the CPU.
> It would be helpful to allocate the writer as a BufferedWriter, so encoding only happens when you flush. JavaDoc for OutputStreamWriter recommends this: "For top efficiency, consider wrapping an OutputStreamWriter within a BufferedWriter so as to avoid frequent converter invocations."
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org