an improvement for memory usage in SimplePostTool

[Li Li <fa...@gmail.com>]

hi all
    when using post.jar to post very large xml files to solr like
"java -Xmx1g -Durl=http://localhost/solr/update -jar post.jar
sample.xml",
it will use many java heap space. e.g. in our case, we will post  xml
files larger than 1GB.
    Because UpdateHandler in solr will do many things. It uses
XMLStreamReader to parse xml file and contruct documents and add them
into indice. while SimplePostTool just simple read the file and post
it using http post protocol. when UpdateHandler is busy. it will not
read data from system input buffer so SimplePostTool can't write data
to system output buffer. Thus, SimplePostTool will write data into
it's own java heap space and waste memory. In c like language in
linux, we can control the speed of sender and reciever.
    After some test, we improved SimplePostTool  when posting large
files, it also use little memory.
    The key statement is:
      urlc.setFixedLengthStreamingMode(len);
    it's added atfter
      urlc.setAllowUserInteraction(false);
      urlc.setRequestProperty("Content-type", "text/xml; charset=" +
POST_ENCODING);

    where len is the length of the xml file by new File("xml file").length()

    because the file is encoded by utf8 and post encoding is also
utf8, so we can set http request header without needing to read the
whole
file into memory to calculate it's size.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

Re: an improvement for memory usage in SimplePostTool

[Chris Hostetter <ho...@fucit.org>]

: hi all
:     when using post.jar to post very large xml files to solr like
: "java -Xmx1g -Durl=http://localhost/solr/update -jar post.jar
: sample.xml",

FWIW: post.jar is not intended/suggested to be a robust solution that 
people should use to index content in Solr -- it's provided solely as a 
simpel way to post example file in the demo/tutorial w/o any external 
dependencies. (ie: it exists only because we don't want to assume "curl" 
is installed everywhere).  If you have realy large amounts of data you are 
indexing, you shouldn't be using post.jar

With that caveat, I was making some fixes to post.jar anyway because of 
Uwe's recent charset fixes to Solr, so i went ahead and imcluded your 
suggestion...

:       urlc.setFixedLengthStreamingMode(len);

...as part of my patch...

https://issues.apache.org/jira/browse/SOLR-2350

Thanks!

-Hoss

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org