[jira] [Resolved] (SOLR-8937) bin/post (SimplePostTool) should stream/chunk if size not known

["David Smiley (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SOLR-8937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

David Smiley resolved SOLR-8937.
--------------------------------
       Resolution: Fixed
    Fix Version/s: 6.1

> bin/post (SimplePostTool) should stream/chunk if size not known
> ---------------------------------------------------------------
>
>                 Key: SOLR-8937
>                 URL: https://issues.apache.org/jira/browse/SOLR-8937
>             Project: Solr
>          Issue Type: Improvement
>          Components: SimplePostTool
>            Reporter: David Smiley
>            Assignee: David Smiley
>             Fix For: 6.1
>
>         Attachments: SOLR_8937_post_streaming.patch
>
>
> {{SimplePostTool.postData()}} takes a {{Long length}} arg that is sometimes null -- for stdin and for processing some web pages for links.  If it's null, we should this on the underlying HttpURLConnection:
> {{urlc.setChunkedStreamingMode(-1);}}.  I have found that I can't use bin/post to stream large files from stdin since it the JDK's implementation wants to put the whole thing in memory first.  Sadly this is true of {{curl}} as well.  When I set chunk mode, it works.  (FYI -1 tells it to use its default chunk size -- 4k).
> FYI I'm using stdin because I've got massive JSON files zipped up.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org