You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Sharmila Thapa <sh...@gmail.com> on 2013/04/23 14:02:29 UTC
what is the maximum XML file size to import?
Hello,
What is the maximum size limit of the XML document file that is allowed to
import into solr to index from java -Durl. As I am testing to import XMLfile
of 5 GB and it throws an error like
SimplePostTool: WARNING: Solr returned an error #400 Bad Request
SimplePostTool: WARNING: IOException while reading response:
java.io.IOException: Server returned HTTP response code: 400 for URL:
http://10.0.1.140:8080/solr/solr1/update
--
View this message in context: http://lucene.472066.n3.nabble.com/what-is-the-maximum-XML-file-size-to-import-tp4058263.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: what is the maximum XML file size to import?
Posted by Alexandre Rafalovitch <ar...@gmail.com>.
DataImportHandler might be a better way to import very large XML files
if it can be loaded from Solr-local file system.
Regards,
Alex.
Personal blog: http://blog.outerthoughts.com/
LinkedIn: http://www.linkedin.com/in/alexandrerafalovitch
- Time is the quality of nature that keeps events from happening all
at once. Lately, it doesn't seem to be working. (Anonymous - via GTD
book)
On Tue, Apr 23, 2013 at 9:46 AM, Shawn Heisey <so...@elyograg.org> wrote:
> On 4/23/2013 6:02 AM, Sharmila Thapa wrote:
>> What is the maximum size limit of the XML document file that is allowed to
>> import into solr to index from java -Durl. As I am testing to import XMLfile
>> of 5 GB and it throws an error like
>> SimplePostTool: WARNING: Solr returned an error #400 Bad Request
>> SimplePostTool: WARNING: IOException while reading response:
>> java.io.IOException: Server returned HTTP response code: 400 for URL:
>> http://10.0.1.140:8080/solr/solr1/update
>
> Unless the simple post tool is capable of breaking the input XML into
> many pieces, you'll run into the POST size limit of your servlet
> container. I don't know if it has this capability, but I would be
> somewhat surprised if it did.
>
> Solr is packaged so the example uses jetty (start.jar), but you may be
> running under tomcat or one of a few other choices. The history of the
> POST limit in Solr is a little complex.
>
> The example jetty config in Solr 3.x (and possibly earlier) used a 1MiB
> POST buffer. You could change that value with no problem. If you used
> another container, you could change it using that container's
> configuration method.
>
> When 4.0 was released, jetty 8.x had a bug and the 1MiB configuration in
> the example wasn't working, so the limit became 200KB, jetty's default.
> Just like earlier versions, if you were using another container, you
> could change the limit using that container's configuration.
>
> The bug in jetty has now been fixed.
> https://bugs.eclipse.org/bugs/show_bug.cgi?id=397130
>
> Solr 4.1 changed things, with SOLR-4265. Now Solr controls the max POST
> size itself, defaulting formdataUploadLimitInKB in solrconfig.xml to 2048.
>
> https://issues.apache.org/jira/browse/SOLR-4265
>
> Thanks,
> Shawn
>
Re: what is the maximum XML file size to import?
Posted by Shawn Heisey <so...@elyograg.org>.
On 4/23/2013 6:02 AM, Sharmila Thapa wrote:
> What is the maximum size limit of the XML document file that is allowed to
> import into solr to index from java -Durl. As I am testing to import XMLfile
> of 5 GB and it throws an error like
> SimplePostTool: WARNING: Solr returned an error #400 Bad Request
> SimplePostTool: WARNING: IOException while reading response:
> java.io.IOException: Server returned HTTP response code: 400 for URL:
> http://10.0.1.140:8080/solr/solr1/update
Unless the simple post tool is capable of breaking the input XML into
many pieces, you'll run into the POST size limit of your servlet
container. I don't know if it has this capability, but I would be
somewhat surprised if it did.
Solr is packaged so the example uses jetty (start.jar), but you may be
running under tomcat or one of a few other choices. The history of the
POST limit in Solr is a little complex.
The example jetty config in Solr 3.x (and possibly earlier) used a 1MiB
POST buffer. You could change that value with no problem. If you used
another container, you could change it using that container's
configuration method.
When 4.0 was released, jetty 8.x had a bug and the 1MiB configuration in
the example wasn't working, so the limit became 200KB, jetty's default.
Just like earlier versions, if you were using another container, you
could change the limit using that container's configuration.
The bug in jetty has now been fixed.
https://bugs.eclipse.org/bugs/show_bug.cgi?id=397130
Solr 4.1 changed things, with SOLR-4265. Now Solr controls the max POST
size itself, defaulting formdataUploadLimitInKB in solrconfig.xml to 2048.
https://issues.apache.org/jira/browse/SOLR-4265
Thanks,
Shawn
Re: what is the maximum XML file size to import?
Posted by Sharmila Thapa <sh...@gmail.com>.
Thanks to all for your suggestions.
--
View this message in context: http://lucene.472066.n3.nabble.com/what-is-the-maximum-XML-file-size-to-import-tp4058263p4059113.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: what is the maximum XML file size to import?
Posted by Otis Gospodnetic <ot...@gmail.com>.
Hi,
Even if you could import giant files, I'd avoid it because it feels
like just asking for trouble. Chunk the file into smaller pieces.
You can index such smaller pieces in parallel, too, and end up with
faster indexing as the result.
Otis
--
Solr & ElasticSearch Support
http://sematext.com/
On Thu, Apr 25, 2013 at 12:10 AM, Sharmila Thapa <sh...@gmail.com> wrote:
> Yes,
> I have again tried to post the XML of size 2.02GB, now it throws a different
> error message
> <http://lucene.472066.n3.nabble.com/file/n4058825/1.png>
>
> While searching the cause for this error message it is found that
> java's setFixedLengthStreamingMode method throws this error.
> Reference:Documentation:http://docs.oracle.com/javase/7/docs/api/java/net/HttpURLConnection.html.
> So we have to limit size of the file to 16 bits i.e. 2GB.
> I have also tried by setting unlimited java heap size, but does not work.
>
> So is there anything that can be done to support upto 6GB xml file size. If
> possible I would like to try to use java -Durl to import the XML data. If
> this does not work, then I will try for other alternatives as you have
> suggested DIH.
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/what-is-the-maximum-XML-file-size-to-import-tp4058263p4058825.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: what is the maximum XML file size to import?
Posted by Sharmila Thapa <sh...@gmail.com>.
Yes,
I have again tried to post the XML of size 2.02GB, now it throws a different
error message
<http://lucene.472066.n3.nabble.com/file/n4058825/1.png>
While searching the cause for this error message it is found that
java's setFixedLengthStreamingMode method throws this error.
Reference:Documentation:http://docs.oracle.com/javase/7/docs/api/java/net/HttpURLConnection.html.
So we have to limit size of the file to 16 bits i.e. 2GB.
I have also tried by setting unlimited java heap size, but does not work.
So is there anything that can be done to support upto 6GB xml file size. If
possible I would like to try to use java -Durl to import the XML data. If
this does not work, then I will try for other alternatives as you have
suggested DIH.
--
View this message in context: http://lucene.472066.n3.nabble.com/what-is-the-maximum-XML-file-size-to-import-tp4058263p4058825.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: what is the maximum XML file size to import?
Posted by Alexandre Rafalovitch <ar...@gmail.com>.
This does not seem to be related to the XML size. Check the exact
error message on the server side. Looks to me like the URL may not be
correct. I think in some cases, post.jar automatically adds /update
handler, so maybe you are doubling it up.
Regards,
Alex.
Personal blog: http://blog.outerthoughts.com/
LinkedIn: http://www.linkedin.com/in/alexandrerafalovitch
- Time is the quality of nature that keeps events from happening all
at once. Lately, it doesn't seem to be working. (Anonymous - via GTD
book)
On Tue, Apr 23, 2013 at 8:02 AM, Sharmila Thapa <sh...@gmail.com> wrote:
> Hello,
>
> What is the maximum size limit of the XML document file that is allowed to
> import into solr to index from java -Durl. As I am testing to import XMLfile
> of 5 GB and it throws an error like
> SimplePostTool: WARNING: Solr returned an error #400 Bad Request
> SimplePostTool: WARNING: IOException while reading response:
> java.io.IOException: Server returned HTTP response code: 400 for URL:
> http://10.0.1.140:8080/solr/solr1/update
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/what-is-the-maximum-XML-file-size-to-import-tp4058263.html
> Sent from the Solr - User mailing list archive at Nabble.com.