You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "EXTERNAL Taminidi Ravi (ETI, AA-AS/PAS-PTS)" <ex...@us.bosch.com> on 2015/07/13 20:56:54 UTC
XML File Size for Post.jar
HI, Where I have to change to support the xml file more than 2GB to Index in Solr, using the simple post tool (post.jar) for Jetty and Tomcat.
Thanks
Ravi
Re: XML File Size for Post.jar
Posted by Upayavira <uv...@odoko.co.uk>.
If you have hundreds of files, the post command (SimplePostTool) can
also push a directory of files up to Solr.
(It is called "Simple" under the hood, but it is far from "simple"!)
Upayavira
On Mon, Jul 13, 2015, at 09:28 PM, Alexandre Rafalovitch wrote:
> Solr ships with XML processing example for DIH in the examples
> directory (RSS core). In your case, you will most probably read the
> filelist or directory list and then run XML processor as a nested
> entity. So, check the nested example at
> https://cwiki.apache.org/confluence/display/solr/Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler
>
> Regards,
> Alex.
> ----
> Solr Analyzers, Tokenizers, Filters, URPs and even a newsletter:
> http://www.solr-start.com/
>
>
> On 13 July 2015 at 15:12, EXTERNAL Taminidi Ravi (ETI, AA-AS/PAS-PTS)
> <ex...@us.bosch.com> wrote:
> > I Can break that into smaller files but for other case the number of files growing in 100s..
> >
> > Can I Parse XML Files to DIH..? Can you refer few examples..?
> >
> > Thanks
> >
> > Ravi
> >
> > -----Original Message-----
> > From: Alexandre Rafalovitch [mailto:arafalov@gmail.com]
> > Sent: Monday, July 13, 2015 3:01 PM
> > To: solr-user
> > Subject: Re: XML File Size for Post.jar
> >
> > I don't think you can do files that big. The memory would blow out.
> > You sure you cannot chunk it into smaller document sets? Or make it a streaming parsing with DIH in a pull fashion?
> >
> > Regards,
> > Alex.
> > ----
> > Solr Analyzers, Tokenizers, Filters, URPs and even a newsletter:
> > http://www.solr-start.com/
> >
> >
> > On 13 July 2015 at 14:56, EXTERNAL Taminidi Ravi (ETI, AA-AS/PAS-PTS) <ex...@us.bosch.com> wrote:
> >>
> >> HI, Where I have to change to support the xml file more than 2GB to Index in Solr, using the simple post tool (post.jar) for Jetty and Tomcat.
> >>
> >> Thanks
> >>
> >> Ravi
Re: XML File Size for Post.jar
Posted by Alexandre Rafalovitch <ar...@gmail.com>.
Solr ships with XML processing example for DIH in the examples
directory (RSS core). In your case, you will most probably read the
filelist or directory list and then run XML processor as a nested
entity. So, check the nested example at
https://cwiki.apache.org/confluence/display/solr/Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler
Regards,
Alex.
----
Solr Analyzers, Tokenizers, Filters, URPs and even a newsletter:
http://www.solr-start.com/
On 13 July 2015 at 15:12, EXTERNAL Taminidi Ravi (ETI, AA-AS/PAS-PTS)
<ex...@us.bosch.com> wrote:
> I Can break that into smaller files but for other case the number of files growing in 100s..
>
> Can I Parse XML Files to DIH..? Can you refer few examples..?
>
> Thanks
>
> Ravi
>
> -----Original Message-----
> From: Alexandre Rafalovitch [mailto:arafalov@gmail.com]
> Sent: Monday, July 13, 2015 3:01 PM
> To: solr-user
> Subject: Re: XML File Size for Post.jar
>
> I don't think you can do files that big. The memory would blow out.
> You sure you cannot chunk it into smaller document sets? Or make it a streaming parsing with DIH in a pull fashion?
>
> Regards,
> Alex.
> ----
> Solr Analyzers, Tokenizers, Filters, URPs and even a newsletter:
> http://www.solr-start.com/
>
>
> On 13 July 2015 at 14:56, EXTERNAL Taminidi Ravi (ETI, AA-AS/PAS-PTS) <ex...@us.bosch.com> wrote:
>>
>> HI, Where I have to change to support the xml file more than 2GB to Index in Solr, using the simple post tool (post.jar) for Jetty and Tomcat.
>>
>> Thanks
>>
>> Ravi
RE: XML File Size for Post.jar
Posted by "EXTERNAL Taminidi Ravi (ETI, AA-AS/PAS-PTS)" <ex...@us.bosch.com>.
I Can break that into smaller files but for other case the number of files growing in 100s..
Can I Parse XML Files to DIH..? Can you refer few examples..?
Thanks
Ravi
-----Original Message-----
From: Alexandre Rafalovitch [mailto:arafalov@gmail.com]
Sent: Monday, July 13, 2015 3:01 PM
To: solr-user
Subject: Re: XML File Size for Post.jar
I don't think you can do files that big. The memory would blow out.
You sure you cannot chunk it into smaller document sets? Or make it a streaming parsing with DIH in a pull fashion?
Regards,
Alex.
----
Solr Analyzers, Tokenizers, Filters, URPs and even a newsletter:
http://www.solr-start.com/
On 13 July 2015 at 14:56, EXTERNAL Taminidi Ravi (ETI, AA-AS/PAS-PTS) <ex...@us.bosch.com> wrote:
>
> HI, Where I have to change to support the xml file more than 2GB to Index in Solr, using the simple post tool (post.jar) for Jetty and Tomcat.
>
> Thanks
>
> Ravi
Re: XML File Size for Post.jar
Posted by Alexandre Rafalovitch <ar...@gmail.com>.
I don't think you can do files that big. The memory would blow out.
You sure you cannot chunk it into smaller document sets? Or make it a
streaming parsing with DIH in a pull fashion?
Regards,
Alex.
----
Solr Analyzers, Tokenizers, Filters, URPs and even a newsletter:
http://www.solr-start.com/
On 13 July 2015 at 14:56, EXTERNAL Taminidi Ravi (ETI, AA-AS/PAS-PTS)
<ex...@us.bosch.com> wrote:
>
> HI, Where I have to change to support the xml file more than 2GB to Index in Solr, using the simple post tool (post.jar) for Jetty and Tomcat.
>
> Thanks
>
> Ravi
Re: XML File Size for Post.jar
Posted by Erik Hatcher <er...@gmail.com>.
Ravi - don’t get hung up on post.jar - it’s just a means to an end. Use good ol’ curl if you’re having issues with post.jar.
But really, 2GB? Is that a single document? Or a bunch of documents? If it’s a bunch of documents, definitely split it up into multiple files.
—
Erik Hatcher, Senior Solutions Architect
http://www.lucidworks.com <http://www.lucidworks.com/>
> On Jul 13, 2015, at 2:56 PM, EXTERNAL Taminidi Ravi (ETI, AA-AS/PAS-PTS) <ex...@us.bosch.com> wrote:
>
>
> HI, Where I have to change to support the xml file more than 2GB to Index in Solr, using the simple post tool (post.jar) for Jetty and Tomcat.
>
> Thanks
>
> Ravi