You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Floyd Wu <fl...@gmail.com> on 2014/06/15 10:44:36 UTC
What is the best approach to send lots of XML Messages to Solr to
build index?
Hi,
I have many XML Message file formatted like this
https://wiki.apache.org/solr/UpdateXmlMessages
These files are generated by my index builder daily.
Currently I am sending these file through http post to Solr but sometimes I
hit OOM exception or pending too many tlog.
Do you have better way to "import" these files to Solr to build index?
Thanks for the suggestion
Floyd
Re: What is the best approach to send lots of XML Messages to Solr to
build index?
Posted by Floyd Wu <fl...@gmail.com>.
Hi Shawn,
I've tried to set 4GB heap for Solr and the OOM exception rellay get reduce
and also performance gained.
Floyd
2014-06-16 0:00 GMT+08:00 Shawn Heisey <so...@elyograg.org>:
> On 6/15/2014 2:54 AM, Floyd Wu wrote:
> > Thank you Alex.
> > I'm doing commit every 100 fiels.
> > Maybe there is a better way to do this job, something like DIH(possible?)
> > Sometimes i have much bigger xml file (2MB) and post to SOLR(jetty
> enabled)
> > may encounter slow or exceed limitation.
>
> If you are getting OOM exceptions on your Solr server, then you need to
> increase your Java heap size. I have never seen the "pending too many"
> error that you mentioned at the beginning of the thread, and Google
> didn't turn up anything useful, so I don't know what needs to be done
> for that. If you can post the entire exception stacktrace for this
> error, perhaps we can figure it out. We would also need the exact Solr
> version.
>
> Solr has a default 2MB limit on POST requests. This can be increased
> with the formdataUploadLimitInKB parameter on the requestDispatcher tag
> in solrconfig.xml -- assuming that you're running 4.1 or later.
> Previous versions required changing the request size in the servlet
> container config, but there was a bug in the example Jetty included in
> 4.0.0 that made it impossible to change the size.
>
> https://bugs.eclipse.org/bugs/show_bug.cgi?id=397130
> https://issues.apache.org/jira/browse/SOLR-4223
>
> Regarding the advice to disable the updateLog: I never like to do this,
> but if you are sending a very large number of updates in a single
> request, it might be advisable until indexing is complete, so that Solr
> restart times are not excessive.
>
> Thanks,
> Shawn
>
>
Re: What is the best approach to send lots of XML Messages to Solr
to build index?
Posted by Shawn Heisey <so...@elyograg.org>.
On 6/15/2014 2:54 AM, Floyd Wu wrote:
> Thank you Alex.
> I'm doing commit every 100 fiels.
> Maybe there is a better way to do this job, something like DIH(possible?)
> Sometimes i have much bigger xml file (2MB) and post to SOLR(jetty enabled)
> may encounter slow or exceed limitation.
If you are getting OOM exceptions on your Solr server, then you need to
increase your Java heap size. I have never seen the "pending too many"
error that you mentioned at the beginning of the thread, and Google
didn't turn up anything useful, so I don't know what needs to be done
for that. If you can post the entire exception stacktrace for this
error, perhaps we can figure it out. We would also need the exact Solr
version.
Solr has a default 2MB limit on POST requests. This can be increased
with the formdataUploadLimitInKB parameter on the requestDispatcher tag
in solrconfig.xml -- assuming that you're running 4.1 or later.
Previous versions required changing the request size in the servlet
container config, but there was a bug in the example Jetty included in
4.0.0 that made it impossible to change the size.
https://bugs.eclipse.org/bugs/show_bug.cgi?id=397130
https://issues.apache.org/jira/browse/SOLR-4223
Regarding the advice to disable the updateLog: I never like to do this,
but if you are sending a very large number of updates in a single
request, it might be advisable until indexing is complete, so that Solr
restart times are not excessive.
Thanks,
Shawn
Re: What is the best approach to send lots of XML Messages to Solr to
build index?
Posted by Floyd Wu <fl...@gmail.com>.
Thank you Alex.
I'm doing commit every 100 fiels.
Maybe there is a better way to do this job, something like DIH(possible?)
Sometimes i have much bigger xml file (2MB) and post to SOLR(jetty enabled)
may encounter slow or exceed limitation.
Floyd
2014-06-15 16:48 GMT+08:00 Alexandre Rafalovitch <ar...@gmail.com>:
> When are you doing commit? You can issue one manually, have one with
> timeout parameter (commitWithin), or you can configure it to happen
> automatically (in solrconfig.xml).
>
> Regards,
> Alex.
> Personal website: http://www.outerthoughts.com/
> Current project: http://www.solr-start.com/ - Accelerating your Solr
> proficiency
>
>
> On Sun, Jun 15, 2014 at 3:44 PM, Floyd Wu <fl...@gmail.com> wrote:
> > Hi,
> > I have many XML Message file formatted like this
> > https://wiki.apache.org/solr/UpdateXmlMessages
> >
> > These files are generated by my index builder daily.
> > Currently I am sending these file through http post to Solr but
> sometimes I
> > hit OOM exception or pending too many tlog.
> >
> > Do you have better way to "import" these files to Solr to build index?
> >
> > Thanks for the suggestion
> >
> > Floyd
>
Re: What is the best approach to send lots of XML Messages to Solr to
build index?
Posted by Alexandre Rafalovitch <ar...@gmail.com>.
When are you doing commit? You can issue one manually, have one with
timeout parameter (commitWithin), or you can configure it to happen
automatically (in solrconfig.xml).
Regards,
Alex.
Personal website: http://www.outerthoughts.com/
Current project: http://www.solr-start.com/ - Accelerating your Solr proficiency
On Sun, Jun 15, 2014 at 3:44 PM, Floyd Wu <fl...@gmail.com> wrote:
> Hi,
> I have many XML Message file formatted like this
> https://wiki.apache.org/solr/UpdateXmlMessages
>
> These files are generated by my index builder daily.
> Currently I am sending these file through http post to Solr but sometimes I
> hit OOM exception or pending too many tlog.
>
> Do you have better way to "import" these files to Solr to build index?
>
> Thanks for the suggestion
>
> Floyd
Re: What is the best approach to send lots of XML Messages to Solr to
build index?
Posted by Floyd Wu <fl...@gmail.com>.
Hi Erick, Thanks for your advice. autoCommit is configured 30 sec in my
environment.
i'm using C# to develop main system and Solr as a service, so using SolrJ
would consider as impossible(for now).
I;m seeking the better way to directly input(import) the offline generated
XML to build index.
Currently i'm using my own C# code to send these xml files one by one
through HTTP but result poor performance. (parallel will hit OOM or
generate lots of tlog files).
Actually a main question is "what is the best(better) way to rebuild whole
index from scratch".
Floyd
2014-06-15 23:59 GMT+08:00 Erick Erickson <er...@gmail.com>:
> A couple of things:
>
> > Consider indexing them with SolrJ, here's a place to get started:
> http://searchhub.org/2012/02/14/indexing-with-solrj/. Especially if you
> use a SAX-based parser you have more control over memory consumption, it's
> on the client after all. And, you can rack together as many clients all
> going to Solr as you need.
>
> > Here's a bunch of information about tlogs and commits that might be
> useful background.
>
> http://searchhub.org/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
> .
> Consider setting your <autoCommit> interval quite short (15 seconds)
> with openSearcher set to false. That'll truncate your tlog, although
> how that relates to your error is something of a mystery to me...
>
> Best,
> Erick
>
> On Sun, Jun 15, 2014 at 3:14 AM, Mikhail Khludnev
> <mk...@griddynamics.com> wrote:
> > Hello Floyd,
> >
> > Did you consider to disable tlog?
> > Does a file consist of many docs?
> > Do you have SolrCloud? Do you use just sh/curl or have a java program?
> > DIH is not really performant so far. Submitting roughly ten huge files in
> > parallel is a way to perform good. Once again, nuke tlog.
> >
> >
> > On Sun, Jun 15, 2014 at 12:44 PM, Floyd Wu <fl...@gmail.com> wrote:
> >
> >> Hi,
> >> I have many XML Message file formatted like this
> >> https://wiki.apache.org/solr/UpdateXmlMessages
> >>
> >> These files are generated by my index builder daily.
> >> Currently I am sending these file through http post to Solr but
> sometimes I
> >> hit OOM exception or pending too many tlog.
> >>
> >> Do you have better way to "import" these files to Solr to build index?
> >>
> >> Thanks for the suggestion
> >>
> >> Floyd
> >>
> >
> >
> >
> > --
> > Sincerely yours
> > Mikhail Khludnev
> > Principal Engineer,
> > Grid Dynamics
> >
> > <http://www.griddynamics.com>
> > <mk...@griddynamics.com>
>
Re: What is the best approach to send lots of XML Messages to Solr to
build index?
Posted by Erick Erickson <er...@gmail.com>.
A couple of things:
> Consider indexing them with SolrJ, here's a place to get started: http://searchhub.org/2012/02/14/indexing-with-solrj/. Especially if you use a SAX-based parser you have more control over memory consumption, it's on the client after all. And, you can rack together as many clients all going to Solr as you need.
> Here's a bunch of information about tlogs and commits that might be useful background.
http://searchhub.org/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/.
Consider setting your <autoCommit> interval quite short (15 seconds)
with openSearcher set to false. That'll truncate your tlog, although
how that relates to your error is something of a mystery to me...
Best,
Erick
On Sun, Jun 15, 2014 at 3:14 AM, Mikhail Khludnev
<mk...@griddynamics.com> wrote:
> Hello Floyd,
>
> Did you consider to disable tlog?
> Does a file consist of many docs?
> Do you have SolrCloud? Do you use just sh/curl or have a java program?
> DIH is not really performant so far. Submitting roughly ten huge files in
> parallel is a way to perform good. Once again, nuke tlog.
>
>
> On Sun, Jun 15, 2014 at 12:44 PM, Floyd Wu <fl...@gmail.com> wrote:
>
>> Hi,
>> I have many XML Message file formatted like this
>> https://wiki.apache.org/solr/UpdateXmlMessages
>>
>> These files are generated by my index builder daily.
>> Currently I am sending these file through http post to Solr but sometimes I
>> hit OOM exception or pending too many tlog.
>>
>> Do you have better way to "import" these files to Solr to build index?
>>
>> Thanks for the suggestion
>>
>> Floyd
>>
>
>
>
> --
> Sincerely yours
> Mikhail Khludnev
> Principal Engineer,
> Grid Dynamics
>
> <http://www.griddynamics.com>
> <mk...@griddynamics.com>
Re: What is the best approach to send lots of XML Messages to Solr to
build index?
Posted by Floyd Wu <fl...@gmail.com>.
Hi Mikhail
Thanks for you suggestions.
Floyd
2014-06-16 17:28 GMT+08:00 Mikhail Khludnev <mk...@griddynamics.com>:
> On Mon, Jun 16, 2014 at 6:57 AM, Floyd Wu <fl...@gmail.com> wrote:
>
> > Hi Mikhail,
> > What is the pros. to disable tlog?
> >
> I consumes the heap much providing the benefits (real-time get, recover
> uncommitted docs on failure) which are not necessary in old-school bulk
> index scenario.
>
>
> > Each of my xml file contained to doc, one is main content and the other
> is
> > acl.
> >
> How I can guess how many of them you have? Once again submitting a few
> (let's say ten) huge files in parallel allows to utilize indexing jvm
> fully, and yields the best performance.
>
>
> > Currently I'm not using SolrCloud due to my poor understanding of this
> > architecture and pros/cons.
> > The main system is developed using .Net C# so using SolrJ won't be a
> > solution.
> >
> anyway, if you submit small requests by C# code via REST, make sure that
> http keep-alive is enabled, and you don't waste time for establishing TCP
> connection. I might be wrong but I've thought Lucid guys provide some C#
> client or just its' scratch for Solr. Don't they?
>
>
> >
> > Floyd
> >
> >
> >
> > 2014-06-15 18:14 GMT+08:00 Mikhail Khludnev <mkhludnev@griddynamics.com
> >:
> >
> > > Hello Floyd,
> > >
> > > Did you consider to disable tlog?
> > > Does a file consist of many docs?
> > > Do you have SolrCloud? Do you use just sh/curl or have a java program?
> > > DIH is not really performant so far. Submitting roughly ten huge files
> in
> > > parallel is a way to perform good. Once again, nuke tlog.
> > >
> > >
> > > On Sun, Jun 15, 2014 at 12:44 PM, Floyd Wu <fl...@gmail.com> wrote:
> > >
> > > > Hi,
> > > > I have many XML Message file formatted like this
> > > > https://wiki.apache.org/solr/UpdateXmlMessages
> > > >
> > > > These files are generated by my index builder daily.
> > > > Currently I am sending these file through http post to Solr but
> > > sometimes I
> > > > hit OOM exception or pending too many tlog.
> > > >
> > > > Do you have better way to "import" these files to Solr to build
> index?
> > > >
> > > > Thanks for the suggestion
> > > >
> > > > Floyd
> > > >
> > >
> > >
> > >
> > > --
> > > Sincerely yours
> > > Mikhail Khludnev
> > > Principal Engineer,
> > > Grid Dynamics
> > >
> > > <http://www.griddynamics.com>
> > > <mk...@griddynamics.com>
> > >
> >
>
>
>
> --
> Sincerely yours
> Mikhail Khludnev
> Principal Engineer,
> Grid Dynamics
>
> <http://www.griddynamics.com>
> <mk...@griddynamics.com>
>
Re: What is the best approach to send lots of XML Messages to Solr to
build index?
Posted by Mikhail Khludnev <mk...@griddynamics.com>.
On Mon, Jun 16, 2014 at 6:57 AM, Floyd Wu <fl...@gmail.com> wrote:
> Hi Mikhail,
> What is the pros. to disable tlog?
>
I consumes the heap much providing the benefits (real-time get, recover
uncommitted docs on failure) which are not necessary in old-school bulk
index scenario.
> Each of my xml file contained to doc, one is main content and the other is
> acl.
>
How I can guess how many of them you have? Once again submitting a few
(let's say ten) huge files in parallel allows to utilize indexing jvm
fully, and yields the best performance.
> Currently I'm not using SolrCloud due to my poor understanding of this
> architecture and pros/cons.
> The main system is developed using .Net C# so using SolrJ won't be a
> solution.
>
anyway, if you submit small requests by C# code via REST, make sure that
http keep-alive is enabled, and you don't waste time for establishing TCP
connection. I might be wrong but I've thought Lucid guys provide some C#
client or just its' scratch for Solr. Don't they?
>
> Floyd
>
>
>
> 2014-06-15 18:14 GMT+08:00 Mikhail Khludnev <mk...@griddynamics.com>:
>
> > Hello Floyd,
> >
> > Did you consider to disable tlog?
> > Does a file consist of many docs?
> > Do you have SolrCloud? Do you use just sh/curl or have a java program?
> > DIH is not really performant so far. Submitting roughly ten huge files in
> > parallel is a way to perform good. Once again, nuke tlog.
> >
> >
> > On Sun, Jun 15, 2014 at 12:44 PM, Floyd Wu <fl...@gmail.com> wrote:
> >
> > > Hi,
> > > I have many XML Message file formatted like this
> > > https://wiki.apache.org/solr/UpdateXmlMessages
> > >
> > > These files are generated by my index builder daily.
> > > Currently I am sending these file through http post to Solr but
> > sometimes I
> > > hit OOM exception or pending too many tlog.
> > >
> > > Do you have better way to "import" these files to Solr to build index?
> > >
> > > Thanks for the suggestion
> > >
> > > Floyd
> > >
> >
> >
> >
> > --
> > Sincerely yours
> > Mikhail Khludnev
> > Principal Engineer,
> > Grid Dynamics
> >
> > <http://www.griddynamics.com>
> > <mk...@griddynamics.com>
> >
>
--
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics
<http://www.griddynamics.com>
<mk...@griddynamics.com>
Re: What is the best approach to send lots of XML Messages to Solr to
build index?
Posted by Floyd Wu <fl...@gmail.com>.
Hi Mikhail,
What is the pros. to disable tlog?
Each of my xml file contained to doc, one is main content and the other is
acl.
Currently I'm not using SolrCloud due to my poor understanding of this
architecture and pros/cons.
The main system is developed using .Net C# so using SolrJ won't be a
solution.
Floyd
2014-06-15 18:14 GMT+08:00 Mikhail Khludnev <mk...@griddynamics.com>:
> Hello Floyd,
>
> Did you consider to disable tlog?
> Does a file consist of many docs?
> Do you have SolrCloud? Do you use just sh/curl or have a java program?
> DIH is not really performant so far. Submitting roughly ten huge files in
> parallel is a way to perform good. Once again, nuke tlog.
>
>
> On Sun, Jun 15, 2014 at 12:44 PM, Floyd Wu <fl...@gmail.com> wrote:
>
> > Hi,
> > I have many XML Message file formatted like this
> > https://wiki.apache.org/solr/UpdateXmlMessages
> >
> > These files are generated by my index builder daily.
> > Currently I am sending these file through http post to Solr but
> sometimes I
> > hit OOM exception or pending too many tlog.
> >
> > Do you have better way to "import" these files to Solr to build index?
> >
> > Thanks for the suggestion
> >
> > Floyd
> >
>
>
>
> --
> Sincerely yours
> Mikhail Khludnev
> Principal Engineer,
> Grid Dynamics
>
> <http://www.griddynamics.com>
> <mk...@griddynamics.com>
>
Re: What is the best approach to send lots of XML Messages to Solr to
build index?
Posted by Mikhail Khludnev <mk...@griddynamics.com>.
Hello Floyd,
Did you consider to disable tlog?
Does a file consist of many docs?
Do you have SolrCloud? Do you use just sh/curl or have a java program?
DIH is not really performant so far. Submitting roughly ten huge files in
parallel is a way to perform good. Once again, nuke tlog.
On Sun, Jun 15, 2014 at 12:44 PM, Floyd Wu <fl...@gmail.com> wrote:
> Hi,
> I have many XML Message file formatted like this
> https://wiki.apache.org/solr/UpdateXmlMessages
>
> These files are generated by my index builder daily.
> Currently I am sending these file through http post to Solr but sometimes I
> hit OOM exception or pending too many tlog.
>
> Do you have better way to "import" these files to Solr to build index?
>
> Thanks for the suggestion
>
> Floyd
>
--
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics
<http://www.griddynamics.com>
<mk...@griddynamics.com>