You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Grant Ingersoll <gs...@apache.org> on 2009/03/28 16:00:41 UTC
RE: [solr-user] Upgrade from 1.2 to 1.3 gives 3x slowdown
Hey Fergus,
Finally got a chance to run your scripts, etc. per the thread:
http://www.lucidimagination.com/search/document/5c3de15a4e61095c/upgrade_from_1_2_to_1_3_gives_3x_slowdown_script#8324a98d8840c623
I can reproduce your slowdown.
One oddity with rev 643465 is:
On the old version, there is an exception during startup:
Mar 28, 2009 10:44:31 AM org.apache.solr.common.SolrException log
SEVERE: java.lang.NullPointerException
at
org
.apache
.solr
.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:
129)
at
org
.apache
.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:
125)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:953)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:968)
at
org
.apache
.solr.core.QuerySenderListener.newSearcher(QuerySenderListener.java:50)
at org.apache.solr.core.SolrCore$3.call(SolrCore.java:797)
at java.util.concurrent.FutureTask
$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:637)
I see two things in CHANGES.txt that might apply, but I'm not sure:
1. I think commons-csv was upgraded
2. The CSV loader stuff was refactored to share common code
I'm still investigating.
-Grant
Re: [solr-user] Upgrade from 1.2 to 1.3 gives 3x slowdown
Posted by Grant Ingersoll <gs...@apache.org>.
svn co -r REV_NUM https://svn.apache.org/repos/asf/lucene/solr/trunk
solr-REV_NUM
-Grant
On Mar 30, 2009, at 4:55 PM, Fergus McMenemie wrote:
>> Can you verify that rev 701485 still performs reasonably well? This
>> is from October 2008 and I get similar results to the earlier rev.
>> Am now trying some other versions between October and when you first
>> reported the issue in November.
>
> OK. Can you tell me how to get a hold of revision 701485. What is the
> magic svn line?
>
>
>> On Mar 30, 2009, at 3:37 PM, Grant Ingersoll wrote:
>>
>>> Fregus,
>>>
>>> Is rev 643465 the absolute latest you tried that still performs?
>>> i.e. every revision after is slower?
>>>
>>> -Grant
>>>
>>> On Mar 30, 2009, at 12:45 PM, Grant Ingersoll wrote:
>>>
>>>> Fergus,
>>>>
>>>> I think the problem may actually be due to something that was
>>>> introduced by a change to Solr's StopFilterFactory and the way it
>>>> loads the stop words set. See https://issues.apache.org/jira/browse/SOLR-1095
>>>>
>>>> I am in the process of testing it out and will let you know.
>>>>
>>>> -Grant
>>>>
>>>> On Mar 28, 2009, at 11:00 AM, Grant Ingersoll wrote:
>>>>
>>>>> Hey Fergus,
>>>>>
>>>>> Finally got a chance to run your scripts, etc. per the thread:
>>>>> http://www.lucidimagination.com/search/document/5c3de15a4e61095c/upgrade_from_1_2_to_1_3_gives_3x_slowdown_script#8324a98d8840c623
>>>>>
>>>>> I can reproduce your slowdown.
>>>>>
>>>>> One oddity with rev 643465 is:
>>>>>
>>>>> On the old version, there is an exception during startup:
>>>>> Mar 28, 2009 10:44:31 AM org.apache.solr.common.SolrException log
>>>>> SEVERE: java.lang.NullPointerException
>>>>> at
>>>>> org
>>>>> .apache
>>>>> .solr
>>>>> .handler
>>>>> .component.SearchHandler.handleRequestBody(SearchHandler.java:129)
>>>>> at
>>>>> org
>>>>> .apache
>>>>> .solr
>>>>> .handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:
>>>>> 125)
>>>>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:953)
>>>>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:968)
>>>>> at
>>>>> org
>>>>> .apache
>>>>> .solr
>>>>> .core.QuerySenderListener.newSearcher(QuerySenderListener.java:50)
>>>>> at org.apache.solr.core.SolrCore$3.call(SolrCore.java:797)
>>>>> at java.util.concurrent.FutureTask
>>>>> $Sync.innerRun(FutureTask.java:303)
>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>>> at java.util.concurrent.ThreadPoolExecutor
>>>>> $Worker.runTask(ThreadPoolExecutor.java:885)
>>>>> at java.util.concurrent.ThreadPoolExecutor
>>>>> $Worker.run(ThreadPoolExecutor.java:907)
>>>>> at java.lang.Thread.run(Thread.java:637)
>>>>>
>>>>> I see two things in CHANGES.txt that might apply, but I'm not
>>>>> sure:
>>>>> 1. I think commons-csv was upgraded
>>>>> 2. The CSV loader stuff was refactored to share common code
>>>>>
>>>>> I'm still investigating.
>>>>>
>>>>> -Grant
>>>>
>>>> --------------------------
>>>> Grant Ingersoll
>>>> http://www.lucidimagination.com/
>>>>
>>>> Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)
>>>> using Solr/Lucene:
>>>> http://www.lucidimagination.com/search
>>>>
>>>
>>> --------------------------
>>> Grant Ingersoll
>>> http://www.lucidimagination.com/
>>>
>>> Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)
>>> using Solr/Lucene:
>>> http://www.lucidimagination.com/search
>>>
>>
>> --------------------------
>> Grant Ingersoll
>> http://www.lucidimagination.com/
>>
>> Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)
>> using Solr/Lucene:
>> http://www.lucidimagination.com/search
>
> --
>
> ===============================================================
> Fergus McMenemie Email:fergus@twig.me.uk
> Techmore Ltd Phone:(UK) 07721 376021
>
> Unix/Mac/Intranets Analyst Programmer
> ===============================================================
--------------------------
Grant Ingersoll
http://www.lucidimagination.com/
Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)
using Solr/Lucene:
http://www.lucidimagination.com/search
Re: [solr-user] Upgrade from 1.2 to 1.3 gives 3x slowdown
Posted by Fergus McMenemie <fe...@twig.me.uk>.
>Can you verify that rev 701485 still performs reasonably well? This
>is from October 2008 and I get similar results to the earlier rev.
>Am now trying some other versions between October and when you first
>reported the issue in November.
OK. Can you tell me how to get a hold of revision 701485. What is the
magic svn line?
>On Mar 30, 2009, at 3:37 PM, Grant Ingersoll wrote:
>
>> Fregus,
>>
>> Is rev 643465 the absolute latest you tried that still performs?
>> i.e. every revision after is slower?
>>
>> -Grant
>>
>> On Mar 30, 2009, at 12:45 PM, Grant Ingersoll wrote:
>>
>>> Fergus,
>>>
>>> I think the problem may actually be due to something that was
>>> introduced by a change to Solr's StopFilterFactory and the way it
>>> loads the stop words set. See https://issues.apache.org/jira/browse/SOLR-1095
>>>
>>> I am in the process of testing it out and will let you know.
>>>
>>> -Grant
>>>
>>> On Mar 28, 2009, at 11:00 AM, Grant Ingersoll wrote:
>>>
>>>> Hey Fergus,
>>>>
>>>> Finally got a chance to run your scripts, etc. per the thread:
>>>> http://www.lucidimagination.com/search/document/5c3de15a4e61095c/upgrade_from_1_2_to_1_3_gives_3x_slowdown_script#8324a98d8840c623
>>>>
>>>> I can reproduce your slowdown.
>>>>
>>>> One oddity with rev 643465 is:
>>>>
>>>> On the old version, there is an exception during startup:
>>>> Mar 28, 2009 10:44:31 AM org.apache.solr.common.SolrException log
>>>> SEVERE: java.lang.NullPointerException
>>>> at
>>>> org
>>>> .apache
>>>> .solr
>>>> .handler
>>>> .component.SearchHandler.handleRequestBody(SearchHandler.java:129)
>>>> at
>>>> org
>>>> .apache
>>>> .solr
>>>> .handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:
>>>> 125)
>>>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:953)
>>>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:968)
>>>> at
>>>> org
>>>> .apache
>>>> .solr
>>>> .core.QuerySenderListener.newSearcher(QuerySenderListener.java:50)
>>>> at org.apache.solr.core.SolrCore$3.call(SolrCore.java:797)
>>>> at java.util.concurrent.FutureTask
>>>> $Sync.innerRun(FutureTask.java:303)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>> at java.util.concurrent.ThreadPoolExecutor
>>>> $Worker.runTask(ThreadPoolExecutor.java:885)
>>>> at java.util.concurrent.ThreadPoolExecutor
>>>> $Worker.run(ThreadPoolExecutor.java:907)
>>>> at java.lang.Thread.run(Thread.java:637)
>>>>
>>>> I see two things in CHANGES.txt that might apply, but I'm not sure:
>>>> 1. I think commons-csv was upgraded
>>>> 2. The CSV loader stuff was refactored to share common code
>>>>
>>>> I'm still investigating.
>>>>
>>>> -Grant
>>>
>>> --------------------------
>>> Grant Ingersoll
>>> http://www.lucidimagination.com/
>>>
>>> Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)
>>> using Solr/Lucene:
>>> http://www.lucidimagination.com/search
>>>
>>
>> --------------------------
>> Grant Ingersoll
>> http://www.lucidimagination.com/
>>
>> Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)
>> using Solr/Lucene:
>> http://www.lucidimagination.com/search
>>
>
>--------------------------
>Grant Ingersoll
>http://www.lucidimagination.com/
>
>Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)
>using Solr/Lucene:
>http://www.lucidimagination.com/search
--
===============================================================
Fergus McMenemie Email:fergus@twig.me.uk
Techmore Ltd Phone:(UK) 07721 376021
Unix/Mac/Intranets Analyst Programmer
===============================================================
Re: [solr-user] Upgrade from 1.2 to 1.3 gives 3x slowdown
Posted by Grant Ingersoll <gs...@apache.org>.
Can you verify that rev 701485 still performs reasonably well? This
is from October 2008 and I get similar results to the earlier rev.
Am now trying some other versions between October and when you first
reported the issue in November.
-Grant
On Mar 30, 2009, at 3:37 PM, Grant Ingersoll wrote:
> Fregus,
>
> Is rev 643465 the absolute latest you tried that still performs?
> i.e. every revision after is slower?
>
> -Grant
>
> On Mar 30, 2009, at 12:45 PM, Grant Ingersoll wrote:
>
>> Fergus,
>>
>> I think the problem may actually be due to something that was
>> introduced by a change to Solr's StopFilterFactory and the way it
>> loads the stop words set. See https://issues.apache.org/jira/browse/SOLR-1095
>>
>> I am in the process of testing it out and will let you know.
>>
>> -Grant
>>
>> On Mar 28, 2009, at 11:00 AM, Grant Ingersoll wrote:
>>
>>> Hey Fergus,
>>>
>>> Finally got a chance to run your scripts, etc. per the thread:
>>> http://www.lucidimagination.com/search/document/5c3de15a4e61095c/upgrade_from_1_2_to_1_3_gives_3x_slowdown_script#8324a98d8840c623
>>>
>>> I can reproduce your slowdown.
>>>
>>> One oddity with rev 643465 is:
>>>
>>> On the old version, there is an exception during startup:
>>> Mar 28, 2009 10:44:31 AM org.apache.solr.common.SolrException log
>>> SEVERE: java.lang.NullPointerException
>>> at
>>> org
>>> .apache
>>> .solr
>>> .handler
>>> .component.SearchHandler.handleRequestBody(SearchHandler.java:129)
>>> at
>>> org
>>> .apache
>>> .solr
>>> .handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:
>>> 125)
>>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:953)
>>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:968)
>>> at
>>> org
>>> .apache
>>> .solr
>>> .core.QuerySenderListener.newSearcher(QuerySenderListener.java:50)
>>> at org.apache.solr.core.SolrCore$3.call(SolrCore.java:797)
>>> at java.util.concurrent.FutureTask
>>> $Sync.innerRun(FutureTask.java:303)
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>> at java.util.concurrent.ThreadPoolExecutor
>>> $Worker.runTask(ThreadPoolExecutor.java:885)
>>> at java.util.concurrent.ThreadPoolExecutor
>>> $Worker.run(ThreadPoolExecutor.java:907)
>>> at java.lang.Thread.run(Thread.java:637)
>>>
>>> I see two things in CHANGES.txt that might apply, but I'm not sure:
>>> 1. I think commons-csv was upgraded
>>> 2. The CSV loader stuff was refactored to share common code
>>>
>>> I'm still investigating.
>>>
>>> -Grant
>>
>> --------------------------
>> Grant Ingersoll
>> http://www.lucidimagination.com/
>>
>> Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)
>> using Solr/Lucene:
>> http://www.lucidimagination.com/search
>>
>
> --------------------------
> Grant Ingersoll
> http://www.lucidimagination.com/
>
> Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)
> using Solr/Lucene:
> http://www.lucidimagination.com/search
>
--------------------------
Grant Ingersoll
http://www.lucidimagination.com/
Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)
using Solr/Lucene:
http://www.lucidimagination.com/search
Re: [solr-user] Upgrade from 1.2 to 1.3 gives 3x slowdown
Posted by Fergus McMenemie <fe...@twig.me.uk>.
Grant,
I am messing with the script, and with your tip I expect I can
make it recurse over as many releases as needed.
I did run it again using the full file, this time using my Imac:-
643465 took 22min 14sec 2008-04-01
734796 73min 58sec 2009-01-15
758795 70min 55sec 2009-03-26
I then ran it again using only the first 1M records:-
643465 took 2m51.516s 2008-04-01
734796 7m29.326s 2009-01-15
758795 8m18.403s 2009-03-26
this time with commit=true.
643465 took 2m49.200s 2008-04-01
734796 8m27.414s 2009-01-15
758795 9m32.459s 2009-03-26
this time with commit=false&overwrite=false.
643465 took 2m46.149s 2008-04-01
734796 3m29.909s 2009-01-15
758795 3m26.248s 2009-03-26
Just read your latest post. I will apply the patches and retest
the above.
>Can you try adding &overwrite=false and running against the latest
>version? My current working theory is that Solr/Lucene has changed
>how deletes are handled such that work that was deferred before is now
>not deferred as often. In fact, you are not seeing this cost paid (or
>at least not noticing it) because you are not committing, but I
>believe you do see it when you are closing down Solr, which is why it
>takes so long to exit.
It can take ages! (>15min to get tomcat to quit). Also my script does
have the separate commit step, which does not take any time!
>I also think that Lucene adding fsync() into
>the equation may cause some slow down, but that is a penalty we are
>willing to pay as it gives us higher data integrity.
Data integrity is always good. However if performance seems
unreasonable, user/customers tend to take things into their
own hands and kill the process or machine. This tends to be
very bad for data integrity.
>So, depending on how you have your data, I think a workaround is to:
>Add a field that contains a single term identifying the data type for
>this particular CSV file, i.e. something like field: type, value:
>fergs-csv
>Then, before indexing, you can issue a Delete By Query: type:fergs-csv
>and then add your CSV file using overwrite=false. This amounts to a
>batch delete followed by a batch add, but without the add having to
>issue deletes for each add.
Ok.. but... for these test cases I am starting off with an empty
index. The script does a "rm -rf solr/data" before tomcat is launched.
So I do not understand how the above helps. UNLESS there are duplicate
gaz entries.
>In the meantime, I'm trying to see if I can pinpoint down a specific
>change and see if there is anything that might help it perform better.
>
>-Grant
>
--
===============================================================
Fergus McMenemie Email:fergus@twig.me.uk
Techmore Ltd Phone:(UK) 07721 376021
Unix/Mac/Intranets Analyst Programmer
===============================================================
Re: [solr-user] Upgrade from 1.2 to 1.3 gives 3x slowdown
Posted by Grant Ingersoll <gs...@apache.org>.
Can you try adding &overwrite=false and running against the latest
version? My current working theory is that Solr/Lucene has changed
how deletes are handled such that work that was deferred before is now
not deferred as often. In fact, you are not seeing this cost paid (or
at least not noticing it) because you are not committing, but I
believe you do see it when you are closing down Solr, which is why it
takes so long to exit. I also think that Lucene adding fsync() into
the equation may cause some slow down, but that is a penalty we are
willing to pay as it gives us higher data integrity.
So, depending on how you have your data, I think a workaround is to:
Add a field that contains a single term identifying the data type for
this particular CSV file, i.e. something like field: type, value:
fergs-csv
Then, before indexing, you can issue a Delete By Query: type:fergs-csv
and then add your CSV file using overwrite=false. This amounts to a
batch delete followed by a batch add, but without the add having to
issue deletes for each add.
In the meantime, I'm trying to see if I can pinpoint down a specific
change and see if there is anything that might help it perform better.
-Grant
On Mar 30, 2009, at 4:52 PM, Fergus McMenemie wrote:
> Grant,
>
> After all my playing about at boot camp, I gave things a rest. It
> was not till months later that got back to looking at solr again.
> So after 643465 (2008-Apr-01) the next version I tried was 694377
> from (2008-Sep-11). Nothing in between. Yep so 643465 is the latest
> version I tried that still performs. Every later revision is slower.
>
> However I need to repeat the tests using 643465, 694377 and whatever
> is the latest version. On my macbook I am only seeing a 2x slowdown
> of 643465 vis today, where as I had been seeing a 3x slowdown using
> my Imac.
>
> Fergus
>
>
>> Fregus,
>>
>> Is rev 643465 the absolute latest you tried that still performs?
>> i.e.
>> every revision after is slower?
>>
>> -Grant
>>
>> On Mar 30, 2009, at 12:45 PM, Grant Ingersoll wrote:
>>
>>> Fergus,
>>>
>>> I think the problem may actually be due to something that was
>>> introduced by a change to Solr's StopFilterFactory and the way it
>>> loads the stop words set. See https://issues.apache.org/jira/browse/SOLR-1095
>>>
>>> I am in the process of testing it out and will let you know.
>>>
>>> -Grant
>>>
>>> On Mar 28, 2009, at 11:00 AM, Grant Ingersoll wrote:
>>>
>>>> Hey Fergus,
>>>>
>>>> Finally got a chance to run your scripts, etc. per the thread:
>>>> http://www.lucidimagination.com/search/document/5c3de15a4e61095c/upgrade_from_1_2_to_1_3_gives_3x_slowdown_script#8324a98d8840c623
>>>>
>>>> I can reproduce your slowdown.
>>>>
>>>> One oddity with rev 643465 is:
>>>>
>>>> On the old version, there is an exception during startup:
>>>> Mar 28, 2009 10:44:31 AM org.apache.solr.common.SolrException log
>>>> SEVERE: java.lang.NullPointerException
>>>> at
>>>> org
>>>> .apache
>>>> .solr
>>>> .handler
>>>> .component.SearchHandler.handleRequestBody(SearchHandler.java:129)
>>>> at
>>>> org
>>>> .apache
>>>> .solr
>>>> .handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:
>>>> 125)
>>>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:953)
>>>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:968)
>>>> at
>>>> org
>>>> .apache
>>>> .solr
>>>> .core.QuerySenderListener.newSearcher(QuerySenderListener.java:
>>>> 50)
>>>> at org.apache.solr.core.SolrCore$3.call(SolrCore.java:797)
>>>> at java.util.concurrent.FutureTask
>>>> $Sync.innerRun(FutureTask.java:303)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>> at java.util.concurrent.ThreadPoolExecutor
>>>> $Worker.runTask(ThreadPoolExecutor.java:885)
>>>> at java.util.concurrent.ThreadPoolExecutor
>>>> $Worker.run(ThreadPoolExecutor.java:907)
>>>> at java.lang.Thread.run(Thread.java:637)
>>>>
>>>> I see two things in CHANGES.txt that might apply, but I'm not sure:
>>>> 1. I think commons-csv was upgraded
>>>> 2. The CSV loader stuff was refactored to share common code
>>>>
>>>> I'm still investigating.
>>>>
>>>> -Grant
>>>
>>> --------------------------
>>> Grant Ingersoll
>>> http://www.lucidimagination.com/
>>>
>>> Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)
>>> using Solr/Lucene:
>>> http://www.lucidimagination.com/search
>
> --
>
> ===============================================================
> Fergus McMenemie Email:fergus@twig.me.uk
> Techmore Ltd Phone:(UK) 07721 376021
>
> Unix/Mac/Intranets Analyst Programmer
> ===============================================================
--------------------------
Grant Ingersoll
http://www.lucidimagination.com/
Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)
using Solr/Lucene:
http://www.lucidimagination.com/search
Re: [solr-user] Upgrade from 1.2 to 1.3 gives 3x slowdown
Posted by Fergus McMenemie <fe...@twig.me.uk>.
Grant,
After all my playing about at boot camp, I gave things a rest. It
was not till months later that got back to looking at solr again.
So after 643465 (2008-Apr-01) the next version I tried was 694377
from (2008-Sep-11). Nothing in between. Yep so 643465 is the latest
version I tried that still performs. Every later revision is slower.
However I need to repeat the tests using 643465, 694377 and whatever
is the latest version. On my macbook I am only seeing a 2x slowdown
of 643465 vis today, where as I had been seeing a 3x slowdown using
my Imac.
Fergus
>Fregus,
>
>Is rev 643465 the absolute latest you tried that still performs? i.e.
>every revision after is slower?
>
>-Grant
>
>On Mar 30, 2009, at 12:45 PM, Grant Ingersoll wrote:
>
>> Fergus,
>>
>> I think the problem may actually be due to something that was
>> introduced by a change to Solr's StopFilterFactory and the way it
>> loads the stop words set. See https://issues.apache.org/jira/browse/SOLR-1095
>>
>> I am in the process of testing it out and will let you know.
>>
>> -Grant
>>
>> On Mar 28, 2009, at 11:00 AM, Grant Ingersoll wrote:
>>
>>> Hey Fergus,
>>>
>>> Finally got a chance to run your scripts, etc. per the thread:
>>> http://www.lucidimagination.com/search/document/5c3de15a4e61095c/upgrade_from_1_2_to_1_3_gives_3x_slowdown_script#8324a98d8840c623
>>>
>>> I can reproduce your slowdown.
>>>
>>> One oddity with rev 643465 is:
>>>
>>> On the old version, there is an exception during startup:
>>> Mar 28, 2009 10:44:31 AM org.apache.solr.common.SolrException log
>>> SEVERE: java.lang.NullPointerException
>>> at
>>> org
>>> .apache
>>> .solr
>>> .handler
>>> .component.SearchHandler.handleRequestBody(SearchHandler.java:129)
>>> at
>>> org
>>> .apache
>>> .solr
>>> .handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:
>>> 125)
>>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:953)
>>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:968)
>>> at
>>> org
>>> .apache
>>> .solr.core.QuerySenderListener.newSearcher(QuerySenderListener.java:
>>> 50)
>>> at org.apache.solr.core.SolrCore$3.call(SolrCore.java:797)
>>> at java.util.concurrent.FutureTask
>>> $Sync.innerRun(FutureTask.java:303)
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>> at java.util.concurrent.ThreadPoolExecutor
>>> $Worker.runTask(ThreadPoolExecutor.java:885)
>>> at java.util.concurrent.ThreadPoolExecutor
>>> $Worker.run(ThreadPoolExecutor.java:907)
>>> at java.lang.Thread.run(Thread.java:637)
>>>
>>> I see two things in CHANGES.txt that might apply, but I'm not sure:
>>> 1. I think commons-csv was upgraded
>>> 2. The CSV loader stuff was refactored to share common code
>>>
>>> I'm still investigating.
>>>
>>> -Grant
>>
>> --------------------------
>> Grant Ingersoll
>> http://www.lucidimagination.com/
>>
>> Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)
>> using Solr/Lucene:
>> http://www.lucidimagination.com/search
--
===============================================================
Fergus McMenemie Email:fergus@twig.me.uk
Techmore Ltd Phone:(UK) 07721 376021
Unix/Mac/Intranets Analyst Programmer
===============================================================
Re: [solr-user] Upgrade from 1.2 to 1.3 gives 3x slowdown
Posted by Grant Ingersoll <gs...@apache.org>.
Fregus,
Is rev 643465 the absolute latest you tried that still performs? i.e.
every revision after is slower?
-Grant
On Mar 30, 2009, at 12:45 PM, Grant Ingersoll wrote:
> Fergus,
>
> I think the problem may actually be due to something that was
> introduced by a change to Solr's StopFilterFactory and the way it
> loads the stop words set. See https://issues.apache.org/jira/browse/SOLR-1095
>
> I am in the process of testing it out and will let you know.
>
> -Grant
>
> On Mar 28, 2009, at 11:00 AM, Grant Ingersoll wrote:
>
>> Hey Fergus,
>>
>> Finally got a chance to run your scripts, etc. per the thread:
>> http://www.lucidimagination.com/search/document/5c3de15a4e61095c/upgrade_from_1_2_to_1_3_gives_3x_slowdown_script#8324a98d8840c623
>>
>> I can reproduce your slowdown.
>>
>> One oddity with rev 643465 is:
>>
>> On the old version, there is an exception during startup:
>> Mar 28, 2009 10:44:31 AM org.apache.solr.common.SolrException log
>> SEVERE: java.lang.NullPointerException
>> at
>> org
>> .apache
>> .solr
>> .handler
>> .component.SearchHandler.handleRequestBody(SearchHandler.java:129)
>> at
>> org
>> .apache
>> .solr
>> .handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:
>> 125)
>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:953)
>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:968)
>> at
>> org
>> .apache
>> .solr.core.QuerySenderListener.newSearcher(QuerySenderListener.java:
>> 50)
>> at org.apache.solr.core.SolrCore$3.call(SolrCore.java:797)
>> at java.util.concurrent.FutureTask
>> $Sync.innerRun(FutureTask.java:303)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> at java.util.concurrent.ThreadPoolExecutor
>> $Worker.runTask(ThreadPoolExecutor.java:885)
>> at java.util.concurrent.ThreadPoolExecutor
>> $Worker.run(ThreadPoolExecutor.java:907)
>> at java.lang.Thread.run(Thread.java:637)
>>
>> I see two things in CHANGES.txt that might apply, but I'm not sure:
>> 1. I think commons-csv was upgraded
>> 2. The CSV loader stuff was refactored to share common code
>>
>> I'm still investigating.
>>
>> -Grant
>
> --------------------------
> Grant Ingersoll
> http://www.lucidimagination.com/
>
> Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)
> using Solr/Lucene:
> http://www.lucidimagination.com/search
>
--------------------------
Grant Ingersoll
http://www.lucidimagination.com/
Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)
using Solr/Lucene:
http://www.lucidimagination.com/search
Re: [solr-user] Upgrade from 1.2 to 1.3 gives 3x slowdown
Posted by Grant Ingersoll <gs...@apache.org>.
Fergus,
I think the problem may actually be due to something that was
introduced by a change to Solr's StopFilterFactory and the way it
loads the stop words set. See https://issues.apache.org/jira/browse/SOLR-1095
I am in the process of testing it out and will let you know.
-Grant
On Mar 28, 2009, at 11:00 AM, Grant Ingersoll wrote:
> Hey Fergus,
>
> Finally got a chance to run your scripts, etc. per the thread:
> http://www.lucidimagination.com/search/document/5c3de15a4e61095c/upgrade_from_1_2_to_1_3_gives_3x_slowdown_script#8324a98d8840c623
>
> I can reproduce your slowdown.
>
> One oddity with rev 643465 is:
>
> On the old version, there is an exception during startup:
> Mar 28, 2009 10:44:31 AM org.apache.solr.common.SolrException log
> SEVERE: java.lang.NullPointerException
> at
> org
> .apache
> .solr
> .handler
> .component.SearchHandler.handleRequestBody(SearchHandler.java:129)
> at
> org
> .apache
> .solr
> .handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:125)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:953)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:968)
> at
> org
> .apache
> .solr.core.QuerySenderListener.newSearcher(QuerySenderListener.java:
> 50)
> at org.apache.solr.core.SolrCore$3.call(SolrCore.java:797)
> at java.util.concurrent.FutureTask
> $Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor
> $Worker.runTask(ThreadPoolExecutor.java:885)
> at java.util.concurrent.ThreadPoolExecutor
> $Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:637)
>
> I see two things in CHANGES.txt that might apply, but I'm not sure:
> 1. I think commons-csv was upgraded
> 2. The CSV loader stuff was refactored to share common code
>
> I'm still investigating.
>
> -Grant
--------------------------
Grant Ingersoll
http://www.lucidimagination.com/
Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)
using Solr/Lucene:
http://www.lucidimagination.com/search