You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Bruno Mannina <bm...@free.fr> on 2012/06/19 09:09:32 UTC
Indexation Speed?
Dear All,
I would like to know if the indexation speed is right.
I have a 40Go file size with around 27 000 docs inside.
I index around 20 fields,
My (old) test server is a DualCore 3.06GHz Intel Xeon with only 1Go Ram
The file takes 40 seconds with the command line:
java -jar post.jar myfile.xml
Could I increase this speed or reduce this time?
Thanks a lot,
PS: Newbie user
Re: Indexation Speed?
Posted by Erik Hatcher <er...@gmail.com>.
I stand corrected. (and I "knew" that, just wasn't thinking fully when replying from my mobile)
I still say there's nothing wrong with using post.jar to send in Solr XML provided commits are managed appropriately. Are there any drawbacks others know of to using post.jar like one would use curl?
Erik
On Jun 22, 2012, at 04:37 , Jan Høydahl wrote:
> Actually, post.jar does not use SolrJ at all, thus it is simple and light with no dependencies. It simply uses HttpURLConnection :)
> Also have a look at SOLR-599 which has an interesting light-weight SolrJ server also without the extra deps.
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
> Solr Training - www.solrtraining.com
>
> On 21. juni 2012, at 00:23, Erik Hatcher wrote:
>
>> I think it's a bit of an "it depends" on whether post.jar is the Right choice for production.
>>
>> It -is- SolrJ inside after all, Erick :) and it's pretty much the same as using curl. Just be sure you control commits as needed.
>>
>> Erik
>>
>> On Jun 20, 2012, at 15:18, Bruno Mannina <bm...@free.fr> wrote:
>>
>>> Hi Erick,
>>>
>>>> I doubt you'll find any significant difference in indexing speed. But the
>>>> post.jar file is really intended as a demo program to quickly get the
>>>> examples working. It was never intended to be a production-ready
>>>> program. I'd think about using something like SolrJ etc. to index the docs.
>>>
>>> ah?! I don't know yet SolrJ :(
>>> I need to know how to program in java?
>>>
>>> I transformed all my xml source files to the xml structure below and I'm using post.jar
>>> I thought it was (post.jar) a standard tool to index docs.
>>>
>>>> And I'm assuming your documents are in the approved Solr format, somthing
>>>> like
>>>> <add>
>>>> <doc>
>>>> <field name="myfield">value for field</field>
>>>> .
>>>> .
>>>> </doc>
>>>> <doc>
>>>> .
>>>> .
>>>> .
>>>> </doc>
>>>> </add>
>>> Yes all my xml docs have this format.
>>>
>>>> solr will not index arbitrary XML. If you're trying to do this, you'll
>>>> need to transform
>>>> your arbitrary XML into the above format, consider SolrJ or something
>>>> like that in
>>>> this case.
>>>
>>> If all my xml docs are in the xml structure above, is it necessary to use SolrJ ?
>>>
>>>
>
Re: Indexation Speed?
Posted by Jan Høydahl <ja...@cominvent.com>.
Actually, post.jar does not use SolrJ at all, thus it is simple and light with no dependencies. It simply uses HttpURLConnection :)
Also have a look at SOLR-599 which has an interesting light-weight SolrJ server also without the extra deps.
--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com
Solr Training - www.solrtraining.com
On 21. juni 2012, at 00:23, Erik Hatcher wrote:
> I think it's a bit of an "it depends" on whether post.jar is the Right choice for production.
>
> It -is- SolrJ inside after all, Erick :) and it's pretty much the same as using curl. Just be sure you control commits as needed.
>
> Erik
>
> On Jun 20, 2012, at 15:18, Bruno Mannina <bm...@free.fr> wrote:
>
>> Hi Erick,
>>
>>> I doubt you'll find any significant difference in indexing speed. But the
>>> post.jar file is really intended as a demo program to quickly get the
>>> examples working. It was never intended to be a production-ready
>>> program. I'd think about using something like SolrJ etc. to index the docs.
>>
>> ah?! I don't know yet SolrJ :(
>> I need to know how to program in java?
>>
>> I transformed all my xml source files to the xml structure below and I'm using post.jar
>> I thought it was (post.jar) a standard tool to index docs.
>>
>>> And I'm assuming your documents are in the approved Solr format, somthing
>>> like
>>> <add>
>>> <doc>
>>> <field name="myfield">value for field</field>
>>> .
>>> .
>>> </doc>
>>> <doc>
>>> .
>>> .
>>> .
>>> </doc>
>>> </add>
>> Yes all my xml docs have this format.
>>
>>> solr will not index arbitrary XML. If you're trying to do this, you'll
>>> need to transform
>>> your arbitrary XML into the above format, consider SolrJ or something
>>> like that in
>>> this case.
>>
>> If all my xml docs are in the xml structure above, is it necessary to use SolrJ ?
>>
>>
Re: Indexation Speed?
Posted by Erik Hatcher <er...@gmail.com>.
I think it's a bit of an "it depends" on whether post.jar is the Right choice for production.
It -is- SolrJ inside after all, Erick :) and it's pretty much the same as using curl. Just be sure you control commits as needed.
Erik
On Jun 20, 2012, at 15:18, Bruno Mannina <bm...@free.fr> wrote:
> Hi Erick,
>
>> I doubt you'll find any significant difference in indexing speed. But the
>> post.jar file is really intended as a demo program to quickly get the
>> examples working. It was never intended to be a production-ready
>> program. I'd think about using something like SolrJ etc. to index the docs.
>
> ah?! I don't know yet SolrJ :(
> I need to know how to program in java?
>
> I transformed all my xml source files to the xml structure below and I'm using post.jar
> I thought it was (post.jar) a standard tool to index docs.
>
>> And I'm assuming your documents are in the approved Solr format, somthing
>> like
>> <add>
>> <doc>
>> <field name="myfield">value for field</field>
>> .
>> .
>> </doc>
>> <doc>
>> .
>> .
>> .
>> </doc>
>> </add>
> Yes all my xml docs have this format.
>
>> solr will not index arbitrary XML. If you're trying to do this, you'll
>> need to transform
>> your arbitrary XML into the above format, consider SolrJ or something
>> like that in
>> this case.
>
> If all my xml docs are in the xml structure above, is it necessary to use SolrJ ?
>
>
Re: Indexation Speed?
Posted by Bruno Mannina <bm...@free.fr>.
Hi Erick,
> I doubt you'll find any significant difference in indexing speed. But the
> post.jar file is really intended as a demo program to quickly get the
> examples working. It was never intended to be a production-ready
> program. I'd think about using something like SolrJ etc. to index the docs.
ah?! I don't know yet SolrJ :(
I need to know how to program in java?
I transformed all my xml source files to the xml structure below and I'm
using post.jar
I thought it was (post.jar) a standard tool to index docs.
> And I'm assuming your documents are in the approved Solr format, somthing
> like
> <add>
> <doc>
> <field name="myfield">value for field</field>
> .
> .
> </doc>
> <doc>
> .
> .
> .
> </doc>
> </add>
Yes all my xml docs have this format.
> solr will not index arbitrary XML. If you're trying to do this, you'll
> need to transform
> your arbitrary XML into the above format, consider SolrJ or something
> like that in
> this case.
If all my xml docs are in the xml structure above, is it necessary to
use SolrJ ?
Re: Indexation Speed?
Posted by Erick Erickson <er...@gmail.com>.
I doubt you'll find any significant difference in indexing speed. But the
post.jar file is really intended as a demo program to quickly get the
examples working. It was never intended to be a production-ready
program. I'd think about using something like SolrJ etc. to index the docs.
And I'm assuming your documents are in the approved Solr format, somthing
like
<add>
<doc>
<field name="myfield">value for field</field>
.
.
</doc>
<doc>
.
.
.
</doc>
</add>
solr will not index arbitrary XML. If you're trying to do this, you'll
need to transform
your arbitrary XML into the above format, consider SolrJ or something
like that in
this case.
Best
Erick
On Wed, Jun 20, 2012 at 10:40 AM, Bruno Mannina <bm...@free.fr> wrote:
> Little question please:
>
> I have directories with around 30 files of 40Mo with around 17 000 doc for
> each files.
>
> is it better to index:
> - file by file with java -jar 1.xml, java -jar 2.xml, etc....
> or
> - all at the same time with java -jar *.xml
>
> All files are verified, so my question is just concerning speed
>
> Thx for your comments,
> Bruno
>
>
>
> Le 20/06/2012 05:44, Lance Norskog a écrit :
>>
>> M. Della Bitta is right- we're not talking about post.jar, but starting
>> Solr:
>>
>>
>> java -xMx300m -jar start.jar
>>
>> On Tue, Jun 19, 2012 at 10:05 AM, Erick Erickson
>> <er...@gmail.com> wrote:
>>>
>>> Well, it _used_ to be defaulted in the code, but on looking at 3.6 it's
>>> seems
>>> like it defaults to Integer.MAX_VALUE, so you're fine....
>>>
>>> And it's all deprecated in 4.x, will be gone
>>>
>>> Best
>>> Erick
>>>
>>> On Tue, Jun 19, 2012 at 7:07 AM, Bruno Mannina<bm...@free.fr> wrote:
>>>>
>>>> Actually -Xmx512m and no effect
>>>>
>>>> Concerning maxFieldLength, no problem it's commented
>>>>
>>>> Le 19/06/2012 13:02, Erick Erickson a écrit :
>>>>
>>>>> Then try -Xmx600M
>>>>> next try -Xmx900M
>>>>>
>>>>>
>>>>> etc. The idea is to bump things on separate runs.
>>>>>
>>>>> But be a little cautious here. Look in your solrconfig.xml file, you'll
>>>>> see
>>>>> a commented-out line
>>>>> <maxFieldLength>10000</maxFieldLength>
>>>>>
>>>>> The default behavior for Solr/Lucene is to index the first 10,000
>>>>> tokens
>>>>> (not characters, think of tokens as words for not) in each
>>>>> document and throw the rest on the floor. At the sizes you're talking
>>>>> about,
>>>>> that's probably not a problem, but do be aware of it.
>>>>>
>>>>> Best
>>>>> Erick
>>>>>
>>>>> On Tue, Jun 19, 2012 at 5:44 AM, Bruno Mannina<bm...@free.fr>
>>>>> wrote:
>>>>>>
>>>>>> Like that?
>>>>>>
>>>>>> java -Xmx300m -jar post.jar myfile.xml
>>>>>>
>>>>>>
>>>>>>
>>>>>> Le 19/06/2012 11:11, Lance Norskog a écrit :
>>>>>>
>>>>>>> Ah! Java memory size is a java command line option:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> http://javahowto.blogspot.com/2006/06/6-common-errors-in-setting-java-heap.html
>>>>>>>
>>>>>>> You would try increasing the memory size in stages up to maybe 300m.
>>>>>>>
>>>>>>> On Tue, Jun 19, 2012 at 2:04 AM, Bruno Mannina<bm...@free.fr>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> Le 19/06/2012 10:51, Lance Norskog a écrit :
>>>>>>>>
>>>>>>>>> 675 doc/s is respectable for that server. You might move the memory
>>>>>>>>> allocated to Java up and down- there is a balance between amount of
>>>>>>>>> memory in Java v.s. the OS disk buffer.
>>>>>>>>
>>>>>>>>
>>>>>>>> How can I do that ? is there an option during my command line or in
>>>>>>>> a
>>>>>>>> config
>>>>>>>> file?
>>>>>>>> sorry for this newbie question :(
>>>>>>>>
>>>>>>>>
>>>>>>>>> And, of course, use the latest trunk.
>>>>>>>>
>>>>>>>> Solr 3.6
>>>>>>>>
>>>>>>>>
>>>>>>>>> On Tue, Jun 19, 2012 at 12:10 AM, Bruno Mannina<bm...@free.fr>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Correction: file size is 40 Mo !!!
>>>>>>>>>>
>>>>>>>>>> Le 19/06/2012 09:09, Bruno Mannina a écrit :
>>>>>>>>>>
>>>>>>>>>>> Dear All,
>>>>>>>>>>>
>>>>>>>>>>> I would like to know if the indexation speed is right.
>>>>>>>>>>>
>>>>>>>>>>> I have a 40Go file size with around 27 000 docs inside.
>>>>>>>>>>> I index around 20 fields,
>>>>>>>>>>>
>>>>>>>>>>> My (old) test server is a DualCore 3.06GHz Intel Xeon with only
>>>>>>>>>>> 1Go
>>>>>>>>>>> Ram
>>>>>>>>>>>
>>>>>>>>>>> The file takes 40 seconds with the command line:
>>>>>>>>>>> java -jar post.jar myfile.xml
>>>>>>>>>>>
>>>>>>>>>>> Could I increase this speed or reduce this time?
>>>>>>>>>>>
>>>>>>>>>>> Thanks a lot,
>>>>>>>>>>> PS: Newbie user
>>>>>>>>>>>
>>>>>>>>>>>
>>
>>
>
Re: Indexation Speed?
Posted by Bruno Mannina <bm...@free.fr>.
Little question please:
I have directories with around 30 files of 40Mo with around 17 000 doc
for each files.
is it better to index:
- file by file with java -jar 1.xml, java -jar 2.xml, etc....
or
- all at the same time with java -jar *.xml
All files are verified, so my question is just concerning speed
Thx for your comments,
Bruno
Le 20/06/2012 05:44, Lance Norskog a écrit :
> M. Della Bitta is right- we're not talking about post.jar, but starting Solr:
>
> java -xMx300m -jar start.jar
>
> On Tue, Jun 19, 2012 at 10:05 AM, Erick Erickson
> <er...@gmail.com> wrote:
>> Well, it _used_ to be defaulted in the code, but on looking at 3.6 it's seems
>> like it defaults to Integer.MAX_VALUE, so you're fine....
>>
>> And it's all deprecated in 4.x, will be gone
>>
>> Best
>> Erick
>>
>> On Tue, Jun 19, 2012 at 7:07 AM, Bruno Mannina<bm...@free.fr> wrote:
>>> Actually -Xmx512m and no effect
>>>
>>> Concerning maxFieldLength, no problem it's commented
>>>
>>> Le 19/06/2012 13:02, Erick Erickson a écrit :
>>>
>>>> Then try -Xmx600M
>>>> next try -Xmx900M
>>>>
>>>>
>>>> etc. The idea is to bump things on separate runs.
>>>>
>>>> But be a little cautious here. Look in your solrconfig.xml file, you'll
>>>> see
>>>> a commented-out line
>>>> <maxFieldLength>10000</maxFieldLength>
>>>>
>>>> The default behavior for Solr/Lucene is to index the first 10,000 tokens
>>>> (not characters, think of tokens as words for not) in each
>>>> document and throw the rest on the floor. At the sizes you're talking
>>>> about,
>>>> that's probably not a problem, but do be aware of it.
>>>>
>>>> Best
>>>> Erick
>>>>
>>>> On Tue, Jun 19, 2012 at 5:44 AM, Bruno Mannina<bm...@free.fr> wrote:
>>>>> Like that?
>>>>>
>>>>> java -Xmx300m -jar post.jar myfile.xml
>>>>>
>>>>>
>>>>>
>>>>> Le 19/06/2012 11:11, Lance Norskog a écrit :
>>>>>
>>>>>> Ah! Java memory size is a java command line option:
>>>>>>
>>>>>>
>>>>>> http://javahowto.blogspot.com/2006/06/6-common-errors-in-setting-java-heap.html
>>>>>>
>>>>>> You would try increasing the memory size in stages up to maybe 300m.
>>>>>>
>>>>>> On Tue, Jun 19, 2012 at 2:04 AM, Bruno Mannina<bm...@free.fr>
>>>>>> wrote:
>>>>>>>
>>>>>>> Le 19/06/2012 10:51, Lance Norskog a écrit :
>>>>>>>
>>>>>>>> 675 doc/s is respectable for that server. You might move the memory
>>>>>>>> allocated to Java up and down- there is a balance between amount of
>>>>>>>> memory in Java v.s. the OS disk buffer.
>>>>>>>
>>>>>>> How can I do that ? is there an option during my command line or in a
>>>>>>> config
>>>>>>> file?
>>>>>>> sorry for this newbie question :(
>>>>>>>
>>>>>>>
>>>>>>>> And, of course, use the latest trunk.
>>>>>>> Solr 3.6
>>>>>>>
>>>>>>>
>>>>>>>> On Tue, Jun 19, 2012 at 12:10 AM, Bruno Mannina<bm...@free.fr>
>>>>>>>> wrote:
>>>>>>>>> Correction: file size is 40 Mo !!!
>>>>>>>>>
>>>>>>>>> Le 19/06/2012 09:09, Bruno Mannina a écrit :
>>>>>>>>>
>>>>>>>>>> Dear All,
>>>>>>>>>>
>>>>>>>>>> I would like to know if the indexation speed is right.
>>>>>>>>>>
>>>>>>>>>> I have a 40Go file size with around 27 000 docs inside.
>>>>>>>>>> I index around 20 fields,
>>>>>>>>>>
>>>>>>>>>> My (old) test server is a DualCore 3.06GHz Intel Xeon with only 1Go
>>>>>>>>>> Ram
>>>>>>>>>>
>>>>>>>>>> The file takes 40 seconds with the command line:
>>>>>>>>>> java -jar post.jar myfile.xml
>>>>>>>>>>
>>>>>>>>>> Could I increase this speed or reduce this time?
>>>>>>>>>>
>>>>>>>>>> Thanks a lot,
>>>>>>>>>> PS: Newbie user
>>>>>>>>>>
>>>>>>>>>>
>
>
Re: Indexation Speed?
Posted by Bruno Mannina <bm...@free.fr>.
Ok thanks for this information,
Le 20/06/2012 05:44, Lance Norskog a écrit :
> M. Della Bitta is right- we're not talking about post.jar, but starting Solr:
>
> java -xMx300m -jar start.jar
>
> On Tue, Jun 19, 2012 at 10:05 AM, Erick Erickson
> <er...@gmail.com> wrote:
>> Well, it _used_ to be defaulted in the code, but on looking at 3.6 it's seems
>> like it defaults to Integer.MAX_VALUE, so you're fine....
>>
>> And it's all deprecated in 4.x, will be gone
>>
>> Best
>> Erick
>>
>> On Tue, Jun 19, 2012 at 7:07 AM, Bruno Mannina<bm...@free.fr> wrote:
>>> Actually -Xmx512m and no effect
>>>
>>> Concerning maxFieldLength, no problem it's commented
>>>
>>> Le 19/06/2012 13:02, Erick Erickson a écrit :
>>>
>>>> Then try -Xmx600M
>>>> next try -Xmx900M
>>>>
>>>>
>>>> etc. The idea is to bump things on separate runs.
>>>>
>>>> But be a little cautious here. Look in your solrconfig.xml file, you'll
>>>> see
>>>> a commented-out line
>>>> <maxFieldLength>10000</maxFieldLength>
>>>>
>>>> The default behavior for Solr/Lucene is to index the first 10,000 tokens
>>>> (not characters, think of tokens as words for not) in each
>>>> document and throw the rest on the floor. At the sizes you're talking
>>>> about,
>>>> that's probably not a problem, but do be aware of it.
>>>>
>>>> Best
>>>> Erick
>>>>
>>>> On Tue, Jun 19, 2012 at 5:44 AM, Bruno Mannina<bm...@free.fr> wrote:
>>>>> Like that?
>>>>>
>>>>> java -Xmx300m -jar post.jar myfile.xml
>>>>>
>>>>>
>>>>>
>>>>> Le 19/06/2012 11:11, Lance Norskog a écrit :
>>>>>
>>>>>> Ah! Java memory size is a java command line option:
>>>>>>
>>>>>>
>>>>>> http://javahowto.blogspot.com/2006/06/6-common-errors-in-setting-java-heap.html
>>>>>>
>>>>>> You would try increasing the memory size in stages up to maybe 300m.
>>>>>>
>>>>>> On Tue, Jun 19, 2012 at 2:04 AM, Bruno Mannina<bm...@free.fr>
>>>>>> wrote:
>>>>>>>
>>>>>>> Le 19/06/2012 10:51, Lance Norskog a écrit :
>>>>>>>
>>>>>>>> 675 doc/s is respectable for that server. You might move the memory
>>>>>>>> allocated to Java up and down- there is a balance between amount of
>>>>>>>> memory in Java v.s. the OS disk buffer.
>>>>>>>
>>>>>>> How can I do that ? is there an option during my command line or in a
>>>>>>> config
>>>>>>> file?
>>>>>>> sorry for this newbie question :(
>>>>>>>
>>>>>>>
>>>>>>>> And, of course, use the latest trunk.
>>>>>>> Solr 3.6
>>>>>>>
>>>>>>>
>>>>>>>> On Tue, Jun 19, 2012 at 12:10 AM, Bruno Mannina<bm...@free.fr>
>>>>>>>> wrote:
>>>>>>>>> Correction: file size is 40 Mo !!!
>>>>>>>>>
>>>>>>>>> Le 19/06/2012 09:09, Bruno Mannina a écrit :
>>>>>>>>>
>>>>>>>>>> Dear All,
>>>>>>>>>>
>>>>>>>>>> I would like to know if the indexation speed is right.
>>>>>>>>>>
>>>>>>>>>> I have a 40Go file size with around 27 000 docs inside.
>>>>>>>>>> I index around 20 fields,
>>>>>>>>>>
>>>>>>>>>> My (old) test server is a DualCore 3.06GHz Intel Xeon with only 1Go
>>>>>>>>>> Ram
>>>>>>>>>>
>>>>>>>>>> The file takes 40 seconds with the command line:
>>>>>>>>>> java -jar post.jar myfile.xml
>>>>>>>>>>
>>>>>>>>>> Could I increase this speed or reduce this time?
>>>>>>>>>>
>>>>>>>>>> Thanks a lot,
>>>>>>>>>> PS: Newbie user
>>>>>>>>>>
>>>>>>>>>>
>
>
Re: Indexation Speed?
Posted by Lance Norskog <go...@gmail.com>.
M. Della Bitta is right- we're not talking about post.jar, but starting Solr:
java -xMx300m -jar start.jar
On Tue, Jun 19, 2012 at 10:05 AM, Erick Erickson
<er...@gmail.com> wrote:
> Well, it _used_ to be defaulted in the code, but on looking at 3.6 it's seems
> like it defaults to Integer.MAX_VALUE, so you're fine....
>
> And it's all deprecated in 4.x, will be gone
>
> Best
> Erick
>
> On Tue, Jun 19, 2012 at 7:07 AM, Bruno Mannina <bm...@free.fr> wrote:
>> Actually -Xmx512m and no effect
>>
>> Concerning maxFieldLength, no problem it's commented
>>
>> Le 19/06/2012 13:02, Erick Erickson a écrit :
>>
>>> Then try -Xmx600M
>>> next try -Xmx900M
>>>
>>>
>>> etc. The idea is to bump things on separate runs.
>>>
>>> But be a little cautious here. Look in your solrconfig.xml file, you'll
>>> see
>>> a commented-out line
>>> <maxFieldLength>10000</maxFieldLength>
>>>
>>> The default behavior for Solr/Lucene is to index the first 10,000 tokens
>>> (not characters, think of tokens as words for not) in each
>>> document and throw the rest on the floor. At the sizes you're talking
>>> about,
>>> that's probably not a problem, but do be aware of it.
>>>
>>> Best
>>> Erick
>>>
>>> On Tue, Jun 19, 2012 at 5:44 AM, Bruno Mannina<bm...@free.fr> wrote:
>>>>
>>>> Like that?
>>>>
>>>> java -Xmx300m -jar post.jar myfile.xml
>>>>
>>>>
>>>>
>>>> Le 19/06/2012 11:11, Lance Norskog a écrit :
>>>>
>>>>> Ah! Java memory size is a java command line option:
>>>>>
>>>>>
>>>>> http://javahowto.blogspot.com/2006/06/6-common-errors-in-setting-java-heap.html
>>>>>
>>>>> You would try increasing the memory size in stages up to maybe 300m.
>>>>>
>>>>> On Tue, Jun 19, 2012 at 2:04 AM, Bruno Mannina<bm...@free.fr>
>>>>> wrote:
>>>>>>
>>>>>>
>>>>>> Le 19/06/2012 10:51, Lance Norskog a écrit :
>>>>>>
>>>>>>> 675 doc/s is respectable for that server. You might move the memory
>>>>>>> allocated to Java up and down- there is a balance between amount of
>>>>>>> memory in Java v.s. the OS disk buffer.
>>>>>>
>>>>>>
>>>>>> How can I do that ? is there an option during my command line or in a
>>>>>> config
>>>>>> file?
>>>>>> sorry for this newbie question :(
>>>>>>
>>>>>>
>>>>>>> And, of course, use the latest trunk.
>>>>>>
>>>>>> Solr 3.6
>>>>>>
>>>>>>
>>>>>>> On Tue, Jun 19, 2012 at 12:10 AM, Bruno Mannina<bm...@free.fr>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Correction: file size is 40 Mo !!!
>>>>>>>>
>>>>>>>> Le 19/06/2012 09:09, Bruno Mannina a écrit :
>>>>>>>>
>>>>>>>>> Dear All,
>>>>>>>>>
>>>>>>>>> I would like to know if the indexation speed is right.
>>>>>>>>>
>>>>>>>>> I have a 40Go file size with around 27 000 docs inside.
>>>>>>>>> I index around 20 fields,
>>>>>>>>>
>>>>>>>>> My (old) test server is a DualCore 3.06GHz Intel Xeon with only 1Go
>>>>>>>>> Ram
>>>>>>>>>
>>>>>>>>> The file takes 40 seconds with the command line:
>>>>>>>>> java -jar post.jar myfile.xml
>>>>>>>>>
>>>>>>>>> Could I increase this speed or reduce this time?
>>>>>>>>>
>>>>>>>>> Thanks a lot,
>>>>>>>>> PS: Newbie user
>>>>>>>>>
>>>>>>>>>
>>>>>
>>>
>>
--
Lance Norskog
goksron@gmail.com
Re: Indexation Speed?
Posted by Erick Erickson <er...@gmail.com>.
Well, it _used_ to be defaulted in the code, but on looking at 3.6 it's seems
like it defaults to Integer.MAX_VALUE, so you're fine....
And it's all deprecated in 4.x, will be gone
Best
Erick
On Tue, Jun 19, 2012 at 7:07 AM, Bruno Mannina <bm...@free.fr> wrote:
> Actually -Xmx512m and no effect
>
> Concerning maxFieldLength, no problem it's commented
>
> Le 19/06/2012 13:02, Erick Erickson a écrit :
>
>> Then try -Xmx600M
>> next try -Xmx900M
>>
>>
>> etc. The idea is to bump things on separate runs.
>>
>> But be a little cautious here. Look in your solrconfig.xml file, you'll
>> see
>> a commented-out line
>> <maxFieldLength>10000</maxFieldLength>
>>
>> The default behavior for Solr/Lucene is to index the first 10,000 tokens
>> (not characters, think of tokens as words for not) in each
>> document and throw the rest on the floor. At the sizes you're talking
>> about,
>> that's probably not a problem, but do be aware of it.
>>
>> Best
>> Erick
>>
>> On Tue, Jun 19, 2012 at 5:44 AM, Bruno Mannina<bm...@free.fr> wrote:
>>>
>>> Like that?
>>>
>>> java -Xmx300m -jar post.jar myfile.xml
>>>
>>>
>>>
>>> Le 19/06/2012 11:11, Lance Norskog a écrit :
>>>
>>>> Ah! Java memory size is a java command line option:
>>>>
>>>>
>>>> http://javahowto.blogspot.com/2006/06/6-common-errors-in-setting-java-heap.html
>>>>
>>>> You would try increasing the memory size in stages up to maybe 300m.
>>>>
>>>> On Tue, Jun 19, 2012 at 2:04 AM, Bruno Mannina<bm...@free.fr>
>>>> wrote:
>>>>>
>>>>>
>>>>> Le 19/06/2012 10:51, Lance Norskog a écrit :
>>>>>
>>>>>> 675 doc/s is respectable for that server. You might move the memory
>>>>>> allocated to Java up and down- there is a balance between amount of
>>>>>> memory in Java v.s. the OS disk buffer.
>>>>>
>>>>>
>>>>> How can I do that ? is there an option during my command line or in a
>>>>> config
>>>>> file?
>>>>> sorry for this newbie question :(
>>>>>
>>>>>
>>>>>> And, of course, use the latest trunk.
>>>>>
>>>>> Solr 3.6
>>>>>
>>>>>
>>>>>> On Tue, Jun 19, 2012 at 12:10 AM, Bruno Mannina<bm...@free.fr>
>>>>>> wrote:
>>>>>>>
>>>>>>> Correction: file size is 40 Mo !!!
>>>>>>>
>>>>>>> Le 19/06/2012 09:09, Bruno Mannina a écrit :
>>>>>>>
>>>>>>>> Dear All,
>>>>>>>>
>>>>>>>> I would like to know if the indexation speed is right.
>>>>>>>>
>>>>>>>> I have a 40Go file size with around 27 000 docs inside.
>>>>>>>> I index around 20 fields,
>>>>>>>>
>>>>>>>> My (old) test server is a DualCore 3.06GHz Intel Xeon with only 1Go
>>>>>>>> Ram
>>>>>>>>
>>>>>>>> The file takes 40 seconds with the command line:
>>>>>>>> java -jar post.jar myfile.xml
>>>>>>>>
>>>>>>>> Could I increase this speed or reduce this time?
>>>>>>>>
>>>>>>>> Thanks a lot,
>>>>>>>> PS: Newbie user
>>>>>>>>
>>>>>>>>
>>>>
>>
>
Re: Indexation Speed?
Posted by Michael Della Bitta <mi...@appinions.com>.
Just an observation...
The OP is tweaking the heap size on post.jar, but wouldn't some tuning
of the container that Solr is running in also be in order?
Michael Della Bitta
------------------------------------------------
Appinions, Inc. -- Where Influence Isn’t a Game.
http://www.appinions.com
On Tue, Jun 19, 2012 at 9:14 AM, François Schiettecatte
<fs...@gmail.com> wrote:
> There is a lot of good information about that on the web, just google for 'ubuntu performance monitor'
>
> Also the ubuntu website has a pretty good help section:
>
> https://help.ubuntu.com/
>
> and a community wiki:
>
> https://help.ubuntu.com/community
>
> Cheers
>
> François
>
> On Jun 19, 2012, at 9:03 AM, Bruno Mannina wrote:
>
>> Linux Ubuntu :) since 2 months ! so I'm a new in this world :)
>>
>> Le 19/06/2012 15:01, François Schiettecatte a écrit :
>>> Well that depends on the platform you are on, you did not mention that.
>>>
>>> If you are using linux, you could use atop ( http://www.atoptool.nl/ ), or top, or iostat or stat, or all four.
>>>
>>> Cheers
>>>
>>> François
>>>
>>> On Jun 19, 2012, at 8:55 AM, Bruno Mannina wrote:
>>>
>>>> CPU is not used, just 50-60% sometimes during the process but How can I check IO HDD ?
>>>>
>>>> Le 19/06/2012 14:13, François Schiettecatte a écrit :
>>>>> Just a suggestion, you might want to monitor CPU usage and disk I/O, there might be a bottleneck.
>>>>>
>>>>> Cheers
>>>>>
>>>>> François
>>>>>
>>>>> On Jun 19, 2012, at 7:07 AM, Bruno Mannina wrote:
>>>>>
>>>>>> Actually -Xmx512m and no effect
>>>>>>
>>>>>> Concerning maxFieldLength, no problem it's commented
>>>>>>
>>>>>> Le 19/06/2012 13:02, Erick Erickson a écrit :
>>>>>>> Then try -Xmx600M
>>>>>>> next try -Xmx900M
>>>>>>>
>>>>>>>
>>>>>>> etc. The idea is to bump things on separate runs.
>>>>>>>
>>>>>>> But be a little cautious here. Look in your solrconfig.xml file, you'll see
>>>>>>> a commented-out line
>>>>>>> <maxFieldLength>10000</maxFieldLength>
>>>>>>>
>>>>>>> The default behavior for Solr/Lucene is to index the first 10,000 tokens
>>>>>>> (not characters, think of tokens as words for not) in each
>>>>>>> document and throw the rest on the floor. At the sizes you're talking about,
>>>>>>> that's probably not a problem, but do be aware of it.
>>>>>>>
>>>>>>> Best
>>>>>>> Erick
>>>>>>>
>>>>>>> On Tue, Jun 19, 2012 at 5:44 AM, Bruno Mannina<bm...@free.fr> wrote:
>>>>>>>> Like that?
>>>>>>>>
>>>>>>>> java -Xmx300m -jar post.jar myfile.xml
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Le 19/06/2012 11:11, Lance Norskog a écrit :
>>>>>>>>
>>>>>>>>> Ah! Java memory size is a java command line option:
>>>>>>>>>
>>>>>>>>> http://javahowto.blogspot.com/2006/06/6-common-errors-in-setting-java-heap.html
>>>>>>>>>
>>>>>>>>> You would try increasing the memory size in stages up to maybe 300m.
>>>>>>>>>
>>>>>>>>> On Tue, Jun 19, 2012 at 2:04 AM, Bruno Mannina<bm...@free.fr> wrote:
>>>>>>>>>> Le 19/06/2012 10:51, Lance Norskog a écrit :
>>>>>>>>>>
>>>>>>>>>>> 675 doc/s is respectable for that server. You might move the memory
>>>>>>>>>>> allocated to Java up and down- there is a balance between amount of
>>>>>>>>>>> memory in Java v.s. the OS disk buffer.
>>>>>>>>>> How can I do that ? is there an option during my command line or in a
>>>>>>>>>> config
>>>>>>>>>> file?
>>>>>>>>>> sorry for this newbie question :(
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> And, of course, use the latest trunk.
>>>>>>>>>> Solr 3.6
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> On Tue, Jun 19, 2012 at 12:10 AM, Bruno Mannina<bm...@free.fr>
>>>>>>>>>>> wrote:
>>>>>>>>>>>> Correction: file size is 40 Mo !!!
>>>>>>>>>>>>
>>>>>>>>>>>> Le 19/06/2012 09:09, Bruno Mannina a écrit :
>>>>>>>>>>>>
>>>>>>>>>>>>> Dear All,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I would like to know if the indexation speed is right.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have a 40Go file size with around 27 000 docs inside.
>>>>>>>>>>>>> I index around 20 fields,
>>>>>>>>>>>>>
>>>>>>>>>>>>> My (old) test server is a DualCore 3.06GHz Intel Xeon with only 1Go
>>>>>>>>>>>>> Ram
>>>>>>>>>>>>>
>>>>>>>>>>>>> The file takes 40 seconds with the command line:
>>>>>>>>>>>>> java -jar post.jar myfile.xml
>>>>>>>>>>>>>
>>>>>>>>>>>>> Could I increase this speed or reduce this time?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks a lot,
>>>>>>>>>>>>> PS: Newbie user
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>
>>>
>>>
>>
>
Re: Indexation Speed?
Posted by François Schiettecatte <fs...@gmail.com>.
There is a lot of good information about that on the web, just google for 'ubuntu performance monitor'
Also the ubuntu website has a pretty good help section:
https://help.ubuntu.com/
and a community wiki:
https://help.ubuntu.com/community
Cheers
François
On Jun 19, 2012, at 9:03 AM, Bruno Mannina wrote:
> Linux Ubuntu :) since 2 months ! so I'm a new in this world :)
>
> Le 19/06/2012 15:01, François Schiettecatte a écrit :
>> Well that depends on the platform you are on, you did not mention that.
>>
>> If you are using linux, you could use atop ( http://www.atoptool.nl/ ), or top, or iostat or stat, or all four.
>>
>> Cheers
>>
>> François
>>
>> On Jun 19, 2012, at 8:55 AM, Bruno Mannina wrote:
>>
>>> CPU is not used, just 50-60% sometimes during the process but How can I check IO HDD ?
>>>
>>> Le 19/06/2012 14:13, François Schiettecatte a écrit :
>>>> Just a suggestion, you might want to monitor CPU usage and disk I/O, there might be a bottleneck.
>>>>
>>>> Cheers
>>>>
>>>> François
>>>>
>>>> On Jun 19, 2012, at 7:07 AM, Bruno Mannina wrote:
>>>>
>>>>> Actually -Xmx512m and no effect
>>>>>
>>>>> Concerning maxFieldLength, no problem it's commented
>>>>>
>>>>> Le 19/06/2012 13:02, Erick Erickson a écrit :
>>>>>> Then try -Xmx600M
>>>>>> next try -Xmx900M
>>>>>>
>>>>>>
>>>>>> etc. The idea is to bump things on separate runs.
>>>>>>
>>>>>> But be a little cautious here. Look in your solrconfig.xml file, you'll see
>>>>>> a commented-out line
>>>>>> <maxFieldLength>10000</maxFieldLength>
>>>>>>
>>>>>> The default behavior for Solr/Lucene is to index the first 10,000 tokens
>>>>>> (not characters, think of tokens as words for not) in each
>>>>>> document and throw the rest on the floor. At the sizes you're talking about,
>>>>>> that's probably not a problem, but do be aware of it.
>>>>>>
>>>>>> Best
>>>>>> Erick
>>>>>>
>>>>>> On Tue, Jun 19, 2012 at 5:44 AM, Bruno Mannina<bm...@free.fr> wrote:
>>>>>>> Like that?
>>>>>>>
>>>>>>> java -Xmx300m -jar post.jar myfile.xml
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Le 19/06/2012 11:11, Lance Norskog a écrit :
>>>>>>>
>>>>>>>> Ah! Java memory size is a java command line option:
>>>>>>>>
>>>>>>>> http://javahowto.blogspot.com/2006/06/6-common-errors-in-setting-java-heap.html
>>>>>>>>
>>>>>>>> You would try increasing the memory size in stages up to maybe 300m.
>>>>>>>>
>>>>>>>> On Tue, Jun 19, 2012 at 2:04 AM, Bruno Mannina<bm...@free.fr> wrote:
>>>>>>>>> Le 19/06/2012 10:51, Lance Norskog a écrit :
>>>>>>>>>
>>>>>>>>>> 675 doc/s is respectable for that server. You might move the memory
>>>>>>>>>> allocated to Java up and down- there is a balance between amount of
>>>>>>>>>> memory in Java v.s. the OS disk buffer.
>>>>>>>>> How can I do that ? is there an option during my command line or in a
>>>>>>>>> config
>>>>>>>>> file?
>>>>>>>>> sorry for this newbie question :(
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> And, of course, use the latest trunk.
>>>>>>>>> Solr 3.6
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On Tue, Jun 19, 2012 at 12:10 AM, Bruno Mannina<bm...@free.fr>
>>>>>>>>>> wrote:
>>>>>>>>>>> Correction: file size is 40 Mo !!!
>>>>>>>>>>>
>>>>>>>>>>> Le 19/06/2012 09:09, Bruno Mannina a écrit :
>>>>>>>>>>>
>>>>>>>>>>>> Dear All,
>>>>>>>>>>>>
>>>>>>>>>>>> I would like to know if the indexation speed is right.
>>>>>>>>>>>>
>>>>>>>>>>>> I have a 40Go file size with around 27 000 docs inside.
>>>>>>>>>>>> I index around 20 fields,
>>>>>>>>>>>>
>>>>>>>>>>>> My (old) test server is a DualCore 3.06GHz Intel Xeon with only 1Go
>>>>>>>>>>>> Ram
>>>>>>>>>>>>
>>>>>>>>>>>> The file takes 40 seconds with the command line:
>>>>>>>>>>>> java -jar post.jar myfile.xml
>>>>>>>>>>>>
>>>>>>>>>>>> Could I increase this speed or reduce this time?
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks a lot,
>>>>>>>>>>>> PS: Newbie user
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>
>>
>>
>
Re: Indexation Speed?
Posted by Bruno Mannina <bm...@free.fr>.
Linux Ubuntu :) since 2 months ! so I'm a new in this world :)
Le 19/06/2012 15:01, François Schiettecatte a écrit :
> Well that depends on the platform you are on, you did not mention that.
>
> If you are using linux, you could use atop ( http://www.atoptool.nl/ ), or top, or iostat or stat, or all four.
>
> Cheers
>
> François
>
> On Jun 19, 2012, at 8:55 AM, Bruno Mannina wrote:
>
>> CPU is not used, just 50-60% sometimes during the process but How can I check IO HDD ?
>>
>> Le 19/06/2012 14:13, François Schiettecatte a écrit :
>>> Just a suggestion, you might want to monitor CPU usage and disk I/O, there might be a bottleneck.
>>>
>>> Cheers
>>>
>>> François
>>>
>>> On Jun 19, 2012, at 7:07 AM, Bruno Mannina wrote:
>>>
>>>> Actually -Xmx512m and no effect
>>>>
>>>> Concerning maxFieldLength, no problem it's commented
>>>>
>>>> Le 19/06/2012 13:02, Erick Erickson a écrit :
>>>>> Then try -Xmx600M
>>>>> next try -Xmx900M
>>>>>
>>>>>
>>>>> etc. The idea is to bump things on separate runs.
>>>>>
>>>>> But be a little cautious here. Look in your solrconfig.xml file, you'll see
>>>>> a commented-out line
>>>>> <maxFieldLength>10000</maxFieldLength>
>>>>>
>>>>> The default behavior for Solr/Lucene is to index the first 10,000 tokens
>>>>> (not characters, think of tokens as words for not) in each
>>>>> document and throw the rest on the floor. At the sizes you're talking about,
>>>>> that's probably not a problem, but do be aware of it.
>>>>>
>>>>> Best
>>>>> Erick
>>>>>
>>>>> On Tue, Jun 19, 2012 at 5:44 AM, Bruno Mannina<bm...@free.fr> wrote:
>>>>>> Like that?
>>>>>>
>>>>>> java -Xmx300m -jar post.jar myfile.xml
>>>>>>
>>>>>>
>>>>>>
>>>>>> Le 19/06/2012 11:11, Lance Norskog a écrit :
>>>>>>
>>>>>>> Ah! Java memory size is a java command line option:
>>>>>>>
>>>>>>> http://javahowto.blogspot.com/2006/06/6-common-errors-in-setting-java-heap.html
>>>>>>>
>>>>>>> You would try increasing the memory size in stages up to maybe 300m.
>>>>>>>
>>>>>>> On Tue, Jun 19, 2012 at 2:04 AM, Bruno Mannina<bm...@free.fr> wrote:
>>>>>>>> Le 19/06/2012 10:51, Lance Norskog a écrit :
>>>>>>>>
>>>>>>>>> 675 doc/s is respectable for that server. You might move the memory
>>>>>>>>> allocated to Java up and down- there is a balance between amount of
>>>>>>>>> memory in Java v.s. the OS disk buffer.
>>>>>>>> How can I do that ? is there an option during my command line or in a
>>>>>>>> config
>>>>>>>> file?
>>>>>>>> sorry for this newbie question :(
>>>>>>>>
>>>>>>>>
>>>>>>>>> And, of course, use the latest trunk.
>>>>>>>> Solr 3.6
>>>>>>>>
>>>>>>>>
>>>>>>>>> On Tue, Jun 19, 2012 at 12:10 AM, Bruno Mannina<bm...@free.fr>
>>>>>>>>> wrote:
>>>>>>>>>> Correction: file size is 40 Mo !!!
>>>>>>>>>>
>>>>>>>>>> Le 19/06/2012 09:09, Bruno Mannina a écrit :
>>>>>>>>>>
>>>>>>>>>>> Dear All,
>>>>>>>>>>>
>>>>>>>>>>> I would like to know if the indexation speed is right.
>>>>>>>>>>>
>>>>>>>>>>> I have a 40Go file size with around 27 000 docs inside.
>>>>>>>>>>> I index around 20 fields,
>>>>>>>>>>>
>>>>>>>>>>> My (old) test server is a DualCore 3.06GHz Intel Xeon with only 1Go
>>>>>>>>>>> Ram
>>>>>>>>>>>
>>>>>>>>>>> The file takes 40 seconds with the command line:
>>>>>>>>>>> java -jar post.jar myfile.xml
>>>>>>>>>>>
>>>>>>>>>>> Could I increase this speed or reduce this time?
>>>>>>>>>>>
>>>>>>>>>>> Thanks a lot,
>>>>>>>>>>> PS: Newbie user
>>>>>>>>>>>
>>>>>>>>>>>
>>>
>
>
Re: Indexation Speed?
Posted by François Schiettecatte <fs...@gmail.com>.
Well that depends on the platform you are on, you did not mention that.
If you are using linux, you could use atop ( http://www.atoptool.nl/ ), or top, or iostat or stat, or all four.
Cheers
François
On Jun 19, 2012, at 8:55 AM, Bruno Mannina wrote:
> CPU is not used, just 50-60% sometimes during the process but How can I check IO HDD ?
>
> Le 19/06/2012 14:13, François Schiettecatte a écrit :
>> Just a suggestion, you might want to monitor CPU usage and disk I/O, there might be a bottleneck.
>>
>> Cheers
>>
>> François
>>
>> On Jun 19, 2012, at 7:07 AM, Bruno Mannina wrote:
>>
>>> Actually -Xmx512m and no effect
>>>
>>> Concerning maxFieldLength, no problem it's commented
>>>
>>> Le 19/06/2012 13:02, Erick Erickson a écrit :
>>>> Then try -Xmx600M
>>>> next try -Xmx900M
>>>>
>>>>
>>>> etc. The idea is to bump things on separate runs.
>>>>
>>>> But be a little cautious here. Look in your solrconfig.xml file, you'll see
>>>> a commented-out line
>>>> <maxFieldLength>10000</maxFieldLength>
>>>>
>>>> The default behavior for Solr/Lucene is to index the first 10,000 tokens
>>>> (not characters, think of tokens as words for not) in each
>>>> document and throw the rest on the floor. At the sizes you're talking about,
>>>> that's probably not a problem, but do be aware of it.
>>>>
>>>> Best
>>>> Erick
>>>>
>>>> On Tue, Jun 19, 2012 at 5:44 AM, Bruno Mannina<bm...@free.fr> wrote:
>>>>> Like that?
>>>>>
>>>>> java -Xmx300m -jar post.jar myfile.xml
>>>>>
>>>>>
>>>>>
>>>>> Le 19/06/2012 11:11, Lance Norskog a écrit :
>>>>>
>>>>>> Ah! Java memory size is a java command line option:
>>>>>>
>>>>>> http://javahowto.blogspot.com/2006/06/6-common-errors-in-setting-java-heap.html
>>>>>>
>>>>>> You would try increasing the memory size in stages up to maybe 300m.
>>>>>>
>>>>>> On Tue, Jun 19, 2012 at 2:04 AM, Bruno Mannina<bm...@free.fr> wrote:
>>>>>>> Le 19/06/2012 10:51, Lance Norskog a écrit :
>>>>>>>
>>>>>>>> 675 doc/s is respectable for that server. You might move the memory
>>>>>>>> allocated to Java up and down- there is a balance between amount of
>>>>>>>> memory in Java v.s. the OS disk buffer.
>>>>>>> How can I do that ? is there an option during my command line or in a
>>>>>>> config
>>>>>>> file?
>>>>>>> sorry for this newbie question :(
>>>>>>>
>>>>>>>
>>>>>>>> And, of course, use the latest trunk.
>>>>>>> Solr 3.6
>>>>>>>
>>>>>>>
>>>>>>>> On Tue, Jun 19, 2012 at 12:10 AM, Bruno Mannina<bm...@free.fr>
>>>>>>>> wrote:
>>>>>>>>> Correction: file size is 40 Mo !!!
>>>>>>>>>
>>>>>>>>> Le 19/06/2012 09:09, Bruno Mannina a écrit :
>>>>>>>>>
>>>>>>>>>> Dear All,
>>>>>>>>>>
>>>>>>>>>> I would like to know if the indexation speed is right.
>>>>>>>>>>
>>>>>>>>>> I have a 40Go file size with around 27 000 docs inside.
>>>>>>>>>> I index around 20 fields,
>>>>>>>>>>
>>>>>>>>>> My (old) test server is a DualCore 3.06GHz Intel Xeon with only 1Go
>>>>>>>>>> Ram
>>>>>>>>>>
>>>>>>>>>> The file takes 40 seconds with the command line:
>>>>>>>>>> java -jar post.jar myfile.xml
>>>>>>>>>>
>>>>>>>>>> Could I increase this speed or reduce this time?
>>>>>>>>>>
>>>>>>>>>> Thanks a lot,
>>>>>>>>>> PS: Newbie user
>>>>>>>>>>
>>>>>>>>>>
>>
>>
>
Re: Indexation Speed?
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Bruno,
Look at SPM for Solr ( http://sematext.com/spm ) - very handy for understanding disk IO vs. CPU vs. JVM GC, etc. during indexing/performance testing.
You could also play with ramBufferSizeMB in solrconfig.xml
Otis
----
Performance Monitoring for Solr / ElasticSearch / HBase - http://sematext.com/spm
>________________________________
> From: Bruno Mannina <bm...@free.fr>
>To: solr-user@lucene.apache.org
>Sent: Tuesday, June 19, 2012 8:55 AM
>Subject: Re: Indexation Speed?
>
>CPU is not used, just 50-60% sometimes during the process but How can I
>check IO HDD ?
>
>Le 19/06/2012 14:13, François Schiettecatte a écrit :
>> Just a suggestion, you might want to monitor CPU usage and disk I/O, there might be a bottleneck.
>>
>> Cheers
>>
>> François
>>
>> On Jun 19, 2012, at 7:07 AM, Bruno Mannina wrote:
>>
>>> Actually -Xmx512m and no effect
>>>
>>> Concerning maxFieldLength, no problem it's commented
>>>
>>> Le 19/06/2012 13:02, Erick Erickson a écrit :
>>>> Then try -Xmx600M
>>>> next try -Xmx900M
>>>>
>>>>
>>>> etc. The idea is to bump things on separate runs.
>>>>
>>>> But be a little cautious here. Look in your solrconfig.xml file, you'll see
>>>> a commented-out line
>>>> <maxFieldLength>10000</maxFieldLength>
>>>>
>>>> The default behavior for Solr/Lucene is to index the first 10,000 tokens
>>>> (not characters, think of tokens as words for not) in each
>>>> document and throw the rest on the floor. At the sizes you're talking about,
>>>> that's probably not a problem, but do be aware of it.
>>>>
>>>> Best
>>>> Erick
>>>>
>>>> On Tue, Jun 19, 2012 at 5:44 AM, Bruno Mannina<bm...@free.fr> wrote:
>>>>> Like that?
>>>>>
>>>>> java -Xmx300m -jar post.jar myfile.xml
>>>>>
>>>>>
>>>>>
>>>>> Le 19/06/2012 11:11, Lance Norskog a écrit :
>>>>>
>>>>>> Ah! Java memory size is a java command line option:
>>>>>>
>>>>>> http://javahowto.blogspot.com/2006/06/6-common-errors-in-setting-java-heap.html
>>>>>>
>>>>>> You would try increasing the memory size in stages up to maybe 300m.
>>>>>>
>>>>>> On Tue, Jun 19, 2012 at 2:04 AM, Bruno Mannina<bm...@free.fr> wrote:
>>>>>>> Le 19/06/2012 10:51, Lance Norskog a écrit :
>>>>>>>
>>>>>>>> 675 doc/s is respectable for that server. You might move the memory
>>>>>>>> allocated to Java up and down- there is a balance between amount of
>>>>>>>> memory in Java v.s. the OS disk buffer.
>>>>>>> How can I do that ? is there an option during my command line or in a
>>>>>>> config
>>>>>>> file?
>>>>>>> sorry for this newbie question :(
>>>>>>>
>>>>>>>
>>>>>>>> And, of course, use the latest trunk.
>>>>>>> Solr 3.6
>>>>>>>
>>>>>>>
>>>>>>>> On Tue, Jun 19, 2012 at 12:10 AM, Bruno Mannina<bm...@free.fr>
>>>>>>>> wrote:
>>>>>>>>> Correction: file size is 40 Mo !!!
>>>>>>>>>
>>>>>>>>> Le 19/06/2012 09:09, Bruno Mannina a écrit :
>>>>>>>>>
>>>>>>>>>> Dear All,
>>>>>>>>>>
>>>>>>>>>> I would like to know if the indexation speed is right.
>>>>>>>>>>
>>>>>>>>>> I have a 40Go file size with around 27 000 docs inside.
>>>>>>>>>> I index around 20 fields,
>>>>>>>>>>
>>>>>>>>>> My (old) test server is a DualCore 3.06GHz Intel Xeon with only 1Go
>>>>>>>>>> Ram
>>>>>>>>>>
>>>>>>>>>> The file takes 40 seconds with the command line:
>>>>>>>>>> java -jar post.jar myfile.xml
>>>>>>>>>>
>>>>>>>>>> Could I increase this speed or reduce this time?
>>>>>>>>>>
>>>>>>>>>> Thanks a lot,
>>>>>>>>>> PS: Newbie user
>>>>>>>>>>
>>>>>>>>>>
>>
>>
>
>
>
>
Re: Indexation Speed?
Posted by Bruno Mannina <bm...@free.fr>.
CPU is not used, just 50-60% sometimes during the process but How can I
check IO HDD ?
Le 19/06/2012 14:13, François Schiettecatte a écrit :
> Just a suggestion, you might want to monitor CPU usage and disk I/O, there might be a bottleneck.
>
> Cheers
>
> François
>
> On Jun 19, 2012, at 7:07 AM, Bruno Mannina wrote:
>
>> Actually -Xmx512m and no effect
>>
>> Concerning maxFieldLength, no problem it's commented
>>
>> Le 19/06/2012 13:02, Erick Erickson a écrit :
>>> Then try -Xmx600M
>>> next try -Xmx900M
>>>
>>>
>>> etc. The idea is to bump things on separate runs.
>>>
>>> But be a little cautious here. Look in your solrconfig.xml file, you'll see
>>> a commented-out line
>>> <maxFieldLength>10000</maxFieldLength>
>>>
>>> The default behavior for Solr/Lucene is to index the first 10,000 tokens
>>> (not characters, think of tokens as words for not) in each
>>> document and throw the rest on the floor. At the sizes you're talking about,
>>> that's probably not a problem, but do be aware of it.
>>>
>>> Best
>>> Erick
>>>
>>> On Tue, Jun 19, 2012 at 5:44 AM, Bruno Mannina<bm...@free.fr> wrote:
>>>> Like that?
>>>>
>>>> java -Xmx300m -jar post.jar myfile.xml
>>>>
>>>>
>>>>
>>>> Le 19/06/2012 11:11, Lance Norskog a écrit :
>>>>
>>>>> Ah! Java memory size is a java command line option:
>>>>>
>>>>> http://javahowto.blogspot.com/2006/06/6-common-errors-in-setting-java-heap.html
>>>>>
>>>>> You would try increasing the memory size in stages up to maybe 300m.
>>>>>
>>>>> On Tue, Jun 19, 2012 at 2:04 AM, Bruno Mannina<bm...@free.fr> wrote:
>>>>>> Le 19/06/2012 10:51, Lance Norskog a écrit :
>>>>>>
>>>>>>> 675 doc/s is respectable for that server. You might move the memory
>>>>>>> allocated to Java up and down- there is a balance between amount of
>>>>>>> memory in Java v.s. the OS disk buffer.
>>>>>> How can I do that ? is there an option during my command line or in a
>>>>>> config
>>>>>> file?
>>>>>> sorry for this newbie question :(
>>>>>>
>>>>>>
>>>>>>> And, of course, use the latest trunk.
>>>>>> Solr 3.6
>>>>>>
>>>>>>
>>>>>>> On Tue, Jun 19, 2012 at 12:10 AM, Bruno Mannina<bm...@free.fr>
>>>>>>> wrote:
>>>>>>>> Correction: file size is 40 Mo !!!
>>>>>>>>
>>>>>>>> Le 19/06/2012 09:09, Bruno Mannina a écrit :
>>>>>>>>
>>>>>>>>> Dear All,
>>>>>>>>>
>>>>>>>>> I would like to know if the indexation speed is right.
>>>>>>>>>
>>>>>>>>> I have a 40Go file size with around 27 000 docs inside.
>>>>>>>>> I index around 20 fields,
>>>>>>>>>
>>>>>>>>> My (old) test server is a DualCore 3.06GHz Intel Xeon with only 1Go
>>>>>>>>> Ram
>>>>>>>>>
>>>>>>>>> The file takes 40 seconds with the command line:
>>>>>>>>> java -jar post.jar myfile.xml
>>>>>>>>>
>>>>>>>>> Could I increase this speed or reduce this time?
>>>>>>>>>
>>>>>>>>> Thanks a lot,
>>>>>>>>> PS: Newbie user
>>>>>>>>>
>>>>>>>>>
>
>
Re: Indexation Speed?
Posted by François Schiettecatte <fs...@gmail.com>.
Just a suggestion, you might want to monitor CPU usage and disk I/O, there might be a bottleneck.
Cheers
François
On Jun 19, 2012, at 7:07 AM, Bruno Mannina wrote:
> Actually -Xmx512m and no effect
>
> Concerning maxFieldLength, no problem it's commented
>
> Le 19/06/2012 13:02, Erick Erickson a écrit :
>> Then try -Xmx600M
>> next try -Xmx900M
>>
>>
>> etc. The idea is to bump things on separate runs.
>>
>> But be a little cautious here. Look in your solrconfig.xml file, you'll see
>> a commented-out line
>> <maxFieldLength>10000</maxFieldLength>
>>
>> The default behavior for Solr/Lucene is to index the first 10,000 tokens
>> (not characters, think of tokens as words for not) in each
>> document and throw the rest on the floor. At the sizes you're talking about,
>> that's probably not a problem, but do be aware of it.
>>
>> Best
>> Erick
>>
>> On Tue, Jun 19, 2012 at 5:44 AM, Bruno Mannina<bm...@free.fr> wrote:
>>> Like that?
>>>
>>> java -Xmx300m -jar post.jar myfile.xml
>>>
>>>
>>>
>>> Le 19/06/2012 11:11, Lance Norskog a écrit :
>>>
>>>> Ah! Java memory size is a java command line option:
>>>>
>>>> http://javahowto.blogspot.com/2006/06/6-common-errors-in-setting-java-heap.html
>>>>
>>>> You would try increasing the memory size in stages up to maybe 300m.
>>>>
>>>> On Tue, Jun 19, 2012 at 2:04 AM, Bruno Mannina<bm...@free.fr> wrote:
>>>>>
>>>>> Le 19/06/2012 10:51, Lance Norskog a écrit :
>>>>>
>>>>>> 675 doc/s is respectable for that server. You might move the memory
>>>>>> allocated to Java up and down- there is a balance between amount of
>>>>>> memory in Java v.s. the OS disk buffer.
>>>>>
>>>>> How can I do that ? is there an option during my command line or in a
>>>>> config
>>>>> file?
>>>>> sorry for this newbie question :(
>>>>>
>>>>>
>>>>>> And, of course, use the latest trunk.
>>>>> Solr 3.6
>>>>>
>>>>>
>>>>>> On Tue, Jun 19, 2012 at 12:10 AM, Bruno Mannina<bm...@free.fr>
>>>>>> wrote:
>>>>>>> Correction: file size is 40 Mo !!!
>>>>>>>
>>>>>>> Le 19/06/2012 09:09, Bruno Mannina a écrit :
>>>>>>>
>>>>>>>> Dear All,
>>>>>>>>
>>>>>>>> I would like to know if the indexation speed is right.
>>>>>>>>
>>>>>>>> I have a 40Go file size with around 27 000 docs inside.
>>>>>>>> I index around 20 fields,
>>>>>>>>
>>>>>>>> My (old) test server is a DualCore 3.06GHz Intel Xeon with only 1Go
>>>>>>>> Ram
>>>>>>>>
>>>>>>>> The file takes 40 seconds with the command line:
>>>>>>>> java -jar post.jar myfile.xml
>>>>>>>>
>>>>>>>> Could I increase this speed or reduce this time?
>>>>>>>>
>>>>>>>> Thanks a lot,
>>>>>>>> PS: Newbie user
>>>>>>>>
>>>>>>>>
>>>>
>>
>
Re: Indexation Speed?
Posted by Bruno Mannina <bm...@free.fr>.
Actually -Xmx512m and no effect
Concerning maxFieldLength, no problem it's commented
Le 19/06/2012 13:02, Erick Erickson a écrit :
> Then try -Xmx600M
> next try -Xmx900M
>
>
> etc. The idea is to bump things on separate runs.
>
> But be a little cautious here. Look in your solrconfig.xml file, you'll see
> a commented-out line
> <maxFieldLength>10000</maxFieldLength>
>
> The default behavior for Solr/Lucene is to index the first 10,000 tokens
> (not characters, think of tokens as words for not) in each
> document and throw the rest on the floor. At the sizes you're talking about,
> that's probably not a problem, but do be aware of it.
>
> Best
> Erick
>
> On Tue, Jun 19, 2012 at 5:44 AM, Bruno Mannina<bm...@free.fr> wrote:
>> Like that?
>>
>> java -Xmx300m -jar post.jar myfile.xml
>>
>>
>>
>> Le 19/06/2012 11:11, Lance Norskog a écrit :
>>
>>> Ah! Java memory size is a java command line option:
>>>
>>> http://javahowto.blogspot.com/2006/06/6-common-errors-in-setting-java-heap.html
>>>
>>> You would try increasing the memory size in stages up to maybe 300m.
>>>
>>> On Tue, Jun 19, 2012 at 2:04 AM, Bruno Mannina<bm...@free.fr> wrote:
>>>>
>>>> Le 19/06/2012 10:51, Lance Norskog a écrit :
>>>>
>>>>> 675 doc/s is respectable for that server. You might move the memory
>>>>> allocated to Java up and down- there is a balance between amount of
>>>>> memory in Java v.s. the OS disk buffer.
>>>>
>>>> How can I do that ? is there an option during my command line or in a
>>>> config
>>>> file?
>>>> sorry for this newbie question :(
>>>>
>>>>
>>>>> And, of course, use the latest trunk.
>>>> Solr 3.6
>>>>
>>>>
>>>>> On Tue, Jun 19, 2012 at 12:10 AM, Bruno Mannina<bm...@free.fr>
>>>>> wrote:
>>>>>> Correction: file size is 40 Mo !!!
>>>>>>
>>>>>> Le 19/06/2012 09:09, Bruno Mannina a écrit :
>>>>>>
>>>>>>> Dear All,
>>>>>>>
>>>>>>> I would like to know if the indexation speed is right.
>>>>>>>
>>>>>>> I have a 40Go file size with around 27 000 docs inside.
>>>>>>> I index around 20 fields,
>>>>>>>
>>>>>>> My (old) test server is a DualCore 3.06GHz Intel Xeon with only 1Go
>>>>>>> Ram
>>>>>>>
>>>>>>> The file takes 40 seconds with the command line:
>>>>>>> java -jar post.jar myfile.xml
>>>>>>>
>>>>>>> Could I increase this speed or reduce this time?
>>>>>>>
>>>>>>> Thanks a lot,
>>>>>>> PS: Newbie user
>>>>>>>
>>>>>>>
>>>
>
Re: Indexation Speed?
Posted by Erick Erickson <er...@gmail.com>.
Then try -Xmx600M
next try -Xmx900M
etc. The idea is to bump things on separate runs.
But be a little cautious here. Look in your solrconfig.xml file, you'll see
a commented-out line
<maxFieldLength>10000</maxFieldLength>
The default behavior for Solr/Lucene is to index the first 10,000 tokens
(not characters, think of tokens as words for not) in each
document and throw the rest on the floor. At the sizes you're talking about,
that's probably not a problem, but do be aware of it.
Best
Erick
On Tue, Jun 19, 2012 at 5:44 AM, Bruno Mannina <bm...@free.fr> wrote:
> Like that?
>
> java -Xmx300m -jar post.jar myfile.xml
>
>
>
> Le 19/06/2012 11:11, Lance Norskog a écrit :
>
>> Ah! Java memory size is a java command line option:
>>
>> http://javahowto.blogspot.com/2006/06/6-common-errors-in-setting-java-heap.html
>>
>> You would try increasing the memory size in stages up to maybe 300m.
>>
>> On Tue, Jun 19, 2012 at 2:04 AM, Bruno Mannina<bm...@free.fr> wrote:
>>>
>>>
>>> Le 19/06/2012 10:51, Lance Norskog a écrit :
>>>
>>>> 675 doc/s is respectable for that server. You might move the memory
>>>> allocated to Java up and down- there is a balance between amount of
>>>> memory in Java v.s. the OS disk buffer.
>>>
>>>
>>> How can I do that ? is there an option during my command line or in a
>>> config
>>> file?
>>> sorry for this newbie question :(
>>>
>>>
>>>> And, of course, use the latest trunk.
>>>
>>> Solr 3.6
>>>
>>>
>>>> On Tue, Jun 19, 2012 at 12:10 AM, Bruno Mannina<bm...@free.fr>
>>>> wrote:
>>>>>
>>>>> Correction: file size is 40 Mo !!!
>>>>>
>>>>> Le 19/06/2012 09:09, Bruno Mannina a écrit :
>>>>>
>>>>>> Dear All,
>>>>>>
>>>>>> I would like to know if the indexation speed is right.
>>>>>>
>>>>>> I have a 40Go file size with around 27 000 docs inside.
>>>>>> I index around 20 fields,
>>>>>>
>>>>>> My (old) test server is a DualCore 3.06GHz Intel Xeon with only 1Go
>>>>>> Ram
>>>>>>
>>>>>> The file takes 40 seconds with the command line:
>>>>>> java -jar post.jar myfile.xml
>>>>>>
>>>>>> Could I increase this speed or reduce this time?
>>>>>>
>>>>>> Thanks a lot,
>>>>>> PS: Newbie user
>>>>>>
>>>>>>
>>>>
>>
>>
>
Re: Indexation Speed?
Posted by Bruno Mannina <bm...@free.fr>.
Like that?
java -Xmx300m -jar post.jar myfile.xml
Le 19/06/2012 11:11, Lance Norskog a écrit :
> Ah! Java memory size is a java command line option:
> http://javahowto.blogspot.com/2006/06/6-common-errors-in-setting-java-heap.html
>
> You would try increasing the memory size in stages up to maybe 300m.
>
> On Tue, Jun 19, 2012 at 2:04 AM, Bruno Mannina<bm...@free.fr> wrote:
>>
>> Le 19/06/2012 10:51, Lance Norskog a écrit :
>>
>>> 675 doc/s is respectable for that server. You might move the memory
>>> allocated to Java up and down- there is a balance between amount of
>>> memory in Java v.s. the OS disk buffer.
>>
>> How can I do that ? is there an option during my command line or in a config
>> file?
>> sorry for this newbie question :(
>>
>>
>>> And, of course, use the latest trunk.
>> Solr 3.6
>>
>>
>>> On Tue, Jun 19, 2012 at 12:10 AM, Bruno Mannina<bm...@free.fr> wrote:
>>>> Correction: file size is 40 Mo !!!
>>>>
>>>> Le 19/06/2012 09:09, Bruno Mannina a écrit :
>>>>
>>>>> Dear All,
>>>>>
>>>>> I would like to know if the indexation speed is right.
>>>>>
>>>>> I have a 40Go file size with around 27 000 docs inside.
>>>>> I index around 20 fields,
>>>>>
>>>>> My (old) test server is a DualCore 3.06GHz Intel Xeon with only 1Go Ram
>>>>>
>>>>> The file takes 40 seconds with the command line:
>>>>> java -jar post.jar myfile.xml
>>>>>
>>>>> Could I increase this speed or reduce this time?
>>>>>
>>>>> Thanks a lot,
>>>>> PS: Newbie user
>>>>>
>>>>>
>>>
>
>
Re: Indexation Speed?
Posted by Lance Norskog <go...@gmail.com>.
Ah! Java memory size is a java command line option:
http://javahowto.blogspot.com/2006/06/6-common-errors-in-setting-java-heap.html
You would try increasing the memory size in stages up to maybe 300m.
On Tue, Jun 19, 2012 at 2:04 AM, Bruno Mannina <bm...@free.fr> wrote:
>
>
> Le 19/06/2012 10:51, Lance Norskog a écrit :
>
>> 675 doc/s is respectable for that server. You might move the memory
>> allocated to Java up and down- there is a balance between amount of
>> memory in Java v.s. the OS disk buffer.
>
>
> How can I do that ? is there an option during my command line or in a config
> file?
> sorry for this newbie question :(
>
>
>> And, of course, use the latest trunk.
>
> Solr 3.6
>
>
>> On Tue, Jun 19, 2012 at 12:10 AM, Bruno Mannina<bm...@free.fr> wrote:
>>>
>>> Correction: file size is 40 Mo !!!
>>>
>>> Le 19/06/2012 09:09, Bruno Mannina a écrit :
>>>
>>>> Dear All,
>>>>
>>>> I would like to know if the indexation speed is right.
>>>>
>>>> I have a 40Go file size with around 27 000 docs inside.
>>>> I index around 20 fields,
>>>>
>>>> My (old) test server is a DualCore 3.06GHz Intel Xeon with only 1Go Ram
>>>>
>>>> The file takes 40 seconds with the command line:
>>>> java -jar post.jar myfile.xml
>>>>
>>>> Could I increase this speed or reduce this time?
>>>>
>>>> Thanks a lot,
>>>> PS: Newbie user
>>>>
>>>>
>>
>>
>
--
Lance Norskog
goksron@gmail.com
Re: Indexation Speed?
Posted by Bruno Mannina <bm...@free.fr>.
Le 19/06/2012 10:51, Lance Norskog a écrit :
> 675 doc/s is respectable for that server. You might move the memory
> allocated to Java up and down- there is a balance between amount of
> memory in Java v.s. the OS disk buffer.
How can I do that ? is there an option during my command line or in a
config file?
sorry for this newbie question :(
> And, of course, use the latest trunk.
Solr 3.6
> On Tue, Jun 19, 2012 at 12:10 AM, Bruno Mannina<bm...@free.fr> wrote:
>> Correction: file size is 40 Mo !!!
>>
>> Le 19/06/2012 09:09, Bruno Mannina a écrit :
>>
>>> Dear All,
>>>
>>> I would like to know if the indexation speed is right.
>>>
>>> I have a 40Go file size with around 27 000 docs inside.
>>> I index around 20 fields,
>>>
>>> My (old) test server is a DualCore 3.06GHz Intel Xeon with only 1Go Ram
>>>
>>> The file takes 40 seconds with the command line:
>>> java -jar post.jar myfile.xml
>>>
>>> Could I increase this speed or reduce this time?
>>>
>>> Thanks a lot,
>>> PS: Newbie user
>>>
>>>
>
>
Re: Indexation Speed?
Posted by Lance Norskog <go...@gmail.com>.
675 doc/s is respectable for that server. You might move the memory
allocated to Java up and down- there is a balance between amount of
memory in Java v.s. the OS disk buffer.
And, of course, use the latest trunk.
On Tue, Jun 19, 2012 at 12:10 AM, Bruno Mannina <bm...@free.fr> wrote:
> Correction: file size is 40 Mo !!!
>
> Le 19/06/2012 09:09, Bruno Mannina a écrit :
>
>> Dear All,
>>
>> I would like to know if the indexation speed is right.
>>
>> I have a 40Go file size with around 27 000 docs inside.
>> I index around 20 fields,
>>
>> My (old) test server is a DualCore 3.06GHz Intel Xeon with only 1Go Ram
>>
>> The file takes 40 seconds with the command line:
>> java -jar post.jar myfile.xml
>>
>> Could I increase this speed or reduce this time?
>>
>> Thanks a lot,
>> PS: Newbie user
>>
>>
>
--
Lance Norskog
goksron@gmail.com
Re: Indexation Speed?
Posted by Bruno Mannina <bm...@free.fr>.
Correction: file size is 40 Mo !!!
Le 19/06/2012 09:09, Bruno Mannina a écrit :
> Dear All,
>
> I would like to know if the indexation speed is right.
>
> I have a 40Go file size with around 27 000 docs inside.
> I index around 20 fields,
>
> My (old) test server is a DualCore 3.06GHz Intel Xeon with only 1Go Ram
>
> The file takes 40 seconds with the command line:
> java -jar post.jar myfile.xml
>
> Could I increase this speed or reduce this time?
>
> Thanks a lot,
> PS: Newbie user
>
>