Re: Solr1.3 / MySql / Tomcat55 multiple delta-import inside a big full-import

[Shalin Shekhar Mangar <sh...@gmail.com>]

On Fri, Oct 31, 2008 at 3:27 PM, sunnyfr <jo...@gmail.com> wrote:

>
> I would like to know if it's very longer to make a limited full import and
> multi delta-import to index all the database.
> If I fire a full-import without limit 4M in my request that will run me OOM
> because I've 8,5M of document.
> If I fire a full-import without limit and a batchsize=-1 I will stock the
> database just for me, and stack other request for 10hours, but it will
> work.
>

A full-import should not stack other requests though response time will be
more because of the heavy processing.

Most users have a dedicated Master instance used only for indexing and many
slaves dedicated for serving search requests. Maybe you can try a
master-slave architecture.

-- 
Regards,
Shalin Shekhar Mangar.

Re: Solr1.3 / MySql / Tomcat55 multiple delta-import inside a big full-import

[sunnyfr <jo...@gmail.com>]

Do you have an idea ? 


sunnyfr wrote:
> 
> Sorry I wasn't clear,
> The stack is not on solr database or index query, stack request are on our
> main database MySql,
> When I do a full import to create indexes for solr, MySql honnor it and
> won't drive it OOM, but with a batchsize -1, it uses MySql memory which
> let less memory for the rest of the request on the dabase like, insert,
> update, delete ...
> 
> :) thanks for your answer,
> 
> 
> Shalin Shekhar Mangar wrote:
>> 
>> On Fri, Oct 31, 2008 at 3:27 PM, sunnyfr <jo...@gmail.com> wrote:
>> 
>>>
>>> I would like to know if it's very longer to make a limited full import
>>> and
>>> multi delta-import to index all the database.
>>> If I fire a full-import without limit 4M in my request that will run me
>>> OOM
>>> because I've 8,5M of document.
>>> If I fire a full-import without limit and a batchsize=-1 I will stock
>>> the
>>> database just for me, and stack other request for 10hours, but it will
>>> work.
>>>
>> 
>> A full-import should not stack other requests though response time will
>> be
>> more because of the heavy processing.
>> 
>> Most users have a dedicated Master instance used only for indexing and
>> many
>> slaves dedicated for serving search requests. Maybe you can try a
>> master-slave architecture.
>> 
>> -- 
>> Regards,
>> Shalin Shekhar Mangar.
>> 
>> 
> 
> 

-- 
View this message in context: http://www.nabble.com/Solr1.3---MySql---Tomcat55--multiple-delta-import-inside-a-big-full-import-tp20262801p20302944.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr1.3 / MySql / Tomcat55 multiple delta-import inside a big full-import

[sunnyfr <jo...@gmail.com>]

Sorry I wasn't clear,
The stack is not on solr database or index query, stack request are on our
main database MySql,
When I do a full import to create indexes for solr, MySql honnor it and
won't drive it OOM, but with a batchsize -1, it uses MySql memory which let
less memory for the rest of the request on the dabase like, insert, update,
delete ...

:) thanks for your answer,


Shalin Shekhar Mangar wrote:
> 
> On Fri, Oct 31, 2008 at 3:27 PM, sunnyfr <jo...@gmail.com> wrote:
> 
>>
>> I would like to know if it's very longer to make a limited full import
>> and
>> multi delta-import to index all the database.
>> If I fire a full-import without limit 4M in my request that will run me
>> OOM
>> because I've 8,5M of document.
>> If I fire a full-import without limit and a batchsize=-1 I will stock the
>> database just for me, and stack other request for 10hours, but it will
>> work.
>>
> 
> A full-import should not stack other requests though response time will be
> more because of the heavy processing.
> 
> Most users have a dedicated Master instance used only for indexing and
> many
> slaves dedicated for serving search requests. Maybe you can try a
> master-slave architecture.
> 
> -- 
> Regards,
> Shalin Shekhar Mangar.
> 
> 

-- 
View this message in context: http://www.nabble.com/Solr1.3---MySql---Tomcat55--multiple-delta-import-inside-a-big-full-import-tp20262801p20264431.html
Sent from the Solr - User mailing list archive at Nabble.com.