You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Marc Sturlese <ma...@gmail.com> on 2009/02/03 09:31:14 UTC
DIH stopping an action
Hey there,
I would like to know if there is any way to stop a dela-import or a
full-import in the middle of the ejecution and free Tomcats memory.
In case not... is there any way to tell Solr to stop all actions and free
all memory that is using?
Is it possible to do one of both things without restarting Tomcat??
Thanks in advance.
--
View this message in context: http://www.nabble.com/DIH-stopping-an-action-tp21805669p21805669.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: DIH stopping an action
Posted by Marc Sturlese <ma...@gmail.com>.
I just opened an issue explaining my solution:
https://issues.apache.org/jira/browse/SOLR-1004
Shalin Shekhar Mangar wrote:
>
> On Tue, Feb 3, 2009 at 4:06 PM, Marc Sturlese
> <ma...@gmail.com>wrote:
>
>>
>> Doing that, once a doc is aborted in DocBuilder, it will not keep
>> checking
>> all other docs and abort will finish soon.
>> I think it could be done in the function deleteAll(deletedKeys); in case
>> the
>> amount of docs to delete is huge aswell.
>>
>> Has to do that any bad consequence? In case not... do you think it would
>> be
>> useful to add it in dataimporthandler for other use cases?
>>
>
> You are right Marc. We should be getting out of that loop (in
> buildDocument
> as well as in collectDelta) if abort is called. Can you please a raise an
> issue in jira?
>
> --
>> View this message in context:
>> http://www.nabble.com/DIH-stopping-an-action-tp21805669p21807365.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
>
--
View this message in context: http://www.nabble.com/DIH-stopping-an-action---possible-improvement-tp21805669p21808760.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: DIH stopping an action
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
On Tue, Feb 3, 2009 at 4:06 PM, Marc Sturlese <ma...@gmail.com>wrote:
>
> Doing that, once a doc is aborted in DocBuilder, it will not keep checking
> all other docs and abort will finish soon.
> I think it could be done in the function deleteAll(deletedKeys); in case
> the
> amount of docs to delete is huge aswell.
>
> Has to do that any bad consequence? In case not... do you think it would be
> useful to add it in dataimporthandler for other use cases?
>
You are right Marc. We should be getting out of that loop (in buildDocument
as well as in collectDelta) if abort is called. Can you please a raise an
issue in jira?
--
> View this message in context:
> http://www.nabble.com/DIH-stopping-an-action-tp21805669p21807365.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
--
Regards,
Shalin Shekhar Mangar.
Re: DIH stopping an action
Posted by Marc Sturlese <ma...@gmail.com>.
Hey Shalin,
I have been testing de abort command and for full-import there's no problem.
In delta-import, at DocBuilder.java I have seen it checks for
if (stop.get())
before executing deleteAll and inside collectDelta (in doDelta function).
The problem is that once you have the Set<Map<String, Object>> with all de
data to modify, it will just check for if (stop.get()) inside the function
BuilDocuement. In my case, I have 300.000 docs to modifiy so, as
BuildDocuement in doDelta is called inside a while it will pass for all
300.000 aborting all of them. What I have done is check if there is abortion
inside the while:
while (pkIter.hasNext()) {
Map<String, Object> map = pkIter.next();
vri.addNamespace(DataConfig.IMPORTER_NS + ".delta", map);
buildDocument(vri, null, map, root, true, null,true);
pkIter.remove();
//#patch checking if abortion
if (stop.get()) { return; }
}
This part of code is from doDelta in DocBuilder.
Doing that, once a doc is aborted in DocBuilder, it will not keep checking
all other docs and abort will finish soon.
I think it could be done in the function deleteAll(deletedKeys); in case the
amount of docs to delete is huge aswell.
Has to do that any bad consequence? In case not... do you think it would be
useful to add it in dataimporthandler for other use cases?
Marc Sturlese wrote:
>
> Thanks, that's exactly what I need.
>
> Shalin Shekhar Mangar wrote:
>>
>> On Tue, Feb 3, 2009 at 2:01 PM, Marc Sturlese
>> <ma...@gmail.com>wrote:
>>
>>>
>>> Hey there,
>>> I would like to know if there is any way to stop a dela-import or a
>>> full-import in the middle of the ejecution and free Tomcats memory.
>>
>>
>> There is an 'abort' command for DIH which should do what you want. Most
>> of
>> the DIH related objects should go out of scope once import is aborted.
>> Then
>> it is upto the garbage collector to free the memory.
>>
>> --
>> Regards,
>> Shalin Shekhar Mangar.
>>
>>
>
>
--
View this message in context: http://www.nabble.com/DIH-stopping-an-action-tp21805669p21807365.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: DIH stopping an action
Posted by Marc Sturlese <ma...@gmail.com>.
Thanks, that's exactly what I need.
Shalin Shekhar Mangar wrote:
>
> On Tue, Feb 3, 2009 at 2:01 PM, Marc Sturlese
> <ma...@gmail.com>wrote:
>
>>
>> Hey there,
>> I would like to know if there is any way to stop a dela-import or a
>> full-import in the middle of the ejecution and free Tomcats memory.
>
>
> There is an 'abort' command for DIH which should do what you want. Most of
> the DIH related objects should go out of scope once import is aborted.
> Then
> it is upto the garbage collector to free the memory.
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
>
--
View this message in context: http://www.nabble.com/DIH-stopping-an-action-tp21805669p21805823.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: DIH stopping an action
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
On Tue, Feb 3, 2009 at 2:01 PM, Marc Sturlese <ma...@gmail.com>wrote:
>
> Hey there,
> I would like to know if there is any way to stop a dela-import or a
> full-import in the middle of the ejecution and free Tomcats memory.
There is an 'abort' command for DIH which should do what you want. Most of
the DIH related objects should go out of scope once import is aborted. Then
it is upto the garbage collector to free the memory.
--
Regards,
Shalin Shekhar Mangar.