You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Nicholas Ball <ni...@nodelay.com> on 2012/06/30 15:50:17 UTC
Atomic Multicore Operations - E.G. Move Docs
Hey all,
Trying to figure out the best way to perform atomic operation across
multiple cores on the same solr instance i.e. a multi-core environment.
An example would be to move a set of docs from one core onto another core
and ensure that a softcommit is done as the exact same time. If one were to
fail so would the other.
Obviously this would probably require some customization but wanted to
know what the best way to tackle this would be and where should I be
looking in the source.
Many thanks for the help in advance,
Nicholas a.k.a. incunix
Re: Atomic Multicore Operations - E.G. Move Docs
Posted by Li Li <fa...@gmail.com>.
在 2012-7-2 傍晚6:37，"Nicholas Ball" <ni...@nodelay.com>写道：
>
>
> That could work, but then how do you ensure commit is called on the two
> cores at the exact same time?
that may needs something like two phrase commit in relational dB. lucene
has prepareCommit, but to implement 2pc, many things need to do.
> Also, any way to commit a specific update rather then all the back-logged
> ones?
>
> Cheers,
> Nicholas
>
> On Sat, 30 Jun 2012 16:19:31 -0700, Lance Norskog <go...@gmail.com>
> wrote:
> > Index all documents to both cores, but do not call commit until both
> > report that indexing worked. If one of the cores throws an exception,
> > call roll back on both cores.
> >
> > On Sat, Jun 30, 2012 at 6:50 AM, Nicholas Ball
> > <ni...@nodelay.com> wrote:
> >>
> >> Hey all,
> >>
> >> Trying to figure out the best way to perform atomic operation across
> >> multiple cores on the same solr instance i.e. a multi-core environment.
> >>
> >> An example would be to move a set of docs from one core onto another
> core
> >> and ensure that a softcommit is done as the exact same time. If one
> were
> >> to
> >> fail so would the other.
> >> Obviously this would probably require some customization but wanted to
> >> know what the best way to tackle this would be and where should I be
> >> looking in the source.
> >>
> >> Many thanks for the help in advance,
> >> Nicholas a.k.a. incunix
Re: Atomic Multicore Operations - E.G. Move Docs
Posted by Lance Norskog <go...@gmail.com>.
Yes, by generations I meant versioning. The problem is that you have
to have a central holder of the current generation number. ZK does
this very well. It is a distributed synchronized file system for very
small files. If you have a more natural place to store the current
generation number, that's fine also.
On Mon, Aug 20, 2012 at 2:47 PM, Nicholas Ball
<ni...@nodelay.com> wrote:
>
> hi lance,
>
> how would that work? generation is essentially versioning right?
> i also don't see why you need to use zk to do this as it's all on a single
> machine, was hoping for a simpler solution :)
>
> On Sun, 19 Aug 2012 19:26:41 -0700, Lance Norskog <go...@gmail.com>
> wrote:
>> I would use generation numbers on documents, and communicate a global
>> generation number in ZK.
>>
>> On Thu, Aug 16, 2012 at 2:22 AM, Nicholas Ball
>> <ni...@nodelay.com> wrote:
>>>
>>> I've been close to implementing a 2PC protocol before for something
> else,
>>> however for this it's not needed.
>>> As the move operation will be done on a single node which has both the
>>> cores, this could be done differently. Just not entirely sure how to do
>>> it.
>>>
>>> When a commit is done at the moment, the core must get locked somehow,
> it
>>> is at this point where we should lock the other core too if a move
>>> operation is being executed.
>>>
>>> Nick
>>>
>>> On Thu, 16 Aug 2012 10:32:10 +0800, Li Li <fa...@gmail.com> wrote:
>>>>
>>>
> http://zookeeper.apache.org/doc/r3.3.6/recipes.html#sc_recipes_twoPhasedCommit
>>>>
>>>> On Thu, Aug 16, 2012 at 7:41 AM, Nicholas Ball
>>>> <ni...@nodelay.com> wrote:
>>>>>
>>>>> Haven't managed to find a good way to do this yet. Does anyone have
> any
>>>>> ideas on how I could implement this feature?
>>>>> Really need to move docs across from one core to another atomically.
>>>>>
>>>>> Many thanks,
>>>>> Nicholas
>>>>>
>>>>> On Mon, 02 Jul 2012 04:37:12 -0600, Nicholas Ball
>>>>> <ni...@nodelay.com> wrote:
>>>>>> That could work, but then how do you ensure commit is called on the
>>> two
>>>>>> cores at the exact same time?
>>>>>>
>>>>>> Cheers,
>>>>>> Nicholas
>>>>>>
>>>>>> On Sat, 30 Jun 2012 16:19:31 -0700, Lance Norskog
> <go...@gmail.com>
>>>>>> wrote:
>>>>>>> Index all documents to both cores, but do not call commit until
> both
>>>>>>> report that indexing worked. If one of the cores throws an
> exception,
>>>>>>> call roll back on both cores.
>>>>>>>
>>>>>>> On Sat, Jun 30, 2012 at 6:50 AM, Nicholas Ball
>>>>>>> <ni...@nodelay.com> wrote:
>>>>>>>>
>>>>>>>> Hey all,
>>>>>>>>
>>>>>>>> Trying to figure out the best way to perform atomic operation
> across
>>>>>>>> multiple cores on the same solr instance i.e. a multi-core
>>>>> environment.
>>>>>>>>
>>>>>>>> An example would be to move a set of docs from one core onto
> another
>>>>>> core
>>>>>>>> and ensure that a softcommit is done as the exact same time. If
> one
>>>>>> were
>>>>>>>> to
>>>>>>>> fail so would the other.
>>>>>>>> Obviously this would probably require some customization but
> wanted
>>> to
>>>>>>>> know what the best way to tackle this would be and where should I
> be
>>>>>>>> looking in the source.
>>>>>>>>
>>>>>>>> Many thanks for the help in advance,
>>>>>>>> Nicholas a.k.a. incunix
--
Lance Norskog
goksron@gmail.com
Re: Atomic Multicore Operations - E.G. Move Docs
Posted by Nicholas Ball <ni...@nodelay.com>.
hi lance,
how would that work? generation is essentially versioning right?
i also don't see why you need to use zk to do this as it's all on a single
machine, was hoping for a simpler solution :)
On Sun, 19 Aug 2012 19:26:41 -0700, Lance Norskog <go...@gmail.com>
wrote:
> I would use generation numbers on documents, and communicate a global
> generation number in ZK.
>
> On Thu, Aug 16, 2012 at 2:22 AM, Nicholas Ball
> <ni...@nodelay.com> wrote:
>>
>> I've been close to implementing a 2PC protocol before for something
else,
>> however for this it's not needed.
>> As the move operation will be done on a single node which has both the
>> cores, this could be done differently. Just not entirely sure how to do
>> it.
>>
>> When a commit is done at the moment, the core must get locked somehow,
it
>> is at this point where we should lock the other core too if a move
>> operation is being executed.
>>
>> Nick
>>
>> On Thu, 16 Aug 2012 10:32:10 +0800, Li Li <fa...@gmail.com> wrote:
>>>
>>
http://zookeeper.apache.org/doc/r3.3.6/recipes.html#sc_recipes_twoPhasedCommit
>>>
>>> On Thu, Aug 16, 2012 at 7:41 AM, Nicholas Ball
>>> <ni...@nodelay.com> wrote:
>>>>
>>>> Haven't managed to find a good way to do this yet. Does anyone have
any
>>>> ideas on how I could implement this feature?
>>>> Really need to move docs across from one core to another atomically.
>>>>
>>>> Many thanks,
>>>> Nicholas
>>>>
>>>> On Mon, 02 Jul 2012 04:37:12 -0600, Nicholas Ball
>>>> <ni...@nodelay.com> wrote:
>>>>> That could work, but then how do you ensure commit is called on the
>> two
>>>>> cores at the exact same time?
>>>>>
>>>>> Cheers,
>>>>> Nicholas
>>>>>
>>>>> On Sat, 30 Jun 2012 16:19:31 -0700, Lance Norskog
<go...@gmail.com>
>>>>> wrote:
>>>>>> Index all documents to both cores, but do not call commit until
both
>>>>>> report that indexing worked. If one of the cores throws an
exception,
>>>>>> call roll back on both cores.
>>>>>>
>>>>>> On Sat, Jun 30, 2012 at 6:50 AM, Nicholas Ball
>>>>>> <ni...@nodelay.com> wrote:
>>>>>>>
>>>>>>> Hey all,
>>>>>>>
>>>>>>> Trying to figure out the best way to perform atomic operation
across
>>>>>>> multiple cores on the same solr instance i.e. a multi-core
>>>> environment.
>>>>>>>
>>>>>>> An example would be to move a set of docs from one core onto
another
>>>>> core
>>>>>>> and ensure that a softcommit is done as the exact same time. If
one
>>>>> were
>>>>>>> to
>>>>>>> fail so would the other.
>>>>>>> Obviously this would probably require some customization but
wanted
>> to
>>>>>>> know what the best way to tackle this would be and where should I
be
>>>>>>> looking in the source.
>>>>>>>
>>>>>>> Many thanks for the help in advance,
>>>>>>> Nicholas a.k.a. incunix
Re: Atomic Multicore Operations - E.G. Move Docs
Posted by Lance Norskog <go...@gmail.com>.
I would use generation numbers on documents, and communicate a global
generation number in ZK.
On Thu, Aug 16, 2012 at 2:22 AM, Nicholas Ball
<ni...@nodelay.com> wrote:
>
> I've been close to implementing a 2PC protocol before for something else,
> however for this it's not needed.
> As the move operation will be done on a single node which has both the
> cores, this could be done differently. Just not entirely sure how to do it.
>
> When a commit is done at the moment, the core must get locked somehow, it
> is at this point where we should lock the other core too if a move
> operation is being executed.
>
> Nick
>
> On Thu, 16 Aug 2012 10:32:10 +0800, Li Li <fa...@gmail.com> wrote:
>>
> http://zookeeper.apache.org/doc/r3.3.6/recipes.html#sc_recipes_twoPhasedCommit
>>
>> On Thu, Aug 16, 2012 at 7:41 AM, Nicholas Ball
>> <ni...@nodelay.com> wrote:
>>>
>>> Haven't managed to find a good way to do this yet. Does anyone have any
>>> ideas on how I could implement this feature?
>>> Really need to move docs across from one core to another atomically.
>>>
>>> Many thanks,
>>> Nicholas
>>>
>>> On Mon, 02 Jul 2012 04:37:12 -0600, Nicholas Ball
>>> <ni...@nodelay.com> wrote:
>>>> That could work, but then how do you ensure commit is called on the
> two
>>>> cores at the exact same time?
>>>>
>>>> Cheers,
>>>> Nicholas
>>>>
>>>> On Sat, 30 Jun 2012 16:19:31 -0700, Lance Norskog <go...@gmail.com>
>>>> wrote:
>>>>> Index all documents to both cores, but do not call commit until both
>>>>> report that indexing worked. If one of the cores throws an exception,
>>>>> call roll back on both cores.
>>>>>
>>>>> On Sat, Jun 30, 2012 at 6:50 AM, Nicholas Ball
>>>>> <ni...@nodelay.com> wrote:
>>>>>>
>>>>>> Hey all,
>>>>>>
>>>>>> Trying to figure out the best way to perform atomic operation across
>>>>>> multiple cores on the same solr instance i.e. a multi-core
>>> environment.
>>>>>>
>>>>>> An example would be to move a set of docs from one core onto another
>>>> core
>>>>>> and ensure that a softcommit is done as the exact same time. If one
>>>> were
>>>>>> to
>>>>>> fail so would the other.
>>>>>> Obviously this would probably require some customization but wanted
> to
>>>>>> know what the best way to tackle this would be and where should I be
>>>>>> looking in the source.
>>>>>>
>>>>>> Many thanks for the help in advance,
>>>>>> Nicholas a.k.a. incunix
--
Lance Norskog
goksron@gmail.com
Re: Atomic Multicore Operations - E.G. Move Docs
Posted by Nicholas Ball <ni...@nodelay.com>.
I've been close to implementing a 2PC protocol before for something else,
however for this it's not needed.
As the move operation will be done on a single node which has both the
cores, this could be done differently. Just not entirely sure how to do it.
When a commit is done at the moment, the core must get locked somehow, it
is at this point where we should lock the other core too if a move
operation is being executed.
Nick
On Thu, 16 Aug 2012 10:32:10 +0800, Li Li <fa...@gmail.com> wrote:
>
http://zookeeper.apache.org/doc/r3.3.6/recipes.html#sc_recipes_twoPhasedCommit
>
> On Thu, Aug 16, 2012 at 7:41 AM, Nicholas Ball
> <ni...@nodelay.com> wrote:
>>
>> Haven't managed to find a good way to do this yet. Does anyone have any
>> ideas on how I could implement this feature?
>> Really need to move docs across from one core to another atomically.
>>
>> Many thanks,
>> Nicholas
>>
>> On Mon, 02 Jul 2012 04:37:12 -0600, Nicholas Ball
>> <ni...@nodelay.com> wrote:
>>> That could work, but then how do you ensure commit is called on the
two
>>> cores at the exact same time?
>>>
>>> Cheers,
>>> Nicholas
>>>
>>> On Sat, 30 Jun 2012 16:19:31 -0700, Lance Norskog <go...@gmail.com>
>>> wrote:
>>>> Index all documents to both cores, but do not call commit until both
>>>> report that indexing worked. If one of the cores throws an exception,
>>>> call roll back on both cores.
>>>>
>>>> On Sat, Jun 30, 2012 at 6:50 AM, Nicholas Ball
>>>> <ni...@nodelay.com> wrote:
>>>>>
>>>>> Hey all,
>>>>>
>>>>> Trying to figure out the best way to perform atomic operation across
>>>>> multiple cores on the same solr instance i.e. a multi-core
>> environment.
>>>>>
>>>>> An example would be to move a set of docs from one core onto another
>>> core
>>>>> and ensure that a softcommit is done as the exact same time. If one
>>> were
>>>>> to
>>>>> fail so would the other.
>>>>> Obviously this would probably require some customization but wanted
to
>>>>> know what the best way to tackle this would be and where should I be
>>>>> looking in the source.
>>>>>
>>>>> Many thanks for the help in advance,
>>>>> Nicholas a.k.a. incunix
Re: Atomic Multicore Operations - E.G. Move Docs
Posted by Li Li <fa...@gmail.com>.
http://zookeeper.apache.org/doc/r3.3.6/recipes.html#sc_recipes_twoPhasedCommit
On Thu, Aug 16, 2012 at 7:41 AM, Nicholas Ball
<ni...@nodelay.com> wrote:
>
> Haven't managed to find a good way to do this yet. Does anyone have any
> ideas on how I could implement this feature?
> Really need to move docs across from one core to another atomically.
>
> Many thanks,
> Nicholas
>
> On Mon, 02 Jul 2012 04:37:12 -0600, Nicholas Ball
> <ni...@nodelay.com> wrote:
>> That could work, but then how do you ensure commit is called on the two
>> cores at the exact same time?
>>
>> Cheers,
>> Nicholas
>>
>> On Sat, 30 Jun 2012 16:19:31 -0700, Lance Norskog <go...@gmail.com>
>> wrote:
>>> Index all documents to both cores, but do not call commit until both
>>> report that indexing worked. If one of the cores throws an exception,
>>> call roll back on both cores.
>>>
>>> On Sat, Jun 30, 2012 at 6:50 AM, Nicholas Ball
>>> <ni...@nodelay.com> wrote:
>>>>
>>>> Hey all,
>>>>
>>>> Trying to figure out the best way to perform atomic operation across
>>>> multiple cores on the same solr instance i.e. a multi-core
> environment.
>>>>
>>>> An example would be to move a set of docs from one core onto another
>> core
>>>> and ensure that a softcommit is done as the exact same time. If one
>> were
>>>> to
>>>> fail so would the other.
>>>> Obviously this would probably require some customization but wanted to
>>>> know what the best way to tackle this would be and where should I be
>>>> looking in the source.
>>>>
>>>> Many thanks for the help in advance,
>>>> Nicholas a.k.a. incunix
Re: Atomic Multicore Operations - E.G. Move Docs
Posted by Li Li <fa...@gmail.com>.
do you really need this？
distributed transaction is a difficult problem. in 2pc, every node could
fail, including coordinator. something like leader election needed to make
sure it works. you maybe try zookeeper.
but if the transaction is not very very important like transfer money in
bank, you can do like this.
coordinator:
在 2012-8-16 上午7:42，"Nicholas Ball" <ni...@nodelay.com>写道：
>
> Haven't managed to find a good way to do this yet. Does anyone have any
> ideas on how I could implement this feature?
> Really need to move docs across from one core to another atomically.
>
> Many thanks,
> Nicholas
>
> On Mon, 02 Jul 2012 04:37:12 -0600, Nicholas Ball
> <ni...@nodelay.com> wrote:
> > That could work, but then how do you ensure commit is called on the two
> > cores at the exact same time?
> >
> > Cheers,
> > Nicholas
> >
> > On Sat, 30 Jun 2012 16:19:31 -0700, Lance Norskog <go...@gmail.com>
> > wrote:
> >> Index all documents to both cores, but do not call commit until both
> >> report that indexing worked. If one of the cores throws an exception,
> >> call roll back on both cores.
> >>
> >> On Sat, Jun 30, 2012 at 6:50 AM, Nicholas Ball
> >> <ni...@nodelay.com> wrote:
> >>>
> >>> Hey all,
> >>>
> >>> Trying to figure out the best way to perform atomic operation across
> >>> multiple cores on the same solr instance i.e. a multi-core
> environment.
> >>>
> >>> An example would be to move a set of docs from one core onto another
> > core
> >>> and ensure that a softcommit is done as the exact same time. If one
> > were
> >>> to
> >>> fail so would the other.
> >>> Obviously this would probably require some customization but wanted to
> >>> know what the best way to tackle this would be and where should I be
> >>> looking in the source.
> >>>
> >>> Many thanks for the help in advance,
> >>> Nicholas a.k.a. incunix
>
Re: Atomic Multicore Operations - E.G. Move Docs
Posted by Nicholas Ball <ni...@nodelay.com>.
Haven't managed to find a good way to do this yet. Does anyone have any
ideas on how I could implement this feature?
Really need to move docs across from one core to another atomically.
Many thanks,
Nicholas
On Mon, 02 Jul 2012 04:37:12 -0600, Nicholas Ball
<ni...@nodelay.com> wrote:
> That could work, but then how do you ensure commit is called on the two
> cores at the exact same time?
>
> Cheers,
> Nicholas
>
> On Sat, 30 Jun 2012 16:19:31 -0700, Lance Norskog <go...@gmail.com>
> wrote:
>> Index all documents to both cores, but do not call commit until both
>> report that indexing worked. If one of the cores throws an exception,
>> call roll back on both cores.
>>
>> On Sat, Jun 30, 2012 at 6:50 AM, Nicholas Ball
>> <ni...@nodelay.com> wrote:
>>>
>>> Hey all,
>>>
>>> Trying to figure out the best way to perform atomic operation across
>>> multiple cores on the same solr instance i.e. a multi-core
environment.
>>>
>>> An example would be to move a set of docs from one core onto another
> core
>>> and ensure that a softcommit is done as the exact same time. If one
> were
>>> to
>>> fail so would the other.
>>> Obviously this would probably require some customization but wanted to
>>> know what the best way to tackle this would be and where should I be
>>> looking in the source.
>>>
>>> Many thanks for the help in advance,
>>> Nicholas a.k.a. incunix
Re: Atomic Multicore Operations - E.G. Move Docs
Posted by Nicholas Ball <ni...@nodelay.com>.
That could work, but then how do you ensure commit is called on the two
cores at the exact same time?
Also, any way to commit a specific update rather then all the back-logged
ones?
Cheers,
Nicholas
On Sat, 30 Jun 2012 16:19:31 -0700, Lance Norskog <go...@gmail.com>
wrote:
> Index all documents to both cores, but do not call commit until both
> report that indexing worked. If one of the cores throws an exception,
> call roll back on both cores.
>
> On Sat, Jun 30, 2012 at 6:50 AM, Nicholas Ball
> <ni...@nodelay.com> wrote:
>>
>> Hey all,
>>
>> Trying to figure out the best way to perform atomic operation across
>> multiple cores on the same solr instance i.e. a multi-core environment.
>>
>> An example would be to move a set of docs from one core onto another
core
>> and ensure that a softcommit is done as the exact same time. If one
were
>> to
>> fail so would the other.
>> Obviously this would probably require some customization but wanted to
>> know what the best way to tackle this would be and where should I be
>> looking in the source.
>>
>> Many thanks for the help in advance,
>> Nicholas a.k.a. incunix
Re: Atomic Multicore Operations - E.G. Move Docs
Posted by Lance Norskog <go...@gmail.com>.
Index all documents to both cores, but do not call commit until both
report that indexing worked. If one of the cores throws an exception,
call roll back on both cores.
On Sat, Jun 30, 2012 at 6:50 AM, Nicholas Ball
<ni...@nodelay.com> wrote:
>
> Hey all,
>
> Trying to figure out the best way to perform atomic operation across
> multiple cores on the same solr instance i.e. a multi-core environment.
>
> An example would be to move a set of docs from one core onto another core
> and ensure that a softcommit is done as the exact same time. If one were to
> fail so would the other.
> Obviously this would probably require some customization but wanted to
> know what the best way to tackle this would be and where should I be
> looking in the source.
>
> Many thanks for the help in advance,
> Nicholas a.k.a. incunix
--
Lance Norskog
goksron@gmail.com