You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@lucy.apache.org by Serkan Mulayim <se...@gmail.com> on 2017/01/06 00:31:15 UTC
[lucy-user] C library - DECREFing PolySearcher
Hi guys,
I have a question regarding the searchers (PolySearcher and IndexSearcher).
I wrote an application to do searching and indexing for 2 cases: 1- unified
index(searched via IndexSearcher) 2- splitted index (there are around 12
indexes to be searched via PolySearcher). Both indexes have the same number
of documents in total.
In order to make the application simpler, for every search query I create
the related searcher, do the search, and then close it (I do closing by
DECREF). So basically a searcher is created and removed.
For the queries that do not have any results, for the IndexSearcher, I do
not see any issues. For the same query with Poly search, I receive the
following LLDB output. If I wait in the LLDB with a breakpoint before
DECREFing the PolySearcher this issue does not happen.
Process 44678 stopped
* thread #1: tid = 0x325b47, 0x0000000100004c0a
test3searchs`cfish_dec_refcount(vself=0x0000000103abe060) + 10 at
clownfish.c:100, queue = 'com.apple.main-thread', stop reason =
EXC_BAD_ACCESS (code=1, address=0x28)
frame #0: 0x0000000100004c0a
test3searchs`cfish_dec_refcount(vself=0x0000000103abe060) + 10 at
clownfish.c:100
97 cfish_dec_refcount(void *vself) {
98 cfish_Obj *self = (Obj*)vself;
99 cfish_Class *klass = self->klass;
-> 100 if (klass->flags & CFISH_fREFCOUNTSPECIAL) {
101 if (SI_immortal(klass)) {
102 return (uint32_t)self->refcount;
103 }
I believe it is not directly related to the number of results returned. But
something happens when we try to DECREF very quickly. I see that we need
pthread. This makes me think that search and DECREF does not happen in the
same thread, but I do not want to be confusing so no more comments
regarding the threads :).
Has anyone seen this issue? Do you have any suggestions? Is there a concern
in creating a new searcher and destroying it by DECREF every time (other
than a possible slowness)? Is there a better way to destroy a searcher?
Thanks,
Serkan
Re: [lucy-user] C library - DECREFing PolySearcher
Posted by Serkan Mulayim <se...@gmail.com>.
Thank you Nick, I think there might be a problem in my code which is not
related to this. I am working on verifying it. I am sorry if I caused if
any troubles. Thanks again...
On Fri, Jan 6, 2017 at 10:08 AM, Nick Wellnhofer <we...@aevum.de>
wrote:
> On 06/01/2017 18:55, Serkan Mulayim wrote:
>
>> Then can there be a bug s.t. you try to decref the searcher after getting
>> 0
>> results?
>>
>
> That's unlikely but I can't tell without a test case.
>
> Nick
>
>
Re: [lucy-user] C library - DECREFing PolySearcher
Posted by Nick Wellnhofer <we...@aevum.de>.
On 06/01/2017 18:55, Serkan Mulayim wrote:
> Then can there be a bug s.t. you try to decref the searcher after getting 0
> results?
That's unlikely but I can't tell without a test case.
Nick
Re: [lucy-user] C library - DECREFing PolySearcher
Posted by Serkan Mulayim <se...@gmail.com>.
Thanks Nick,
Then can there be a bug s.t. you try to decref the searcher after getting 0
results?
Thanks again,
Serkan
On Fri, Jan 6, 2017 at 4:20 AM, Nick Wellnhofer <we...@aevum.de> wrote:
> On 06/01/2017 01:31, Serkan Mulayim wrote:
>
>> I believe it is not directly related to the number of results returned.
>> But
>> something happens when we try to DECREF very quickly. I see that we need
>> pthread. This makes me think that search and DECREF does not happen in the
>> same thread, but I do not want to be confusing so no more comments
>> regarding the threads :).
>>
>
> Lucy doesn't use threads internally.
>
> Has anyone seen this issue? Do you have any suggestions? Is there a concern
>> in creating a new searcher and destroying it by DECREF every time (other
>> than a possible slowness)? Is there a better way to destroy a searcher?
>>
>
> Recreating searchers is perfectly fine and DECREF is the only way to
> destroy a searcher. The only suggestion I have is to test with Valgrind or
> ASan.
>
> Nick
>
>
Re: [lucy-user] C library - DECREFing PolySearcher
Posted by Nick Wellnhofer <we...@aevum.de>.
On 06/01/2017 01:31, Serkan Mulayim wrote:
> I believe it is not directly related to the number of results returned. But
> something happens when we try to DECREF very quickly. I see that we need
> pthread. This makes me think that search and DECREF does not happen in the
> same thread, but I do not want to be confusing so no more comments
> regarding the threads :).
Lucy doesn't use threads internally.
> Has anyone seen this issue? Do you have any suggestions? Is there a concern
> in creating a new searcher and destroying it by DECREF every time (other
> than a possible slowness)? Is there a better way to destroy a searcher?
Recreating searchers is perfectly fine and DECREF is the only way to destroy a
searcher. The only suggestion I have is to test with Valgrind or ASan.
Nick
[lucy-user] Re: C library - DECREFing PolySearcher
Posted by Serkan Mulayim <se...@gmail.com>.
Sorry guys, 1 more update: when I build the app with -O2 and without -g,
the issue is gone. This makes me think that there is something wrong with
the timing, and related to threads. Just speculating :)
On Thu, Jan 5, 2017 at 5:08 PM, Serkan Mulayim <se...@gmail.com>
wrote:
> Update: I can reproduce it with IndexSearcher too. Previously I had said
> that it was only happening in PolySearcher...
>
> On Thu, Jan 5, 2017 at 4:31 PM, Serkan Mulayim <se...@gmail.com>
> wrote:
>
>> Hi guys,
>>
>> I have a question regarding the searchers (PolySearcher and
>> IndexSearcher).
>>
>> I wrote an application to do searching and indexing for 2 cases: 1-
>> unified index(searched via IndexSearcher) 2- splitted index (there are
>> around 12 indexes to be searched via PolySearcher). Both indexes have the
>> same number of documents in total.
>>
>> In order to make the application simpler, for every search query I create
>> the related searcher, do the search, and then close it (I do closing by
>> DECREF). So basically a searcher is created and removed.
>>
>> For the queries that do not have any results, for the IndexSearcher, I do
>> not see any issues. For the same query with Poly search, I receive the
>> following LLDB output. If I wait in the LLDB with a breakpoint before
>> DECREFing the PolySearcher this issue does not happen.
>>
>> Process 44678 stopped
>> * thread #1: tid = 0x325b47, 0x0000000100004c0a
>> test3searchs`cfish_dec_refcount(vself=0x0000000103abe060) + 10 at
>> clownfish.c:100, queue = 'com.apple.main-thread', stop reason =
>> EXC_BAD_ACCESS (code=1, address=0x28)
>> frame #0: 0x0000000100004c0a test3searchs`cfish_dec_refcount(vself=0x0000000103abe060)
>> + 10 at clownfish.c:100
>> 97 cfish_dec_refcount(void *vself) {
>> 98 cfish_Obj *self = (Obj*)vself;
>> 99 cfish_Class *klass = self->klass;
>> -> 100 if (klass->flags & CFISH_fREFCOUNTSPECIAL) {
>> 101 if (SI_immortal(klass)) {
>> 102 return (uint32_t)self->refcount;
>> 103 }
>>
>>
>> I believe it is not directly related to the number of results returned.
>> But something happens when we try to DECREF very quickly. I see that we
>> need pthread. This makes me think that search and DECREF does not happen in
>> the same thread, but I do not want to be confusing so no more comments
>> regarding the threads :).
>>
>> Has anyone seen this issue? Do you have any suggestions? Is there a
>> concern in creating a new searcher and destroying it by DECREF every time
>> (other than a possible slowness)? Is there a better way to destroy a
>> searcher?
>>
>> Thanks,
>> Serkan
>>
>>
>
[lucy-user] Re: C library - DECREFing PolySearcher
Posted by Serkan Mulayim <se...@gmail.com>.
Update: I can reproduce it with IndexSearcher too. Previously I had said
that it was only happening in PolySearcher...
On Thu, Jan 5, 2017 at 4:31 PM, Serkan Mulayim <se...@gmail.com>
wrote:
> Hi guys,
>
> I have a question regarding the searchers (PolySearcher and IndexSearcher).
>
> I wrote an application to do searching and indexing for 2 cases: 1-
> unified index(searched via IndexSearcher) 2- splitted index (there are
> around 12 indexes to be searched via PolySearcher). Both indexes have the
> same number of documents in total.
>
> In order to make the application simpler, for every search query I create
> the related searcher, do the search, and then close it (I do closing by
> DECREF). So basically a searcher is created and removed.
>
> For the queries that do not have any results, for the IndexSearcher, I do
> not see any issues. For the same query with Poly search, I receive the
> following LLDB output. If I wait in the LLDB with a breakpoint before
> DECREFing the PolySearcher this issue does not happen.
>
> Process 44678 stopped
> * thread #1: tid = 0x325b47, 0x0000000100004c0a test3searchs`cfish_dec_
> refcount(vself=0x0000000103abe060) + 10 at clownfish.c:100, queue =
> 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x28)
> frame #0: 0x0000000100004c0a test3searchs`cfish_dec_refcount(vself=0x0000000103abe060)
> + 10 at clownfish.c:100
> 97 cfish_dec_refcount(void *vself) {
> 98 cfish_Obj *self = (Obj*)vself;
> 99 cfish_Class *klass = self->klass;
> -> 100 if (klass->flags & CFISH_fREFCOUNTSPECIAL) {
> 101 if (SI_immortal(klass)) {
> 102 return (uint32_t)self->refcount;
> 103 }
>
>
> I believe it is not directly related to the number of results returned.
> But something happens when we try to DECREF very quickly. I see that we
> need pthread. This makes me think that search and DECREF does not happen in
> the same thread, but I do not want to be confusing so no more comments
> regarding the threads :).
>
> Has anyone seen this issue? Do you have any suggestions? Is there a
> concern in creating a new searcher and destroying it by DECREF every time
> (other than a possible slowness)? Is there a better way to destroy a
> searcher?
>
> Thanks,
> Serkan
>
>