You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@asterixdb.apache.org by abdullah alamoudi <ba...@gmail.com> on 2016/01/07 14:48:08 UTC
Deadlock issue
Today, as I was working on fixing handling of duplicate keys with feeds,
everything seemed to work fine. here is what we do when we encounter a
duplicate key exception.
1. we remove the tuple causing the exception.
2. we continue from where we stopped.
The problem is that when I try to query the dataset after that to check and
see which records made it into the dataset, I get a deadlock.
I have looked at the stack trace (attached) and I think the threads in the
file are the relevant ones. Please, have a look and let me know if you have
a possible cause in mind.
The threads are related to:
1. BufferCache.
2. Logging.
3. Locking.
Let me know what you think. I can reproduce this bug. it happened on 100%
of my test runs.
I will let you know when I solve it but it is taking longer than I thought.
Amoudi, Abdullah.
Re: Deadlock issue
Posted by abdullah alamoudi <ba...@gmail.com>.
Hi Ildar,
No, my proposed change has nothing to do with this issue. In fact, it is
not clear from the issue description what the expected behavior is.
However, it is clear that fixing the issue should be very easy (once we
know what should be done.)
Cheers,
Abdullah.
On Fri, Feb 19, 2016 at 10:55 AM, Ildar Absalyamov <
ildar.absalyamov@gmail.com> wrote:
> Abdullah,
>
> Do you think that issue is somehow related to the patch that you proposed?
> https://issues.apache.org/jira/browse/ASTERIXDB-1302 <
> https://issues.apache.org/jira/browse/ASTERIXDB-1302>
> It does not involve duplicates per se, but has the same issue with feed
> lock management.
>
> > On Jan 7, 2016, at 09:12, abdullah alamoudi <ba...@gmail.com> wrote:
> >
> > So I have pushed the new change (Still under review).
> >
> > @Vignesh,
> > This change has the fix for the twitter adapter.
> >
> > @Jianfeng,
> > This change has the fix for the duplicate key issue.
> >
> > It can be checked out using
> > git fetch https://asterix-gerrit.ics.uci.edu/asterixdb
> > refs/changes/74/574/2 && git checkout FETCH_HEAD
> >
> > Amoudi, Abdullah.
> >
> > On Thu, Jan 7, 2016 at 7:42 PM, abdullah alamoudi <ba...@gmail.com>
> > wrote:
> >
> >> Found the issue. It had nothing to do with any of the three components.
> >>
> >> The problem was that:
> >>> 1. we remove the tuple causing the exception.
> >> didn't work as I expected and instead it removed all the tuples upto the
> >> offending record.
> >> I have fixed it and now it works like a charm.
> >>
> >> Pushing the fix in a few minutes.
> >> Abdullah.
> >>
> >> Amoudi, Abdullah.
> >>
> >> On Thu, Jan 7, 2016 at 6:16 PM, Mike Carey <dt...@gmail.com> wrote:
> >>
> >>> The general transaction handling for such an exception wrt locking and
> >>> aborts probably assumes that total bailouts are the answer. Thus, it
> may
> >>> leave messes that rollbacks are otherwise the answer to. Feeds and
> >>> transactions don't mix super well, it seems.... Watching how duplicate
> >>> keys work for insert from query statements may help you debug. If we
> >>> change
> >>> things to allow those to succeed for all non duplicate keys - which
> might
> >>> make more sense for that anyway.
> >>> On Jan 7, 2016 5:48 AM, "abdullah alamoudi" <ba...@gmail.com>
> wrote:
> >>>
> >>>> Today, as I was working on fixing handling of duplicate keys with
> feeds,
> >>>> everything seemed to work fine. here is what we do when we encounter a
> >>>> duplicate key exception.
> >>>>
> >>>> 1. we remove the tuple causing the exception.
> >>>> 2. we continue from where we stopped.
> >>>>
> >>>> The problem is that when I try to query the dataset after that to
> check
> >>>> and see which records made it into the dataset, I get a deadlock.
> >>>>
> >>>> I have looked at the stack trace (attached) and I think the threads in
> >>> the
> >>>> file are the relevant ones. Please, have a look and let me know if you
> >>> have
> >>>> a possible cause in mind.
> >>>>
> >>>> The threads are related to:
> >>>> 1. BufferCache.
> >>>> 2. Logging.
> >>>> 3. Locking.
> >>>>
> >>>> Let me know what you think. I can reproduce this bug. it happened on
> >>> 100%
> >>>> of my test runs.
> >>>>
> >>>> I will let you know when I solve it but it is taking longer than I
> >>> thought.
> >>>>
> >>>> Amoudi, Abdullah.
> >>>>
> >>>
> >>
> >>
>
> Best regards,
> Ildar
>
>
Re: Deadlock issue
Posted by Ildar Absalyamov <il...@gmail.com>.
Abdullah,
Do you think that issue is somehow related to the patch that you proposed? https://issues.apache.org/jira/browse/ASTERIXDB-1302 <https://issues.apache.org/jira/browse/ASTERIXDB-1302>
It does not involve duplicates per se, but has the same issue with feed lock management.
> On Jan 7, 2016, at 09:12, abdullah alamoudi <ba...@gmail.com> wrote:
>
> So I have pushed the new change (Still under review).
>
> @Vignesh,
> This change has the fix for the twitter adapter.
>
> @Jianfeng,
> This change has the fix for the duplicate key issue.
>
> It can be checked out using
> git fetch https://asterix-gerrit.ics.uci.edu/asterixdb
> refs/changes/74/574/2 && git checkout FETCH_HEAD
>
> Amoudi, Abdullah.
>
> On Thu, Jan 7, 2016 at 7:42 PM, abdullah alamoudi <ba...@gmail.com>
> wrote:
>
>> Found the issue. It had nothing to do with any of the three components.
>>
>> The problem was that:
>>> 1. we remove the tuple causing the exception.
>> didn't work as I expected and instead it removed all the tuples upto the
>> offending record.
>> I have fixed it and now it works like a charm.
>>
>> Pushing the fix in a few minutes.
>> Abdullah.
>>
>> Amoudi, Abdullah.
>>
>> On Thu, Jan 7, 2016 at 6:16 PM, Mike Carey <dt...@gmail.com> wrote:
>>
>>> The general transaction handling for such an exception wrt locking and
>>> aborts probably assumes that total bailouts are the answer. Thus, it may
>>> leave messes that rollbacks are otherwise the answer to. Feeds and
>>> transactions don't mix super well, it seems.... Watching how duplicate
>>> keys work for insert from query statements may help you debug. If we
>>> change
>>> things to allow those to succeed for all non duplicate keys - which might
>>> make more sense for that anyway.
>>> On Jan 7, 2016 5:48 AM, "abdullah alamoudi" <ba...@gmail.com> wrote:
>>>
>>>> Today, as I was working on fixing handling of duplicate keys with feeds,
>>>> everything seemed to work fine. here is what we do when we encounter a
>>>> duplicate key exception.
>>>>
>>>> 1. we remove the tuple causing the exception.
>>>> 2. we continue from where we stopped.
>>>>
>>>> The problem is that when I try to query the dataset after that to check
>>>> and see which records made it into the dataset, I get a deadlock.
>>>>
>>>> I have looked at the stack trace (attached) and I think the threads in
>>> the
>>>> file are the relevant ones. Please, have a look and let me know if you
>>> have
>>>> a possible cause in mind.
>>>>
>>>> The threads are related to:
>>>> 1. BufferCache.
>>>> 2. Logging.
>>>> 3. Locking.
>>>>
>>>> Let me know what you think. I can reproduce this bug. it happened on
>>> 100%
>>>> of my test runs.
>>>>
>>>> I will let you know when I solve it but it is taking longer than I
>>> thought.
>>>>
>>>> Amoudi, Abdullah.
>>>>
>>>
>>
>>
Best regards,
Ildar
Re: Deadlock issue
Posted by abdullah alamoudi <ba...@gmail.com>.
So I have pushed the new change (Still under review).
@Vignesh,
This change has the fix for the twitter adapter.
@Jianfeng,
This change has the fix for the duplicate key issue.
It can be checked out using
git fetch https://asterix-gerrit.ics.uci.edu/asterixdb
refs/changes/74/574/2 && git checkout FETCH_HEAD
Amoudi, Abdullah.
On Thu, Jan 7, 2016 at 7:42 PM, abdullah alamoudi <ba...@gmail.com>
wrote:
> Found the issue. It had nothing to do with any of the three components.
>
> The problem was that:
> > 1. we remove the tuple causing the exception.
> didn't work as I expected and instead it removed all the tuples upto the
> offending record.
> I have fixed it and now it works like a charm.
>
> Pushing the fix in a few minutes.
> Abdullah.
>
> Amoudi, Abdullah.
>
> On Thu, Jan 7, 2016 at 6:16 PM, Mike Carey <dt...@gmail.com> wrote:
>
>> The general transaction handling for such an exception wrt locking and
>> aborts probably assumes that total bailouts are the answer. Thus, it may
>> leave messes that rollbacks are otherwise the answer to. Feeds and
>> transactions don't mix super well, it seems.... Watching how duplicate
>> keys work for insert from query statements may help you debug. If we
>> change
>> things to allow those to succeed for all non duplicate keys - which might
>> make more sense for that anyway.
>> On Jan 7, 2016 5:48 AM, "abdullah alamoudi" <ba...@gmail.com> wrote:
>>
>> > Today, as I was working on fixing handling of duplicate keys with feeds,
>> > everything seemed to work fine. here is what we do when we encounter a
>> > duplicate key exception.
>> >
>> > 1. we remove the tuple causing the exception.
>> > 2. we continue from where we stopped.
>> >
>> > The problem is that when I try to query the dataset after that to check
>> > and see which records made it into the dataset, I get a deadlock.
>> >
>> > I have looked at the stack trace (attached) and I think the threads in
>> the
>> > file are the relevant ones. Please, have a look and let me know if you
>> have
>> > a possible cause in mind.
>> >
>> > The threads are related to:
>> > 1. BufferCache.
>> > 2. Logging.
>> > 3. Locking.
>> >
>> > Let me know what you think. I can reproduce this bug. it happened on
>> 100%
>> > of my test runs.
>> >
>> > I will let you know when I solve it but it is taking longer than I
>> thought.
>> >
>> > Amoudi, Abdullah.
>> >
>>
>
>
Re: Deadlock issue
Posted by abdullah alamoudi <ba...@gmail.com>.
Found the issue. It had nothing to do with any of the three components.
The problem was that:
> 1. we remove the tuple causing the exception.
didn't work as I expected and instead it removed all the tuples upto the
offending record.
I have fixed it and now it works like a charm.
Pushing the fix in a few minutes.
Abdullah.
Amoudi, Abdullah.
On Thu, Jan 7, 2016 at 6:16 PM, Mike Carey <dt...@gmail.com> wrote:
> The general transaction handling for such an exception wrt locking and
> aborts probably assumes that total bailouts are the answer. Thus, it may
> leave messes that rollbacks are otherwise the answer to. Feeds and
> transactions don't mix super well, it seems.... Watching how duplicate
> keys work for insert from query statements may help you debug. If we change
> things to allow those to succeed for all non duplicate keys - which might
> make more sense for that anyway.
> On Jan 7, 2016 5:48 AM, "abdullah alamoudi" <ba...@gmail.com> wrote:
>
> > Today, as I was working on fixing handling of duplicate keys with feeds,
> > everything seemed to work fine. here is what we do when we encounter a
> > duplicate key exception.
> >
> > 1. we remove the tuple causing the exception.
> > 2. we continue from where we stopped.
> >
> > The problem is that when I try to query the dataset after that to check
> > and see which records made it into the dataset, I get a deadlock.
> >
> > I have looked at the stack trace (attached) and I think the threads in
> the
> > file are the relevant ones. Please, have a look and let me know if you
> have
> > a possible cause in mind.
> >
> > The threads are related to:
> > 1. BufferCache.
> > 2. Logging.
> > 3. Locking.
> >
> > Let me know what you think. I can reproduce this bug. it happened on 100%
> > of my test runs.
> >
> > I will let you know when I solve it but it is taking longer than I
> thought.
> >
> > Amoudi, Abdullah.
> >
>
Re: Deadlock issue
Posted by Mike Carey <dt...@gmail.com>.
The general transaction handling for such an exception wrt locking and
aborts probably assumes that total bailouts are the answer. Thus, it may
leave messes that rollbacks are otherwise the answer to. Feeds and
transactions don't mix super well, it seems.... Watching how duplicate
keys work for insert from query statements may help you debug. If we change
things to allow those to succeed for all non duplicate keys - which might
make more sense for that anyway.
On Jan 7, 2016 5:48 AM, "abdullah alamoudi" <ba...@gmail.com> wrote:
> Today, as I was working on fixing handling of duplicate keys with feeds,
> everything seemed to work fine. here is what we do when we encounter a
> duplicate key exception.
>
> 1. we remove the tuple causing the exception.
> 2. we continue from where we stopped.
>
> The problem is that when I try to query the dataset after that to check
> and see which records made it into the dataset, I get a deadlock.
>
> I have looked at the stack trace (attached) and I think the threads in the
> file are the relevant ones. Please, have a look and let me know if you have
> a possible cause in mind.
>
> The threads are related to:
> 1. BufferCache.
> 2. Logging.
> 3. Locking.
>
> Let me know what you think. I can reproduce this bug. it happened on 100%
> of my test runs.
>
> I will let you know when I solve it but it is taking longer than I thought.
>
> Amoudi, Abdullah.
>