You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@arrow.apache.org by Jeremy Dyer <jd...@gmail.com> on 2023/03/28 16:02:39 UTC

Optimizer - LogicalPlanSignature hasher deadlock

Hello,

After updating to Datafusion 21.0.0 I have noticed that codebase has
deadlocks anytime that the `optimize` method is called. I have tracked down
the location where the deadlock is happening to [1]. The call to `plan.hash(
&mut hasher);` never returns. Does anyone have any ideas on what might be
going on here?

Thanks,
Jeremy Dyer

[1] -
https://github.com/apache/arrow-datafusion/blob/main/datafusion/optimizer/src/plan_signature.rs#L64

Re: Optimizer - LogicalPlanSignature hasher deadlock

Posted by Daniël Heres <da...@gmail.com>.

Thanks Jeremy for reporting back, glad you solved the issue.

Best regards,

Daniël Heres

On Wed, Mar 29, 2023, 20:02 Jeremy Dyer <jd...@gmail.com> wrote:

> Wanted to follow up on this and close the discussion. We have several
> custom optimizers in Dask-SQL and what was causing this issue was a silly
> mistake I had made in which dyn_hash and hash were calling into each other
> in an infinite loop. Sorry for the confusion.
>
> Thanks,
> Jeremy Dyer
>
> On Tue, Mar 28, 2023 at 12:02 PM Jeremy Dyer <jd...@gmail.com> wrote:
>
> > Hello,
> >
> > After updating to Datafusion 21.0.0 I have noticed that codebase has
> > deadlocks anytime that the `optimize` method is called. I have tracked
> down
> > the location where the deadlock is happening to [1]. The call to `plan.
> > hash(&mut hasher);` never returns. Does anyone have any ideas on what
> > might be going on here?
> >
> > Thanks,
> > Jeremy Dyer
> >
> > [1] -
> >
> https://github.com/apache/arrow-datafusion/blob/main/datafusion/optimizer/src/plan_signature.rs#L64
> >
>

Re: Optimizer - LogicalPlanSignature hasher deadlock

Posted by Jeremy Dyer <jd...@gmail.com>.

Wanted to follow up on this and close the discussion. We have several
custom optimizers in Dask-SQL and what was causing this issue was a silly
mistake I had made in which dyn_hash and hash were calling into each other
in an infinite loop. Sorry for the confusion.

Thanks,
Jeremy Dyer

On Tue, Mar 28, 2023 at 12:02 PM Jeremy Dyer <jd...@gmail.com> wrote:

> Hello,
>
> After updating to Datafusion 21.0.0 I have noticed that codebase has
> deadlocks anytime that the `optimize` method is called. I have tracked down
> the location where the deadlock is happening to [1]. The call to `plan.
> hash(&mut hasher);` never returns. Does anyone have any ideas on what
> might be going on here?
>
> Thanks,
> Jeremy Dyer
>
> [1] -
> https://github.com/apache/arrow-datafusion/blob/main/datafusion/optimizer/src/plan_signature.rs#L64
>