You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by Abdel Hakim Deneche <ad...@maprtech.com> on 2015/05/04 18:47:53 UTC
what is the correct behavior when an upstream operator returns STOP
What is the correct behavior for an operator when it receives an
IterOutcome.STOP from upstream:
1. should it just stop right away and return STOP downstream ?, or
2. should it cleanup it's resources before returning STOP downstream ?
Some operators follow 1 and others (like OrderedPartitionRecordBatch)
follow 2.
--
Abdelhakim Deneche
Software Engineer
<http://www.mapr.com/>
Now Available - Free Hadoop On-Demand Training
<http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available>
Re: what is the correct behavior when an upstream operator returns STOP
Posted by Steven Phillips <sp...@maprtech.com>.
The *current* correct behavior is to return STOP downstream. That is
because the ownership of batches created by that operator's allocator may
have been transferred downstream, and if we try to cleanup right away, it
will result in apparent memory leaks.
Like Jacques mentioned, I am working on a patch that will change this
behavior, and the reason we will be able to clean up right away is because
there is work in progress to move to a model where ownership of buffers is
transferred as well.
On Mon, May 4, 2015 at 10:11 AM, Jacques Nadeau <ja...@apache.org> wrote:
> In general, an operator should release its resources as soon as it can. I
> believe Steven is working on a patch to make all resource intensive
> operators do just this.
>
> On Mon, May 4, 2015 at 9:47 AM, Abdel Hakim Deneche <adeneche@maprtech.com
> >
> wrote:
>
> > What is the correct behavior for an operator when it receives an
> > IterOutcome.STOP from upstream:
> > 1. should it just stop right away and return STOP downstream ?, or
> > 2. should it cleanup it's resources before returning STOP downstream ?
> >
> > Some operators follow 1 and others (like OrderedPartitionRecordBatch)
> > follow 2.
> >
> > --
> >
> > Abdelhakim Deneche
> >
> > Software Engineer
> >
> > <http://www.mapr.com/>
> >
> >
> > Now Available - Free Hadoop On-Demand Training
> > <
> >
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> > >
> >
>
--
Steven Phillips
Software Engineer
mapr.com
Re: what is the correct behavior when an upstream operator returns STOP
Posted by Jacques Nadeau <ja...@apache.org>.
In general, an operator should release its resources as soon as it can. I
believe Steven is working on a patch to make all resource intensive
operators do just this.
On Mon, May 4, 2015 at 9:47 AM, Abdel Hakim Deneche <ad...@maprtech.com>
wrote:
> What is the correct behavior for an operator when it receives an
> IterOutcome.STOP from upstream:
> 1. should it just stop right away and return STOP downstream ?, or
> 2. should it cleanup it's resources before returning STOP downstream ?
>
> Some operators follow 1 and others (like OrderedPartitionRecordBatch)
> follow 2.
>
> --
>
> Abdelhakim Deneche
>
> Software Engineer
>
> <http://www.mapr.com/>
>
>
> Now Available - Free Hadoop On-Demand Training
> <
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >
>