You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apex.apache.org by Bryan Bende <bb...@gmail.com> on 2016/01/25 23:13:09 UTC

Input Operator Question

Hello,

Quick beginner question regarding the life-cycle of operators...

Can emitTuples() get called concurrently by multiple threads? or is it
always called sequentially for a given window until endWindow() is called?

Thanks,

Bryan

Re: Input Operator Question

Posted by Munagala Ramanath <ra...@datatorrent.com>.
An example of an operator starting a separate thread is the use of
WindowBoundedService in:
https://github.com/apache/incubator-apex-malhar/blob/master/library/src/main/java/com/datatorrent/lib/io/PubSubWebSocketAppDataQuery.java

Ram

On Mon, Jan 25, 2016 at 3:03 PM, Amol Kekre <am...@datatorrent.com> wrote:

> Smith,
> The only way to get into a multi-thread situation is for the user to fork
> off threads from the operator code. The platform guarantees that the
> operator api calls will happen in a single thread.
>
> Thks
> Amol
>
>
> On Mon, Jan 25, 2016 at 2:58 PM, Ashwin Chandra Putta <
> ashwinchandrap@gmail.com> wrote:
>
> > Smith,
> >
> > Within a yarn container, we can start multiple operators. Each such
> > operator runs in one thread. All the calls on the operator happen in that
> > thread. So the operator is not shared.
> >
> > Regards,
> > Ashwin.
> >
> > On Mon, Jan 25, 2016 at 2:33 PM, Smith, Kent <Kent.Smith2@capitalone.com
> >
> > wrote:
> >
> > > Wait, additional newbee question.  So an Operator runs in own thread.
> Do
> > > you mean a given Operator is shared or could multiple threads, external
> > to
> > > the Operator thread call the same operator and each get an operator
> > thread
> > > of their own (within their context) or would each thread call within
> the
> > > Operator thread one at a time so each context shares the Operator
> thread?
> > >
> > > Thanks.
> > >
> > > -----Original Message-----
> > > From: Bryan Bende [mailto:bbende@gmail.com]
> > > Sent: Monday, January 25, 2016 5:27 PM
> > > To: dev@apex.incubator.apache.org
> > > Subject: Re: Input Operator Question
> > >
> > > Thanks! Appreciate the quick response.
> > >
> > > On Mon, Jan 25, 2016 at 5:22 PM, Ashwin Chandra Putta <
> > > ashwinchandrap@gmail.com> wrote:
> > >
> > > > It always called sequentially in a single thread. A given operator
> > > > runs in its main thread and all the calls on the operator are called
> > > > with in the same thread.
> > > >
> > > > Regards,
> > > > Ashwin.
> > > >
> > > > On Mon, Jan 25, 2016 at 2:13 PM, Bryan Bende <bb...@gmail.com>
> wrote:
> > > >
> > > > > Hello,
> > > > >
> > > > > Quick beginner question regarding the life-cycle of operators...
> > > > >
> > > > > Can emitTuples() get called concurrently by multiple threads? or is
> > > > > it always called sequentially for a given window until endWindow()
> > > > > is
> > > > called?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Bryan
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Regards,
> > > > Ashwin.
> > > >
> > > ________________________________________________________
> > >
> > > The information contained in this e-mail is confidential and/or
> > > proprietary to Capital One and/or its affiliates and may only be used
> > > solely in performance of work or services for Capital One. The
> > information
> > > transmitted herewith is intended only for use by the individual or
> entity
> > > to which it is addressed. If the reader of this message is not the
> > intended
> > > recipient, you are hereby notified that any review, retransmission,
> > > dissemination, distribution, copying or other use of, or taking of any
> > > action in reliance upon this information is strictly prohibited. If you
> > > have received this communication in error, please contact the sender
> and
> > > delete the material from your computer.
> > >
> >
> >
> >
> > --
> >
> > Regards,
> > Ashwin.
> >
>

Re: Input Operator Question

Posted by Amol Kekre <am...@datatorrent.com>.
Smith,
The only way to get into a multi-thread situation is for the user to fork
off threads from the operator code. The platform guarantees that the
operator api calls will happen in a single thread.

Thks
Amol


On Mon, Jan 25, 2016 at 2:58 PM, Ashwin Chandra Putta <
ashwinchandrap@gmail.com> wrote:

> Smith,
>
> Within a yarn container, we can start multiple operators. Each such
> operator runs in one thread. All the calls on the operator happen in that
> thread. So the operator is not shared.
>
> Regards,
> Ashwin.
>
> On Mon, Jan 25, 2016 at 2:33 PM, Smith, Kent <Ke...@capitalone.com>
> wrote:
>
> > Wait, additional newbee question.  So an Operator runs in own thread.  Do
> > you mean a given Operator is shared or could multiple threads, external
> to
> > the Operator thread call the same operator and each get an operator
> thread
> > of their own (within their context) or would each thread call within the
> > Operator thread one at a time so each context shares the Operator thread?
> >
> > Thanks.
> >
> > -----Original Message-----
> > From: Bryan Bende [mailto:bbende@gmail.com]
> > Sent: Monday, January 25, 2016 5:27 PM
> > To: dev@apex.incubator.apache.org
> > Subject: Re: Input Operator Question
> >
> > Thanks! Appreciate the quick response.
> >
> > On Mon, Jan 25, 2016 at 5:22 PM, Ashwin Chandra Putta <
> > ashwinchandrap@gmail.com> wrote:
> >
> > > It always called sequentially in a single thread. A given operator
> > > runs in its main thread and all the calls on the operator are called
> > > with in the same thread.
> > >
> > > Regards,
> > > Ashwin.
> > >
> > > On Mon, Jan 25, 2016 at 2:13 PM, Bryan Bende <bb...@gmail.com> wrote:
> > >
> > > > Hello,
> > > >
> > > > Quick beginner question regarding the life-cycle of operators...
> > > >
> > > > Can emitTuples() get called concurrently by multiple threads? or is
> > > > it always called sequentially for a given window until endWindow()
> > > > is
> > > called?
> > > >
> > > > Thanks,
> > > >
> > > > Bryan
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > Regards,
> > > Ashwin.
> > >
> > ________________________________________________________
> >
> > The information contained in this e-mail is confidential and/or
> > proprietary to Capital One and/or its affiliates and may only be used
> > solely in performance of work or services for Capital One. The
> information
> > transmitted herewith is intended only for use by the individual or entity
> > to which it is addressed. If the reader of this message is not the
> intended
> > recipient, you are hereby notified that any review, retransmission,
> > dissemination, distribution, copying or other use of, or taking of any
> > action in reliance upon this information is strictly prohibited. If you
> > have received this communication in error, please contact the sender and
> > delete the material from your computer.
> >
>
>
>
> --
>
> Regards,
> Ashwin.
>

Re: Input Operator Question

Posted by Ashwin Chandra Putta <as...@gmail.com>.
Smith,

Within a yarn container, we can start multiple operators. Each such
operator runs in one thread. All the calls on the operator happen in that
thread. So the operator is not shared.

Regards,
Ashwin.

On Mon, Jan 25, 2016 at 2:33 PM, Smith, Kent <Ke...@capitalone.com>
wrote:

> Wait, additional newbee question.  So an Operator runs in own thread.  Do
> you mean a given Operator is shared or could multiple threads, external to
> the Operator thread call the same operator and each get an operator thread
> of their own (within their context) or would each thread call within the
> Operator thread one at a time so each context shares the Operator thread?
>
> Thanks.
>
> -----Original Message-----
> From: Bryan Bende [mailto:bbende@gmail.com]
> Sent: Monday, January 25, 2016 5:27 PM
> To: dev@apex.incubator.apache.org
> Subject: Re: Input Operator Question
>
> Thanks! Appreciate the quick response.
>
> On Mon, Jan 25, 2016 at 5:22 PM, Ashwin Chandra Putta <
> ashwinchandrap@gmail.com> wrote:
>
> > It always called sequentially in a single thread. A given operator
> > runs in its main thread and all the calls on the operator are called
> > with in the same thread.
> >
> > Regards,
> > Ashwin.
> >
> > On Mon, Jan 25, 2016 at 2:13 PM, Bryan Bende <bb...@gmail.com> wrote:
> >
> > > Hello,
> > >
> > > Quick beginner question regarding the life-cycle of operators...
> > >
> > > Can emitTuples() get called concurrently by multiple threads? or is
> > > it always called sequentially for a given window until endWindow()
> > > is
> > called?
> > >
> > > Thanks,
> > >
> > > Bryan
> > >
> >
> >
> >
> > --
> >
> > Regards,
> > Ashwin.
> >
> ________________________________________________________
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>



-- 

Regards,
Ashwin.

Re: Input Operator Question

Posted by Sandesh Hegde <sa...@datatorrent.com>.
Atmost one thread is responsible for one operator.

On Mon, Jan 25, 2016 at 2:33 PM Smith, Kent <Ke...@capitalone.com>
wrote:

> Wait, additional newbee question.  So an Operator runs in own thread.  Do
> you mean a given Operator is shared or could multiple threads, external to
> the Operator thread call the same operator and each get an operator thread
> of their own (within their context) or would each thread call within the
> Operator thread one at a time so each context shares the Operator thread?
>
> Thanks.
>
> -----Original Message-----
> From: Bryan Bende [mailto:bbende@gmail.com]
> Sent: Monday, January 25, 2016 5:27 PM
> To: dev@apex.incubator.apache.org
> Subject: Re: Input Operator Question
>
> Thanks! Appreciate the quick response.
>
> On Mon, Jan 25, 2016 at 5:22 PM, Ashwin Chandra Putta <
> ashwinchandrap@gmail.com> wrote:
>
> > It always called sequentially in a single thread. A given operator
> > runs in its main thread and all the calls on the operator are called
> > with in the same thread.
> >
> > Regards,
> > Ashwin.
> >
> > On Mon, Jan 25, 2016 at 2:13 PM, Bryan Bende <bb...@gmail.com> wrote:
> >
> > > Hello,
> > >
> > > Quick beginner question regarding the life-cycle of operators...
> > >
> > > Can emitTuples() get called concurrently by multiple threads? or is
> > > it always called sequentially for a given window until endWindow()
> > > is
> > called?
> > >
> > > Thanks,
> > >
> > > Bryan
> > >
> >
> >
> >
> > --
> >
> > Regards,
> > Ashwin.
> >
> ________________________________________________________
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>

RE: Input Operator Question

Posted by "Smith, Kent" <Ke...@capitalone.com>.
Wait, additional newbee question.  So an Operator runs in own thread.  Do you mean a given Operator is shared or could multiple threads, external to the Operator thread call the same operator and each get an operator thread of their own (within their context) or would each thread call within the Operator thread one at a time so each context shares the Operator thread?

Thanks.

-----Original Message-----
From: Bryan Bende [mailto:bbende@gmail.com] 
Sent: Monday, January 25, 2016 5:27 PM
To: dev@apex.incubator.apache.org
Subject: Re: Input Operator Question

Thanks! Appreciate the quick response.

On Mon, Jan 25, 2016 at 5:22 PM, Ashwin Chandra Putta < ashwinchandrap@gmail.com> wrote:

> It always called sequentially in a single thread. A given operator 
> runs in its main thread and all the calls on the operator are called 
> with in the same thread.
>
> Regards,
> Ashwin.
>
> On Mon, Jan 25, 2016 at 2:13 PM, Bryan Bende <bb...@gmail.com> wrote:
>
> > Hello,
> >
> > Quick beginner question regarding the life-cycle of operators...
> >
> > Can emitTuples() get called concurrently by multiple threads? or is 
> > it always called sequentially for a given window until endWindow() 
> > is
> called?
> >
> > Thanks,
> >
> > Bryan
> >
>
>
>
> --
>
> Regards,
> Ashwin.
>
________________________________________________________

The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.

Re: Input Operator Question

Posted by Bryan Bende <bb...@gmail.com>.
Thanks! Appreciate the quick response.

On Mon, Jan 25, 2016 at 5:22 PM, Ashwin Chandra Putta <
ashwinchandrap@gmail.com> wrote:

> It always called sequentially in a single thread. A given operator runs in
> its main thread and all the calls on the operator are called with in the
> same thread.
>
> Regards,
> Ashwin.
>
> On Mon, Jan 25, 2016 at 2:13 PM, Bryan Bende <bb...@gmail.com> wrote:
>
> > Hello,
> >
> > Quick beginner question regarding the life-cycle of operators...
> >
> > Can emitTuples() get called concurrently by multiple threads? or is it
> > always called sequentially for a given window until endWindow() is
> called?
> >
> > Thanks,
> >
> > Bryan
> >
>
>
>
> --
>
> Regards,
> Ashwin.
>

Re: Input Operator Question

Posted by Ashwin Chandra Putta <as...@gmail.com>.
It always called sequentially in a single thread. A given operator runs in
its main thread and all the calls on the operator are called with in the
same thread.

Regards,
Ashwin.

On Mon, Jan 25, 2016 at 2:13 PM, Bryan Bende <bb...@gmail.com> wrote:

> Hello,
>
> Quick beginner question regarding the life-cycle of operators...
>
> Can emitTuples() get called concurrently by multiple threads? or is it
> always called sequentially for a given window until endWindow() is called?
>
> Thanks,
>
> Bryan
>



-- 

Regards,
Ashwin.