You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@apex.apache.org by Junguk Cho <jm...@gmail.com> on 2016/06/29 19:55:55 UTC

Does Apex use kryo and netty internally?

Hi,

I have some questions for Apex architecture.
Does it use netty for transport layer?
In addition, if two operators are in different processes in one machine,
does they use local TCP connection?

Also does it use kryo serialziation for object between operators?


Thanks in advance.
Junguk

Re: Does Apex use kryo and netty internally?

Posted by Junguk Cho <jm...@gmail.com>.
Hi.

I already tried to Apex with docker.
I just want to know high level structure.

Thanks,
Junguk

2016-06-29 19:08 GMT-04:00 Sandesh Hegde <sa...@datatorrent.com>:

> Deployment policies THREAD_LOCAL & CONTAINER_LOCAL are strict, if it
> cannot be satisfied application won't be launched. Otherwise, it is a best
> effort to satisfy user requirement.
> Apex also has anti-affinity feature.
>
> How are you planning to use Apex?
>
> On Wed, Jun 29, 2016 at 3:27 PM Junguk Cho <jm...@gmail.com> wrote:
>
>> Hi, Sandesh.
>>
>> Thank you for reply.
>>
>> One following question from second one, as far as I know we can give
>> hints to deployment policies, but that is just hint and APEX decides the
>> deployments.
>> Is it right?
>> If so, does it check current cluster states and deploy topologies?
>>
>> Thanks,
>> Junguk
>>
>>
>> 2016-06-29 16:55 GMT-04:00 Sandesh Hegde <sa...@datatorrent.com>:
>>
>>> 1. Not Netty but https://github.com/DataTorrent/Netlet
>>> 2. Operators can be deployed in the same container or in different
>>> containers and the containers can be in the same machine or in different
>>> machines.
>>>
>>>     Within Container:
>>>        Between the operators, it is either is a queue or function call.
>>>         (CONTAINER_LOCAL, THREAD_LOCAL)
>>>        Every container has a BufferServer, to communicate with
>>> downstream operators.            Communication between BufferServer and
>>> operators in that container is local loop back..
>>>
>>>    Different Containers in the same machine -> Local loop back
>>>    Different Containers in different machines -> All bets are off.
>>>
>>> 3. Kryo - by default. Users can use different codec for different
>>> streams.
>>>        (take a look at StreamCodec)
>>>
>>> On Wed, Jun 29, 2016 at 12:55 PM Junguk Cho <jm...@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I have some questions for Apex architecture.
>>>> Does it use netty for transport layer?
>>>> In addition, if two operators are in different processes in one
>>>> machine, does they use local TCP connection?
>>>>
>>>> Also does it use kryo serialziation for object between operators?
>>>>
>>>>
>>>> Thanks in advance.
>>>> Junguk
>>>>
>>>
>>

Re: Does Apex use kryo and netty internally?

Posted by Sandesh Hegde <sa...@datatorrent.com>.
Deployment policies THREAD_LOCAL & CONTAINER_LOCAL are strict, if it cannot
be satisfied application won't be launched. Otherwise, it is a best effort
to satisfy user requirement.
Apex also has anti-affinity feature.

How are you planning to use Apex?

On Wed, Jun 29, 2016 at 3:27 PM Junguk Cho <jm...@gmail.com> wrote:

> Hi, Sandesh.
>
> Thank you for reply.
>
> One following question from second one, as far as I know we can give hints
> to deployment policies, but that is just hint and APEX decides the
> deployments.
> Is it right?
> If so, does it check current cluster states and deploy topologies?
>
> Thanks,
> Junguk
>
>
> 2016-06-29 16:55 GMT-04:00 Sandesh Hegde <sa...@datatorrent.com>:
>
>> 1. Not Netty but https://github.com/DataTorrent/Netlet
>> 2. Operators can be deployed in the same container or in different
>> containers and the containers can be in the same machine or in different
>> machines.
>>
>>     Within Container:
>>        Between the operators, it is either is a queue or function call.
>>         (CONTAINER_LOCAL, THREAD_LOCAL)
>>        Every container has a BufferServer, to communicate with downstream
>> operators.            Communication between BufferServer and operators in
>> that container is local loop back..
>>
>>    Different Containers in the same machine -> Local loop back
>>    Different Containers in different machines -> All bets are off.
>>
>> 3. Kryo - by default. Users can use different codec for different streams.
>>        (take a look at StreamCodec)
>>
>> On Wed, Jun 29, 2016 at 12:55 PM Junguk Cho <jm...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I have some questions for Apex architecture.
>>> Does it use netty for transport layer?
>>> In addition, if two operators are in different processes in one machine,
>>> does they use local TCP connection?
>>>
>>> Also does it use kryo serialziation for object between operators?
>>>
>>>
>>> Thanks in advance.
>>> Junguk
>>>
>>
>

Re: Does Apex use kryo and netty internally?

Posted by Junguk Cho <jm...@gmail.com>.
Hi, Sandesh.

Thank you for reply.

One following question from second one, as far as I know we can give hints
to deployment policies, but that is just hint and APEX decides the
deployments.
Is it right?
If so, does it check current cluster states and deploy topologies?

Thanks,
Junguk


2016-06-29 16:55 GMT-04:00 Sandesh Hegde <sa...@datatorrent.com>:

> 1. Not Netty but https://github.com/DataTorrent/Netlet
> 2. Operators can be deployed in the same container or in different
> containers and the containers can be in the same machine or in different
> machines.
>
>     Within Container:
>        Between the operators, it is either is a queue or function call.
>         (CONTAINER_LOCAL, THREAD_LOCAL)
>        Every container has a BufferServer, to communicate with downstream
> operators.            Communication between BufferServer and operators in
> that container is local loop back..
>
>    Different Containers in the same machine -> Local loop back
>    Different Containers in different machines -> All bets are off.
>
> 3. Kryo - by default. Users can use different codec for different streams.
>        (take a look at StreamCodec)
>
> On Wed, Jun 29, 2016 at 12:55 PM Junguk Cho <jm...@gmail.com> wrote:
>
>> Hi,
>>
>> I have some questions for Apex architecture.
>> Does it use netty for transport layer?
>> In addition, if two operators are in different processes in one machine,
>> does they use local TCP connection?
>>
>> Also does it use kryo serialziation for object between operators?
>>
>>
>> Thanks in advance.
>> Junguk
>>
>

Re: Does Apex use kryo and netty internally?

Posted by Sandesh Hegde <sa...@datatorrent.com>.
1. Not Netty but https://github.com/DataTorrent/Netlet
2. Operators can be deployed in the same container or in different
containers and the containers can be in the same machine or in different
machines.

    Within Container:
       Between the operators, it is either is a queue or function call.
        (CONTAINER_LOCAL, THREAD_LOCAL)
       Every container has a BufferServer, to communicate with downstream
operators.            Communication between BufferServer and operators in
that container is local loop back..

   Different Containers in the same machine -> Local loop back
   Different Containers in different machines -> All bets are off.

3. Kryo - by default. Users can use different codec for different streams.
       (take a look at StreamCodec)

On Wed, Jun 29, 2016 at 12:55 PM Junguk Cho <jm...@gmail.com> wrote:

> Hi,
>
> I have some questions for Apex architecture.
> Does it use netty for transport layer?
> In addition, if two operators are in different processes in one machine,
> does they use local TCP connection?
>
> Also does it use kryo serialziation for object between operators?
>
>
> Thanks in advance.
> Junguk
>