You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@apex.apache.org by Junguk Cho <jm...@gmail.com> on 2016/06/07 13:49:37 UTC
Capabilities for Dynamic Application Modifications
Hi, all.
I read tutorial in
https://apex.apache.org/docs/apex/application_development/#running-a-test-application.
There are some capabilities from apache apex called "Dynamic
Application Modifications".
I would like to explore "Dynamic Partitioning" and "Modification of
DAG structure".
How do I test both capabilities?
In addition, are there some example to explore them?
Thanks in advance.
- Junguk
Re: Capabilities for Dynamic Application Modifications
Posted by Munagala Ramanath <ra...@datatorrent.com>.
Yes, I believe that is correct.
Ram
On Tue, Jun 7, 2016 at 11:32 AM, Junguk Cho <jm...@gmail.com> wrote:
> Hi, Ram.
>
> Thank you for reply.
>
> I have one following question.
>
> Apex manual said that
>
> *"provided the code of the operator being added is already in the
> classpath of the running application master. This enables application
> developers to add or remove processing pipelines on the fly without having
> to restart the application."*
> Which means when we build the application, all classes which we use now or
> later are in application.apa file.
> Is it right?
> And the using below commands (apexcli) , we can modify the DAG on the fly.
>
> create-operator operator-name class-name Create an operator create-stream
> stream-name from-operator-name from-port-name to-operator-name to-port-name
> Create a stream
>
>
> Thanks,
> Junguk
>
> 2016-06-07 12:37 GMT-04:00 Munagala Ramanath <ra...@datatorrent.com>:
>
>> Hi,
>>
>> For Dynamic Partitioning, please take a look at the example at:
>>
>> https://github.com/DataTorrent/examples/tree/master/tutorials/dynamic-partition
>>
>> as well as the Malhar class:
>>
>> https://github.com/apache/apex-malhar/blob/master/library/src/main/java/com/datatorrent/lib/partitioner/StatelessThroughputBasedPartitioner.java
>>
>> For dynamically modifying the DAG, you should be able to use the
>> *apexcli* (aka *dtcli*) commandline tool documented
>> here: http://docs.datatorrent.com/apexcli/
>>
>> Roughly speaking, the process is:
>> 1. Connect to your application id using the *connect* command
>> 2. *begin-logical-plan-change*
>> 3. Use some combination of *remove-stream*, *remove-operator*,
>> *create-operator,* and *create-stream*
>> 4. *submit*
>>
>> Ram
>>
>> On Tue, Jun 7, 2016 at 6:49 AM, Junguk Cho <jm...@gmail.com> wrote:
>>
>>> Hi, all.
>>>
>>> I read tutorial in
>>>
>>> https://apex.apache.org/docs/apex/application_development/#running-a-test-application
>>> .
>>>
>>> There are some capabilities from apache apex called "Dynamic
>>> Application Modifications".
>>>
>>> I would like to explore "Dynamic Partitioning" and "Modification of
>>> DAG structure".
>>> How do I test both capabilities?
>>> In addition, are there some example to explore them?
>>>
>>> Thanks in advance.
>>> - Junguk
>>>
>>
>>
>
Re: Capabilities for Dynamic Application Modifications
Posted by Junguk Cho <jm...@gmail.com>.
Hi, Ram.
Thank you for reply.
I have one following question.
Apex manual said that
*"provided the code of the operator being added is already in the classpath
of the running application master. This enables application developers to
add or remove processing pipelines on the fly without having to restart the
application."*
Which means when we build the application, all classes which we use now or
later are in application.apa file.
Is it right?
And the using below commands (apexcli) , we can modify the DAG on the fly.
create-operator operator-name class-name Create an operator create-stream
stream-name from-operator-name from-port-name to-operator-name to-port-name
Create a stream
Thanks,
Junguk
2016-06-07 12:37 GMT-04:00 Munagala Ramanath <ra...@datatorrent.com>:
> Hi,
>
> For Dynamic Partitioning, please take a look at the example at:
>
> https://github.com/DataTorrent/examples/tree/master/tutorials/dynamic-partition
>
> as well as the Malhar class:
>
> https://github.com/apache/apex-malhar/blob/master/library/src/main/java/com/datatorrent/lib/partitioner/StatelessThroughputBasedPartitioner.java
>
> For dynamically modifying the DAG, you should be able to use the *apexcli*
> (aka *dtcli*) commandline tool documented
> here: http://docs.datatorrent.com/apexcli/
>
> Roughly speaking, the process is:
> 1. Connect to your application id using the *connect* command
> 2. *begin-logical-plan-change*
> 3. Use some combination of *remove-stream*, *remove-operator*,
> *create-operator,* and *create-stream*
> 4. *submit*
>
> Ram
>
> On Tue, Jun 7, 2016 at 6:49 AM, Junguk Cho <jm...@gmail.com> wrote:
>
>> Hi, all.
>>
>> I read tutorial in
>>
>> https://apex.apache.org/docs/apex/application_development/#running-a-test-application
>> .
>>
>> There are some capabilities from apache apex called "Dynamic
>> Application Modifications".
>>
>> I would like to explore "Dynamic Partitioning" and "Modification of
>> DAG structure".
>> How do I test both capabilities?
>> In addition, are there some example to explore them?
>>
>> Thanks in advance.
>> - Junguk
>>
>
>
Re: Capabilities for Dynamic Application Modifications
Posted by Munagala Ramanath <ra...@datatorrent.com>.
Hi,
For Dynamic Partitioning, please take a look at the example at:
https://github.com/DataTorrent/examples/tree/master/tutorials/dynamic-partition
as well as the Malhar class:
https://github.com/apache/apex-malhar/blob/master/library/src/main/java/com/datatorrent/lib/partitioner/StatelessThroughputBasedPartitioner.java
For dynamically modifying the DAG, you should be able to use the *apexcli*
(aka *dtcli*) commandline tool documented
here: http://docs.datatorrent.com/apexcli/
Roughly speaking, the process is:
1. Connect to your application id using the *connect* command
2. *begin-logical-plan-change*
3. Use some combination of *remove-stream*, *remove-operator*,
*create-operator,* and *create-stream*
4. *submit*
Ram
On Tue, Jun 7, 2016 at 6:49 AM, Junguk Cho <jm...@gmail.com> wrote:
> Hi, all.
>
> I read tutorial in
>
> https://apex.apache.org/docs/apex/application_development/#running-a-test-application
> .
>
> There are some capabilities from apache apex called "Dynamic
> Application Modifications".
>
> I would like to explore "Dynamic Partitioning" and "Modification of
> DAG structure".
> How do I test both capabilities?
> In addition, are there some example to explore them?
>
> Thanks in advance.
> - Junguk
>