You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apex.apache.org by "Kottapalli, Venkatesh" <VK...@DIRECTV.com> on 2015/11/06 02:03:20 UTC

Reg. partitioning operator

Hi,

                I have tried partitioning block reader by setting the following properties.

<property>
              <name>dt.application.appname.operator.reader.attr.MEMORY_MB
              </name>
              <value>24576</value>
       </property>

       property>
              <name>dt.application.appname.operator.reader.prop.PARTITIONER</name>
              <value>com.datatorrent.lib.partitioner.StatelessPartitioner:4</value>
       </property>

<!- Only Downstream operator of reader, It has a port 'input' listening on output port of reader - This will not unify data -->
       <property>
              <name>dt.application.appname.operator.console.port.input.attr.PARTITION_PARALLEL</name>
              <value>true</value>
       </property>

This is reading data into a single operator and not partitioning reader operator, do I need to add something else for the partitioning to be enabled. BlockReader has not implemented Partitioner since it is already implemented in one of its parent classes. Please suggest.

Regards,
Venkatesh.

Re: Reg. partitioning operator

Posted by Chandni Singh <ch...@datatorrent.com>.
Hi Venkatesh,

      property>
              <name>dt.application.appname.operator.reader.prop.
PARTITIONER</name>
              <value>com.datatorrent.lib.partitioner.
StatelessPartitioner:4</value>
       </property>

In the above config, PARTITIONER should be an attribute:
      property>
              <name>dt.application.appname.operator.reader.attr.
PARTITIONER</name>
              <value>com.datatorrent.common.partitioner.StatelessPartitioner
:4</value>
       </property>

I believe with release 3.0 the StatelessPartitioner was moved to Apex
common package so you should be using that as well.

Chandni



On Thu, Nov 5, 2015 at 5:03 PM, Kottapalli, Venkatesh <
VKottapalli@directv.com> wrote:

> Hi,
>
>                 I have tried partitioning block reader by setting the
> following properties.
>
> <property>
>               <name>dt.application.appname.operator.reader.attr.MEMORY_MB
>               </name>
>               <value>24576</value>
>        </property>
>
>        property>
>
> <name>dt.application.appname.operator.reader.prop.PARTITIONER</name>
>
> <value>com.datatorrent.lib.partitioner.StatelessPartitioner:4</value>
>        </property>
>
> <!- Only Downstream operator of reader, It has a port 'input' listening on
> output port of reader - This will not unify data -->
>        <property>
>
> <name>dt.application.appname.operator.console.port.input.attr.PARTITION_PARALLEL</name>
>               <value>true</value>
>        </property>
>
> This is reading data into a single operator and not partitioning reader
> operator, do I need to add something else for the partitioning to be
> enabled. BlockReader has not implemented Partitioner since it is already
> implemented in one of its parent classes. Please suggest.
>
> Regards,
> Venkatesh.
>
>
>
>
>
>