You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apex.apache.org by "Ganelin, Ilya" <Il...@capitalone.com> on 2015/09/03 20:54:21 UTC
Can Partitioners/StatsListeners be shared between Operators?
Hello – I’m wondering if the following code is safe. Are there any potential issues with sharing a partitioner or StatsListener in this manner?:
FeatureGeneration featureGenOne =
dag.addOperator("FeatureGenerator I", FeatureGeneration.class);
// setup dynamic partitioning for the FeatureGeneration
StatelessThroughputBasedPartitioner<FeatureGeneration> featureGeneratorPartitioner =
new StatelessThroughputBasedPartitioner<FeatureGeneration>();
featureGeneratorPartitioner.setMaximumEvents(1000);
featureGeneratorPartitioner.setMinimumEvents(100);
featureGeneratorPartitioner.setCooldownMillis(10000);
featureGeneratorPartitioner.setInitialPartitionCount(16);
dag.setAttribute(featureGenOne, Context.OperatorContext.PARTITIONER, featureGeneratorPartitioner);
Collection<StatsListener> featureGeneratorListeners = new ArrayList<StatsListener>();
featureGeneratorListeners.add(featureGeneratorPartitioner);
dag.setAttribute(featureGenOne, Context.OperatorContext.STATS_LISTENERS, featureGeneratorListeners);
FeatureGeneration featureGenTwo =
dag.addOperator("FeatureGenerator II", FeatureGeneration.class);
// setup dynamic partitioning for the FeatureGeneration
dag.setAttribute(featureGenTwo, Context.OperatorContext.PARTITIONER, featureGeneratorPartitioner)
dag.setAttribute(featureGenTwo, Context.OperatorContext.STATS_LISTENERS, featureGeneratorListeners);
________________________________________________________
The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
Re: Can Partitioners/StatsListeners be shared between Operators?
Posted by Chetan Narsude <ch...@datatorrent.com>.
Yes! As a matter of fact - we do use this pattern frequently as stats
frequently influence need for partitioning.
--
Chetan
On Thu, Sep 3, 2015 at 11:54 AM, Ganelin, Ilya <Il...@capitalone.com>
wrote:
> Hello – I’m wondering if the following code is safe. Are there any
> potential issues with sharing a partitioner or StatsListener in this
> manner?:
>
> FeatureGeneration featureGenOne =
> dag.addOperator("FeatureGenerator I", FeatureGeneration.class);
> // setup dynamic partitioning for the FeatureGeneration
> StatelessThroughputBasedPartitioner<FeatureGeneration>
> featureGeneratorPartitioner =
> new StatelessThroughputBasedPartitioner<FeatureGeneration>();
> featureGeneratorPartitioner.setMaximumEvents(1000);
> featureGeneratorPartitioner.setMinimumEvents(100);
> featureGeneratorPartitioner.setCooldownMillis(10000);
> featureGeneratorPartitioner.setInitialPartitionCount(16);
> dag.setAttribute(featureGenOne, Context.OperatorContext.PARTITIONER,
> featureGeneratorPartitioner);
> Collection<StatsListener> featureGeneratorListeners = new
> ArrayList<StatsListener>();
> featureGeneratorListeners.add(featureGeneratorPartitioner);
> dag.setAttribute(featureGenOne, Context.OperatorContext.STATS_LISTENERS,
> featureGeneratorListeners);
>
> FeatureGeneration featureGenTwo =
> dag.addOperator("FeatureGenerator II", FeatureGeneration.class);
> // setup dynamic partitioning for the FeatureGeneration
> dag.setAttribute(featureGenTwo, Context.OperatorContext.PARTITIONER,
> featureGeneratorPartitioner)
>
> dag.setAttribute(featureGenTwo, Context.OperatorContext.STATS_LISTENERS,
> featureGeneratorListeners);
> ________________________________________________________
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>