Re: [EXTERNAL] Re: How to Change memory setting for a unifier?

["Raja.Aravapalli" <Ra...@target.com>]

Thanks a lot for the response.

Do we any web link that gives details about all the possibile configurations and properties ?



Thanks
Raja.


From: Sandesh Hegde <sa...@datatorrent.com>
Reply-To: "users@apex.apache.org" <us...@apex.apache.org>
Date: Tuesday, June 20, 2017 at 3:23 PM
To: "users@apex.apache.org" <us...@apex.apache.org>
Subject: [EXTERNAL] Re: How to Change memory setting for a unifier?


 Example below.






<property>

<name>dt.operator.redisJoin.outputport.output.unifier.attr.MEMORY_MB</name>

<value>4072</value>

</property>



https://github.com/DataTorrent/streaming-benchmarks/blob/master/conf/apex.xml

On Tue, Jun 20, 2017 at 1:19 PM Raja.Aravapalli <Ra...@target.com>> wrote:

Hi,

Can someone please help me with configuration property on how to increase memory setting for Unifier?

I am using the below way, to increase the memory for Application Master and opertors of a DAG:

<property>
    <name>dt.attr.MASTER_MEMORY_MB</name>
    <value>10240</value>
</property>

<property>
    <name>dt.operator.*.attr.MEMORY_MB</name>
    <value>10240</value>
</property>


UseCase/Scenario:

I am having an apex application that reads from Kafka in 3 partitions with the setting “ONE_TO_MANY”, and sends the messages to following operators. But, the problem is, even though the messages are being read thru multiple parititons, but I see a unifier operation happening after the messages read in multiple kafka partitions. Which is becoming a bottle nect with the number of increase in the number of messages being floated.

I have configured the memory all the operators to 10GB, with above given setting, but, the unifier which has been cropped up, was taking only 2GB by default, and failing with heap space problems.

Hence I wish to increase the memory for the unifier.

Thanks a lot in advance.


Regards,
Raja.