You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@apex.apache.org by "Raja.Aravapalli" <Ra...@target.com> on 2017/06/20 20:13:06 UTC
How to Change memory setting for a unifier?
Hi,
Can someone please help me with configuration property on how to increase memory setting for Unifier?
I am using the below way, to increase the memory for Application Master and opertors of a DAG:
<property>
<name>dt.attr.MASTER_MEMORY_MB</name>
<value>10240</value>
</property>
<property>
<name>dt.operator.*.attr.MEMORY_MB</name>
<value>10240</value>
</property>
UseCase/Scenario:
I am having an apex application that reads from Kafka in 3 partitions with the setting “ONE_TO_MANY”, and sends the messages to following operators. But, the problem is, even though the messages are being read thru multiple parititons, but I see a unifier operation happening after the messages read in multiple kafka partitions. Which is becoming a bottle nect with the number of increase in the number of messages being floated.
I have configured the memory all the operators to 10GB, with above given setting, but, the unifier which has been cropped up, was taking only 2GB by default, and failing with heap space problems.
Hence I wish to increase the memory for the unifier.
Thanks a lot in advance.
Regards,
Raja.
Re: [EXTERNAL] Re: How to Change memory setting for a unifier?
Posted by "Raja.Aravapalli" <Ra...@target.com>.
Thanks a lot for the response.
Do we any web link that gives details about all the possibile configurations and properties ?
Thanks
Raja.
From: Sandesh Hegde <sa...@datatorrent.com>
Reply-To: "users@apex.apache.org" <us...@apex.apache.org>
Date: Tuesday, June 20, 2017 at 3:23 PM
To: "users@apex.apache.org" <us...@apex.apache.org>
Subject: [EXTERNAL] Re: How to Change memory setting for a unifier?
Example below.
<property>
<name>dt.operator.redisJoin.outputport.output.unifier.attr.MEMORY_MB</name>
<value>4072</value>
</property>
https://github.com/DataTorrent/streaming-benchmarks/blob/master/conf/apex.xml
On Tue, Jun 20, 2017 at 1:19 PM Raja.Aravapalli <Ra...@target.com>> wrote:
Hi,
Can someone please help me with configuration property on how to increase memory setting for Unifier?
I am using the below way, to increase the memory for Application Master and opertors of a DAG:
<property>
<name>dt.attr.MASTER_MEMORY_MB</name>
<value>10240</value>
</property>
<property>
<name>dt.operator.*.attr.MEMORY_MB</name>
<value>10240</value>
</property>
UseCase/Scenario:
I am having an apex application that reads from Kafka in 3 partitions with the setting “ONE_TO_MANY”, and sends the messages to following operators. But, the problem is, even though the messages are being read thru multiple parititons, but I see a unifier operation happening after the messages read in multiple kafka partitions. Which is becoming a bottle nect with the number of increase in the number of messages being floated.
I have configured the memory all the operators to 10GB, with above given setting, but, the unifier which has been cropped up, was taking only 2GB by default, and failing with heap space problems.
Hence I wish to increase the memory for the unifier.
Thanks a lot in advance.
Regards,
Raja.
Re: How to Change memory setting for a unifier?
Posted by Sandesh Hegde <sa...@datatorrent.com>.
Example below.
<property>
<name>dt.operator.redisJoin.outputport.output.unifier.attr.MEMORY_MB</name>
<value>4072</value>
</property>
https://github.com/DataTorrent/streaming-benchmarks/blob/master/conf/apex.xml
On Tue, Jun 20, 2017 at 1:19 PM Raja.Aravapalli <Ra...@target.com>
wrote:
>
>
> Hi,
>
>
>
> Can someone please help me with configuration property on how to increase
> memory setting for Unifier?
>
>
>
> I am using the below way, to increase the memory for Application Master
> and opertors of a DAG:
>
>
>
> <*property*>
> <*name*>dt.attr.MASTER_MEMORY_MB</*name*>
> <*value*>10240</*value*>
> </*property*>
>
> <*property*>
> <*name*>dt.operator.*.attr.MEMORY_MB</*name*>
> <*value*>10240</*value*>
> </*property*>
>
>
>
>
>
> *UseCase/Scenario:*
>
>
>
> I am having an apex application that reads from Kafka in 3 partitions with
> the setting “*ONE_TO_MANY*”, and sends the messages to following
> operators. But, the problem is, even though the messages are being read
> thru multiple parititons, but I see a unifier operation happening after the
> messages read in multiple kafka partitions. Which is becoming a bottle nect
> with the number of increase in the number of messages being floated.
>
>
>
> I have configured the memory all the operators to 10GB, with above given
> setting, but, the unifier which has been cropped up, was taking only 2GB by
> default, and failing with heap space problems.
>
>
>
> Hence I wish to increase the memory for the unifier.
>
>
>
> Thanks a lot in advance.
>
>
>
>
>
> Regards,
>
> Raja.
>
>
>
>
>
>
>
Re: [EXTERNAL] Re: How to Change memory setting for a unifier?
Posted by "Raja.Aravapalli" <Ra...@target.com>.
Thanks Sandesh. That helps.
Regards,
Raja.
From: Sandesh Hegde <sa...@datatorrent.com>
Reply-To: "users@apex.apache.org" <us...@apex.apache.org>
Date: Tuesday, June 20, 2017 at 6:00 PM
To: "users@apex.apache.org" <us...@apex.apache.org>
Subject: Re: [EXTERNAL] Re: How to Change memory setting for a unifier?
To find (almost)everything, source is the answer
https://github.com/apache/apex-core/blob/master/api/src/main/java/com/datatorrent/api/Context.java
On Tue, Jun 20, 2017 at 2:13 PM Raja.Aravapalli <Ra...@target.com>> wrote:
Thanks a lot for the response.
Do we any web link that gives details about all the possibile configurations and properties ?
Thanks
Raja.
From: Pramod Immaneni <pr...@datatorrent.com>>
Reply-To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Date: Tuesday, June 20, 2017 at 3:22 PM
To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Subject: [EXTERNAL] Re: How to Change memory setting for a unifier?
Hi Raja,
To set the memory for the unifier you would use the port to identify the unifier, so something like
dt.operator.oper1.port.port1.unifier.MEMORY_MB
or use the setUnifierAttribute in DAG if setting it from java.
Thanks
On Tue, Jun 20, 2017 at 1:13 PM, Raja.Aravapalli <Ra...@target.com>> wrote:
Hi,
Can someone please help me with configuration property on how to increase memory setting for Unifier?
I am using the below way, to increase the memory for Application Master and opertors of a DAG:
<property>
<name>dt.attr.MASTER_MEMORY_MB</name>
<value>10240</value>
</property>
<property>
<name>dt.operator.*.attr.MEMORY_MB</name>
<value>10240</value>
</property>
UseCase/Scenario:
I am having an apex application that reads from Kafka in 3 partitions with the setting “ONE_TO_MANY”, and sends the messages to following operators. But, the problem is, even though the messages are being read thru multiple parititons, but I see a unifier operation happening after the messages read in multiple kafka partitions. Which is becoming a bottle nect with the number of increase in the number of messages being floated.
I have configured the memory all the operators to 10GB, with above given setting, but, the unifier which has been cropped up, was taking only 2GB by default, and failing with heap space problems.
Hence I wish to increase the memory for the unifier.
Thanks a lot in advance.
Regards,
Raja.
Re: [EXTERNAL] Re: How to Change memory setting for a unifier?
Posted by Sandesh Hegde <sa...@datatorrent.com>.
To find (almost)everything, source is the answer
https://github.com/apache/apex-core/blob/master/api/src/main/java/com/datatorrent/api/Context.java
On Tue, Jun 20, 2017 at 2:13 PM Raja.Aravapalli <Ra...@target.com>
wrote:
> Thanks a lot for the response.
>
>
>
> Do we any web link that gives details about all the possibile
> configurations and properties ?
>
>
>
>
>
>
>
> Thanks
>
> Raja.
>
>
>
>
>
> *From: *Pramod Immaneni <pr...@datatorrent.com>
> *Reply-To: *"users@apex.apache.org" <us...@apex.apache.org>
> *Date: *Tuesday, June 20, 2017 at 3:22 PM
>
>
> *To: *"users@apex.apache.org" <us...@apex.apache.org>
> *Subject: *[EXTERNAL] Re: How to Change memory setting for a unifier?
>
>
>
> Hi Raja,
>
>
>
> To set the memory for the unifier you would use the port to identify the
> unifier, so something like
>
>
>
> dt.operator.oper1.port.port1.unifier.MEMORY_MB
>
>
>
> or use the setUnifierAttribute in DAG if setting it from java.
>
>
>
> Thanks
>
>
>
> On Tue, Jun 20, 2017 at 1:13 PM, Raja.Aravapalli <
> Raja.Aravapalli@target.com> wrote:
>
>
>
> Hi,
>
>
>
> Can someone please help me with configuration property on how to increase
> memory setting for Unifier?
>
>
>
> I am using the below way, to increase the memory for Application Master
> and opertors of a DAG:
>
>
>
> <*property*>
> <*name*>dt.attr.MASTER_MEMORY_MB</*name*>
> <*value*>10240</*value*>
> </*property*>
>
> <*property*>
> <*name*>dt.operator.*.attr.MEMORY_MB</*name*>
> <*value*>10240</*value*>
> </*property*>
>
>
>
>
>
> *UseCase/Scenario:*
>
>
>
> I am having an apex application that reads from Kafka in 3 partitions with
> the setting “*ONE_TO_MANY*”, and sends the messages to following
> operators. But, the problem is, even though the messages are being read
> thru multiple parititons, but I see a unifier operation happening after the
> messages read in multiple kafka partitions. Which is becoming a bottle nect
> with the number of increase in the number of messages being floated.
>
>
>
> I have configured the memory all the operators to 10GB, with above given
> setting, but, the unifier which has been cropped up, was taking only 2GB by
> default, and failing with heap space problems.
>
>
>
> Hence I wish to increase the memory for the unifier.
>
>
>
> Thanks a lot in advance.
>
>
>
>
>
> Regards,
>
> Raja.
>
>
>
>
>
>
>
>
>
Re: [EXTERNAL] Re: How to Change memory setting for a unifier?
Posted by "Raja.Aravapalli" <Ra...@target.com>.
Thanks a lot for the response.
Do we any web link that gives details about all the possibile configurations and properties ?
Thanks
Raja.
From: Pramod Immaneni <pr...@datatorrent.com>
Reply-To: "users@apex.apache.org" <us...@apex.apache.org>
Date: Tuesday, June 20, 2017 at 3:22 PM
To: "users@apex.apache.org" <us...@apex.apache.org>
Subject: [EXTERNAL] Re: How to Change memory setting for a unifier?
Hi Raja,
To set the memory for the unifier you would use the port to identify the unifier, so something like
dt.operator.oper1.port.port1.unifier.MEMORY_MB
or use the setUnifierAttribute in DAG if setting it from java.
Thanks
On Tue, Jun 20, 2017 at 1:13 PM, Raja.Aravapalli <Ra...@target.com>> wrote:
Hi,
Can someone please help me with configuration property on how to increase memory setting for Unifier?
I am using the below way, to increase the memory for Application Master and opertors of a DAG:
<property>
<name>dt.attr.MASTER_MEMORY_MB</name>
<value>10240</value>
</property>
<property>
<name>dt.operator.*.attr.MEMORY_MB</name>
<value>10240</value>
</property>
UseCase/Scenario:
I am having an apex application that reads from Kafka in 3 partitions with the setting “ONE_TO_MANY”, and sends the messages to following operators. But, the problem is, even though the messages are being read thru multiple parititons, but I see a unifier operation happening after the messages read in multiple kafka partitions. Which is becoming a bottle nect with the number of increase in the number of messages being floated.
I have configured the memory all the operators to 10GB, with above given setting, but, the unifier which has been cropped up, was taking only 2GB by default, and failing with heap space problems.
Hence I wish to increase the memory for the unifier.
Thanks a lot in advance.
Regards,
Raja.
Re: How to Change memory setting for a unifier?
Posted by Pramod Immaneni <pr...@datatorrent.com>.
Hi Raja,
To set the memory for the unifier you would use the port to identify the
unifier, so something like
dt.operator.oper1.port.port1.unifier.MEMORY_MB
or use the setUnifierAttribute in DAG if setting it from java.
Thanks
On Tue, Jun 20, 2017 at 1:13 PM, Raja.Aravapalli <Raja.Aravapalli@target.com
> wrote:
>
>
> Hi,
>
>
>
> Can someone please help me with configuration property on how to increase
> memory setting for Unifier?
>
>
>
> I am using the below way, to increase the memory for Application Master
> and opertors of a DAG:
>
>
>
> <*property*>
> <*name*>dt.attr.MASTER_MEMORY_MB</*name*>
> <*value*>10240</*value*>
> </*property*>
>
> <*property*>
> <*name*>dt.operator.*.attr.MEMORY_MB</*name*>
> <*value*>10240</*value*>
> </*property*>
>
>
>
>
>
> *UseCase/Scenario:*
>
>
>
> I am having an apex application that reads from Kafka in 3 partitions with
> the setting “*ONE_TO_MANY*”, and sends the messages to following
> operators. But, the problem is, even though the messages are being read
> thru multiple parititons, but I see a unifier operation happening after the
> messages read in multiple kafka partitions. Which is becoming a bottle nect
> with the number of increase in the number of messages being floated.
>
>
>
> I have configured the memory all the operators to 10GB, with above given
> setting, but, the unifier which has been cropped up, was taking only 2GB by
> default, and failing with heap space problems.
>
>
>
> Hence I wish to increase the memory for the unifier.
>
>
>
> Thanks a lot in advance.
>
>
>
>
>
> Regards,
>
> Raja.
>
>
>
>
>
>
>