You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@apex.apache.org by Ananth Gundabattula <ag...@gmail.com> on 2016/05/12 08:05:12 UTC
YARN memory settings and the Apex memory model
Hello All,
I am seeing the following log from the web ui ocassionally when my
operators are getting killed. Is there any way I can control the memory
settings that are used to communicate with YARN when negotiating a
container ?
How does the typical yarn settings for a container heap and max memory
relate to the Apex memory allocation model.
The info messages I see in the web UI are as follows:
Container [pid=14699,containerID=container_1462863487071_0015_01_000012]
is running beyond physical memory limits. Current usage: 1.5 GB of 1.5
GB physical memory used; 6.1 GB of 3.1 GB virtual memory used. Killing
container.
Dump of the process-tree for container_1462863487071_0015_01_000012 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES)
FULL_CMD_LINE
|- 14817 14699 14699 14699 (java) 1584 1654 6426968064 393896
/usr/java/default/bin/java -Xmx4429185024
-Ddt.attr.APPLICATION_PATH=hdfs://dwh109.qaperf2.sac.int.threatmetrix.com:8020/user/dtadmin/datatorrent/apps/application_1462863487071_0015
-Djava.io.tmpdir=/data3/yarn/nm/usercache/root/appcache/application_1462863487071_0015/container_1462863487071_0015_01_000012/tmp
-Ddt.cid=container_1462863487071_0015_01_000012
-Dhadoop.root.logger=INFO,RFA
-Dhadoop.log.dir=/data3/yarn/container-logs/application_1462863487071_0015/container_1462863487071_0015_01_000012
-Ddt.loggers.level=com.datatorrent.*:INFO,org.apache.*:INFO
com.datatorrent.stram.engine.StreamingContainer
|- 14699 14697 14699 14699 (bash) 1 2 108646400 303 /bin/bash -c
/usr/java/default/bin/java -Xmx4429185024
-Ddt.attr.APPLICATION_PATH=hdfs://dwh109.qaperf2.sac.int.threatmetrix.com:8020/user/dtadmin/datatorrent/apps/application_1462863487071_0015
-Djava.io.tmpdir=/data3/yarn/nm/usercache/root/appcache/application_1462863487071_0015/container_1462863487071_0015_01_000012/tmp
-Ddt.cid=container_1462863487071_0015_01_000012
-Dhadoop.root.logger=INFO,RFA
-Dhadoop.log.dir=/data3/yarn/container-logs/application_1462863487071_0015/container_1462863487071_0015_01_000012
-Ddt.loggers.level=com.datatorrent.*:INFO,org.apache.*:INFO
com.datatorrent.stram.engine.StreamingContainer
1>/data3/yarn/container-logs/application_1462863487071_0015/container_1462863487071_0015_01_000012/stdout
2>/data3/yarn/container-logs/application_1462863487071_0015/container_1462863487071_0015_01_000012/stderr
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Regards,
Ananth
Re: YARN memory settings and the Apex memory model
Posted by Sandesh Hegde <sa...@datatorrent.com>.
Yes, that's correct.
Also, there are other things that affects the container size as well,
examples: Unifiers, Max memory and Min memory.
If you are adventurous take look at the Physical Plan preparation code.
https://github.com/apache/incubator-apex-core/blob/ddb7471edd37ef228432c7d80e1e118368e68450/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java
On Thu, May 12, 2016 at 4:00 AM Ananth Gundabattula <ag...@gmail.com>
wrote:
> Thanks Shubham. I shall bump up the memory a bit more.
>
> I was wondering how the operator memory relates to the YARN container
> memory settings ? Or it depends on the deployment models ?
>
> For example , if the deployment model is thread local, the YARN container
> needs to be ( considering above example ) configured for atleast memory of
> 2048 * number of operators + Buffer Server Size ?
>
> If the deployment model were not Thread local, it would make YARN
> container requirements for memory lower per container ?
>
> Regards,
> Ananth
>
> On Thu, May 12, 2016 at 7:19 PM, Shubham Pathak <sh...@datatorrent.com>
> wrote:
>
>> Hello Ananth,
>>
>> Looks like operator requires more memory.
>> You may add this property to have more memory allocated to the container.
>>
>> In properties.xml , for operator O in the application you may specify the
>> property :
>>
>> <property>
>> <name>dt.operator.*O*.attr.MEMORY_MB</name>
>> <value>2048</value>
>> </property>
>>
>> Thanks,
>> Shubham
>>
>> On Thu, May 12, 2016 at 1:35 PM, Ananth Gundabattula <
>> agundabattula@gmail.com> wrote:
>>
>>> Hello All,
>>>
>>> I am seeing the following log from the web ui ocassionally when my
>>> operators are getting killed. Is there any way I can control the memory
>>> settings that are used to communicate with YARN when negotiating a
>>> container ?
>>>
>>> How does the typical yarn settings for a container heap and max memory
>>> relate to the Apex memory allocation model.
>>>
>>> The info messages I see in the web UI are as follows:
>>>
>>> Container [pid=14699,containerID=container_1462863487071_0015_01_000012] is running beyond physical memory limits. Current usage: 1.5 GB of 1.5 GB physical memory used; 6.1 GB of 3.1 GB virtual memory used. Killing container.
>>> Dump of the process-tree for container_1462863487071_0015_01_000012 :
>>> |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
>>> |- 14817 14699 14699 14699 (java) 1584 1654 6426968064 393896 /usr/java/default/bin/java -Xmx4429185024 -Ddt.attr.APPLICATION_PATH=hdfs://dwh109.qaperf2.sac.int.threatmetrix.com:8020/user/dtadmin/datatorrent/apps/application_1462863487071_0015 -Djava.io.tmpdir=/data3/yarn/nm/usercache/root/appcache/application_1462863487071_0015/container_1462863487071_0015_01_000012/tmp -Ddt.cid=container_1462863487071_0015_01_000012 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=/data3/yarn/container-logs/application_1462863487071_0015/container_1462863487071_0015_01_000012 -Ddt.loggers.level=com.datatorrent.*:INFO,org.apache.*:INFO com.datatorrent.stram.engine.StreamingContainer
>>> |- 14699 14697 14699 14699 (bash) 1 2 108646400 303 /bin/bash -c /usr/java/default/bin/java -Xmx4429185024 -Ddt.attr.APPLICATION_PATH=hdfs://dwh109.qaperf2.sac.int.threatmetrix.com:8020/user/dtadmin/datatorrent/apps/application_1462863487071_0015 -Djava.io.tmpdir=/data3/yarn/nm/usercache/root/appcache/application_1462863487071_0015/container_1462863487071_0015_01_000012/tmp -Ddt.cid=container_1462863487071_0015_01_000012 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=/data3/yarn/container-logs/application_1462863487071_0015/container_1462863487071_0015_01_000012 -Ddt.loggers.level=com.datatorrent.*:INFO,org.apache.*:INFO com.datatorrent.stram.engine.StreamingContainer 1>/data3/yarn/container-logs/application_1462863487071_0015/container_1462863487071_0015_01_000012/stdout 2>/data3/yarn/container-logs/application_1462863487071_0015/container_1462863487071_0015_01_000012/stderr
>>>
>>> Container killed on request. Exit code is 143
>>> Container exited with a non-zero exit code 143
>>>
>>>
>>> Regards,
>>>
>>> Ananth
>>>
>>>
>>>
>>
>
Re: YARN memory settings and the Apex memory model
Posted by Ananth Gundabattula <ag...@gmail.com>.
Thanks a lot Thomas & Sandesh.
I somehow missed the "troubleshooting" section of the docs. This is what I
was looking for.
Regards,
Ananth
On Fri, May 13, 2016 at 1:30 AM, Thomas Weise <th...@gmail.com>
wrote:
> Ananth,
>
> Please have a look at:
>
> http://docs.datatorrent.com/troubleshooting/#configuring-memory
>
> Thanks,
> Thomas
>
>
> On Thu, May 12, 2016 at 4:00 AM, Ananth Gundabattula <
> agundabattula@gmail.com> wrote:
>
>> Thanks Shubham. I shall bump up the memory a bit more.
>>
>> I was wondering how the operator memory relates to the YARN container
>> memory settings ? Or it depends on the deployment models ?
>>
>> For example , if the deployment model is thread local, the YARN container
>> needs to be ( considering above example ) configured for atleast memory of
>> 2048 * number of operators + Buffer Server Size ?
>>
>> If the deployment model were not Thread local, it would make YARN
>> container requirements for memory lower per container ?
>>
>> Regards,
>> Ananth
>>
>> On Thu, May 12, 2016 at 7:19 PM, Shubham Pathak <sh...@datatorrent.com>
>> wrote:
>>
>>> Hello Ananth,
>>>
>>> Looks like operator requires more memory.
>>> You may add this property to have more memory allocated to the container.
>>>
>>> In properties.xml , for operator O in the application you may specify
>>> the property :
>>>
>>> <property>
>>> <name>dt.operator.*O*.attr.MEMORY_MB</name>
>>> <value>2048</value>
>>> </property>
>>>
>>> Thanks,
>>> Shubham
>>>
>>> On Thu, May 12, 2016 at 1:35 PM, Ananth Gundabattula <
>>> agundabattula@gmail.com> wrote:
>>>
>>>> Hello All,
>>>>
>>>> I am seeing the following log from the web ui ocassionally when my
>>>> operators are getting killed. Is there any way I can control the memory
>>>> settings that are used to communicate with YARN when negotiating a
>>>> container ?
>>>>
>>>> How does the typical yarn settings for a container heap and max memory
>>>> relate to the Apex memory allocation model.
>>>>
>>>> The info messages I see in the web UI are as follows:
>>>>
>>>> Container [pid=14699,containerID=container_1462863487071_0015_01_000012] is running beyond physical memory limits. Current usage: 1.5 GB of 1.5 GB physical memory used; 6.1 GB of 3.1 GB virtual memory used. Killing container.
>>>> Dump of the process-tree for container_1462863487071_0015_01_000012 :
>>>> |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
>>>> |- 14817 14699 14699 14699 (java) 1584 1654 6426968064 393896 /usr/java/default/bin/java -Xmx4429185024 -Ddt.attr.APPLICATION_PATH=hdfs://dwh109.qaperf2.sac.int.threatmetrix.com:8020/user/dtadmin/datatorrent/apps/application_1462863487071_0015 -Djava.io.tmpdir=/data3/yarn/nm/usercache/root/appcache/application_1462863487071_0015/container_1462863487071_0015_01_000012/tmp -Ddt.cid=container_1462863487071_0015_01_000012 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=/data3/yarn/container-logs/application_1462863487071_0015/container_1462863487071_0015_01_000012 -Ddt.loggers.level=com.datatorrent.*:INFO,org.apache.*:INFO com.datatorrent.stram.engine.StreamingContainer
>>>> |- 14699 14697 14699 14699 (bash) 1 2 108646400 303 /bin/bash -c /usr/java/default/bin/java -Xmx4429185024 -Ddt.attr.APPLICATION_PATH=hdfs://dwh109.qaperf2.sac.int.threatmetrix.com:8020/user/dtadmin/datatorrent/apps/application_1462863487071_0015 -Djava.io.tmpdir=/data3/yarn/nm/usercache/root/appcache/application_1462863487071_0015/container_1462863487071_0015_01_000012/tmp -Ddt.cid=container_1462863487071_0015_01_000012 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=/data3/yarn/container-logs/application_1462863487071_0015/container_1462863487071_0015_01_000012 -Ddt.loggers.level=com.datatorrent.*:INFO,org.apache.*:INFO com.datatorrent.stram.engine.StreamingContainer 1>/data3/yarn/container-logs/application_1462863487071_0015/container_1462863487071_0015_01_000012/stdout 2>/data3/yarn/container-logs/application_1462863487071_0015/container_1462863487071_0015_01_000012/stderr
>>>>
>>>> Container killed on request. Exit code is 143
>>>> Container exited with a non-zero exit code 143
>>>>
>>>>
>>>> Regards,
>>>>
>>>> Ananth
>>>>
>>>>
>>>>
>>>
>>
>
Re: YARN memory settings and the Apex memory model
Posted by Thomas Weise <th...@gmail.com>.
Ananth,
Please have a look at:
http://docs.datatorrent.com/troubleshooting/#configuring-memory
Thanks,
Thomas
On Thu, May 12, 2016 at 4:00 AM, Ananth Gundabattula <
agundabattula@gmail.com> wrote:
> Thanks Shubham. I shall bump up the memory a bit more.
>
> I was wondering how the operator memory relates to the YARN container
> memory settings ? Or it depends on the deployment models ?
>
> For example , if the deployment model is thread local, the YARN container
> needs to be ( considering above example ) configured for atleast memory of
> 2048 * number of operators + Buffer Server Size ?
>
> If the deployment model were not Thread local, it would make YARN
> container requirements for memory lower per container ?
>
> Regards,
> Ananth
>
> On Thu, May 12, 2016 at 7:19 PM, Shubham Pathak <sh...@datatorrent.com>
> wrote:
>
>> Hello Ananth,
>>
>> Looks like operator requires more memory.
>> You may add this property to have more memory allocated to the container.
>>
>> In properties.xml , for operator O in the application you may specify the
>> property :
>>
>> <property>
>> <name>dt.operator.*O*.attr.MEMORY_MB</name>
>> <value>2048</value>
>> </property>
>>
>> Thanks,
>> Shubham
>>
>> On Thu, May 12, 2016 at 1:35 PM, Ananth Gundabattula <
>> agundabattula@gmail.com> wrote:
>>
>>> Hello All,
>>>
>>> I am seeing the following log from the web ui ocassionally when my
>>> operators are getting killed. Is there any way I can control the memory
>>> settings that are used to communicate with YARN when negotiating a
>>> container ?
>>>
>>> How does the typical yarn settings for a container heap and max memory
>>> relate to the Apex memory allocation model.
>>>
>>> The info messages I see in the web UI are as follows:
>>>
>>> Container [pid=14699,containerID=container_1462863487071_0015_01_000012] is running beyond physical memory limits. Current usage: 1.5 GB of 1.5 GB physical memory used; 6.1 GB of 3.1 GB virtual memory used. Killing container.
>>> Dump of the process-tree for container_1462863487071_0015_01_000012 :
>>> |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
>>> |- 14817 14699 14699 14699 (java) 1584 1654 6426968064 393896 /usr/java/default/bin/java -Xmx4429185024 -Ddt.attr.APPLICATION_PATH=hdfs://dwh109.qaperf2.sac.int.threatmetrix.com:8020/user/dtadmin/datatorrent/apps/application_1462863487071_0015 -Djava.io.tmpdir=/data3/yarn/nm/usercache/root/appcache/application_1462863487071_0015/container_1462863487071_0015_01_000012/tmp -Ddt.cid=container_1462863487071_0015_01_000012 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=/data3/yarn/container-logs/application_1462863487071_0015/container_1462863487071_0015_01_000012 -Ddt.loggers.level=com.datatorrent.*:INFO,org.apache.*:INFO com.datatorrent.stram.engine.StreamingContainer
>>> |- 14699 14697 14699 14699 (bash) 1 2 108646400 303 /bin/bash -c /usr/java/default/bin/java -Xmx4429185024 -Ddt.attr.APPLICATION_PATH=hdfs://dwh109.qaperf2.sac.int.threatmetrix.com:8020/user/dtadmin/datatorrent/apps/application_1462863487071_0015 -Djava.io.tmpdir=/data3/yarn/nm/usercache/root/appcache/application_1462863487071_0015/container_1462863487071_0015_01_000012/tmp -Ddt.cid=container_1462863487071_0015_01_000012 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=/data3/yarn/container-logs/application_1462863487071_0015/container_1462863487071_0015_01_000012 -Ddt.loggers.level=com.datatorrent.*:INFO,org.apache.*:INFO com.datatorrent.stram.engine.StreamingContainer 1>/data3/yarn/container-logs/application_1462863487071_0015/container_1462863487071_0015_01_000012/stdout 2>/data3/yarn/container-logs/application_1462863487071_0015/container_1462863487071_0015_01_000012/stderr
>>>
>>> Container killed on request. Exit code is 143
>>> Container exited with a non-zero exit code 143
>>>
>>>
>>> Regards,
>>>
>>> Ananth
>>>
>>>
>>>
>>
>
Re: YARN memory settings and the Apex memory model
Posted by Ananth Gundabattula <ag...@gmail.com>.
Thanks Shubham. I shall bump up the memory a bit more.
I was wondering how the operator memory relates to the YARN container
memory settings ? Or it depends on the deployment models ?
For example , if the deployment model is thread local, the YARN container
needs to be ( considering above example ) configured for atleast memory of
2048 * number of operators + Buffer Server Size ?
If the deployment model were not Thread local, it would make YARN container
requirements for memory lower per container ?
Regards,
Ananth
On Thu, May 12, 2016 at 7:19 PM, Shubham Pathak <sh...@datatorrent.com>
wrote:
> Hello Ananth,
>
> Looks like operator requires more memory.
> You may add this property to have more memory allocated to the container.
>
> In properties.xml , for operator O in the application you may specify the
> property :
>
> <property>
> <name>dt.operator.*O*.attr.MEMORY_MB</name>
> <value>2048</value>
> </property>
>
> Thanks,
> Shubham
>
> On Thu, May 12, 2016 at 1:35 PM, Ananth Gundabattula <
> agundabattula@gmail.com> wrote:
>
>> Hello All,
>>
>> I am seeing the following log from the web ui ocassionally when my
>> operators are getting killed. Is there any way I can control the memory
>> settings that are used to communicate with YARN when negotiating a
>> container ?
>>
>> How does the typical yarn settings for a container heap and max memory
>> relate to the Apex memory allocation model.
>>
>> The info messages I see in the web UI are as follows:
>>
>> Container [pid=14699,containerID=container_1462863487071_0015_01_000012] is running beyond physical memory limits. Current usage: 1.5 GB of 1.5 GB physical memory used; 6.1 GB of 3.1 GB virtual memory used. Killing container.
>> Dump of the process-tree for container_1462863487071_0015_01_000012 :
>> |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
>> |- 14817 14699 14699 14699 (java) 1584 1654 6426968064 393896 /usr/java/default/bin/java -Xmx4429185024 -Ddt.attr.APPLICATION_PATH=hdfs://dwh109.qaperf2.sac.int.threatmetrix.com:8020/user/dtadmin/datatorrent/apps/application_1462863487071_0015 -Djava.io.tmpdir=/data3/yarn/nm/usercache/root/appcache/application_1462863487071_0015/container_1462863487071_0015_01_000012/tmp -Ddt.cid=container_1462863487071_0015_01_000012 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=/data3/yarn/container-logs/application_1462863487071_0015/container_1462863487071_0015_01_000012 -Ddt.loggers.level=com.datatorrent.*:INFO,org.apache.*:INFO com.datatorrent.stram.engine.StreamingContainer
>> |- 14699 14697 14699 14699 (bash) 1 2 108646400 303 /bin/bash -c /usr/java/default/bin/java -Xmx4429185024 -Ddt.attr.APPLICATION_PATH=hdfs://dwh109.qaperf2.sac.int.threatmetrix.com:8020/user/dtadmin/datatorrent/apps/application_1462863487071_0015 -Djava.io.tmpdir=/data3/yarn/nm/usercache/root/appcache/application_1462863487071_0015/container_1462863487071_0015_01_000012/tmp -Ddt.cid=container_1462863487071_0015_01_000012 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=/data3/yarn/container-logs/application_1462863487071_0015/container_1462863487071_0015_01_000012 -Ddt.loggers.level=com.datatorrent.*:INFO,org.apache.*:INFO com.datatorrent.stram.engine.StreamingContainer 1>/data3/yarn/container-logs/application_1462863487071_0015/container_1462863487071_0015_01_000012/stdout 2>/data3/yarn/container-logs/application_1462863487071_0015/container_1462863487071_0015_01_000012/stderr
>>
>> Container killed on request. Exit code is 143
>> Container exited with a non-zero exit code 143
>>
>>
>> Regards,
>>
>> Ananth
>>
>>
>>
>
Re: YARN memory settings and the Apex memory model
Posted by Shubham Pathak <sh...@datatorrent.com>.
Hello Ananth,
Looks like operator requires more memory.
You may add this property to have more memory allocated to the container.
In properties.xml , for operator O in the application you may specify the
property :
<property>
<name>dt.operator.*O*.attr.MEMORY_MB</name>
<value>2048</value>
</property>
Thanks,
Shubham
On Thu, May 12, 2016 at 1:35 PM, Ananth Gundabattula <
agundabattula@gmail.com> wrote:
> Hello All,
>
> I am seeing the following log from the web ui ocassionally when my
> operators are getting killed. Is there any way I can control the memory
> settings that are used to communicate with YARN when negotiating a
> container ?
>
> How does the typical yarn settings for a container heap and max memory
> relate to the Apex memory allocation model.
>
> The info messages I see in the web UI are as follows:
>
> Container [pid=14699,containerID=container_1462863487071_0015_01_000012] is running beyond physical memory limits. Current usage: 1.5 GB of 1.5 GB physical memory used; 6.1 GB of 3.1 GB virtual memory used. Killing container.
> Dump of the process-tree for container_1462863487071_0015_01_000012 :
> |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
> |- 14817 14699 14699 14699 (java) 1584 1654 6426968064 393896 /usr/java/default/bin/java -Xmx4429185024 -Ddt.attr.APPLICATION_PATH=hdfs://dwh109.qaperf2.sac.int.threatmetrix.com:8020/user/dtadmin/datatorrent/apps/application_1462863487071_0015 -Djava.io.tmpdir=/data3/yarn/nm/usercache/root/appcache/application_1462863487071_0015/container_1462863487071_0015_01_000012/tmp -Ddt.cid=container_1462863487071_0015_01_000012 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=/data3/yarn/container-logs/application_1462863487071_0015/container_1462863487071_0015_01_000012 -Ddt.loggers.level=com.datatorrent.*:INFO,org.apache.*:INFO com.datatorrent.stram.engine.StreamingContainer
> |- 14699 14697 14699 14699 (bash) 1 2 108646400 303 /bin/bash -c /usr/java/default/bin/java -Xmx4429185024 -Ddt.attr.APPLICATION_PATH=hdfs://dwh109.qaperf2.sac.int.threatmetrix.com:8020/user/dtadmin/datatorrent/apps/application_1462863487071_0015 -Djava.io.tmpdir=/data3/yarn/nm/usercache/root/appcache/application_1462863487071_0015/container_1462863487071_0015_01_000012/tmp -Ddt.cid=container_1462863487071_0015_01_000012 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=/data3/yarn/container-logs/application_1462863487071_0015/container_1462863487071_0015_01_000012 -Ddt.loggers.level=com.datatorrent.*:INFO,org.apache.*:INFO com.datatorrent.stram.engine.StreamingContainer 1>/data3/yarn/container-logs/application_1462863487071_0015/container_1462863487071_0015_01_000012/stdout 2>/data3/yarn/container-logs/application_1462863487071_0015/container_1462863487071_0015_01_000012/stderr
>
> Container killed on request. Exit code is 143
> Container exited with a non-zero exit code 143
>
>
> Regards,
>
> Ananth
>
>
>