You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@apex.apache.org by Ambarish Pande <am...@gmail.com> on 2017/02/17 07:11:48 UTC
Debugging operator latency.
Hello,
I wanted to know why my operator latency is increasing. Which logs should I
check to get any idea about that. I have checked the container dt.log ,
stderr and stdout, but I cannot find anything there.
Thank You
Re: Debugging operator latency.
Posted by Vlad Rozov <v....@datatorrent.com>.
I would start with taking few stack traces (the latest Datatorrent RTS
allows to take a stack trace otherwise use jstack) of the container
where the operator is deployed and trying to identify what operator's
thread is doing. If there is nothing obvious there, proceed as Ashwin
suggested.
Thank you,
Vlad
/Join us at Apex Big Data World-San Jose
<http://www.apexbigdata.com/san-jose.html>, April 4, 2017/
http://www.apexbigdata.com/san-jose-register.html
<http://www.apexbigdata.com/san-jose-register.html>
On 2/16/17 23:50, Ashwin Chandra Putta wrote:
> It is probably not able to keep up with the throughput and might need
> more partitions. Check the CPU and memory utilization of its
> container. If memory allocated to container is too low, it might be
> hitting GC too often. You can enable GC logging and check GC logs.
>
> Regards,
> Ashwin.
>
>
> On Feb 16, 2017 11:12 PM, "Ambarish Pande"
> <ambarish.pande2011@gmail.com <ma...@gmail.com>>
> wrote:
>
> Hello,
>
> I wanted to know why my operator latency is increasing. Which logs
> should I check to get any idea about that. I have checked the
> container dt.log , stderr and stdout, but I cannot find anything
> there.
>
> Thank You
>
>
Re: Debugging operator latency.
Posted by Vlad Rozov <v....@datatorrent.com>.
The name of the property is dt.attr. CONTAINER_JVM_OPTIONS. You don't
need to replace <LOG_DIR> with the actual path as it is not known till
an application and a containers are allocated by Yarn (Apex will replace
<LOG_DIR> with the container log directory). You may also set the
attribute at the launch time:
-Ddt.attr.CONTAINER_JVM_OPTIONS="-Xloggc:<LOG_DIR>/gc.log -verbose:gc
-XX:+PrintGCDateStamps" or using DT RTS console.
Thank you,
Vlad
/Join us at Apex Big Data World-San Jose
<http://www.apexbigdata.com/san-jose.html>, April 4, 2017/
http://www.apexbigdata.com/san-jose-register.html
<http://www.apexbigdata.com/san-jose-register.html>
On 2/21/17 00:44, Ambarish Pande wrote:
> I cant find the gc.log file in container logs folder. Do i have to
> replace the LOG_DIR with a path?
>
> On Tue, Feb 21, 2017 at 12:21 PM, Tushar Gosavi
> <tushar@datatorrent.com <ma...@datatorrent.com>> wrote:
>
> Hi Ambarish,
>
> you could add following property in your applications
> properties.xml file.
>
> <property>
> <name>dt.application.*.attr.containerJvmOpts</name>
> <value>-Xloggc:<LOG_DIR>/gc.log -verbose:gc
> -XX:+PrintGCDateStamps</value>
> </property>
>
> when this property is used, you will see gc.log file in container
> directory.
>
> - Tushar.
>
>
> On Tue, Feb 21, 2017 at 12:06 PM, Ambarish Pande
> <ambarish.pande2011@gmail.com
> <ma...@gmail.com>> wrote:
> > Hello,
> >
> > I tried enabling gclogs in hadoop configurations. Do I need to
> enable it in
> > Datatorrent RTS somewhere or in my app?. If so, how should I do it?
> >
> > Thank You
> >
> > On Fri, Feb 17, 2017 at 1:20 PM, Ashwin Chandra Putta
> > <ashwinchandrap@gmail.com <ma...@gmail.com>> wrote:
> >>
> >> It is probably not able to keep up with the throughput and
> might need more
> >> partitions. Check the CPU and memory utilization of its
> container. If memory
> >> allocated to container is too low, it might be hitting GC too
> often. You can
> >> enable GC logging and check GC logs.
> >>
> >> Regards,
> >> Ashwin.
> >>
> >>
> >> On Feb 16, 2017 11:12 PM, "Ambarish Pande"
> <ambarish.pande2011@gmail.com <ma...@gmail.com>>
> >> wrote:
> >>
> >> Hello,
> >>
> >> I wanted to know why my operator latency is increasing. Which
> logs should
> >> I check to get any idea about that. I have checked the
> container dt.log ,
> >> stderr and stdout, but I cannot find anything there.
> >>
> >> Thank You
> >>
> >>
> >
>
>
Re: Debugging operator latency.
Posted by Ambarish Pande <am...@gmail.com>.
I cant find the gc.log file in container logs folder. Do i have to replace
the LOG_DIR with a path?
On Tue, Feb 21, 2017 at 12:21 PM, Tushar Gosavi <tu...@datatorrent.com>
wrote:
> Hi Ambarish,
>
> you could add following property in your applications properties.xml file.
>
> <property>
> <name>dt.application.*.attr.containerJvmOpts</name>
> <value>-Xloggc:<LOG_DIR>/gc.log -verbose:gc
> -XX:+PrintGCDateStamps</value>
> </property>
>
> when this property is used, you will see gc.log file in container
> directory.
>
> - Tushar.
>
>
> On Tue, Feb 21, 2017 at 12:06 PM, Ambarish Pande
> <am...@gmail.com> wrote:
> > Hello,
> >
> > I tried enabling gclogs in hadoop configurations. Do I need to enable it
> in
> > Datatorrent RTS somewhere or in my app?. If so, how should I do it?
> >
> > Thank You
> >
> > On Fri, Feb 17, 2017 at 1:20 PM, Ashwin Chandra Putta
> > <as...@gmail.com> wrote:
> >>
> >> It is probably not able to keep up with the throughput and might need
> more
> >> partitions. Check the CPU and memory utilization of its container. If
> memory
> >> allocated to container is too low, it might be hitting GC too often.
> You can
> >> enable GC logging and check GC logs.
> >>
> >> Regards,
> >> Ashwin.
> >>
> >>
> >> On Feb 16, 2017 11:12 PM, "Ambarish Pande" <
> ambarish.pande2011@gmail.com>
> >> wrote:
> >>
> >> Hello,
> >>
> >> I wanted to know why my operator latency is increasing. Which logs
> should
> >> I check to get any idea about that. I have checked the container dt.log
> ,
> >> stderr and stdout, but I cannot find anything there.
> >>
> >> Thank You
> >>
> >>
> >
>
Re: Debugging operator latency.
Posted by Tushar Gosavi <tu...@datatorrent.com>.
Hi Ambarish,
- Benchmark the operator separately, you can write a test which calls
operator lifecycle methods in a loop and find out time taken to process fix
number of records. You could use this information to find out the actual
time required to sustain your input rate.
operator = new Operator();
operator.setup();
operator.beingWindow(1);
long start = System.currentTimeMillis();
for (int i = 0; i < count; i++) {
operator.port.process(inputport)
}
operator.endWindow()
end = System.currentTimeMillis();
Log.info("time taken to process {} items is {} millis ", count, (end -
start))
This test does not consider time taken for serialization and
de-serialization of tuples. you need to separately account for that.
- If your operator communicates with external systems, then check if there
are any latency introduced by external systems.
- GC pauses also contributes towards latency of the operator. monitor your
operator gc stats by enabling gc logging. You could add following property
in application's properties.xml file to enable gc logging.
xml <property> <name>dt.application.*.attr.containerJvmOpts</name>
<value>-Xloggc:<LOG_DIR>/gc.log -verbose:gc
-XX:+PrintGCDateStamps</value> </property>
- If the system on which operator runs is slow because many other processes
are waiting for CPU resources. This normally happens if more CPU heavy
operators are deployed on single node / container. Or other hadoop
applications are running such as hive query. you could use system utilities
such as top to monitor system cpu usage. or could you RTS ui to check all
opeartors deployed on a node and CPU usage for each.
- Tushar.
On Tue, Feb 21, 2017 at 12:21 PM, Tushar Gosavi <tu...@datatorrent.com>
wrote:
> Hi Ambarish,
>
> you could add following property in your applications properties.xml file.
>
> <property>
> <name>dt.application.*.attr.containerJvmOpts</name>
> <value>-Xloggc:<LOG_DIR>/gc.log -verbose:gc
> -XX:+PrintGCDateStamps</value>
> </property>
>
> when this property is used, you will see gc.log file in container
> directory.
>
> - Tushar.
>
>
> On Tue, Feb 21, 2017 at 12:06 PM, Ambarish Pande
> <am...@gmail.com> wrote:
> > Hello,
> >
> > I tried enabling gclogs in hadoop configurations. Do I need to enable it
> in
> > Datatorrent RTS somewhere or in my app?. If so, how should I do it?
> >
> > Thank You
> >
> > On Fri, Feb 17, 2017 at 1:20 PM, Ashwin Chandra Putta
> > <as...@gmail.com> wrote:
> >>
> >> It is probably not able to keep up with the throughput and might need
> more
> >> partitions. Check the CPU and memory utilization of its container. If
> memory
> >> allocated to container is too low, it might be hitting GC too often.
> You can
> >> enable GC logging and check GC logs.
> >>
> >> Regards,
> >> Ashwin.
> >>
> >>
> >> On Feb 16, 2017 11:12 PM, "Ambarish Pande" <
> ambarish.pande2011@gmail.com>
> >> wrote:
> >>
> >> Hello,
> >>
> >> I wanted to know why my operator latency is increasing. Which logs
> should
> >> I check to get any idea about that. I have checked the container dt.log
> ,
> >> stderr and stdout, but I cannot find anything there.
> >>
> >> Thank You
> >>
> >>
> >
>
Re: Debugging operator latency.
Posted by Tushar Gosavi <tu...@datatorrent.com>.
Hi Ambarish,
you could add following property in your applications properties.xml file.
<property>
<name>dt.application.*.attr.containerJvmOpts</name>
<value>-Xloggc:<LOG_DIR>/gc.log -verbose:gc
-XX:+PrintGCDateStamps</value>
</property>
when this property is used, you will see gc.log file in container directory.
- Tushar.
On Tue, Feb 21, 2017 at 12:06 PM, Ambarish Pande
<am...@gmail.com> wrote:
> Hello,
>
> I tried enabling gclogs in hadoop configurations. Do I need to enable it in
> Datatorrent RTS somewhere or in my app?. If so, how should I do it?
>
> Thank You
>
> On Fri, Feb 17, 2017 at 1:20 PM, Ashwin Chandra Putta
> <as...@gmail.com> wrote:
>>
>> It is probably not able to keep up with the throughput and might need more
>> partitions. Check the CPU and memory utilization of its container. If memory
>> allocated to container is too low, it might be hitting GC too often. You can
>> enable GC logging and check GC logs.
>>
>> Regards,
>> Ashwin.
>>
>>
>> On Feb 16, 2017 11:12 PM, "Ambarish Pande" <am...@gmail.com>
>> wrote:
>>
>> Hello,
>>
>> I wanted to know why my operator latency is increasing. Which logs should
>> I check to get any idea about that. I have checked the container dt.log ,
>> stderr and stdout, but I cannot find anything there.
>>
>> Thank You
>>
>>
>
Re: Debugging operator latency.
Posted by Ambarish Pande <am...@gmail.com>.
Hello,
I tried enabling gclogs in hadoop configurations. Do I need to enable it in
Datatorrent RTS somewhere or in my app?. If so, how should I do it?
Thank You
On Fri, Feb 17, 2017 at 1:20 PM, Ashwin Chandra Putta <
ashwinchandrap@gmail.com> wrote:
> It is probably not able to keep up with the throughput and might need more
> partitions. Check the CPU and memory utilization of its container. If
> memory allocated to container is too low, it might be hitting GC too often.
> You can enable GC logging and check GC logs.
>
> Regards,
> Ashwin.
>
>
> On Feb 16, 2017 11:12 PM, "Ambarish Pande" <am...@gmail.com>
> wrote:
>
> Hello,
>
> I wanted to know why my operator latency is increasing. Which logs should
> I check to get any idea about that. I have checked the container dt.log ,
> stderr and stdout, but I cannot find anything there.
>
> Thank You
>
>
>
Re: Debugging operator latency.
Posted by Ashwin Chandra Putta <as...@gmail.com>.
It is probably not able to keep up with the throughput and might need more
partitions. Check the CPU and memory utilization of its container. If
memory allocated to container is too low, it might be hitting GC too often.
You can enable GC logging and check GC logs.
Regards,
Ashwin.
On Feb 16, 2017 11:12 PM, "Ambarish Pande" <am...@gmail.com>
wrote:
Hello,
I wanted to know why my operator latency is increasing. Which logs should I
check to get any idea about that. I have checked the container dt.log ,
stderr and stdout, but I cannot find anything there.
Thank You