You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by "Nick R. Katsipoulakis" <ni...@gmail.com> on 2015/03/19 14:07:51 UTC
Distributed logs coordination
Hello all,
I came across the following issue in my cluster and I would like to share
it with you, in case you have any proposals/solutions:
I am running my Storm (0.9.2) application in a 5 node cluster. Each bolt is
assigned randomly to one of the available workers and during execution,
each worker logs performance information. However, it seems that the clocks
of the machines I am using are skewed (some of them have minutes difference
between them). For instance, each bolt logs a number of events and also the
some custom performance metrics. However, when I try to plot all of the
bolts' metrics in one graph (let's say latency), I see that the time lines
are not identical.
How can I overcome the skewed clocks problem in order to have a single time
line of events happening in my cluster? Is there a way that I can have all
my bolts get a time-stamp from a single source? Any tutorials on using the
metrics framework that comes with Storm?
Thanks,
Nikos
--
Nikolaos Romanos Katsipoulakis,
University of Pittsburgh, PhD candidate
Re: Distributed logs coordination
Posted by "Nick R. Katsipoulakis" <ni...@gmail.com>.
Actually Grant, I just figured out that is not difficult to setup NTP on
those machines. I will go with that for now. Thank you very much
Nikos
2015-03-19 10:26 GMT-04:00 Nick R. Katsipoulakis <ni...@gmail.com>:
> Hello Grant and thanks for your reply. I am aware of NTP, but I would not
> like to go through the process of setting it up in the cluster. Is there
> any tool inside Storm that can help me overcome this issue?
>
> Thank you,
> Nick
>
> 2015-03-19 10:22 GMT-04:00 Grant Overby (groverby) <gr...@cisco.com>:
>
> NTP?
>> http://en.wikipedia.org/wiki/Network_Time_Protocol
>>
>>
>> *Grant Overby*
>> Software Engineer
>> Cisco.com <http://www.cisco.com/>
>> groverby@cisco.com
>> Mobile: *865 724 4910 <865%20724%204910>*
>>
>>
>>
>> Think before you print.
>>
>> This email may contain confidential and privileged material for the sole
>> use of the intended recipient. Any review, use, distribution or disclosure
>> by others is strictly prohibited. If you are not the intended recipient (or
>> authorized to receive for the recipient), please contact the sender by
>> reply email and delete all copies of this message.
>>
>> Please click here
>> <http://www.cisco.com/web/about/doing_business/legal/cri/index.html> for
>> Company Registration Information.
>>
>>
>>
>>
>> From: "Nick R. Katsipoulakis" <ni...@gmail.com>
>> Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
>> Date: Thursday, March 19, 2015 at 9:07 AM
>> To: "user@storm.apache.org" <us...@storm.apache.org>
>> Subject: Distributed logs coordination
>>
>> Hello all,
>>
>> I came across the following issue in my cluster and I would like to
>> share it with you, in case you have any proposals/solutions:
>>
>> I am running my Storm (0.9.2) application in a 5 node cluster. Each bolt
>> is assigned randomly to one of the available workers and during execution,
>> each worker logs performance information. However, it seems that the clocks
>> of the machines I am using are skewed (some of them have minutes difference
>> between them). For instance, each bolt logs a number of events and also the
>> some custom performance metrics. However, when I try to plot all of the
>> bolts' metrics in one graph (let's say latency), I see that the time lines
>> are not identical.
>>
>>
>> How can I overcome the skewed clocks problem in order to have a single
>> time line of events happening in my cluster? Is there a way that I can have
>> all my bolts get a time-stamp from a single source? Any tutorials on using
>> the metrics framework that comes with Storm?
>>
>> Thanks,
>> Nikos
>>
>> --
>> Nikolaos Romanos Katsipoulakis,
>> University of Pittsburgh, PhD candidate
>>
>
>
>
> --
> Nikolaos Romanos Katsipoulakis,
> University of Pittsburgh, PhD candidate
>
--
Nikolaos Romanos Katsipoulakis,
University of Pittsburgh, PhD candidate
Re: Distributed logs coordination
Posted by "Nick R. Katsipoulakis" <ni...@gmail.com>.
Hello Grant and thanks for your reply. I am aware of NTP, but I would not
like to go through the process of setting it up in the cluster. Is there
any tool inside Storm that can help me overcome this issue?
Thank you,
Nick
2015-03-19 10:22 GMT-04:00 Grant Overby (groverby) <gr...@cisco.com>:
> NTP?
> http://en.wikipedia.org/wiki/Network_Time_Protocol
>
>
> *Grant Overby*
> Software Engineer
> Cisco.com <http://www.cisco.com/>
> groverby@cisco.com
> Mobile: *865 724 4910 <865%20724%204910>*
>
>
>
> Think before you print.
>
> This email may contain confidential and privileged material for the sole
> use of the intended recipient. Any review, use, distribution or disclosure
> by others is strictly prohibited. If you are not the intended recipient (or
> authorized to receive for the recipient), please contact the sender by
> reply email and delete all copies of this message.
>
> Please click here
> <http://www.cisco.com/web/about/doing_business/legal/cri/index.html> for
> Company Registration Information.
>
>
>
>
> From: "Nick R. Katsipoulakis" <ni...@gmail.com>
> Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
> Date: Thursday, March 19, 2015 at 9:07 AM
> To: "user@storm.apache.org" <us...@storm.apache.org>
> Subject: Distributed logs coordination
>
> Hello all,
>
> I came across the following issue in my cluster and I would like to share
> it with you, in case you have any proposals/solutions:
>
> I am running my Storm (0.9.2) application in a 5 node cluster. Each bolt
> is assigned randomly to one of the available workers and during execution,
> each worker logs performance information. However, it seems that the clocks
> of the machines I am using are skewed (some of them have minutes difference
> between them). For instance, each bolt logs a number of events and also the
> some custom performance metrics. However, when I try to plot all of the
> bolts' metrics in one graph (let's say latency), I see that the time lines
> are not identical.
>
>
> How can I overcome the skewed clocks problem in order to have a single
> time line of events happening in my cluster? Is there a way that I can have
> all my bolts get a time-stamp from a single source? Any tutorials on using
> the metrics framework that comes with Storm?
>
> Thanks,
> Nikos
>
> --
> Nikolaos Romanos Katsipoulakis,
> University of Pittsburgh, PhD candidate
>
--
Nikolaos Romanos Katsipoulakis,
University of Pittsburgh, PhD candidate
Re: Distributed logs coordination
Posted by "Grant Overby (groverby)" <gr...@cisco.com>.
NTP?
http://en.wikipedia.org/wiki/Network_Time_Protocol
[http://www.cisco.com/web/europe/images/email/signature/est2014/logo_06.png?ct=1398192119726]
Grant Overby
Software Engineer
Cisco.com<http://www.cisco.com/>
groverby@cisco.com<ma...@cisco.com>
Mobile: 865 724 4910
[http://www.cisco.com/assets/swa/img/thinkbeforeyouprint.gif] Think before you print.
This email may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply email and delete all copies of this message.
Please click here<http://www.cisco.com/web/about/doing_business/legal/cri/index.html> for Company Registration Information.
From: "Nick R. Katsipoulakis" <ni...@gmail.com>>
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Date: Thursday, March 19, 2015 at 9:07 AM
To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Subject: Distributed logs coordination
Hello all,
I came across the following issue in my cluster and I would like to share it with you, in case you have any proposals/solutions:
I am running my Storm (0.9.2) application in a 5 node cluster. Each bolt is assigned randomly to one of the available workers and during execution, each worker logs performance information. However, it seems that the clocks of the machines I am using are skewed (some of them have minutes difference between them). For instance, each bolt logs a number of events and also the some custom performance metrics. However, when I try to plot all of the bolts' metrics in one graph (let's say latency), I see that the time lines are not identical.
How can I overcome the skewed clocks problem in order to have a single time line of events happening in my cluster? Is there a way that I can have all my bolts get a time-stamp from a single source? Any tutorials on using the metrics framework that comes with Storm?
Thanks,
Nikos
--
Nikolaos Romanos Katsipoulakis,
University of Pittsburgh, PhD candidate