You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Saurav Lahiri <sa...@sungard.com> on 2013/12/04 15:26:37 UTC
[DISCUSS](CLOUDSTACK-3471)Provide an API to extract the log
statements of a given jobid
All,
This is to initiate discussion towards implementing an api to extract the
log messages
specific to a jobid.
As has been described in the bug descrition as of cloudstack 4.3 there is
no api that can
aggregate log messages by the job id. An api to extract logs by the jobid
would make it
easier to identify the sequence of steps that could have caused any issue
at hand. Manually
sifting through the huge log files is always an inconvenience.
To resolve this issue I am looking to implement the api which will extract
log messages by
job id. This will be available for use by any required consumers like UI,
cloudmonkey etc.
The api would in almost all cases be used to identify the sequence of steps
which have
occurred in relation to a job. In cases of failure this could potentially
point out the steps
which caused the issue. In other scenarios it could to an extent point out
how much a job has
progressed. Though progress indicator is not the purpose of the api.
Being an api, it will follow the standard api invocation procedures. It can
be invoked programatically
using the desired programming languages.
Possible inputs to the api could be filter parameters like jobid, start
timestap, end timestamp,
severity of the message. It will produce a response which will comprise of
all the lines from the
logfiles which correspond to this jobid and satisfy the specified
filtering criteria. The api would be
available for root admin, domain admin and users. The security
consideration should be that users
should only be able to query log messages for jobs that have been
triggerred by them.Since the
user is expecting an output it would be a good idea to implement it as a a
blocking/synchronous
API.
[QUESTION] Does anybody other than root admin need this API?
To filter and search the log messages instead of implementing the
filtering code in cloudstack the
bulk of the job can be done through a opensource tool like
logstash+elasticsearch or fluentd. From
preliminary investigation it appears that logstash+elasticsearch could be a
feasible option.
Inputs on any other tool which can better accomplish the task is also
welcome. A point of consideration
with logstash+elasticsearch is that if the log files grow very large then
to function optimally it could
require a dedicated hardware since these tools tend to cache large amount
of data in memory.
Currently I do not have any data to point to what large means but I shall
be running further tests during
evaluation of this combination.
Above is a brief description on what is being proposed, do let me know if
above sounds workable.
Suggestions/Objections are welcome.
Thanks
Saurav
Re: [DISCUSS](CLOUDSTACK-3471)Provide an API to extract the log
statements of a given jobid
Posted by Saurav Lahiri <sa...@sungard.com>.
Prasanna,
This is regard to the ticket 3471 that you had raised. Your suggestion was
to have a cloudstack API. Based on the discussion and the way logging
is generally handled, it was felt that a API might not be the best thing.
I have documented a process to get a logging service enabled. I have also
attached two scripts to the wiki doc which will enable setting up a template
with the various services enabled and which can be used to spawn the
guests with the services. I have tested it for CentOS templates and
Ubuntu templates. If there are any issues do let me know and I can fix
them. Do give it a try and let me know if this serves the purpose and the
defect
can be closed.
Thanks
Saurav
On Tue, Dec 17, 2013 at 2:05 PM, Saurav Lahiri <sa...@sungard.com>wrote:
> Hi Chiradeep,
> Appreciate your feedback. Sure I am ok with creating and running a seperate
> service. Just to ensure that I got you correctly, so you are saying that
> the
> "LogSearchAsAservice" would be queried when required to search for log
> messages
> related to a jobid and a seperate API is not called for. Also if the
> service requires reusing the
> systemvms, the ACS API layer would be used to start,scaleup,stop the
> systemvm
> instances. Also I understand that there are no objections around using
> the systemvms
> to host the elasticsearch instance?
>
> Do let me know If there are any differences.
>
> Thanks
> Saurav
>
>
> On Sat, Dec 14, 2013 at 5:07 AM, Chiradeep Vittal <
> Chiradeep.Vittal@citrix.com> wrote:
>
>> Hi Saurav,
>> It looks like the feature is a little orthogonal to ACS. It is more an
>> adjunct. The only reason to add an API to CloudStack is to re-use the
>> authentication layer. Otherwise, the feature does not interact with other
>> parts of ACS at all.
>>
>> If there was a way for a service to re-use the API layer, that would work?
>> So, the "LogSearchAsaService" service would be exposed on say a different
>> port, and run as a different app/service/servlet
>>
>> On 12/13/13 12:38 PM, "Saurav Lahiri" <sa...@sungard.com> wrote:
>>
>> >Thanks Dan.
>> >
>> >All,
>> >I have added the first draft of Design at the below location.
>> >
>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/API+to+extract+the+
>> >log+statements+by+jobid
>> >
>> >
>> >Thanks
>> >Saurav
>> >
>> >
>> >On Fri, Dec 13, 2013 at 3:18 PM, Daan Hoogland
>> ><da...@gmail.com>wrote:
>> >
>> >> On Fri, Dec 13, 2013 at 6:21 AM, Saurav Lahiri
>> >> <sa...@sungard.com> wrote:
>> >> > saurav.lahiri
>> >>
>> >>
>> >> is added
>> >>
>> >>
>>
>>
>>
>
Re: [DISCUSS](CLOUDSTACK-3471)Provide an API to extract the log
statements of a given jobid
Posted by Saurav Lahiri <sa...@sungard.com>.
Hi Chiradeep,
Appreciate your feedback. Sure I am ok with creating and running a seperate
service. Just to ensure that I got you correctly, so you are saying that
the
"LogSearchAsAservice" would be queried when required to search for log
messages
related to a jobid and a seperate API is not called for. Also if the
service requires reusing the
systemvms, the ACS API layer would be used to start,scaleup,stop the
systemvm
instances. Also I understand that there are no objections around using the
systemvms
to host the elasticsearch instance?
Do let me know If there are any differences.
Thanks
Saurav
On Sat, Dec 14, 2013 at 5:07 AM, Chiradeep Vittal <
Chiradeep.Vittal@citrix.com> wrote:
> Hi Saurav,
> It looks like the feature is a little orthogonal to ACS. It is more an
> adjunct. The only reason to add an API to CloudStack is to re-use the
> authentication layer. Otherwise, the feature does not interact with other
> parts of ACS at all.
>
> If there was a way for a service to re-use the API layer, that would work?
> So, the "LogSearchAsaService" service would be exposed on say a different
> port, and run as a different app/service/servlet
>
> On 12/13/13 12:38 PM, "Saurav Lahiri" <sa...@sungard.com> wrote:
>
> >Thanks Dan.
> >
> >All,
> >I have added the first draft of Design at the below location.
> >
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/API+to+extract+the+
> >log+statements+by+jobid
> >
> >
> >Thanks
> >Saurav
> >
> >
> >On Fri, Dec 13, 2013 at 3:18 PM, Daan Hoogland
> ><da...@gmail.com>wrote:
> >
> >> On Fri, Dec 13, 2013 at 6:21 AM, Saurav Lahiri
> >> <sa...@sungard.com> wrote:
> >> > saurav.lahiri
> >>
> >>
> >> is added
> >>
> >>
>
>
>
Re: [DISCUSS](CLOUDSTACK-3471)Provide an API to extract the log
statements of a given jobid
Posted by Chiradeep Vittal <Ch...@citrix.com>.
Hi Saurav,
It looks like the feature is a little orthogonal to ACS. It is more an
adjunct. The only reason to add an API to CloudStack is to re-use the
authentication layer. Otherwise, the feature does not interact with other
parts of ACS at all.
If there was a way for a service to re-use the API layer, that would work?
So, the "LogSearchAsaService" service would be exposed on say a different
port, and run as a different app/service/servlet
On 12/13/13 12:38 PM, "Saurav Lahiri" <sa...@sungard.com> wrote:
>Thanks Dan.
>
>All,
>I have added the first draft of Design at the below location.
>https://cwiki.apache.org/confluence/display/CLOUDSTACK/API+to+extract+the+
>log+statements+by+jobid
>
>
>Thanks
>Saurav
>
>
>On Fri, Dec 13, 2013 at 3:18 PM, Daan Hoogland
><da...@gmail.com>wrote:
>
>> On Fri, Dec 13, 2013 at 6:21 AM, Saurav Lahiri
>> <sa...@sungard.com> wrote:
>> > saurav.lahiri
>>
>>
>> is added
>>
>>
Re: [DISCUSS](CLOUDSTACK-3471)Provide an API to extract the log
statements of a given jobid
Posted by Saurav Lahiri <sa...@sungard.com>.
Thanks Dan.
All,
I have added the first draft of Design at the below location.
https://cwiki.apache.org/confluence/display/CLOUDSTACK/API+to+extract+the+log+statements+by+jobid
Thanks
Saurav
On Fri, Dec 13, 2013 at 3:18 PM, Daan Hoogland <da...@gmail.com>wrote:
> On Fri, Dec 13, 2013 at 6:21 AM, Saurav Lahiri
> <sa...@sungard.com> wrote:
> > saurav.lahiri
>
>
> is added
>
>
Re: [DISCUSS](CLOUDSTACK-3471)Provide an API to extract the log
statements of a given jobid
Posted by Daan Hoogland <da...@gmail.com>.
On Fri, Dec 13, 2013 at 6:21 AM, Saurav Lahiri
<sa...@sungard.com> wrote:
> saurav.lahiri
is added
Re: [DISCUSS](CLOUDSTACK-3471)Provide an API to extract the log
statements of a given jobid
Posted by Saurav Lahiri <sa...@sungard.com>.
Hi All,
I hope some of us have been able to read through the proposal. Awaiting
thoughts/suggestion/feedback. I tried to upload more details on this
proposal to the wiki but unable to add pages.
Can somebody please add permission so that i could add a design doc to
wiki. wiki User - (saurav.lahiri@sungard.com)
Thanks
Saurav
On Wed, Dec 4, 2013 at 7:56 PM, Saurav Lahiri <sa...@sungard.com>wrote:
> All,
> This is to initiate discussion towards implementing an api to extract the
> log messages
> specific to a jobid.
>
> As has been described in the bug descrition as of cloudstack 4.3 there is
> no api that can
> aggregate log messages by the job id. An api to extract logs by the jobid
> would make it
> easier to identify the sequence of steps that could have caused any issue
> at hand. Manually
> sifting through the huge log files is always an inconvenience.
>
> To resolve this issue I am looking to implement the api which will extract
> log messages by
> job id. This will be available for use by any required consumers like UI,
> cloudmonkey etc.
> The api would in almost all cases be used to identify the sequence of
> steps which have
> occurred in relation to a job. In cases of failure this could potentially
> point out the steps
> which caused the issue. In other scenarios it could to an extent point out
> how much a job has
> progressed. Though progress indicator is not the purpose of the api.
>
> Being an api, it will follow the standard api invocation procedures. It
> can be invoked programatically
> using the desired programming languages.
>
> Possible inputs to the api could be filter parameters like jobid, start
> timestap, end timestamp,
> severity of the message. It will produce a response which will comprise of
> all the lines from the
> logfiles which correspond to this jobid and satisfy the specified
> filtering criteria. The api would be
> available for root admin, domain admin and users. The security
> consideration should be that users
> should only be able to query log messages for jobs that have been
> triggerred by them.Since the
> user is expecting an output it would be a good idea to implement it as a a
> blocking/synchronous
> API.
>
> [QUESTION] Does anybody other than root admin need this API?
>
> To filter and search the log messages instead of implementing the
> filtering code in cloudstack the
> bulk of the job can be done through a opensource tool like
> logstash+elasticsearch or fluentd. From
> preliminary investigation it appears that logstash+elasticsearch could be
> a feasible option.
> Inputs on any other tool which can better accomplish the task is also
> welcome. A point of consideration
> with logstash+elasticsearch is that if the log files grow very large then
> to function optimally it could
> require a dedicated hardware since these tools tend to cache large amount
> of data in memory.
> Currently I do not have any data to point to what large means but I shall
> be running further tests during
> evaluation of this combination.
>
> Above is a brief description on what is being proposed, do let me know if
> above sounds workable.
> Suggestions/Objections are welcome.
>
>
> Thanks
> Saurav
>