You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by Jason Smith <jh...@iriscouch.com> on 2011/12/07 02:51:37 UTC
The perfect logger for development
Hi, all. Iris Couch urgently needs improved logging facilities in
CouchDB. My goal is to make something we all love and get it accepted
upstream, God willing. Or committers willing. But I repeat myself!
This is the brainstorming and requirements gathering phase. In the
CouchDB of your dreams, logging system fits you like an old pair of
sneakers. It's perfect. Now, what characteristics does that system
exhibit? I will compile feedback into a spec on the wiki.
I hope to avoid bikeshedding. Seriously, please don't even mention a
product or project by name. At this phase I hope to stick to
descriptions of functionality, goals, and non-goals. I want to
evaluate tools later.
To start the discussion: logging is viewed differently based on your
relationship with CouchDB:
1. Developers of CouchDB
2. Administrators of a couch
3. Application developers
My roles are administration, and a little bit of development.
Requirements, in no order.
* Idiomatic Erlang
* Is a compelling place for new people to contribute. Miguel de Icaza
talks about this. It's not enough that the source code is public. You
have to provide a smooth on-ramp, where people people get great bang
for their buck. They write a modest feature, and are rewarded by
seeing its effects immediately. In other words: plugins. Or maybe a
behaviour. Or some way to swap in formatters and data sinks. I don't
want to write a Loggly target (http://loggly.com). Loggly should be
begging me to merge their module.
* 1st cut, no change to the default behavior. You still get the that
peculiar log file you know and love. People are parsing their log
files, and might expect version 1.x not to change.
* Existing code still works. No sweeping changes hitting every
?LOG_INFO and ?LOG_DEBUG.
(Filipe, would you please share your thoughts on these? I think you
struggled with the conflict between them recently.)
* No performance impact (non-blocking)...
* ... but also, impossible or difficult to overwhelm or crash or lose logs.
(The next few points sort of fit together)
* Logs are not strings, but data structures, with data (the log
message) and metadata (severity, line number, maybe the call stack,
etc.)
* More log levels. Roughly: trace, debug, info, warn, error, fatal
* Maybe automatic trace logs upon function entry/exit with return
values. Not sure if this is doable. Maybe a compile option, for `make
dev`
* When you log something, your module, line number, and maybe PID are known
* "Components" or categories, or tags, where multiple .erl files or
individual log events can share a common property ("http", "views",
* A policy decides what to do with logs based on level, module, or
component. You can set the policy either via configuration or
programatically.
* There is a formatter API to serialize log data. Built-in formatters
include the legacy format, and Jan's Apache httpd combined format.
* There is a transport API to receive logs and DTRT.
* I know this is insane, but kill -HUP <pid> should make couch reopen
its log files. Okay, I'll settle down now.
= Non Goals =
* Log rotation. I have never seen a rotation feature in an application
which was better than the OS tools. And you can get problem where both
the server and the OS are rotating the same logs. I have seen that
happen, twice. Or was it once? Of course, people could write a
rotating file transport.
--
Iris Couch
Re: The perfect logger for development
Posted by Gregor Martynus <gr...@martynus.net>.
As an app developer, I just appreciate that you discuss this topic. It's
really, really hard to me to get useful information out of the current
logs. Doesn't make it easy to "relax" ;-)
On Wed, Dec 7, 2011 at 10:52 AM, Benoit Chesneau <bc...@gmail.com>wrote:
> >
> > ok so you're speaking about log arguments?
>
> s/arguments/attributes. (if yes you should really have a look in the
> lagger readme ;)
>
> - benoît
>
Re: The perfect logger for development
Posted by Benoit Chesneau <bc...@gmail.com>.
>
> ok so you're speaking about log arguments?
s/arguments/attributes. (if yes you should really have a look in the
lagger readme ;)
- benoît
Re: The perfect logger for development
Posted by Randall Leeds <ra...@gmail.com>.
On Wed, Dec 7, 2011 at 12:52, Randall Leeds <ra...@gmail.com> wrote:
> On Wed, Dec 7, 2011 at 08:38, Benoit Chesneau <bc...@gmail.com> wrote:
>> On Wed, Dec 7, 2011 at 5:33 PM, Benoit Chesneau <bc...@gmail.com> wrote:
>>> On Wed, Dec 7, 2011 at 4:11 PM, Adam Kocoloski <ko...@apache.org> wrote:
>>>> On Dec 7, 2011, at 4:51 AM, Benoit Chesneau wrote:
>>>>
>>>>> On Wed, Dec 7, 2011 at 10:38 AM, Jason Smith <jh...@iriscouch.com> wrote:
>>>>>> Thanks, Benoit.
>>>>>>
>>>>>> I would like to move the discussion back, one hundred percent, to
>>>>>> identifying exactly what features are good and bad for CouchDB. We are
>>>>>> in no hurry. IMHO, CouchDB does not need a patch soon. It needs
>>>>>> thoughtful, deliberate planning.
>>>>>
>>>>>
>>>>> yes and no . We don't need to be 100% to start implementation. For
>>>>> example we could just replace our current logger by lagger which would
>>>>> give us automatically support for different backend, log rotations &
>>>>> co and prepare for more attributes as well.
>>>>
>>>> Hi Benoit, let's respect Jason's request when he started this thread:
>>>>
>>>>> Seriously, please don't even mention a product or project by name.
>>>
>
> I'm on board with this. Let's just whiteboard a little more.
> I pretty much agree with everything specified so far.
>
> One point I would add (I didn't see it) is that it should be possible
> to wholesale swap out the logging system with a compatible one via a
> config change.
> That's kind of in line with my thinking about couchdb in general these days.
That may actually be an unreasonable request and I'm not sure exactly
what I meant by it.
>
>>
>> Also I think we need some kind of CEP (couchdb enhancement proposal)
>> like EEP but for couchdb to discuss about new features like this. So
>> the people that propose that would have to do the job to be really
>> clear, (without any mystic) and such thing. With the versionning
>> advantage and comment on particular points.
>
> That could be interesting. Want to start a thread about it? I think
> this ties in nicely with the momentum around documentation
> improvements.
>
> -Randall
Re: The perfect logger for development
Posted by Klaus Trainer <kl...@posteo.de>.
On Wed, 2011-12-07 at 12:52 -0800, Randall Leeds wrote:
> One point I would add (I didn't see it) is that it should be possible
> to wholesale swap out the logging system with a compatible one via a
> config change.
+ 1
> That's kind of in line with my thinking about couchdb in general these
> days.
I'd just like to say that I appreciate the efforts that have been done
and are being done to get the project toward that direction.
Re: The perfect logger for development
Posted by Randall Leeds <ra...@gmail.com>.
On Wed, Dec 7, 2011 at 08:38, Benoit Chesneau <bc...@gmail.com> wrote:
> On Wed, Dec 7, 2011 at 5:33 PM, Benoit Chesneau <bc...@gmail.com> wrote:
>> On Wed, Dec 7, 2011 at 4:11 PM, Adam Kocoloski <ko...@apache.org> wrote:
>>> On Dec 7, 2011, at 4:51 AM, Benoit Chesneau wrote:
>>>
>>>> On Wed, Dec 7, 2011 at 10:38 AM, Jason Smith <jh...@iriscouch.com> wrote:
>>>>> Thanks, Benoit.
>>>>>
>>>>> I would like to move the discussion back, one hundred percent, to
>>>>> identifying exactly what features are good and bad for CouchDB. We are
>>>>> in no hurry. IMHO, CouchDB does not need a patch soon. It needs
>>>>> thoughtful, deliberate planning.
>>>>
>>>>
>>>> yes and no . We don't need to be 100% to start implementation. For
>>>> example we could just replace our current logger by lagger which would
>>>> give us automatically support for different backend, log rotations &
>>>> co and prepare for more attributes as well.
>>>
>>> Hi Benoit, let's respect Jason's request when he started this thread:
>>>
>>>> Seriously, please don't even mention a product or project by name.
>>
I'm on board with this. Let's just whiteboard a little more.
I pretty much agree with everything specified so far.
One point I would add (I didn't see it) is that it should be possible
to wholesale swap out the logging system with a compatible one via a
config change.
That's kind of in line with my thinking about couchdb in general these days.
>
> Also I think we need some kind of CEP (couchdb enhancement proposal)
> like EEP but for couchdb to discuss about new features like this. So
> the people that propose that would have to do the job to be really
> clear, (without any mystic) and such thing. With the versionning
> advantage and comment on particular points.
That could be interesting. Want to start a thread about it? I think
this ties in nicely with the momentum around documentation
improvements.
-Randall
Re: The perfect logger for development
Posted by Benoit Chesneau <bc...@gmail.com>.
On Wed, Dec 7, 2011 at 5:33 PM, Benoit Chesneau <bc...@gmail.com> wrote:
> On Wed, Dec 7, 2011 at 4:11 PM, Adam Kocoloski <ko...@apache.org> wrote:
>> On Dec 7, 2011, at 4:51 AM, Benoit Chesneau wrote:
>>
>>> On Wed, Dec 7, 2011 at 10:38 AM, Jason Smith <jh...@iriscouch.com> wrote:
>>>> Thanks, Benoit.
>>>>
>>>> I would like to move the discussion back, one hundred percent, to
>>>> identifying exactly what features are good and bad for CouchDB. We are
>>>> in no hurry. IMHO, CouchDB does not need a patch soon. It needs
>>>> thoughtful, deliberate planning.
>>>
>>>
>>> yes and no . We don't need to be 100% to start implementation. For
>>> example we could just replace our current logger by lagger which would
>>> give us automatically support for different backend, log rotations &
>>> co and prepare for more attributes as well.
>>
>> Hi Benoit, let's respect Jason's request when he started this thread:
>>
>>> Seriously, please don't even mention a product or project by name.
>
> Well this mail is mixing 2 issues. So let's separate the problems.
>
> 1. managing multi transport, improving file logging (rotation),
> allowing the logger to handle attributes . This is a technical point
> and need a technical solution
>
> 2. deciding what to log, and which format.
>
> I don't really care about the second right now. It will need some time
> to be solved. and I'm pretty sure all expectations can't be solve
> easily.
>
> While the first can be solved now. And this is the more urgent imo. We
> indeed haven't a good solution to log actually. logging to files isn't
> enough, and it's not easy to change that. Abstracting the logging in
> fact may be indeed the first thing to do before choosing any solution.
> Then people can log whatever they want from any part in the code that
> need log.
>
> - benoît
Also I think we need some kind of CEP (couchdb enhancement proposal)
like EEP but for couchdb to discuss about new features like this. So
the people that propose that would have to do the job to be really
clear, (without any mystic) and such thing. With the versionning
advantage and comment on particular points.
- benoît
Re: The perfect logger for development
Posted by Benoit Chesneau <bc...@gmail.com>.
On Wed, Dec 7, 2011 at 4:11 PM, Adam Kocoloski <ko...@apache.org> wrote:
> On Dec 7, 2011, at 4:51 AM, Benoit Chesneau wrote:
>
>> On Wed, Dec 7, 2011 at 10:38 AM, Jason Smith <jh...@iriscouch.com> wrote:
>>> Thanks, Benoit.
>>>
>>> I would like to move the discussion back, one hundred percent, to
>>> identifying exactly what features are good and bad for CouchDB. We are
>>> in no hurry. IMHO, CouchDB does not need a patch soon. It needs
>>> thoughtful, deliberate planning.
>>
>>
>> yes and no . We don't need to be 100% to start implementation. For
>> example we could just replace our current logger by lagger which would
>> give us automatically support for different backend, log rotations &
>> co and prepare for more attributes as well.
>
> Hi Benoit, let's respect Jason's request when he started this thread:
>
>> Seriously, please don't even mention a product or project by name.
Well this mail is mixing 2 issues. So let's separate the problems.
1. managing multi transport, improving file logging (rotation),
allowing the logger to handle attributes . This is a technical point
and need a technical solution
2. deciding what to log, and which format.
I don't really care about the second right now. It will need some time
to be solved. and I'm pretty sure all expectations can't be solve
easily.
While the first can be solved now. And this is the more urgent imo. We
indeed haven't a good solution to log actually. logging to files isn't
enough, and it's not easy to change that. Abstracting the logging in
fact may be indeed the first thing to do before choosing any solution.
Then people can log whatever they want from any part in the code that
need log.
- benoît
Re: The perfect logger for development
Posted by Adam Kocoloski <ko...@apache.org>.
On Dec 7, 2011, at 4:51 AM, Benoit Chesneau wrote:
> On Wed, Dec 7, 2011 at 10:38 AM, Jason Smith <jh...@iriscouch.com> wrote:
>> Thanks, Benoit.
>>
>> I would like to move the discussion back, one hundred percent, to
>> identifying exactly what features are good and bad for CouchDB. We are
>> in no hurry. IMHO, CouchDB does not need a patch soon. It needs
>> thoughtful, deliberate planning.
>
>
> yes and no . We don't need to be 100% to start implementation. For
> example we could just replace our current logger by lagger which would
> give us automatically support for different backend, log rotations &
> co and prepare for more attributes as well.
Hi Benoit, let's respect Jason's request when he started this thread:
> Seriously, please don't even mention a product or project by name.
Jason, thanks for this thread. I think you're absolutely on the right track with your current wish list. Hopefully I'll find some time to write a more thorough reply later today. Cheers,
Adam
Re: The perfect logger for development
Posted by Benoit Chesneau <bc...@gmail.com>.
On Wed, Dec 7, 2011 at 10:38 AM, Jason Smith <jh...@iriscouch.com> wrote:
> Thanks, Benoit.
>
> I would like to move the discussion back, one hundred percent, to
> identifying exactly what features are good and bad for CouchDB. We are
> in no hurry. IMHO, CouchDB does not need a patch soon. It needs
> thoughtful, deliberate planning.
yes and no . We don't need to be 100% to start implementation. For
example we could just replace our current logger by lagger which would
give us automatically support for different backend, log rotations &
co and prepare for more attributes as well.
>
> More responses inline.
>
> On Wed, Dec 7, 2011 at 3:06 PM, Benoit Chesneau <bc...@gmail.com> wrote:
>> I fully agree, we need better logging in couch. I don't really like
>> the idea of saving any data structures, but why not. Imo such things
>> could be saved as binaries strings and then parse but ...
>
> I agree. For the first milestone I propose no change to the logs: the
> same ?LOG_DEBUG() macros, the same text file; the same log format only
> a mother could love.
>
> When I say "data structures," I simply mean that more useful
> information is available to be logged.
>
ok so you're speaking about log arguments?
> A first milestone might to output to the same log file as before.
> However, I hope it would be easy to write different log formatters for
> different needs:
>
> * Web developer format: client IP, method, headers, log message
> * Erlang troubleshooting format: PID, supervision tree, heap usage, log message
well such infos should be natural since the code doesn't hangs at the
same place. you don't have the same info then.
>
> For the first milestone, though, something modest:
>
> * Traditional format: timestamp, log level, pid, log message // i.e.
> what Couch does now
>
again lagger do a lot of thing about that. Technically I really like
its possibilities. And we don't have to reinvent the wheel . I woul
dbe curious anyway how cloudant is comparing twig to it.
- benoît
Re: The perfect logger for development
Posted by Jason Smith <jh...@iriscouch.com>.
Thanks, Benoit.
I would like to move the discussion back, one hundred percent, to
identifying exactly what features are good and bad for CouchDB. We are
in no hurry. IMHO, CouchDB does not need a patch soon. It needs
thoughtful, deliberate planning.
More responses inline.
On Wed, Dec 7, 2011 at 3:06 PM, Benoit Chesneau <bc...@gmail.com> wrote:
> I fully agree, we need better logging in couch. I don't really like
> the idea of saving any data structures, but why not. Imo such things
> could be saved as binaries strings and then parse but ...
I agree. For the first milestone I propose no change to the logs: the
same ?LOG_DEBUG() macros, the same text file; the same log format only
a mother could love.
When I say "data structures," I simply mean that more useful
information is available to be logged.
A first milestone might to output to the same log file as before.
However, I hope it would be easy to write different log formatters for
different needs:
* Web developer format: client IP, method, headers, log message
* Erlang troubleshooting format: PID, supervision tree, heap usage, log message
For the first milestone, though, something modest:
* Traditional format: timestamp, log level, pid, log message // i.e.
what Couch does now
--
Iris Couch
Re: The perfect logger for development
Posted by Benoit Chesneau <bc...@gmail.com>.
On Wed, Dec 7, 2011 at 2:51 AM, Jason Smith <jh...@iriscouch.com> wrote:
> Hi, all. Iris Couch urgently needs improved logging facilities in
> CouchDB. My goal is to make something we all love and get it accepted
> upstream, God willing. Or committers willing. But I repeat myself!
>
> This is the brainstorming and requirements gathering phase. In the
> CouchDB of your dreams, logging system fits you like an old pair of
> sneakers. It's perfect. Now, what characteristics does that system
> exhibit? I will compile feedback into a spec on the wiki.
>
> I hope to avoid bikeshedding. Seriously, please don't even mention a
> product or project by name. At this phase I hope to stick to
> descriptions of functionality, goals, and non-goals. I want to
> evaluate tools later.
>
> To start the discussion: logging is viewed differently based on your
> relationship with CouchDB:
>
> 1. Developers of CouchDB
> 2. Administrators of a couch
> 3. Application developers
>
> My roles are administration, and a little bit of development.
> Requirements, in no order.
>
> * Idiomatic Erlang
>
> * Is a compelling place for new people to contribute. Miguel de Icaza
> talks about this. It's not enough that the source code is public. You
> have to provide a smooth on-ramp, where people people get great bang
> for their buck. They write a modest feature, and are rewarded by
> seeing its effects immediately. In other words: plugins. Or maybe a
> behaviour. Or some way to swap in formatters and data sinks. I don't
> want to write a Loggly target (http://loggly.com). Loggly should be
> begging me to merge their module.
>
> * 1st cut, no change to the default behavior. You still get the that
> peculiar log file you know and love. People are parsing their log
> files, and might expect version 1.x not to change.
>
> * Existing code still works. No sweeping changes hitting every
> ?LOG_INFO and ?LOG_DEBUG.
>
> (Filipe, would you please share your thoughts on these? I think you
> struggled with the conflict between them recently.)
> * No performance impact (non-blocking)...
> * ... but also, impossible or difficult to overwhelm or crash or lose logs.
>
> (The next few points sort of fit together)
>
> * Logs are not strings, but data structures, with data (the log
> message) and metadata (severity, line number, maybe the call stack,
> etc.)
>
> * More log levels. Roughly: trace, debug, info, warn, error, fatal
>
> * Maybe automatic trace logs upon function entry/exit with return
> values. Not sure if this is doable. Maybe a compile option, for `make
> dev`
>
> * When you log something, your module, line number, and maybe PID are known
>
> * "Components" or categories, or tags, where multiple .erl files or
> individual log events can share a common property ("http", "views",
>
> * A policy decides what to do with logs based on level, module, or
> component. You can set the policy either via configuration or
> programatically.
>
> * There is a formatter API to serialize log data. Built-in formatters
> include the legacy format, and Jan's Apache httpd combined format.
>
> * There is a transport API to receive logs and DTRT.
>
> * I know this is insane, but kill -HUP <pid> should make couch reopen
> its log files. Okay, I'll settle down now.
>
> = Non Goals =
>
> * Log rotation. I have never seen a rotation feature in an application
> which was better than the OS tools. And you can get problem where both
> the server and the OS are rotating the same logs. I have seen that
> happen, twice. Or was it once? Of course, people could write a
> rotating file transport.
>
> --
> Iris Couch
I fully agree, we need better logging in couch. I don't really like
the idea of saving any data structures, but why not. Imo such things
could be saved as binaries strings and then parse but ...
Technically all of these things can be covered wby lagger I think :h
https://github.com/basho/lager
It handle different backend, attribute supports, logs rotation,
tracing, loggers integation .. More on their page. We expect to use it
in refuge, and I think I can provide a patch for couch soon as well.
- benoît
Re: The perfect logger for development
Posted by Volker Mische <vo...@gmail.com>.
Don't forgot the request body when it was a PUT/DELETE.
Cheers,
Volker
On 12/07/2011 05:24 AM, Jason Smith wrote:
> Brilliant, thanks!
>
> I think this is possible if the Req object is part of the log object.
> Then a formatter can access it there. Off the top of my head, it would
> have access to the source IP address, the HTTP method, the path and
> query string, and headers.
>
> On Wed, Dec 7, 2011 at 10:44 AM, kowsik <ko...@gmail.com> wrote:
>> As a CouchDB administrator, I would want *all* exception dumps to be
>> prefaced by the inbound request URL with the query parameters
>> (assuming it's a web request that caused the exception). There are
>> case where I've seen a stack trace but couldn't tell which inbound
>> request caused the problem.
>>
>> K.
>> ---
>> http://blitz.io
>> @pcapr
>>
>> On Tue, Dec 6, 2011 at 5:51 PM, Jason Smith <jh...@iriscouch.com> wrote:
>>> Hi, all. Iris Couch urgently needs improved logging facilities in
>>> CouchDB. My goal is to make something we all love and get it accepted
>>> upstream, God willing. Or committers willing. But I repeat myself!
>>>
>>> This is the brainstorming and requirements gathering phase. In the
>>> CouchDB of your dreams, logging system fits you like an old pair of
>>> sneakers. It's perfect. Now, what characteristics does that system
>>> exhibit? I will compile feedback into a spec on the wiki.
>>>
>>> I hope to avoid bikeshedding. Seriously, please don't even mention a
>>> product or project by name. At this phase I hope to stick to
>>> descriptions of functionality, goals, and non-goals. I want to
>>> evaluate tools later.
>>>
>>> To start the discussion: logging is viewed differently based on your
>>> relationship with CouchDB:
>>>
>>> 1. Developers of CouchDB
>>> 2. Administrators of a couch
>>> 3. Application developers
>>>
>>> My roles are administration, and a little bit of development.
>>> Requirements, in no order.
>>>
>>> * Idiomatic Erlang
>>>
>>> * Is a compelling place for new people to contribute. Miguel de Icaza
>>> talks about this. It's not enough that the source code is public. You
>>> have to provide a smooth on-ramp, where people people get great bang
>>> for their buck. They write a modest feature, and are rewarded by
>>> seeing its effects immediately. In other words: plugins. Or maybe a
>>> behaviour. Or some way to swap in formatters and data sinks. I don't
>>> want to write a Loggly target (http://loggly.com). Loggly should be
>>> begging me to merge their module.
>>>
>>> * 1st cut, no change to the default behavior. You still get the that
>>> peculiar log file you know and love. People are parsing their log
>>> files, and might expect version 1.x not to change.
>>>
>>> * Existing code still works. No sweeping changes hitting every
>>> ?LOG_INFO and ?LOG_DEBUG.
>>>
>>> (Filipe, would you please share your thoughts on these? I think you
>>> struggled with the conflict between them recently.)
>>> * No performance impact (non-blocking)...
>>> * ... but also, impossible or difficult to overwhelm or crash or lose logs.
>>>
>>> (The next few points sort of fit together)
>>>
>>> * Logs are not strings, but data structures, with data (the log
>>> message) and metadata (severity, line number, maybe the call stack,
>>> etc.)
>>>
>>> * More log levels. Roughly: trace, debug, info, warn, error, fatal
>>>
>>> * Maybe automatic trace logs upon function entry/exit with return
>>> values. Not sure if this is doable. Maybe a compile option, for `make
>>> dev`
>>>
>>> * When you log something, your module, line number, and maybe PID are known
>>>
>>> * "Components" or categories, or tags, where multiple .erl files or
>>> individual log events can share a common property ("http", "views",
>>>
>>> * A policy decides what to do with logs based on level, module, or
>>> component. You can set the policy either via configuration or
>>> programatically.
>>>
>>> * There is a formatter API to serialize log data. Built-in formatters
>>> include the legacy format, and Jan's Apache httpd combined format.
>>>
>>> * There is a transport API to receive logs and DTRT.
>>>
>>> * I know this is insane, but kill -HUP <pid> should make couch reopen
>>> its log files. Okay, I'll settle down now.
>>>
>>> = Non Goals =
>>>
>>> * Log rotation. I have never seen a rotation feature in an application
>>> which was better than the OS tools. And you can get problem where both
>>> the server and the OS are rotating the same logs. I have seen that
>>> happen, twice. Or was it once? Of course, people could write a
>>> rotating file transport.
>>>
>>> --
>>> Iris Couch
>
>
>
Re: The perfect logger for development
Posted by Robert Dionne <di...@dionne-associates.com>.
One of the things we do in BigCouch is attach a unique identifier to the request also, so that we can correlate a given request with other log messages that may appear from other internal components. We call it an X-Request-ID or some such thing and users can "curl -v" and tell us what that is. It's great for debugging
On Dec 6, 2011, at 11:24 PM, Jason Smith wrote:
> Brilliant, thanks!
>
> I think this is possible if the Req object is part of the log object.
> Then a formatter can access it there. Off the top of my head, it would
> have access to the source IP address, the HTTP method, the path and
> query string, and headers.
>
> On Wed, Dec 7, 2011 at 10:44 AM, kowsik <ko...@gmail.com> wrote:
>> As a CouchDB administrator, I would want *all* exception dumps to be
>> prefaced by the inbound request URL with the query parameters
>> (assuming it's a web request that caused the exception). There are
>> case where I've seen a stack trace but couldn't tell which inbound
>> request caused the problem.
>>
>> K.
>> ---
>> http://blitz.io
>> @pcapr
>>
>> On Tue, Dec 6, 2011 at 5:51 PM, Jason Smith <jh...@iriscouch.com> wrote:
>>> Hi, all. Iris Couch urgently needs improved logging facilities in
>>> CouchDB. My goal is to make something we all love and get it accepted
>>> upstream, God willing. Or committers willing. But I repeat myself!
>>>
>>> This is the brainstorming and requirements gathering phase. In the
>>> CouchDB of your dreams, logging system fits you like an old pair of
>>> sneakers. It's perfect. Now, what characteristics does that system
>>> exhibit? I will compile feedback into a spec on the wiki.
>>>
>>> I hope to avoid bikeshedding. Seriously, please don't even mention a
>>> product or project by name. At this phase I hope to stick to
>>> descriptions of functionality, goals, and non-goals. I want to
>>> evaluate tools later.
>>>
>>> To start the discussion: logging is viewed differently based on your
>>> relationship with CouchDB:
>>>
>>> 1. Developers of CouchDB
>>> 2. Administrators of a couch
>>> 3. Application developers
>>>
>>> My roles are administration, and a little bit of development.
>>> Requirements, in no order.
>>>
>>> * Idiomatic Erlang
>>>
>>> * Is a compelling place for new people to contribute. Miguel de Icaza
>>> talks about this. It's not enough that the source code is public. You
>>> have to provide a smooth on-ramp, where people people get great bang
>>> for their buck. They write a modest feature, and are rewarded by
>>> seeing its effects immediately. In other words: plugins. Or maybe a
>>> behaviour. Or some way to swap in formatters and data sinks. I don't
>>> want to write a Loggly target (http://loggly.com). Loggly should be
>>> begging me to merge their module.
>>>
>>> * 1st cut, no change to the default behavior. You still get the that
>>> peculiar log file you know and love. People are parsing their log
>>> files, and might expect version 1.x not to change.
>>>
>>> * Existing code still works. No sweeping changes hitting every
>>> ?LOG_INFO and ?LOG_DEBUG.
>>>
>>> (Filipe, would you please share your thoughts on these? I think you
>>> struggled with the conflict between them recently.)
>>> * No performance impact (non-blocking)...
>>> * ... but also, impossible or difficult to overwhelm or crash or lose logs.
>>>
>>> (The next few points sort of fit together)
>>>
>>> * Logs are not strings, but data structures, with data (the log
>>> message) and metadata (severity, line number, maybe the call stack,
>>> etc.)
>>>
>>> * More log levels. Roughly: trace, debug, info, warn, error, fatal
>>>
>>> * Maybe automatic trace logs upon function entry/exit with return
>>> values. Not sure if this is doable. Maybe a compile option, for `make
>>> dev`
>>>
>>> * When you log something, your module, line number, and maybe PID are known
>>>
>>> * "Components" or categories, or tags, where multiple .erl files or
>>> individual log events can share a common property ("http", "views",
>>>
>>> * A policy decides what to do with logs based on level, module, or
>>> component. You can set the policy either via configuration or
>>> programatically.
>>>
>>> * There is a formatter API to serialize log data. Built-in formatters
>>> include the legacy format, and Jan's Apache httpd combined format.
>>>
>>> * There is a transport API to receive logs and DTRT.
>>>
>>> * I know this is insane, but kill -HUP <pid> should make couch reopen
>>> its log files. Okay, I'll settle down now.
>>>
>>> = Non Goals =
>>>
>>> * Log rotation. I have never seen a rotation feature in an application
>>> which was better than the OS tools. And you can get problem where both
>>> the server and the OS are rotating the same logs. I have seen that
>>> happen, twice. Or was it once? Of course, people could write a
>>> rotating file transport.
>>>
>>> --
>>> Iris Couch
>
>
>
> --
> Iris Couch
Re: The perfect logger for development
Posted by Jason Smith <jh...@iriscouch.com>.
Brilliant, thanks!
I think this is possible if the Req object is part of the log object.
Then a formatter can access it there. Off the top of my head, it would
have access to the source IP address, the HTTP method, the path and
query string, and headers.
On Wed, Dec 7, 2011 at 10:44 AM, kowsik <ko...@gmail.com> wrote:
> As a CouchDB administrator, I would want *all* exception dumps to be
> prefaced by the inbound request URL with the query parameters
> (assuming it's a web request that caused the exception). There are
> case where I've seen a stack trace but couldn't tell which inbound
> request caused the problem.
>
> K.
> ---
> http://blitz.io
> @pcapr
>
> On Tue, Dec 6, 2011 at 5:51 PM, Jason Smith <jh...@iriscouch.com> wrote:
>> Hi, all. Iris Couch urgently needs improved logging facilities in
>> CouchDB. My goal is to make something we all love and get it accepted
>> upstream, God willing. Or committers willing. But I repeat myself!
>>
>> This is the brainstorming and requirements gathering phase. In the
>> CouchDB of your dreams, logging system fits you like an old pair of
>> sneakers. It's perfect. Now, what characteristics does that system
>> exhibit? I will compile feedback into a spec on the wiki.
>>
>> I hope to avoid bikeshedding. Seriously, please don't even mention a
>> product or project by name. At this phase I hope to stick to
>> descriptions of functionality, goals, and non-goals. I want to
>> evaluate tools later.
>>
>> To start the discussion: logging is viewed differently based on your
>> relationship with CouchDB:
>>
>> 1. Developers of CouchDB
>> 2. Administrators of a couch
>> 3. Application developers
>>
>> My roles are administration, and a little bit of development.
>> Requirements, in no order.
>>
>> * Idiomatic Erlang
>>
>> * Is a compelling place for new people to contribute. Miguel de Icaza
>> talks about this. It's not enough that the source code is public. You
>> have to provide a smooth on-ramp, where people people get great bang
>> for their buck. They write a modest feature, and are rewarded by
>> seeing its effects immediately. In other words: plugins. Or maybe a
>> behaviour. Or some way to swap in formatters and data sinks. I don't
>> want to write a Loggly target (http://loggly.com). Loggly should be
>> begging me to merge their module.
>>
>> * 1st cut, no change to the default behavior. You still get the that
>> peculiar log file you know and love. People are parsing their log
>> files, and might expect version 1.x not to change.
>>
>> * Existing code still works. No sweeping changes hitting every
>> ?LOG_INFO and ?LOG_DEBUG.
>>
>> (Filipe, would you please share your thoughts on these? I think you
>> struggled with the conflict between them recently.)
>> * No performance impact (non-blocking)...
>> * ... but also, impossible or difficult to overwhelm or crash or lose logs.
>>
>> (The next few points sort of fit together)
>>
>> * Logs are not strings, but data structures, with data (the log
>> message) and metadata (severity, line number, maybe the call stack,
>> etc.)
>>
>> * More log levels. Roughly: trace, debug, info, warn, error, fatal
>>
>> * Maybe automatic trace logs upon function entry/exit with return
>> values. Not sure if this is doable. Maybe a compile option, for `make
>> dev`
>>
>> * When you log something, your module, line number, and maybe PID are known
>>
>> * "Components" or categories, or tags, where multiple .erl files or
>> individual log events can share a common property ("http", "views",
>>
>> * A policy decides what to do with logs based on level, module, or
>> component. You can set the policy either via configuration or
>> programatically.
>>
>> * There is a formatter API to serialize log data. Built-in formatters
>> include the legacy format, and Jan's Apache httpd combined format.
>>
>> * There is a transport API to receive logs and DTRT.
>>
>> * I know this is insane, but kill -HUP <pid> should make couch reopen
>> its log files. Okay, I'll settle down now.
>>
>> = Non Goals =
>>
>> * Log rotation. I have never seen a rotation feature in an application
>> which was better than the OS tools. And you can get problem where both
>> the server and the OS are rotating the same logs. I have seen that
>> happen, twice. Or was it once? Of course, people could write a
>> rotating file transport.
>>
>> --
>> Iris Couch
--
Iris Couch
Re: The perfect logger for development
Posted by kowsik <ko...@gmail.com>.
As a CouchDB administrator, I would want *all* exception dumps to be
prefaced by the inbound request URL with the query parameters
(assuming it's a web request that caused the exception). There are
case where I've seen a stack trace but couldn't tell which inbound
request caused the problem.
K.
---
http://blitz.io
@pcapr
On Tue, Dec 6, 2011 at 5:51 PM, Jason Smith <jh...@iriscouch.com> wrote:
> Hi, all. Iris Couch urgently needs improved logging facilities in
> CouchDB. My goal is to make something we all love and get it accepted
> upstream, God willing. Or committers willing. But I repeat myself!
>
> This is the brainstorming and requirements gathering phase. In the
> CouchDB of your dreams, logging system fits you like an old pair of
> sneakers. It's perfect. Now, what characteristics does that system
> exhibit? I will compile feedback into a spec on the wiki.
>
> I hope to avoid bikeshedding. Seriously, please don't even mention a
> product or project by name. At this phase I hope to stick to
> descriptions of functionality, goals, and non-goals. I want to
> evaluate tools later.
>
> To start the discussion: logging is viewed differently based on your
> relationship with CouchDB:
>
> 1. Developers of CouchDB
> 2. Administrators of a couch
> 3. Application developers
>
> My roles are administration, and a little bit of development.
> Requirements, in no order.
>
> * Idiomatic Erlang
>
> * Is a compelling place for new people to contribute. Miguel de Icaza
> talks about this. It's not enough that the source code is public. You
> have to provide a smooth on-ramp, where people people get great bang
> for their buck. They write a modest feature, and are rewarded by
> seeing its effects immediately. In other words: plugins. Or maybe a
> behaviour. Or some way to swap in formatters and data sinks. I don't
> want to write a Loggly target (http://loggly.com). Loggly should be
> begging me to merge their module.
>
> * 1st cut, no change to the default behavior. You still get the that
> peculiar log file you know and love. People are parsing their log
> files, and might expect version 1.x not to change.
>
> * Existing code still works. No sweeping changes hitting every
> ?LOG_INFO and ?LOG_DEBUG.
>
> (Filipe, would you please share your thoughts on these? I think you
> struggled with the conflict between them recently.)
> * No performance impact (non-blocking)...
> * ... but also, impossible or difficult to overwhelm or crash or lose logs.
>
> (The next few points sort of fit together)
>
> * Logs are not strings, but data structures, with data (the log
> message) and metadata (severity, line number, maybe the call stack,
> etc.)
>
> * More log levels. Roughly: trace, debug, info, warn, error, fatal
>
> * Maybe automatic trace logs upon function entry/exit with return
> values. Not sure if this is doable. Maybe a compile option, for `make
> dev`
>
> * When you log something, your module, line number, and maybe PID are known
>
> * "Components" or categories, or tags, where multiple .erl files or
> individual log events can share a common property ("http", "views",
>
> * A policy decides what to do with logs based on level, module, or
> component. You can set the policy either via configuration or
> programatically.
>
> * There is a formatter API to serialize log data. Built-in formatters
> include the legacy format, and Jan's Apache httpd combined format.
>
> * There is a transport API to receive logs and DTRT.
>
> * I know this is insane, but kill -HUP <pid> should make couch reopen
> its log files. Okay, I'll settle down now.
>
> = Non Goals =
>
> * Log rotation. I have never seen a rotation feature in an application
> which was better than the OS tools. And you can get problem where both
> the server and the OS are rotating the same logs. I have seen that
> happen, twice. Or was it once? Of course, people could write a
> rotating file transport.
>
> --
> Iris Couch