You are viewing a plain text version of this content. The canonical link for it is here.
Posted to log4j-dev@logging.apache.org by Paul Smith <ps...@aconex.com> on 2007/06/19 06:26:57 UTC
pinpoint - a proposed companion to log4j
Hi,
I have been in the background working on a companion to log4j that
I'm initially calling 'pinpoint'. It is designed as a log context
tool to facilitate production queries.
Logging events to a DB for querying is fine, except SQL queries,
IHMO, on that are time consuming and unsatisfying, and miss some
important features such as full text indexing. I find for my own use
I tend to rely on grep/fgrep to analyse logs files because the regexp
is handy, but that often trims off important stack traces. Usually
you have to dice the file several times to work with it. It seemed
to make sense to index these log files and be able to analyse them
using Lucene.
pinpoint is designed to create a Context which is a container of
indexed LoggingEvents. The Context contains an Lucene index, and a
serialized version of the events. searches against the index then
return hits that contain offset+length pointers to the serialization
file. That way the full LoggingEvent can be brought back for display
in the results, formatted anyway you'd like.
Using a CLI client, one can open that index and do arbitrary queries.
An example is worth a thousand words:
$ sh pinpoint-cli/target/appassembler/bin/pinpoint -h /temp/pinpoint-
home/ -c testlog -s
pinpoint>search level:INFO factory -english
+message:factory -message:english
[2007-06-08 12:13:00,122 INFO ][eNotificationsAction][Maintenance
Tasks Scheduler_Worker-9] Executing
OverdueCorrespondenceNotificationsAction: exclusionDate=2006-06-08,
maxEmailSendRate=240, maxNumEmails=2147483647, sendEmailEnabled=TRUE
[2007-06-08 12:13:00,553 ERROR][urityInfoRetrieveCmd][Maintenance
Tasks Scheduler_Worker-9][][] Could not find class for command
factory. Class com.aconex.security.command.SecuredAssetListCmd
.......
pinpoint>
Only the basics are there right now, with a simple importer utility
that uses LogFilePatternReceiver to soak events from text-based log
files, but obviously eventually any Receiver might be usable to soak
events into a context. The CLI module has handy command-line
completion support, but it could definitely do with some extensions.
Pinpoint is made up of:
* pinpoint-core
* pinpoint-importer
* pinpoint-search
* pinpoint-cli
and lives in a maven multi-project module setup. It relies on the
log4j companion 1.0-SNAPSHOTs and log4j 1.2.15-SNAPSHOT.
I'd like to eventually add a pinpoint-service module allowing
standard socket/JMS-based receivers to be configured out of the box
to allow easy connection to it as a remote service. Right now
indexing the logs is likely to be slower than the rate of incoming
events, so we'd need to build in some asynch behaviour to allow
pinpoint to buffer some logs (say via an internal JMS implementation
using ActiveMQ) as it processes them. This allows the producing
application to not be slowed down by the indexing.
I'd also like to create pinpoint-webapp to create a nice and easy
deployable webapp than can browse the pinpoint contexts and easily
search logs based on MDC variables etc. (Imagine browsing all logs
generated by a User when using, say, MDC), and once found an
interesting logging event quickly see more context lines from around
that time.
I'd be happy to have this added as a log4j module if other people are
interested. Otherwise I'm just as happy to tinker with it and use it
for my own use. Perhaps I'm the only one that wants to use logging
this way?
cheers,
Paul
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-dev-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-dev-help@logging.apache.org
Re: pinpoint - a proposed companion to log4j
Posted by Paul Smith <ps...@aconex.com>.
>
> If this was a project proposal, I'd have no issue with you starting
> a sandbox project and allow it to develop out in the open.
> However, based on your discussion it appears to be pretty far along
> and it looks like it should be at least reviewed by the Incubator
> PMC before going into Logging Services. I'd suggest putting it in
> Apache Labs. If it gains traction in the Labs, then we can migrate
> it over to LS with appropriate Incubator PMC oversight.
>
"Far along" is probably a stretch... :) the basic project structure
is there, and the ability to perform extremely basic operations as
proof of concept. It's no where near ready for production usage as yet.
What I could is put a tar ball of the project available for review,
and then people can comment on what we should do with it:
http://people.apache.org/~psmith/pinpoint.1.0.0.5.tar.gz
I'll check up on the Apache Labs process in the mean time just so I
know what might be coming.
cheers,
Paul
Re: pinpoint - a proposed companion to log4j
Posted by Curt Arnold <ca...@apache.org>.
On Jun 18, 2007, at 11:26 PM, Paul Smith wrote:
> Hi,
>
> I have been in the background working on a companion to log4j that
> I'm initially calling 'pinpoint'. It is designed as a log context
> tool to facilitate production queries.
>
> Logging events to a DB for querying is fine, except SQL queries,
> IHMO, on that are time consuming and unsatisfying, and miss some
> important features such as full text indexing. I find for my own
> use I tend to rely on grep/fgrep to analyse logs files because the
> regexp is handy, but that often trims off important stack traces.
> Usually you have to dice the file several times to work with it.
> It seemed to make sense to index these log files and be able to
> analyse them using Lucene.
>
> pinpoint is designed to create a Context which is a container of
> indexed LoggingEvents. The Context contains an Lucene index, and
> a serialized version of the events. searches against the index
> then return hits that contain offset+length pointers to the
> serialization file. That way the full LoggingEvent can be brought
> back for display in the results, formatted anyway you'd like.
>
> Using a CLI client, one can open that index and do arbitrary queries.
>
> An example is worth a thousand words:
>
> $ sh pinpoint-cli/target/appassembler/bin/pinpoint -h /temp/
> pinpoint-home/ -c testlog -s
> pinpoint>search level:INFO factory -english
> +message:factory -message:english
> [2007-06-08 12:13:00,122 INFO ][eNotificationsAction][Maintenance
> Tasks Scheduler_Worker-9] Executing
> OverdueCorrespondenceNotificationsAction: exclusionDate=2006-06-08,
> maxEmailSendRate=240, maxNumEmails=2147483647, sendEmailEnabled=TRUE
> [2007-06-08 12:13:00,553 ERROR][urityInfoRetrieveCmd][Maintenance
> Tasks Scheduler_Worker-9][][] Could not find class for command
> factory. Class com.aconex.security.command.SecuredAssetListCmd
> .......
>
> pinpoint>
>
> Only the basics are there right now, with a simple importer utility
> that uses LogFilePatternReceiver to soak events from text-based log
> files, but obviously eventually any Receiver might be usable to
> soak events into a context. The CLI module has handy command-
> line completion support, but it could definitely do with some
> extensions.
>
> Pinpoint is made up of:
>
> * pinpoint-core
> * pinpoint-importer
> * pinpoint-search
> * pinpoint-cli
>
> and lives in a maven multi-project module setup. It relies on the
> log4j companion 1.0-SNAPSHOTs and log4j 1.2.15-SNAPSHOT.
>
> I'd like to eventually add a pinpoint-service module allowing
> standard socket/JMS-based receivers to be configured out of the box
> to allow easy connection to it as a remote service. Right now
> indexing the logs is likely to be slower than the rate of incoming
> events, so we'd need to build in some asynch behaviour to allow
> pinpoint to buffer some logs (say via an internal JMS
> implementation using ActiveMQ) as it processes them. This allows
> the producing application to not be slowed down by the indexing.
>
> I'd also like to create pinpoint-webapp to create a nice and easy
> deployable webapp than can browse the pinpoint contexts and easily
> search logs based on MDC variables etc. (Imagine browsing all logs
> generated by a User when using, say, MDC), and once found an
> interesting logging event quickly see more context lines from
> around that time.
>
> I'd be happy to have this added as a log4j module if other people
> are interested. Otherwise I'm just as happy to tinker with it and
> use it for my own use. Perhaps I'm the only one that wants to use
> logging this way?
>
> cheers,
>
> Paul
>
If this was a project proposal, I'd have no issue with you starting a
sandbox project and allow it to develop out in the open. However,
based on your discussion it appears to be pretty far along and it
looks like it should be at least reviewed by the Incubator PMC before
going into Logging Services. I'd suggest putting it in Apache Labs.
If it gains traction in the Labs, then we can migrate it over to LS
with appropriate Incubator PMC oversight.
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-dev-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-dev-help@logging.apache.org