You are viewing a plain text version of this content. The canonical link for it is here.
Posted to log4j-user@logging.apache.org by Reza Razavipour <re...@gmail.com> on 2008/09/17 00:38:27 UTC
log4j files and parsing and presentation
All,
I am tasked with writing a log4j file aggregator/viewer. Search
criteria such as date ranges and etc...
Any support for log4j file parsing? Combining a set of log files?
Any pointers are appreciated.
Thanks
Reza
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-user-help@logging.apache.org
RE: log4j files and parsing and presentation
Posted by Michael Erskine <mi...@ketech.com>.
> I am tasked with writing a log4j file aggregator/viewer. Search
> criteria such as date ranges and etc...
>
> Any support for log4j file parsing? Combining a set of log files?
>
> Any pointers are appreciated.
Hi Reza,
Just use the best tools for the job: Perl, cat, sed, awk, and bash. Of course some tools need getting used to before becoming the most useful :) .
Regards,
Michael Erskine.
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-user-help@logging.apache.org
Re: log4j files and parsing and presentation
Posted by Paul Smith <ps...@aconex.com>.
Interesting, that's sort of what I was intending to do with my Apache
Lab project called "Pinpoint":
http://svn.apache.org/repos/asf/labs/pinpoint/
The goal of that project is to be able to become a central logging
repository. It has support to soak events via network sockets (Socket/
SocketHub Receivers) and has an importer that can use
LogFilePatternReceiver to import as well.
Pinpoint creates a Context, which is a collection of logging events.
The default context selector routes events received based on date at a
day-based context location, and indexes them using Lucene. there is a
basic command-line interface to search and I am building reporting
elements (Pie graphs etc) to show context summary.
Is that the sort of thing you're looking for? It's not complete, but
might be worth reviewing?
cheers,
Paul
On 17/09/2008, at 8:38 AM, Reza Razavipour wrote:
> All,
>
> I am tasked with writing a log4j file aggregator/viewer. Search
> criteria such as date ranges and etc...
>
> Any support for log4j file parsing? Combining a set of log files?
>
> Any pointers are appreciated.
>
> Thanks
> Reza
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
> For additional commands, e-mail: log4j-user-help@logging.apache.org
>
Paul Smith
Core Engineering Manager
Aconex
The easy way to save time and money on your project
696 Bourke Street, Melbourne,
VIC 3000, Australia
Tel: +61 3 9240 0200 Fax: +61 3 9240 0299
Email: psmith@aconex.com www.aconex.com
This email and any attachments are intended solely for the addressee.
The contents may be privileged, confidential and/or subject to
copyright or other applicable law. No confidentiality or privilege is
lost by an erroneous transmission. If you have received this e-mail in
error, please let us know by reply e-mail and delete or destroy this
mail and all copies. If you are not the intended recipient of this
message you must not disseminate, copy or take any action in reliance
on it. The sender takes no responsibility for the effect of this
message upon the recipient's computer system.
Re: log4j files and parsing and presentation
Posted by Paul Smith <ps...@aconex.com>.
On 17/09/2008, at 11:35 AM, Reza Razavipour wrote:
> Yes, this is a great tool. What I need to do is to do this
> programatically. So, the application, at any point in time, will have
> targets that have log4j log files. At user's request, which would be a
> get of all files, with some date/time constraints, the application
> will have to combine the log files, filter on date/time and present
> the log entries in "some tbd" format...
>
> What I was wondering is that:
> 1. Does log4j has anything to support such things as concatenation of
> log files, searching and filtering of log files?
> 2. If not, is there any other package that does similar things.
The pinpoint-importer layer can import a log file into an indexed
'context' area. You could then point (pardon the pun) the pinpoint-
search layer at that generated context and do programatic queries
using Lucene (date ranges, free text etc) to get a
PinpointSearchResults. You can then iterate over the results to get
the raw LoggingEvents that have matched to then export to some format
of your choosing.
I'm planning on adding a class that converts the search results->Excel
via the POI library at some point. My main focus has been trying to
design the query mechanism to be able to scan quickly for production
triage purposes.
I'm not exactly sure what your use cases are, but I would have thought
a background mechanism to soak the log files into a Pinpoint context
so that at the point the user needs to search you have all the data
there ready to go would be better.
My goal is to produce a Web application that allows an Engineer or QA
person to search logs for problems based on User reports, find their
logs, and work out what they were doing. It's quite a difficult task
when one has Gb's worth of logs to find specific information in a sea
of logs (yes there's a lot that can be done with grep/sed/awk, but a
data-mining tool is much more friendly..)
cheers,
Paul
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-user-help@logging.apache.org
Re: log4j files and parsing and presentation
Posted by Reza Razavipour <re...@gmail.com>.
Yes, this is a great tool. What I need to do is to do this
programatically. So, the application, at any point in time, will have
targets that have log4j log files. At user's request, which would be a
get of all files, with some date/time constraints, the application
will have to combine the log files, filter on date/time and present
the log entries in "some tbd" format...
What I was wondering is that:
1. Does log4j has anything to support such things as concatenation of
log files, searching and filtering of log files?
2. If not, is there any other package that does similar things.
It sounds like the ChainSaw approach is not a programmatic way to do
this. Correct me if I am wrong.
Thanks
Reza
On Tue, Sep 16, 2008 at 4:11 PM, Scott Deboy <sd...@comotivsystems.com> wrote:
> You can use Chainsaw V2 along with a LogFilePatternReceiver to view and
> aggregate log files.
>
> You can filter/search and colorize using the built-in expression syntax,
> which does support date ranges, but it's a bit verbose. Example:
>
> TIMESTAMP >= '2008/09/08 09:41:23' && TIMESTAMP <= '2008/09/08 09:41:24'
>
> To aggregate log files, I'd load all of the log files into Chainsaw and
> then use the 'View-Create custom expression LogPanel' menu item, and
> then specify an expression defining the events you want to see in the
> new view (a new tab will be created containing those aggregated events).
>
> You can find Chainsaw V2 here:
>
> http://logging.apache.org/chainsaw/
>
>
>
> Scott Deboy
> Principal Engineer
> COMOTIV SYSTEMS
> 111 SW Columbia Street Ste. 950
> Portland, OR 97201
> Office: 503.224.7496
> Direct Line: 503.821.6482
> Cell: 503.997.1367
> Fax: 503.222.0185
> sdeboy@comotivsystems.com
> www.comotivsystems.com
>
> -----Original Message-----
> From: Reza Razavipour [mailto:reza.razavipour@gmail.com]
> Sent: Tuesday, September 16, 2008 3:38 PM
> To: Log4J Users List
> Subject: log4j files and parsing and presentation
>
> All,
>
> I am tasked with writing a log4j file aggregator/viewer. Search
> criteria such as date ranges and etc...
>
> Any support for log4j file parsing? Combining a set of log files?
>
> Any pointers are appreciated.
>
> Thanks
> Reza
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
> For additional commands, e-mail: log4j-user-help@logging.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
> For additional commands, e-mail: log4j-user-help@logging.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-user-help@logging.apache.org
RE: log4j files and parsing and presentation
Posted by Scott Deboy <sd...@comotivsystems.com>.
You can use Chainsaw V2 along with a LogFilePatternReceiver to view and
aggregate log files.
You can filter/search and colorize using the built-in expression syntax,
which does support date ranges, but it's a bit verbose. Example:
TIMESTAMP >= '2008/09/08 09:41:23' && TIMESTAMP <= '2008/09/08 09:41:24'
To aggregate log files, I'd load all of the log files into Chainsaw and
then use the 'View-Create custom expression LogPanel' menu item, and
then specify an expression defining the events you want to see in the
new view (a new tab will be created containing those aggregated events).
You can find Chainsaw V2 here:
http://logging.apache.org/chainsaw/
Scott Deboy
Principal Engineer
COMOTIV SYSTEMS
111 SW Columbia Street Ste. 950
Portland, OR 97201
Office: 503.224.7496
Direct Line: 503.821.6482
Cell: 503.997.1367
Fax: 503.222.0185
sdeboy@comotivsystems.com
www.comotivsystems.com
-----Original Message-----
From: Reza Razavipour [mailto:reza.razavipour@gmail.com]
Sent: Tuesday, September 16, 2008 3:38 PM
To: Log4J Users List
Subject: log4j files and parsing and presentation
All,
I am tasked with writing a log4j file aggregator/viewer. Search
criteria such as date ranges and etc...
Any support for log4j file parsing? Combining a set of log files?
Any pointers are appreciated.
Thanks
Reza
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-user-help@logging.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-user-help@logging.apache.org