You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-dev@db.apache.org by Danesh <da...@gmail.com> on 2005/08/20 00:05:00 UTC
Re: regarding derby.log being overwritten on startup. [was Re: [jira] Commented: (DERBY-515) Network Server should log server start and shutdown time to derby.log]
I agree with going the delimited format route. Having the database
write out XML would probably impact preformance quite a bit. I also
wonder what would happen if the database were to die (power failure or
the like) and the XML file ended up invalid due to it missing those
crucial few closing tags. To simplify things and satisfy both parties
someone could write a log conversion tool i.e. DEL -> XML.
~Danesh
On 8/19/05, David Van Couvering <Da...@sun.com> wrote:
> Please, God, no :). Machines like reading XML but people (at least
> certain people) find it a bit horrific.
>
> What I have seen done in the past is using a simple delimiter format, e.g.
>
> |Fri Aug 19 14:06:31 PST
> 2005|org.apache.derby.engine.impl.jdbc.EmbedConnection|Thread
> 275|Invalid user, please try again| | |
>
> then you can write tools that slurp this right into Excel, load into a
> database, etc. We did this with an app I worked on at Sybase and it was
> quite successful.
>
> David
>
> Rick Hillegas wrote:
>
> > At the risk of overloading this topic: While we're talking about
> > improvements to our error logging, I'd like to see us emit more
> > structured logs. This is one of the things that xml is actually good
> > for. It would be pretty easy to turn our log records into xml entries.
> > This would make it easy to write or use off-the-shelf tools for
> > viewing the logs and for filtering signal out of the noise.
> >
> > Cheers,
> > -Rick
> >
> > David Van Couvering wrote:
> >
> >> Well, the most common solution is log rotation. On restart, you
> >> rename the old log to derby.log.1 and then derby.log.2 and so on.
> >> After <n> log files, you start over again at derby.log.1.
> >> This is also useful for long-running systems so you don't run out of
> >> disk space because of your error logs. When the error log gets a
> >> configurable X MBs, the system copies the log to derby.log.1,
> >> derby.log.2, etc., and starts a fresh log. After N log files it
> >> starts back at 1 and overwrites the old log file. Similar but
> >> different from above, but both are great to reduce administrative
> >> overhead for the user.
> >>
> >> David
> >>
> >> Sunitha Kambhampati wrote:
> >>
> >>> Øystein Grøvlen wrote:
> >>>
> >>>>>>>>> "KM" == Kathey Marsden <km...@sbcglobal.net> writes:
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>
> >>>> KM> Øystein Grøvlen (JIRA) wrote:
> >>>> >> [ >> This would be even more helpful if the derby.log file
> >>>> was not overwritten on the next startup. (Probably a separate issue).
> >>>> >> >> >> KM> The derby.infolog.append property is
> >>>> helpful in this regard.
> >>>>
> >>>> KM>
> >>>> http://db.apache.org/derby/docs/10.1/tuning/rtunproper13217.html
> >>>>
> >>>> Thanks, Kathey, that is really what I need.
> >>>>
> >>>>
> >>>>
> >>> One concern I have is that by default, the derby.log gets
> >>> overwritten on subsequent boot. From my experience with customers,
> >>> whenever there is an issue, I always end up telling them to add this
> >>> property almost every time to see the debug logs. It is very common
> >>> to see users reboot the system if they hit an error and in this
> >>> case, it is probable that important relevant information in the
> >>> derby.log is lost because it gets overwritten ( ex - recovery issues
> >>> , deadlock etc ).
> >>>
> >>> Maybe we could do something smart for derby.log , a balance between
> >>> keeping history across boots and disk space taken by the derby.logs.
> >>>
> >>> Any comments ?
> >>>
> >>> Sunitha.
> >>
> >>
> >>
> >
>
>
>
Re: regarding derby.log being overwritten on startup. [was Re: [jira] Commented: (DERBY-515) Network Server should log server start and shutdown time to derby.log]
Posted by Francois Orsini <fr...@gmail.com>.
I worked on some Output LOG Mgr years ago - it had support for various
types of formatted-parameterizable log messages as well as being able
to redirect certain mesage types (i.e. INFO, SEVERE, FATAL, DEBUG,
TRACE, etc) to additional types of storage (FS, Table, etc) - One
feature users like besides parameterizable log msg formatting is being
able to redirect "severe" classified types of messages down to an
_additional_ storage (dedicated file or log table) which they could
monitor - we did not let the user manipulate XML directly, instead we
used basic java properties to configure the log format and other
settings...However, we had the capability of Log.toXML (generating XML
output) down to an additional storage/feed....
--francois
On 8/22/05, Mike Matrigali <mi...@sbcglobal.net> wrote:
> I believe that the current log file format has been written to
> be easy to parse - but not as easy as looking for a single
> delimiter. Remember that the log file can contain arbitrary
> text - so not sure if a single delimiter will work. The worst
> case is log statement text which will print out values of fields,
> which can contain anything and are not under our control.
> At one point there was a tool that could take
> the log file and present it like a read only table, similar to
> the lock table. Adding some delimiters might make it easier.
>
> I also like having the log file at least sad somewhat user readable.
>
> Having said that I believe that if someone really wanted the XML
> version, an alternate error logging module implementation could
> be provided which could either produce XML instead of text or
> could provide 2 different files. If someone wanted to write a
> cool tool that parsed and presented the error log file, or needed
> xml to adapt it to an existing tool then I could see XML form
> being more important.
>
> There was hope to be able to be able to do thing like take the out
> of log statement text and then turn around and use that to drive
> a testing framework which could execute those statments.
>
> Danesh wrote:
> > I agree with going the delimited format route. Having the database
> > write out XML would probably impact preformance quite a bit. I also
> > wonder what would happen if the database were to die (power failure or
> > the like) and the XML file ended up invalid due to it missing those
> > crucial few closing tags. To simplify things and satisfy both parties
> > someone could write a log conversion tool i.e. DEL -> XML.
> >
> > ~Danesh
> >
> > On 8/19/05, David Van Couvering <Da...@sun.com> wrote:
> >
> >>Please, God, no :). Machines like reading XML but people (at least
> >>certain people) find it a bit horrific.
> >>
> >>What I have seen done in the past is using a simple delimiter format, e.g.
> >>
> >>|Fri Aug 19 14:06:31 PST
> >>2005|org.apache.derby.engine.impl.jdbc.EmbedConnection|Thread
> >>275|Invalid user, please try again| | |
> >>
> >>then you can write tools that slurp this right into Excel, load into a
> >>database, etc. We did this with an app I worked on at Sybase and it was
> >>quite successful.
> >>
> >>David
> >>
> >>Rick Hillegas wrote:
> >>
> >>
> >>>At the risk of overloading this topic: While we're talking about
> >>>improvements to our error logging, I'd like to see us emit more
> >>>structured logs. This is one of the things that xml is actually good
> >>>for. It would be pretty easy to turn our log records into xml entries.
> >>>This would make it easy to write or use off-the-shelf tools for
> >>>viewing the logs and for filtering signal out of the noise.
> >>>
> >>>Cheers,
> >>>-Rick
> >>>
> >>>David Van Couvering wrote:
> >>>
> >>>
> >>>>Well, the most common solution is log rotation. On restart, you
> >>>>rename the old log to derby.log.1 and then derby.log.2 and so on.
> >>>>After <n> log files, you start over again at derby.log.1.
> >>>>This is also useful for long-running systems so you don't run out of
> >>>>disk space because of your error logs. When the error log gets a
> >>>>configurable X MBs, the system copies the log to derby.log.1,
> >>>>derby.log.2, etc., and starts a fresh log. After N log files it
> >>>>starts back at 1 and overwrites the old log file. Similar but
> >>>>different from above, but both are great to reduce administrative
> >>>>overhead for the user.
> >>>>
> >>>>David
> >>>>
> >>>>Sunitha Kambhampati wrote:
> >>>>
> >>>>
> >>>>>Øystein Grøvlen wrote:
> >>>>>
> >>>>>
> >>>>>>>>>>>"KM" == Kathey Marsden <km...@sbcglobal.net> writes:
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>> KM> Øystein Grøvlen (JIRA) wrote:
> >>>>>> >> [ >> This would be even more helpful if the derby.log file
> >>>>>>was not overwritten on the next startup. (Probably a separate issue).
> >>>>>> >> >> >> KM> The derby.infolog.append property is
> >>>>>>helpful in this regard.
> >>>>>>
> >>>>>> KM>
> >>>>>>http://db.apache.org/derby/docs/10.1/tuning/rtunproper13217.html
> >>>>>>
> >>>>>>Thanks, Kathey, that is really what I need.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>One concern I have is that by default, the derby.log gets
> >>>>>overwritten on subsequent boot. From my experience with customers,
> >>>>>whenever there is an issue, I always end up telling them to add this
> >>>>>property almost every time to see the debug logs. It is very common
> >>>>>to see users reboot the system if they hit an error and in this
> >>>>>case, it is probable that important relevant information in the
> >>>>>derby.log is lost because it gets overwritten ( ex - recovery issues
> >>>>>, deadlock etc ).
> >>>>>
> >>>>>Maybe we could do something smart for derby.log , a balance between
> >>>>>keeping history across boots and disk space taken by the derby.logs.
> >>>>>
> >>>>>Any comments ?
> >>>>>
> >>>>>Sunitha.
> >>>>
> >>>>
> >>>>
> >>
> >>
>
>
Re: regarding derby.log being overwritten on startup. [was Re: [jira]
Commented: (DERBY-515) Network Server should log server start and shutdown
time to derby.log]
Posted by Rick Hillegas <Ri...@Sun.COM>.
Derby still has an error log reading VTI: ErrorLogReader. It shows how a
reasonable engineer, in a pre-xml world, would approach the problem of
parsing a log file and presenting it in tabular form.
I retract my hasty suggestion that it would be easy to convert Derby
logs to a more structured format. Although BasicGetLogHeader
standardizes the leading fields of a log record, often the trailing
fields are tastier...and today these come to the logging subsystem as a
single, unstructured string.
-Rick
Mike Matrigali wrote:
> I believe that the current log file format has been written to
> be easy to parse - but not as easy as looking for a single
> delimiter. Remember that the log file can contain arbitrary
> text - so not sure if a single delimiter will work. The worst
> case is log statement text which will print out values of fields,
> which can contain anything and are not under our control.
> At one point there was a tool that could take
> the log file and present it like a read only table, similar to
> the lock table. Adding some delimiters might make it easier.
>
> I also like having the log file at least somewhat user readable.
>
> Having said that I believe that if someone really wanted the XML
> version, an alternate error logging module implementation could
> be provided which could either produce XML instead of text or
> could provide 2 different files. If someone wanted to write a
> cool tool that parsed and presented the error log file, or needed
> xml to adapt it to an existing tool then I could see XML form
> being more important.
>
> There was hope to be able to be able to do thing like take the out
> of log statement text and then turn around and use that to drive
> a testing framework which could execute those statments.
>
> Danesh wrote:
>
>> I agree with going the delimited format route. Having the database
>> write out XML would probably impact preformance quite a bit. I also
>> wonder what would happen if the database were to die (power failure or
>> the like) and the XML file ended up invalid due to it missing those
>> crucial few closing tags. To simplify things and satisfy both parties
>> someone could write a log conversion tool i.e. DEL -> XML.
>>
>> ~Danesh
>>
>> On 8/19/05, David Van Couvering <Da...@sun.com> wrote:
>>
>>> Please, God, no :). Machines like reading XML but people (at least
>>> certain people) find it a bit horrific.
>>>
>>> What I have seen done in the past is using a simple delimiter
>>> format, e.g.
>>>
>>> |Fri Aug 19 14:06:31 PST
>>> 2005|org.apache.derby.engine.impl.jdbc.EmbedConnection|Thread
>>> 275|Invalid user, please try again| | |
>>>
>>> then you can write tools that slurp this right into Excel, load into a
>>> database, etc. We did this with an app I worked on at Sybase and it
>>> was
>>> quite successful.
>>>
>>> David
>>>
>>> Rick Hillegas wrote:
>>>
>>>
>>>> At the risk of overloading this topic: While we're talking about
>>>> improvements to our error logging, I'd like to see us emit more
>>>> structured logs. This is one of the things that xml is actually good
>>>> for. It would be pretty easy to turn our log records into xml entries.
>>>> This would make it easy to write or use off-the-shelf tools for
>>>> viewing the logs and for filtering signal out of the noise.
>>>>
>>>> Cheers,
>>>> -Rick
>>>>
>>>> David Van Couvering wrote:
>>>>
>>>>
>>>>> Well, the most common solution is log rotation. On restart, you
>>>>> rename the old log to derby.log.1 and then derby.log.2 and so on.
>>>>> After <n> log files, you start over again at derby.log.1.
>>>>> This is also useful for long-running systems so you don't run out of
>>>>> disk space because of your error logs. When the error log gets a
>>>>> configurable X MBs, the system copies the log to derby.log.1,
>>>>> derby.log.2, etc., and starts a fresh log. After N log files it
>>>>> starts back at 1 and overwrites the old log file. Similar but
>>>>> different from above, but both are great to reduce administrative
>>>>> overhead for the user.
>>>>>
>>>>> David
>>>>>
>>>>> Sunitha Kambhampati wrote:
>>>>>
>>>>>
>>>>>> Øystein Grøvlen wrote:
>>>>>>
>>>>>>
>>>>>>>>>>>> "KM" == Kathey Marsden <km...@sbcglobal.net> writes:
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>> KM> Øystein Grøvlen (JIRA) wrote:
>>>>>>> >> [ >> This would be even more helpful if the derby.log file
>>>>>>> was not overwritten on the next startup. (Probably a separate
>>>>>>> issue).
>>>>>>> >> >> >> KM> The derby.infolog.append property is
>>>>>>> helpful in this regard.
>>>>>>>
>>>>>>> KM>
>>>>>>> http://db.apache.org/derby/docs/10.1/tuning/rtunproper13217.html
>>>>>>>
>>>>>>> Thanks, Kathey, that is really what I need.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> One concern I have is that by default, the derby.log gets
>>>>>> overwritten on subsequent boot. From my experience with customers,
>>>>>> whenever there is an issue, I always end up telling them to add this
>>>>>> property almost every time to see the debug logs. It is very common
>>>>>> to see users reboot the system if they hit an error and in this
>>>>>> case, it is probable that important relevant information in the
>>>>>> derby.log is lost because it gets overwritten ( ex - recovery issues
>>>>>> , deadlock etc ).
>>>>>>
>>>>>> Maybe we could do something smart for derby.log , a balance between
>>>>>> keeping history across boots and disk space taken by the derby.logs.
>>>>>>
>>>>>> Any comments ?
>>>>>>
>>>>>> Sunitha.
>>>>>
>>>>>
>>>>>
>>>>>
>>>
>>>
>
Re: regarding derby.log being overwritten on startup. [was Re: [jira]
Commented: (DERBY-515) Network Server should log server start and shutdown
time to derby.log]
Posted by Mike Matrigali <mi...@sbcglobal.net>.
I believe that the current log file format has been written to
be easy to parse - but not as easy as looking for a single
delimiter. Remember that the log file can contain arbitrary
text - so not sure if a single delimiter will work. The worst
case is log statement text which will print out values of fields,
which can contain anything and are not under our control.
At one point there was a tool that could take
the log file and present it like a read only table, similar to
the lock table. Adding some delimiters might make it easier.
I also like having the log file at least somewhat user readable.
Having said that I believe that if someone really wanted the XML
version, an alternate error logging module implementation could
be provided which could either produce XML instead of text or
could provide 2 different files. If someone wanted to write a
cool tool that parsed and presented the error log file, or needed
xml to adapt it to an existing tool then I could see XML form
being more important.
There was hope to be able to be able to do thing like take the out
of log statement text and then turn around and use that to drive
a testing framework which could execute those statments.
Danesh wrote:
> I agree with going the delimited format route. Having the database
> write out XML would probably impact preformance quite a bit. I also
> wonder what would happen if the database were to die (power failure or
> the like) and the XML file ended up invalid due to it missing those
> crucial few closing tags. To simplify things and satisfy both parties
> someone could write a log conversion tool i.e. DEL -> XML.
>
> ~Danesh
>
> On 8/19/05, David Van Couvering <Da...@sun.com> wrote:
>
>>Please, God, no :). Machines like reading XML but people (at least
>>certain people) find it a bit horrific.
>>
>>What I have seen done in the past is using a simple delimiter format, e.g.
>>
>>|Fri Aug 19 14:06:31 PST
>>2005|org.apache.derby.engine.impl.jdbc.EmbedConnection|Thread
>>275|Invalid user, please try again| | |
>>
>>then you can write tools that slurp this right into Excel, load into a
>>database, etc. We did this with an app I worked on at Sybase and it was
>>quite successful.
>>
>>David
>>
>>Rick Hillegas wrote:
>>
>>
>>>At the risk of overloading this topic: While we're talking about
>>>improvements to our error logging, I'd like to see us emit more
>>>structured logs. This is one of the things that xml is actually good
>>>for. It would be pretty easy to turn our log records into xml entries.
>>>This would make it easy to write or use off-the-shelf tools for
>>>viewing the logs and for filtering signal out of the noise.
>>>
>>>Cheers,
>>>-Rick
>>>
>>>David Van Couvering wrote:
>>>
>>>
>>>>Well, the most common solution is log rotation. On restart, you
>>>>rename the old log to derby.log.1 and then derby.log.2 and so on.
>>>>After <n> log files, you start over again at derby.log.1.
>>>>This is also useful for long-running systems so you don't run out of
>>>>disk space because of your error logs. When the error log gets a
>>>>configurable X MBs, the system copies the log to derby.log.1,
>>>>derby.log.2, etc., and starts a fresh log. After N log files it
>>>>starts back at 1 and overwrites the old log file. Similar but
>>>>different from above, but both are great to reduce administrative
>>>>overhead for the user.
>>>>
>>>>David
>>>>
>>>>Sunitha Kambhampati wrote:
>>>>
>>>>
>>>>>Øystein Grøvlen wrote:
>>>>>
>>>>>
>>>>>>>>>>>"KM" == Kathey Marsden <km...@sbcglobal.net> writes:
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>> KM> Øystein Grøvlen (JIRA) wrote:
>>>>>> >> [ >> This would be even more helpful if the derby.log file
>>>>>>was not overwritten on the next startup. (Probably a separate issue).
>>>>>> >> >> >> KM> The derby.infolog.append property is
>>>>>>helpful in this regard.
>>>>>>
>>>>>> KM>
>>>>>>http://db.apache.org/derby/docs/10.1/tuning/rtunproper13217.html
>>>>>>
>>>>>>Thanks, Kathey, that is really what I need.
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>One concern I have is that by default, the derby.log gets
>>>>>overwritten on subsequent boot. From my experience with customers,
>>>>>whenever there is an issue, I always end up telling them to add this
>>>>>property almost every time to see the debug logs. It is very common
>>>>>to see users reboot the system if they hit an error and in this
>>>>>case, it is probable that important relevant information in the
>>>>>derby.log is lost because it gets overwritten ( ex - recovery issues
>>>>>, deadlock etc ).
>>>>>
>>>>>Maybe we could do something smart for derby.log , a balance between
>>>>>keeping history across boots and disk space taken by the derby.logs.
>>>>>
>>>>>Any comments ?
>>>>>
>>>>>Sunitha.
>>>>
>>>>
>>>>
>>
>>
Re: regarding derby.log being overwritten on startup. [was Re: [jira]
Commented: (DERBY-515) Network Server should log server start and shutdown
time to derby.log]
Posted by Sunitha Kambhampati <ks...@gmail.com>.
Actually, when derby.language.logStatementText=true is set, messages
that are currently printed to derby.log uses a ":" delimiter format.
e.g.
..... Begin compiling prepared statement : SELECT * from tbl1 : end
compiling prepared statement
etc...
For really huge derby.log traces, I usually use this delimiter option
and pull it up in Excel to view the queries etc.
Sunitha.
Danesh wrote:
>I agree with going the delimited format route. Having the database
>write out XML would probably impact preformance quite a bit. I also
>wonder what would happen if the database were to die (power failure or
>the like) and the XML file ended up invalid due to it missing those
>crucial few closing tags. To simplify things and satisfy both parties
>someone could write a log conversion tool i.e. DEL -> XML.
>
>~Danesh
>
>On 8/19/05, David Van Couvering <Da...@sun.com> wrote:
>
>
>>Please, God, no :). Machines like reading XML but people (at least
>>certain people) find it a bit horrific.
>>
>>What I have seen done in the past is using a simple delimiter format, e.g.
>>
>>|Fri Aug 19 14:06:31 PST
>>2005|org.apache.derby.engine.impl.jdbc.EmbedConnection|Thread
>>275|Invalid user, please try again| | |
>>
>>then you can write tools that slurp this right into Excel, load into a
>>database, etc. We did this with an app I worked on at Sybase and it was
>>quite successful.
>>
>>David
>>
>>Rick Hillegas wrote:
>>
>>
>>
>>>At the risk of overloading this topic: While we're talking about
>>>improvements to our error logging, I'd like to see us emit more
>>>structured logs. This is one of the things that xml is actually good
>>>for. It would be pretty easy to turn our log records into xml entries.
>>>This would make it easy to write or use off-the-shelf tools for
>>>viewing the logs and for filtering signal out of the noise.
>>>
>>>Cheers,
>>>-Rick
>>>
>>>David Van Couvering wrote:
>>>
>>>
>>>
>>>>Well, the most common solution is log rotation. On restart, you
>>>>rename the old log to derby.log.1 and then derby.log.2 and so on.
>>>>After <n> log files, you start over again at derby.log.1.
>>>>This is also useful for long-running systems so you don't run out of
>>>>disk space because of your error logs. When the error log gets a
>>>>configurable X MBs, the system copies the log to derby.log.1,
>>>>derby.log.2, etc., and starts a fresh log. After N log files it
>>>>starts back at 1 and overwrites the old log file. Similar but
>>>>different from above, but both are great to reduce administrative
>>>>overhead for the user.
>>>>
>>>>David
>>>>
>>>>Sunitha Kambhampati wrote:
>>>>
>>>>
>>>>
>>>>>Øystein Grøvlen wrote:
>>>>>
>>>>>
>>>>>
>>>>>>>>>>>"KM" == Kathey Marsden <km...@sbcglobal.net> writes:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>> KM> Øystein Grøvlen (JIRA) wrote:
>>>>>> >> [ >> This would be even more helpful if the derby.log file
>>>>>>was not overwritten on the next startup. (Probably a separate issue).
>>>>>> >> >> >> KM> The derby.infolog.append property is
>>>>>>helpful in this regard.
>>>>>>
>>>>>> KM>
>>>>>>http://db.apache.org/derby/docs/10.1/tuning/rtunproper13217.html
>>>>>>
>>>>>>Thanks, Kathey, that is really what I need.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>One concern I have is that by default, the derby.log gets
>>>>>overwritten on subsequent boot. From my experience with customers,
>>>>>whenever there is an issue, I always end up telling them to add this
>>>>>property almost every time to see the debug logs. It is very common
>>>>>to see users reboot the system if they hit an error and in this
>>>>>case, it is probable that important relevant information in the
>>>>>derby.log is lost because it gets overwritten ( ex - recovery issues
>>>>>, deadlock etc ).
>>>>>
>>>>>Maybe we could do something smart for derby.log , a balance between
>>>>>keeping history across boots and disk space taken by the derby.logs.
>>>>>
>>>>>Any comments ?
>>>>>
>>>>>Sunitha.
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>
>>
>>
Re: regarding derby.log being overwritten on startup. [was Re: [jira] Commented: (DERBY-515) Network Server should log server start and shutdown time to derby.log]
Posted by RPost <rp...@pacbell.net>.
And yet, to paraphrase a recent comment, 'Machines like reading delimited
format but people (at least certain people) find it a bit horrific'.
When I read Rick Hillegas's comment ("I'd like to see us emit more
structured logs.") the first question that came to my mind was just how
structured, in terms of content, are the current log messages?
Is the log record based with each message being represented by one record?
If so, is each record composed of the same number of items? That is, is item
one always a date, item 2 a class (or process) name, etc.
If there is no consistent structure to the messages I don't see how it would
be helpful to wrap the messages components in XML tags.
If there are a different number of components in each message record and no
defined order to the components I also don't see how it would be useful to
'slurp this right into Excel, load into a database, etc.
Perhaps it would make sense, as a first step, to try to standardize the
writing of the logs so that the structure of each message or component is
more consistent. A standardized content would then lend itself to a
different format for possible automation.
Can any of you comment of the degree to which the current log messages are,
or are not, consistent or lend themselves to 'record' type of content with
each record having the same number of data items?
----- Original Message -----
From: "Danesh" <da...@gmail.com>
To: "Derby Development" <de...@db.apache.org>
Sent: Friday, August 19, 2005 3:05 PM
Subject: Re: regarding derby.log being overwritten on startup. [was Re:
[jira] Commented: (DERBY-515) Network Server should log server start and
shutdown time to derby.log]
> I agree with going the delimited format route. Having the database
> write out XML would probably impact preformance quite a bit. I also
> wonder what would happen if the database were to die (power failure or
> the like) and the XML file ended up invalid due to it missing those
> crucial few closing tags. To simplify things and satisfy both parties
> someone could write a log conversion tool i.e. DEL -> XML.
>
> ~Danesh
>
> On 8/19/05, David Van Couvering <Da...@sun.com> wrote:
> > Please, God, no :). Machines like reading XML but people (at least
> > certain people) find it a bit horrific.
> >
> > What I have seen done in the past is using a simple delimiter format,
e.g.
> >
> > |Fri Aug 19 14:06:31 PST
> > 2005|org.apache.derby.engine.impl.jdbc.EmbedConnection|Thread
> > 275|Invalid user, please try again| | |
> >
> > then you can write tools that slurp this right into Excel, load into a
> > database, etc. We did this with an app I worked on at Sybase and it was
> > quite successful.
> >
> > David
> >
> > Rick Hillegas wrote:
> >
> > > At the risk of overloading this topic: While we're talking about
> > > improvements to our error logging, I'd like to see us emit more
> > > structured logs. This is one of the things that xml is actually good
> > > for. It would be pretty easy to turn our log records into xml entries.
> > > This would make it easy to write or use off-the-shelf tools for
> > > viewing the logs and for filtering signal out of the noise.
> > >
> > > Cheers,
> > > -Rick
> > >
> > > David Van Couvering wrote:
> > >
> > >> Well, the most common solution is log rotation. On restart, you
> > >> rename the old log to derby.log.1 and then derby.log.2 and so on.
> > >> After <n> log files, you start over again at derby.log.1.
> > >> This is also useful for long-running systems so you don't run out of
> > >> disk space because of your error logs. When the error log gets a
> > >> configurable X MBs, the system copies the log to derby.log.1,
> > >> derby.log.2, etc., and starts a fresh log. After N log files it
> > >> starts back at 1 and overwrites the old log file. Similar but
> > >> different from above, but both are great to reduce administrative
> > >> overhead for the user.
> > >>
> > >> David
> > >>
> > >> Sunitha Kambhampati wrote:
> > >>
> > >>> Øystein Grøvlen wrote:
> > >>>
> > >>>>>>>>> "KM" == Kathey Marsden <km...@sbcglobal.net> writes:
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>
> > >>>> KM> Øystein Grøvlen (JIRA) wrote:
> > >>>> >> [ >> This would be even more helpful if the derby.log file
> > >>>> was not overwritten on the next startup. (Probably a separate
issue).
> > >>>> >> >> >> KM> The derby.infolog.append property is
> > >>>> helpful in this regard.
> > >>>>
> > >>>> KM>
> > >>>> http://db.apache.org/derby/docs/10.1/tuning/rtunproper13217.html
> > >>>>
> > >>>> Thanks, Kathey, that is really what I need.
> > >>>>
> > >>>>
> > >>>>
> > >>> One concern I have is that by default, the derby.log gets
> > >>> overwritten on subsequent boot. From my experience with customers,
> > >>> whenever there is an issue, I always end up telling them to add this
> > >>> property almost every time to see the debug logs. It is very common
> > >>> to see users reboot the system if they hit an error and in this
> > >>> case, it is probable that important relevant information in the
> > >>> derby.log is lost because it gets overwritten ( ex - recovery issues
> > >>> , deadlock etc ).
> > >>>
> > >>> Maybe we could do something smart for derby.log , a balance between
> > >>> keeping history across boots and disk space taken by the derby.logs.
> > >>>
> > >>> Any comments ?
> > >>>
> > >>> Sunitha.
> > >>
> > >>
> > >>
> > >
> >
> >
> >
>