You are viewing a plain text version of this content. The canonical link for it is here.
Posted to log4cxx-dev@logging.apache.org by Jack Strohm <js...@perpetual.com> on 2004/08/27 18:46:07 UTC
Logging of arbitrary length binary data
I was wondering if there has been any thought to the ability to log a
binary stream of data? With log4j you can log any object, but log4cxx
only handles strings (in both properties and the message). Thank for
any input you can give me as to how difficult this might be if I wanted
to do this myself.
- jack
Re: Logging of arbitrary length binary data
Posted by Jack Strohm <js...@perpetual.com>.
Ceki Gülcü wrote:
> At 07:21 PM 8/27/2004, you wrote:
>
>> In my application I might be generating a very large number of log
>> events. Most times I don't want to know what's in the blob, but for
>> a small percentage of these messages I may decide that I want to
>> convert the blob into a string (or use some viewer to display what is
>> in the blob). So while I may end up converting it to a human
>> readable string I can delay that until I know I want to look at the
>> event.
>
>
> OK, you are maybe thinking about a slightly different aspect of the
> problem. You would like to defer the transformation of the message
> objet up until the last possible moment. In the case of log4j, it
> actually converts the message object into a string only when the
> message needs to be written out. Also, it keeps the reference to the
> message object for duration of the life cycle of the LoggingEvent object.
In my case I'm thinking about dumping these log events to another
machine where they can be manipulatd if necessary. So having to convert
the blob into a human readable form initially is not only extra CPU
overhead but also network bandwidth.
>
>> As for binary properties I could see that would make writing filters
>> more efficnet, instead of doing a ton of string compares I can
>> compare primitive data types.
>
>
> I don't understand what you mean by binary properties not setProperty.
Sorry, what I meant was currently an event only supports strings in the
setProperty method. If it supported other forms of properties (like
int, long, float) then filters could be written that would compare
against these primitive data types instead of always doing string
comparisons. You save CPU time on both the primitive data type to
string conversion and in the compare used in the filter.
Re: Logging of arbitrary length binary data
Posted by Ceki Gülcü <ce...@qos.ch>.
At 07:21 PM 8/27/2004, you wrote:
>In my application I might be generating a very large number of log
>events. Most times I don't want to know what's in the blob, but for a
>small percentage of these messages I may decide that I want to convert the
>blob into a string (or use some viewer to display what is in the
>blob). So while I may end up converting it to a human readable string I
>can delay that until I know I want to look at the event.
OK, you are maybe thinking about a slightly different aspect of the
problem. You would like to defer the transformation of the message objet up
until the last possible moment. In the case of log4j, it actually converts
the message object into a string only when the message needs to be written
out. Also, it keeps the reference to the message object for duration of the
life cycle of the LoggingEvent object.
>As for binary properties I could see that would make writing filters more
>efficnet, instead of doing a ton of string compares I can compare
>primitive data types.
I don't understand what you mean by binary properties not setProperty.
--
Ceki Gülcü
Re: Logging of arbitrary length binary data
Posted by Jack Strohm <js...@perpetual.com>.
Curt Arnold wrote:
>> For the blob events I would build my own appender but if it was
>> derived from LoggingEvent and didn't overide or change any of the
>> functionality I would think it would work fine.
>
>
> Why derive a class If it doesn't override or change functionality?
Sorry, I mean I wouldn't change the existing functionality, just add a
new data member.
>
>> Even if it was transported, everything in the log event (minus the
>> blob) should transport fine. So I could have some appenders sending
>> it to files without the blob and other appenders that could inspect
>> the class, realize it's a BlobEvent and extract and handle the blob.
>>
>
> Inspecting the class would require RTTI. There is not a virtual
> destructor, so if the log event was deleted by log4cxx, the new fields
> would not be destructed. Copy constructors are used to buffer logging
> events for asynchronous logging.
>
> Too much of the framework depends on a concrete implementation of
> LoggingEvent to be easily circumvented.
Hmmm, that could be a problem.
Re: Logging of arbitrary length binary data
Posted by Curt Arnold <ca...@houston.rr.com>.
> For the blob events I would build my own appender but if it was
> derived from LoggingEvent and didn't overide or change any of the
> functionality I would think it would work fine.
Why derive a class If it doesn't override or change functionality?
> Even if it was transported, everything in the log event (minus the
> blob) should transport fine. So I could have some appenders sending
> it to files without the blob and other appenders that could inspect
> the class, realize it's a BlobEvent and extract and handle the blob.
>
Inspecting the class would require RTTI. There is not a virtual
destructor, so if the log event was deleted by log4cxx, the new fields
would not be destructed. Copy constructors are used to buffer logging
events for asynchronous logging.
Too much of the framework depends on a concrete implementation of
LoggingEvent to be easily circumvented.
The LoggingEvent constructor in response to a logging request is where
the request gets all packed up assuming that it will be appended to
something. Obviously, you want to delay that constructor until you
are reasonably certain that an appender will actually do something with
it. However, there has to be some threshold point where that cost is
incurred (at least to keep the dispatch and appending code sane) and
once you've reached that point, you might as well get it done with.
The current implementation of log4cxx might be doing the constructor a
little too early, but those things can be improved without affecting
the API. However, I don't think that you can ever eliminate a
threshold point where you really have to decide if you are logging or
not.
Re: Logging of arbitrary length binary data
Posted by Jack Strohm <js...@perpetual.com>.
Curt Arnold wrote:
>
> On Aug 27, 2004, at 1:21 PM, Jack Strohm wrote:
>
>> I was thinking about making a customized LoggingEvent (a BlobEvent?)
>> that supported what I need (blobs and properties with primitive data
>> types) and then my own Appender to transport it. I guess it wouldn't
>> be something anyone else would want to use, but I think it would get
>> the job done for me. So the normal appenders could treat it normally
>> but the blob specific portion of the event could be handled by our
>> speciallized appender.
>>
>> Yeah, I'm already using the macro's, but the concern is to reduce the
>> time it takes to generate a message if possible (so no int to string
>> conversions if it is not necessary).
>>
>
> Pain, much pain. To the best of my knowledge, the logging event is a
> concrete class in all the log4j variants and not ameniable to
> extension. log4j does things like serialize the LoggingEvent in the
> socket appender and transport the bits across the wire to Chainsaw.
> For log4cxx especially, the internals are unfrozen and will be
> changing and your fork could very easily become unsupportable.
For the blob events I would build my own appender but if it was derived
from LoggingEvent and didn't overide or change any of the functionality
I would think it would work fine. Even if it was transported,
everything in the log event (minus the blob) should transport fine. So
I could have some appenders sending it to files without the blob and
other appenders that could inspect the class, realize it's a BlobEvent
and extract and handle the blob.
>
> Some things, like floating point numbers and dates, are very expensive
> to format in legible forms (aka 3.1415926 or 27 Aug 2004). However,
> spitting out a hex representation, which contains the same info in a
> much less legible form, can be pretty darn cheap.
Yes, I've also thought about hexing the binary data and putting it into
a string for transport, but I have a fealing the developers I'm working
with would frown on that. But it still remains an option.
>
> You might consider logging the blobs to a separate logger (or
> hiearchy). That would allow the blob log to be disabled without
> disabling the general application log.
>
Yes, that's another option I've thought about.
Thanks a lot for all your suggestions
- jack
Re: Logging of arbitrary length binary data
Posted by Curt Arnold <ca...@houston.rr.com>.
On Aug 27, 2004, at 1:21 PM, Jack Strohm wrote:
> I was thinking about making a customized LoggingEvent (a BlobEvent?)
> that supported what I need (blobs and properties with primitive data
> types) and then my own Appender to transport it. I guess it wouldn't
> be something anyone else would want to use, but I think it would get
> the job done for me. So the normal appenders could treat it normally
> but the blob specific portion of the event could be handled by our
> speciallized appender.
>
> Yeah, I'm already using the macro's, but the concern is to reduce the
> time it takes to generate a message if possible (so no int to string
> conversions if it is not necessary).
>
Pain, much pain. To the best of my knowledge, the logging event is a
concrete class in all the log4j variants and not ameniable to
extension. log4j does things like serialize the LoggingEvent in the
socket appender and transport the bits across the wire to Chainsaw.
For log4cxx especially, the internals are unfrozen and will be changing
and your fork could very easily become unsupportable.
Some things, like floating point numbers and dates, are very expensive
to format in legible forms (aka 3.1415926 or 27 Aug 2004). However,
spitting out a hex representation, which contains the same info in a
much less legible form, can be pretty darn cheap.
You might consider logging the blobs to a separate logger (or
hiearchy). That would allow the blob log to be disabled without
disabling the general application log.
Re: Logging of arbitrary length binary data
Posted by Jack Strohm <js...@perpetual.com>.
I was thinking about making a customized LoggingEvent (a BlobEvent?)
that supported what I need (blobs and properties with primitive data
types) and then my own Appender to transport it. I guess it wouldn't be
something anyone else would want to use, but I think it would get the
job done for me. So the normal appenders could treat it normally but
the blob specific portion of the event could be handled by our
speciallized appender.
Yeah, I'm already using the macro's, but the concern is to reduce the
time it takes to generate a message if possible (so no int to string
conversions if it is not necessary).
Curt Arnold wrote:
> Every part in the chain believes that it can treat the messages as
> text. A file appender, for example, would possibly apply a character
> set encoding conversion before outputting a file. It would be
> unlikely that your binary extraction code would work reliably if the
> user reconfigured the output encoding. Whatever you put in the
> message has to be unaffected by encoding changes.
>
> If your concern is avoid unnecessary conversions . Use the
> LOG4CXX_DEBUG etc macros and a ToString method.
>
>
> // some stringizing method for Foo
> std::string ToString(const Foo&) {
> ...
> }
>
> ...
>
> Foo myFoo();
>
> // ToString is not executed unless logger->isDebugEnabled() == true
> LOG4CXX_DEBUG(logger, ToString(myFoo));
>
>
>
>
>
>
>
> On Aug 27, 2004, at 12:21 PM, Jack Strohm wrote:
>
>> In my application I might be generating a very large number of log
>> events. Most times I don't want to know what's in the blob, but for
>> a small percentage of these messages I may decide that I want to
>> convert the blob into a string (or use some viewer to display what is
>> in the blob). So while I may end up converting it to a human
>> readable string I can delay that until I know I want to look at the
>> event.
>>
>> As for binary properties I could see that would make writing filters
>> more efficnet, instead of doing a ton of string compares I can
>> compare primitive data types.
>>
>> What do you think? Am I approaching this wrong?
>>
>> Ceki Gülcü wrote:
>
>
>
Re: Logging of arbitrary length binary data
Posted by Curt Arnold <ca...@houston.rr.com>.
Every part in the chain believes that it can treat the messages as
text. A file appender, for example, would possibly apply a character
set encoding conversion before outputting a file. It would be unlikely
that your binary extraction code would work reliably if the user
reconfigured the output encoding. Whatever you put in the message has
to be unaffected by encoding changes.
If your concern is avoid unnecessary conversions . Use the
LOG4CXX_DEBUG etc macros and a ToString method.
// some stringizing method for Foo
std::string ToString(const Foo&) {
...
}
...
Foo myFoo();
// ToString is not executed unless logger->isDebugEnabled() == true
LOG4CXX_DEBUG(logger, ToString(myFoo));
On Aug 27, 2004, at 12:21 PM, Jack Strohm wrote:
> In my application I might be generating a very large number of log
> events. Most times I don't want to know what's in the blob, but for a
> small percentage of these messages I may decide that I want to convert
> the blob into a string (or use some viewer to display what is in the
> blob). So while I may end up converting it to a human readable string
> I can delay that until I know I want to look at the event.
>
> As for binary properties I could see that would make writing filters
> more efficnet, instead of doing a ton of string compares I can compare
> primitive data types.
>
> What do you think? Am I approaching this wrong?
>
> Ceki Gülcü wrote:
Re: Logging of arbitrary length binary data
Posted by Jack Strohm <js...@perpetual.com>.
In my application I might be generating a very large number of log
events. Most times I don't want to know what's in the blob, but for a
small percentage of these messages I may decide that I want to convert
the blob into a string (or use some viewer to display what is in the
blob). So while I may end up converting it to a human readable string I
can delay that until I know I want to look at the event.
As for binary properties I could see that would make writing filters
more efficnet, instead of doing a ton of string compares I can compare
primitive data types.
What do you think? Am I approaching this wrong?
Ceki Gülcü wrote:
> At 07:12 PM 8/27/2004, you wrote:
>
>> I'm not really interested in logging any object, I'm more interested
>> in putting a blob into a logging event that I can extract later.
>
>
> Don't you need to convert the blob into a string eventually? What good
> is a logging API if you can't read its output? What am I missing?
>
>
Re: Logging of arbitrary length binary data
Posted by Ceki Gülcü <ce...@qos.ch>.
At 07:12 PM 8/27/2004, you wrote:
>I'm not really interested in logging any object, I'm more interested in
>putting a blob into a logging event that I can extract later.
Don't you need to convert the blob into a string eventually? What good is a
logging API if you can't read its output? What am I missing?
--
Ceki Gülcü
For log4j documentation consider "The complete log4j manual"
ISBN: 2970036908 http://www.qos.ch/shop/products/clm_t.jsp
Re: Logging of arbitrary length binary data
Posted by Jack Strohm <js...@perpetual.com>.
I'm not really interested in logging any object, I'm more interested in
putting a blob into a logging event that I can extract later. It would
also be nice if the setProperty supported primitive data types (int,
long, float) without having to convert them into a string.
Ceki Gülcü wrote:
>
> Log4j's ability to log any object relies on only one premise: any
> java object can be transformed into a String. The Object class
> contains a toString method and every java object is an instance of the
> Object class. However, the toString() method does not always yield the
> desired results, which is where ObjectRenderers come into play.
>
> The code that selects the appropriate ObjectRenderer for a given
> object (to be logged) relies on two Java primitives. First, the code
> must be able to determine the class of the object being
> rendered. Second, it must be able to walk the class hierarchy of the
> object in question, including interfaces as well as super-classes. The
> rest is pretty straight forward.
>
> I have not coded in C++ for along time so I can't remember whether
> any of the above is available in C++.
>
> At 06:46 PM 8/27/2004, Jack Strohm wrote:
>
>> I was wondering if there has been any thought to the ability to log a
>> binary stream of data? With log4j you can log any object, but
>> log4cxx only handles strings (in both properties and the message).
>> Thank for any input you can give me as to how difficult this might be
>> if I wanted to do this myself.
>>
>> - jack
>
>
Re: Logging of arbitrary length binary data
Posted by Ceki Gülcü <ce...@qos.ch>.
Log4j's ability to log any object relies on only one premise: any
java object can be transformed into a String. The Object class
contains a toString method and every java object is an instance of the
Object class. However, the toString() method does not always yield the
desired results, which is where ObjectRenderers come into play.
The code that selects the appropriate ObjectRenderer for a given
object (to be logged) relies on two Java primitives. First, the code
must be able to determine the class of the object being
rendered. Second, it must be able to walk the class hierarchy of the
object in question, including interfaces as well as super-classes. The
rest is pretty straight forward.
I have not coded in C++ for along time so I can't remember whether
any of the above is available in C++.
At 06:46 PM 8/27/2004, Jack Strohm wrote:
>I was wondering if there has been any thought to the ability to log a
>binary stream of data? With log4j you can log any object, but log4cxx
>only handles strings (in both properties and the message). Thank for any
>input you can give me as to how difficult this might be if I wanted to do
>this myself.
>
>- jack
--
Ceki Gülcü
For log4j documentation consider "The complete log4j manual"
ISBN: 2970036908 http://www.qos.ch/shop/products/clm_t.jsp