You are viewing a plain text version of this content. The canonical link for it is here.
Posted to log4j-dev@logging.apache.org by Ceki Gülcü <ce...@qos.ch> on 2002/03/21 17:09:53 UTC
An alternative JTable
Hi Oliver,
I have just added a small program called TableAppender in
contribs/CekiGulcu/TableAppender.java to our CVS repository. This
program illustrates a possible algorithm to manage a large number of
logging events. This particular implementation uses a fixed sized
buffer although a dynamically growing buffer would work just as well.
My tests show that TableAppender can handle the insertion of 100'000
events in about 4 seconds. These 4 seconds include the painting of those
100'000 elements in the table. Let me know what you think of it.
On a related but different register, chainsaw as it exists today has
3 buffers. A buffer for pending events that chainsaw just
received from the client but has not yet processed, a second buffer
containing all the elements received so far, and a third buffer that
contains the events that remain after filtering by the various
filters.
I would suggest an alternative architecture where only two buffers are
needed. One for collecting events and another buffer to contain the
events after filtering. The latter buffer would also serve as the
underlying data structure for the TableModel. I also believe that
there is no need to have a separate thread to process the events such
as the MyTableModel.Processor. Let me know if you wish to pursue this
further.
As for introducing Checkstyle (http://checkstyle.sourceforge.net/) in
log4j, I am favorable to the idea although I do not yet see a burning
need for it. Being the only active committer I do not have much
trouble sticking to the log4j coding style which happens to be my
favorite. A while back, there was a long and somewhat unpleasant
discussion on general@jakarta mailing list on coding conventions. It
did not go anywhere. As a result of this discussion however, I along
numerous other people concluded that the only way to enforce coding
standards is using an automated process.
Gump (http://jakarta.apache.org/gump/) is a concrete example of the
power of such an automated process. It detects backward incompatible
changes in project A that breaks project B which depends on A. It has
been an extremely useful tool to preserve log4j's backward
compatibility.
I think Checkstyle can prove to be an indispensable tool. As long as it
doesn't bring development to a screeching halt, you are welcome to add
style checks to log4j. You might also want to suggest it to others on
the general@jakarta list.
--
Ceki
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: An alternative JTable
Posted by Ceki Gülcü <ce...@qos.ch>.
Hi Oliver,
Applying checkstyle first to log4j before attempting wider
distribution is most reasonable. You have got my unconditional
support.
Pluggable receivers (as suggested by Mark) are clearly a good idea.
Not those are out of the way, I would like to focus on two particular
issues raised in your email: 1) ordering of events and 2) pausing
updates.
In light of the requirement to order events based on time, the
existing code makes much more sense. I would like to make two
observations. First, two computers will usually not have their clocks
synchronized. Second, given that network delays can be unbounded, the
algorithm to support timed-based event ordering is bound to be
inefficient. I do not think there is any cheap way to *reliably* get
around this problem. (I will nevertheless suggest a cheat in a
moment.)
There is a corollary to the first observation: maintaining
(time-based) event ordering has no practical meaning or
usefulness. This may sound outrageous but I believe it to be a correct
statement nonetheless. I think dropping the ordering requirement makes
sense and will considerably simplify implementation.
The second issue you raised, that is pausing updates, admits an easy
solution even with two buffers. Let us call these two buffers A and
B. (Forgive my lack of imagination.)
Buffer A contains unfiltered events while buffer B contains filtered
events. B also serves as the underlying data structure for the
TableModel.
The arrival of a new event, say e, is handled as follows:
1) insert e to buffer A
2) apply the filters on e. If e is filtered out, then no further
processing is necessary. Otherwise, insert e into B, then invoke
fireUptadeTable() method.
Support for update pausing can be easily added by suspending step 2 as
long as the pause is in effect. When updating is resumed, then we can
clear buffer B and rebuild it from the contents of buffer A.
For the sake of efficiency, we could record the latest element in
buffer A when the pause was ordered. Let us call this element k. We
can later update B (without clearing it) by considering only the
elements in A that came after k.
A poor man's solution to the event ordering problem is simply to sort
buffer B (by time) at the user's request. (This is the cheat I
mentioned earlier.)
Note that the only requirement for these algorithms is for buffers A
and B to be linear. At no time was it required for A and B to be
bounded. Cyclic buffers could do, but an unbounded data structure, say
Vector, would do just as well.
<SIDE-NOTE>
If java.util.Vector becomes inefficient for very large amounts of data
we can use data structures consisting of fixed size sub-buffers, say
of size 10'000. Each time we exceed the available space we allocate a
new sub-buffer. Sub-buffers are managed with a small Vector. Here is
some pseudo code to make the point clearer:
class DynamicButStillLinearBuffer {
int SUBBUFFER_SIZE = 10000;
Vector subbuffers = new Vector();
int elements = 0;
int capacity = 0;
void insertAtEnd(Object o) {
elements++;
if(elements > capacity) {
subbuffers.add(new Object[SUBBUFFER_SIZE]);
capacity += SUBBUFFER_SIZE;
}
Object[] array = subbuffers.elementAt(elements/SUBBUFFER_SIZE);
array[elements%SUBBUFFER_SIZE] = o;
}
Object getElement(int i) {
if((i <0) || (i > elements)) {
throw new GiveMeABreakException("This is just pseudo code.");
}
Object[] array = subbuffers(i/SUBBUFFER_SIZE);
return array[i%SUBBUFFER_SIZE];
}
}
If SUBBUFFER_SIZE is chosen to be a power of 2, then division can be
replaced by right shifts and remainder computations replaced by &
masks.
</SIDE-NOTE>
In summary, I think
1) the ordering requirement has many deep ramifications
2) two buffers are sufficient to support pausing.
I hope you find the above comments useful.
--
Ceki
At 23:29 22.03.2002 +1100, you wrote:
>Hi Ceki,
>
>I thought I would explain the current design on Chainsaw and how it
>uses the threads and buffers. This will also help with people understanding
>Chainsaw and making improvements. :-)
>
>At the moment Chainsaw supports collecting events from multiple servers
>using the SocketAppender. The LoggingReceiver class creates a thread for
>accepting connections from SocketAppenders. For each connection received,
>an instance of LoggingReceiver.Slurper is created with a separate thread.
>The purpose of this class is to receive the sent LoggingEvent objects as
>quickly as possible. Remember that Checkstyle supports pausing the updating
>of the display. However, it should continue to collect events so as to
>not cause grief for the sending SocketAppenders.
>
>The events received by LoggingReceiver.Slurper are added to the model,
>which is an instance of MyTableModel. Note, because Chainsaw is receiving
>events from multiple clients, it must order the events based on time. It
>cannot assume that events will be received in chronological order from
>clients due to network latency issues. The ordering is done by the use of
>the class MyTableModel.Comparator. I feel that for Chainsaw to use
>CyclicBuffer, then it must support also support ordering (which may cause
>problems?).
>
>To support the ability to pause the updating of the display and to filter
>events, the MyTableModel class has the following three buffers:
>
>- mPendingEvents - holds the incoming events whilst updating is paused;
>
>- mAllEvents - events are transferred from mPendingEvents into this buffer
> when not paused. This is done periodically (currently every
> second) by a thread. The reason for the delay
> was to minimise number screen updated, but your email
> indicates this may not be necessary.
>
>- mFilteredEvents - holds the events after apply the filter criteria. These
> are the events that are displayed in the UI. The model
> attempts to efficiently keep this buffer up to date.
>
>The rest of the code in Chainsaw is to display the events in a Swing table,
>and allow the updating of the filter criteria. Oh, and to also display
>events logged to a file in XML format.
>
>My plan is to add version 1.1 of Chainsaw to Log4J, plus a couple of patches
>I have received as well. After that, I suggest that Chainsaw is updated
>based on the input from this mailing list. Mark Womack and I have already
>had discussions and he has some good ideas. I particularly like the idea of
>allowing different receivers being plugged in.
>
>Please find my other comments embedded below.
>
>Regards,
>Oliver
>
>PS: I look forward to contributing to Log4J.
>
> > -----Original Message-----
> > From: Ceki Gulcu [mailto:ceki@qos.ch]
> > Sent: Friday, 22 March 2002 03:10
> > To: log4j-dev@jakarta.apache.org
> > Subject: An alternative JTable
> >
> >
> > Hi Oliver,
> >
> > I have just added a small program called TableAppender in
> > contribs/CekiGulcu/TableAppender.java to our CVS repository. This
> > program illustrates a possible algorithm to manage a large number of
> > logging events. This particular implementation uses a fixed sized
> > buffer although a dynamically growing buffer would work just as well.
> > My tests show that TableAppender can handle the insertion of 100'000
> > events in about 4 seconds. These 4 seconds include the painting of those
> > 100'000 elements in the table. Let me know what you think of it.
>
>I like it! I agree that Chainsaw should not have unbounded buffers and
>use something like CyclicBuffer. However, I feel that Chainsaw need to
>continue to support ordering of events based on time, so it is not able
>to use CyclicBuffer as is.
>
>The use of a single renderer is interesting. This maybe a way to change
>Chainsaw to be more flexible in what fields are displayed....
>
> >
> > On a related but different register, chainsaw as it exists today has
> > 3 buffers. A buffer for pending events that chainsaw just
> > received from the client but has not yet processed, a second buffer
> > containing all the elements received so far, and a third buffer that
> > contains the events that remain after filtering by the various
> > filters.
> >
> > I would suggest an alternative architecture where only two buffers are
> > needed. One for collecting events and another buffer to contain the
> > events after filtering. The latter buffer would also serve as the
> > underlying data structure for the TableModel. I also believe that
> > there is no need to have a separate thread to process the events such
> > as the MyTableModel.Processor. Let me know if you wish to pursue this
> > further.
>
>It is not clear to me how this scheme could support the ability to pause
>updating the display. If people feel this feature is not required, then
>I am happy to remove it and only have two buffers.
>
> >
> > As for introducing Checkstyle (http://checkstyle.sourceforge.net/) in
> > log4j, I am favorable to the idea although I do not yet see a burning
> > need for it. Being the only active committer I do not have much
> > trouble sticking to the log4j coding style which happens to be my
> > favorite. A while back, there was a long and somewhat unpleasant
> > discussion on general@jakarta mailing list on coding conventions. It
> > did not go anywhere. As a result of this discussion however, I along
> > numerous other people concluded that the only way to enforce coding
> > standards is using an automated process.
>
>I agree, that is why I wrote Checkstyle. :-)
>
>Seriously, I propose adding Checkstyle initially for just checking Chainsaw.
>This is as an aid for people (read me initially) to conform to your coding
>style. If it proves successful, then it can be extended to cover the rest
>of the Log4J source code.
>
> >
> > Gump (http://jakarta.apache.org/gump/) is a concrete example of the
> > power of such an automated process. It detects backward incompatible
> > changes in project A that breaks project B which depends on A. It has
> > been an extremely useful tool to preserve log4j's backward
> > compatibility.
> >
> > I think Checkstyle can prove to be an indispensable tool. As long as it
> > doesn't bring development to a screeching halt, you are welcome to add
> > style checks to log4j. You might also want to suggest it to others on
> > the general@jakarta list.
>
>I'd suggest using it on Log4J first to prove it's usefulness on a real
>Jakarta project first.
>
>
>--
>To unsubscribe, e-mail: <ma...@jakarta.apache.org>
>For additional commands, e-mail: <ma...@jakarta.apache.org>
--
Ceki
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: An alternative JTable
Posted by Oliver Burn <ob...@yahoo.com>.
Hi Ceki,
I thought I would explain the current design on Chainsaw and how it
uses the threads and buffers. This will also help with people understanding
Chainsaw and making improvements. :-)
At the moment Chainsaw supports collecting events from multiple servers
using the SocketAppender. The LoggingReceiver class creates a thread for
accepting connections from SocketAppenders. For each connection received,
an instance of LoggingReceiver.Slurper is created with a separate thread.
The purpose of this class is to receive the sent LoggingEvent objects as
quickly as possible. Remember that Checkstyle supports pausing the updating
of the display. However, it should continue to collect events so as to
not cause grief for the sending SocketAppenders.
The events received by LoggingReceiver.Slurper are added to the model,
which is an instance of MyTableModel. Note, because Chainsaw is receiving
events from multiple clients, it must order the events based on time. It
cannot assume that events will be received in chronological order from
clients due to network latency issues. The ordering is done by the use of
the class MyTableModel.Comparator. I feel that for Chainsaw to use
CyclicBuffer, then it must support also support ordering (which may cause
problems?).
To support the ability to pause the updating of the display and to filter
events, the MyTableModel class has the following three buffers:
- mPendingEvents - holds the incoming events whilst updating is paused;
- mAllEvents - events are transferred from mPendingEvents into this buffer
when not paused. This is done periodically (currently every
second) by a thread. The reason for the delay
was to minimise number screen updated, but your email
indicates this may not be necessary.
- mFilteredEvents - holds the events after apply the filter criteria. These
are the events that are displayed in the UI. The model
attempts to efficiently keep this buffer up to date.
The rest of the code in Chainsaw is to display the events in a Swing table,
and allow the updating of the filter criteria. Oh, and to also display
events logged to a file in XML format.
My plan is to add version 1.1 of Chainsaw to Log4J, plus a couple of patches
I have received as well. After that, I suggest that Chainsaw is updated
based on the input from this mailing list. Mark Womack and I have already
had discussions and he has some good ideas. I particularly like the idea of
allowing different receivers being plugged in.
Please find my other comments embedded below.
Regards,
Oliver
PS: I look forward to contributing to Log4J.
> -----Original Message-----
> From: Ceki Gulcu [mailto:ceki@qos.ch]
> Sent: Friday, 22 March 2002 03:10
> To: log4j-dev@jakarta.apache.org
> Subject: An alternative JTable
>
>
> Hi Oliver,
>
> I have just added a small program called TableAppender in
> contribs/CekiGulcu/TableAppender.java to our CVS repository. This
> program illustrates a possible algorithm to manage a large number of
> logging events. This particular implementation uses a fixed sized
> buffer although a dynamically growing buffer would work just as well.
> My tests show that TableAppender can handle the insertion of 100'000
> events in about 4 seconds. These 4 seconds include the painting of those
> 100'000 elements in the table. Let me know what you think of it.
I like it! I agree that Chainsaw should not have unbounded buffers and
use something like CyclicBuffer. However, I feel that Chainsaw need to
continue to support ordering of events based on time, so it is not able
to use CyclicBuffer as is.
The use of a single renderer is interesting. This maybe a way to change
Chainsaw to be more flexible in what fields are displayed....
>
> On a related but different register, chainsaw as it exists today has
> 3 buffers. A buffer for pending events that chainsaw just
> received from the client but has not yet processed, a second buffer
> containing all the elements received so far, and a third buffer that
> contains the events that remain after filtering by the various
> filters.
>
> I would suggest an alternative architecture where only two buffers are
> needed. One for collecting events and another buffer to contain the
> events after filtering. The latter buffer would also serve as the
> underlying data structure for the TableModel. I also believe that
> there is no need to have a separate thread to process the events such
> as the MyTableModel.Processor. Let me know if you wish to pursue this
> further.
It is not clear to me how this scheme could support the ability to pause
updating the display. If people feel this feature is not required, then
I am happy to remove it and only have two buffers.
>
> As for introducing Checkstyle (http://checkstyle.sourceforge.net/) in
> log4j, I am favorable to the idea although I do not yet see a burning
> need for it. Being the only active committer I do not have much
> trouble sticking to the log4j coding style which happens to be my
> favorite. A while back, there was a long and somewhat unpleasant
> discussion on general@jakarta mailing list on coding conventions. It
> did not go anywhere. As a result of this discussion however, I along
> numerous other people concluded that the only way to enforce coding
> standards is using an automated process.
I agree, that is why I wrote Checkstyle. :-)
Seriously, I propose adding Checkstyle initially for just checking Chainsaw.
This is as an aid for people (read me initially) to conform to your coding
style. If it proves successful, then it can be extended to cover the rest
of the Log4J source code.
>
> Gump (http://jakarta.apache.org/gump/) is a concrete example of the
> power of such an automated process. It detects backward incompatible
> changes in project A that breaks project B which depends on A. It has
> been an extremely useful tool to preserve log4j's backward
> compatibility.
>
> I think Checkstyle can prove to be an indispensable tool. As long as it
> doesn't bring development to a screeching halt, you are welcome to add
> style checks to log4j. You might also want to suggest it to others on
> the general@jakarta list.
I'd suggest using it on Log4J first to prove it's usefulness on a real
Jakarta project first.
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>