[Log4j] includeNullDelimiter

[Mikael Ståldal <mi...@apache.org>]

Now JsonLayout, XmlLayout, YamlLayout and GelfLayout has an 
includeNullDelimiter option to terminate each log event with a 0-byte.

Would it be useful to have the same in any other layout?

I could think of HtmlLayout, SyslogLayout and RFC5424Layout.

Re: [Log4j] includeNullDelimiter

[Gary Gregory <ga...@gmail.com>]

Don't bait me Mr. Swift! :-)

Gary

On Sat, Jul 22, 2017 at 12:54 PM, Matt Sicker <bo...@gmail.com> wrote:

> I usually stick to big endian encoding while on the JVM thanks to
> ByteBuffer making it easy. If you write your own byte buffer style class,
> then it doesn't really matter which endianness you choose as you'll have to
> split up larger primitives into bytes regardless.
>
> On 21 July 2017 at 15:10, Gary Gregory <ga...@gmail.com> wrote:
>
> > The trick is to define "length": what is the size and endianness for this
> > integer and are either configurable? I think JMeter has some code for
> that.
> >
> > But at that point, why not use the BSON for uint64 (little-endian ?
> >
> > Gary
> >
> > On Fri, Jul 21, 2017 at 12:43 PM, Matt Sicker <bo...@gmail.com> wrote:
> >
> > > Exactly! I've seen similar encodings used in simple TCP framing. It
> > should
> > > be robust enough for logging (no need for additional metadata).
> > >
> > > On 21 July 2017 at 14:39, Gary Gregory <ga...@gmail.com> wrote:
> > >
> > > > Like Pascal strings, lengh-data-length-data...
> > > >
> > > > On Fri, Jul 21, 2017 at 11:31 AM, Matt Sicker <bo...@gmail.com>
> > wrote:
> > > >
> > > > > If the size of the message is known ahead of time, you can create a
> > > > simple
> > > > > framing protocol around messages as well.
> > > > >
> > > > > On 21 July 2017 at 07:23, Mikael Ståldal <mi...@apache.org> wrote:
> > > > >
> > > > > > I don't quite understand.
> > > > > >
> > > > > > This is something which is needed now. When you use a streaming
> > > > appender
> > > > > > (Console, any File or Socket with protocol=tcp/ssl), the layout
> > needs
> > > > to
> > > > > > provide some way to delimit log events.
> > > > > >
> > > > > >
> > > > > >
> > > > > > On 2017-07-20 22:28, Matt Sicker wrote:
> > > > > >
> > > > > >> I'm not a fan of the null termination as it is. Do you know how
> > many
> > > > > >> millions or billions of dollars have been wasted due to buggy C
> > and
> > > > C++
> > > > > >> programs thanks to null-terminated strings? Almost as bad as
> null
> > > > > itself.
> > > > > >> ;)
> > > > > >>
> > > > > >> Having some way to delimit log events could be handy later on
> for
> > > > > >> garbage-free networked streaming appenders that want to break up
> > log
> > > > > >> events
> > > > > >> into individual messages as opposed to how the file-type
> streaming
> > > > > >> appenders work.
> > > > > >>
> > > > > >> On 20 July 2017 at 14:53, Mikael Ståldal <mi...@apache.org>
> > wrote:
> > > > > >>
> > > > > >> Now JsonLayout, XmlLayout, YamlLayout and GelfLayout has an
> > > > > >>> includeNullDelimiter option to terminate each log event with a
> > > > 0-byte.
> > > > > >>>
> > > > > >>> Would it be useful to have the same in any other layout?
> > > > > >>>
> > > > > >>> I could think of HtmlLayout, SyslogLayout and RFC5424Layout.
> > > > > >>>
> > > > > >>
> > > > >
> > > > >
> > > > > --
> > > > > Matt Sicker <bo...@gmail.com>
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Matt Sicker <bo...@gmail.com>
> > >
> >
>
>
>
> --
> Matt Sicker <bo...@gmail.com>
>

Re: [Log4j] includeNullDelimiter

[Matt Sicker <bo...@gmail.com>]

I usually stick to big endian encoding while on the JVM thanks to
ByteBuffer making it easy. If you write your own byte buffer style class,
then it doesn't really matter which endianness you choose as you'll have to
split up larger primitives into bytes regardless.

On 21 July 2017 at 15:10, Gary Gregory <ga...@gmail.com> wrote:

> The trick is to define "length": what is the size and endianness for this
> integer and are either configurable? I think JMeter has some code for that.
>
> But at that point, why not use the BSON for uint64 (little-endian ?
>
> Gary
>
> On Fri, Jul 21, 2017 at 12:43 PM, Matt Sicker <bo...@gmail.com> wrote:
>
> > Exactly! I've seen similar encodings used in simple TCP framing. It
> should
> > be robust enough for logging (no need for additional metadata).
> >
> > On 21 July 2017 at 14:39, Gary Gregory <ga...@gmail.com> wrote:
> >
> > > Like Pascal strings, lengh-data-length-data...
> > >
> > > On Fri, Jul 21, 2017 at 11:31 AM, Matt Sicker <bo...@gmail.com>
> wrote:
> > >
> > > > If the size of the message is known ahead of time, you can create a
> > > simple
> > > > framing protocol around messages as well.
> > > >
> > > > On 21 July 2017 at 07:23, Mikael Ståldal <mi...@apache.org> wrote:
> > > >
> > > > > I don't quite understand.
> > > > >
> > > > > This is something which is needed now. When you use a streaming
> > > appender
> > > > > (Console, any File or Socket with protocol=tcp/ssl), the layout
> needs
> > > to
> > > > > provide some way to delimit log events.
> > > > >
> > > > >
> > > > >
> > > > > On 2017-07-20 22:28, Matt Sicker wrote:
> > > > >
> > > > >> I'm not a fan of the null termination as it is. Do you know how
> many
> > > > >> millions or billions of dollars have been wasted due to buggy C
> and
> > > C++
> > > > >> programs thanks to null-terminated strings? Almost as bad as null
> > > > itself.
> > > > >> ;)
> > > > >>
> > > > >> Having some way to delimit log events could be handy later on for
> > > > >> garbage-free networked streaming appenders that want to break up
> log
> > > > >> events
> > > > >> into individual messages as opposed to how the file-type streaming
> > > > >> appenders work.
> > > > >>
> > > > >> On 20 July 2017 at 14:53, Mikael Ståldal <mi...@apache.org>
> wrote:
> > > > >>
> > > > >> Now JsonLayout, XmlLayout, YamlLayout and GelfLayout has an
> > > > >>> includeNullDelimiter option to terminate each log event with a
> > > 0-byte.
> > > > >>>
> > > > >>> Would it be useful to have the same in any other layout?
> > > > >>>
> > > > >>> I could think of HtmlLayout, SyslogLayout and RFC5424Layout.
> > > > >>>
> > > > >>
> > > >
> > > >
> > > > --
> > > > Matt Sicker <bo...@gmail.com>
> > > >
> > >
> >
> >
> >
> > --
> > Matt Sicker <bo...@gmail.com>
> >
>



-- 
Matt Sicker <bo...@gmail.com>

Re: [Log4j] includeNullDelimiter

[Gary Gregory <ga...@gmail.com>]

The trick is to define "length": what is the size and endianness for this
integer and are either configurable? I think JMeter has some code for that.

But at that point, why not use the BSON for uint64 (little-endian ?

Gary

On Fri, Jul 21, 2017 at 12:43 PM, Matt Sicker <bo...@gmail.com> wrote:

> Exactly! I've seen similar encodings used in simple TCP framing. It should
> be robust enough for logging (no need for additional metadata).
>
> On 21 July 2017 at 14:39, Gary Gregory <ga...@gmail.com> wrote:
>
> > Like Pascal strings, lengh-data-length-data...
> >
> > On Fri, Jul 21, 2017 at 11:31 AM, Matt Sicker <bo...@gmail.com> wrote:
> >
> > > If the size of the message is known ahead of time, you can create a
> > simple
> > > framing protocol around messages as well.
> > >
> > > On 21 July 2017 at 07:23, Mikael Ståldal <mi...@apache.org> wrote:
> > >
> > > > I don't quite understand.
> > > >
> > > > This is something which is needed now. When you use a streaming
> > appender
> > > > (Console, any File or Socket with protocol=tcp/ssl), the layout needs
> > to
> > > > provide some way to delimit log events.
> > > >
> > > >
> > > >
> > > > On 2017-07-20 22:28, Matt Sicker wrote:
> > > >
> > > >> I'm not a fan of the null termination as it is. Do you know how many
> > > >> millions or billions of dollars have been wasted due to buggy C and
> > C++
> > > >> programs thanks to null-terminated strings? Almost as bad as null
> > > itself.
> > > >> ;)
> > > >>
> > > >> Having some way to delimit log events could be handy later on for
> > > >> garbage-free networked streaming appenders that want to break up log
> > > >> events
> > > >> into individual messages as opposed to how the file-type streaming
> > > >> appenders work.
> > > >>
> > > >> On 20 July 2017 at 14:53, Mikael Ståldal <mi...@apache.org> wrote:
> > > >>
> > > >> Now JsonLayout, XmlLayout, YamlLayout and GelfLayout has an
> > > >>> includeNullDelimiter option to terminate each log event with a
> > 0-byte.
> > > >>>
> > > >>> Would it be useful to have the same in any other layout?
> > > >>>
> > > >>> I could think of HtmlLayout, SyslogLayout and RFC5424Layout.
> > > >>>
> > > >>
> > >
> > >
> > > --
> > > Matt Sicker <bo...@gmail.com>
> > >
> >
>
>
>
> --
> Matt Sicker <bo...@gmail.com>
>

Re: [Log4j] includeNullDelimiter

[Matt Sicker <bo...@gmail.com>]

Exactly! I've seen similar encodings used in simple TCP framing. It should
be robust enough for logging (no need for additional metadata).

On 21 July 2017 at 14:39, Gary Gregory <ga...@gmail.com> wrote:

> Like Pascal strings, lengh-data-length-data...
>
> On Fri, Jul 21, 2017 at 11:31 AM, Matt Sicker <bo...@gmail.com> wrote:
>
> > If the size of the message is known ahead of time, you can create a
> simple
> > framing protocol around messages as well.
> >
> > On 21 July 2017 at 07:23, Mikael Ståldal <mi...@apache.org> wrote:
> >
> > > I don't quite understand.
> > >
> > > This is something which is needed now. When you use a streaming
> appender
> > > (Console, any File or Socket with protocol=tcp/ssl), the layout needs
> to
> > > provide some way to delimit log events.
> > >
> > >
> > >
> > > On 2017-07-20 22:28, Matt Sicker wrote:
> > >
> > >> I'm not a fan of the null termination as it is. Do you know how many
> > >> millions or billions of dollars have been wasted due to buggy C and
> C++
> > >> programs thanks to null-terminated strings? Almost as bad as null
> > itself.
> > >> ;)
> > >>
> > >> Having some way to delimit log events could be handy later on for
> > >> garbage-free networked streaming appenders that want to break up log
> > >> events
> > >> into individual messages as opposed to how the file-type streaming
> > >> appenders work.
> > >>
> > >> On 20 July 2017 at 14:53, Mikael Ståldal <mi...@apache.org> wrote:
> > >>
> > >> Now JsonLayout, XmlLayout, YamlLayout and GelfLayout has an
> > >>> includeNullDelimiter option to terminate each log event with a
> 0-byte.
> > >>>
> > >>> Would it be useful to have the same in any other layout?
> > >>>
> > >>> I could think of HtmlLayout, SyslogLayout and RFC5424Layout.
> > >>>
> > >>
> >
> >
> > --
> > Matt Sicker <bo...@gmail.com>
> >
>



-- 
Matt Sicker <bo...@gmail.com>

Re: [Log4j] includeNullDelimiter

[Gary Gregory <ga...@gmail.com>]

Like Pascal strings, lengh-data-length-data...

On Fri, Jul 21, 2017 at 11:31 AM, Matt Sicker <bo...@gmail.com> wrote:

> If the size of the message is known ahead of time, you can create a simple
> framing protocol around messages as well.
>
> On 21 July 2017 at 07:23, Mikael Ståldal <mi...@apache.org> wrote:
>
> > I don't quite understand.
> >
> > This is something which is needed now. When you use a streaming appender
> > (Console, any File or Socket with protocol=tcp/ssl), the layout needs to
> > provide some way to delimit log events.
> >
> >
> >
> > On 2017-07-20 22:28, Matt Sicker wrote:
> >
> >> I'm not a fan of the null termination as it is. Do you know how many
> >> millions or billions of dollars have been wasted due to buggy C and C++
> >> programs thanks to null-terminated strings? Almost as bad as null
> itself.
> >> ;)
> >>
> >> Having some way to delimit log events could be handy later on for
> >> garbage-free networked streaming appenders that want to break up log
> >> events
> >> into individual messages as opposed to how the file-type streaming
> >> appenders work.
> >>
> >> On 20 July 2017 at 14:53, Mikael Ståldal <mi...@apache.org> wrote:
> >>
> >> Now JsonLayout, XmlLayout, YamlLayout and GelfLayout has an
> >>> includeNullDelimiter option to terminate each log event with a 0-byte.
> >>>
> >>> Would it be useful to have the same in any other layout?
> >>>
> >>> I could think of HtmlLayout, SyslogLayout and RFC5424Layout.
> >>>
> >>
>
>
> --
> Matt Sicker <bo...@gmail.com>
>

Re: [Log4j] includeNullDelimiter

[Matt Sicker <bo...@gmail.com>]

If the size of the message is known ahead of time, you can create a simple
framing protocol around messages as well.

On 21 July 2017 at 07:23, Mikael Ståldal <mi...@apache.org> wrote:

> I don't quite understand.
>
> This is something which is needed now. When you use a streaming appender
> (Console, any File or Socket with protocol=tcp/ssl), the layout needs to
> provide some way to delimit log events.
>
>
>
> On 2017-07-20 22:28, Matt Sicker wrote:
>
>> I'm not a fan of the null termination as it is. Do you know how many
>> millions or billions of dollars have been wasted due to buggy C and C++
>> programs thanks to null-terminated strings? Almost as bad as null itself.
>> ;)
>>
>> Having some way to delimit log events could be handy later on for
>> garbage-free networked streaming appenders that want to break up log
>> events
>> into individual messages as opposed to how the file-type streaming
>> appenders work.
>>
>> On 20 July 2017 at 14:53, Mikael Ståldal <mi...@apache.org> wrote:
>>
>> Now JsonLayout, XmlLayout, YamlLayout and GelfLayout has an
>>> includeNullDelimiter option to terminate each log event with a 0-byte.
>>>
>>> Would it be useful to have the same in any other layout?
>>>
>>> I could think of HtmlLayout, SyslogLayout and RFC5424Layout.
>>>
>>


-- 
Matt Sicker <bo...@gmail.com>

Re: [Log4j] includeNullDelimiter

[Mikael Ståldal <mi...@apache.org>]

I don't quite understand.

This is something which is needed now. When you use a streaming appender 
(Console, any File or Socket with protocol=tcp/ssl), the layout needs to 
provide some way to delimit log events.


On 2017-07-20 22:28, Matt Sicker wrote:
> I'm not a fan of the null termination as it is. Do you know how many
> millions or billions of dollars have been wasted due to buggy C and C++
> programs thanks to null-terminated strings? Almost as bad as null itself. ;)
> 
> Having some way to delimit log events could be handy later on for
> garbage-free networked streaming appenders that want to break up log events
> into individual messages as opposed to how the file-type streaming
> appenders work.
> 
> On 20 July 2017 at 14:53, Mikael Ståldal <mi...@apache.org> wrote:
> 
>> Now JsonLayout, XmlLayout, YamlLayout and GelfLayout has an
>> includeNullDelimiter option to terminate each log event with a 0-byte.
>>
>> Would it be useful to have the same in any other layout?
>>
>> I could think of HtmlLayout, SyslogLayout and RFC5424Layout.

Re: [Log4j] includeNullDelimiter

[Matt Sicker <bo...@gmail.com>]

I'm not a fan of the null termination as it is. Do you know how many
millions or billions of dollars have been wasted due to buggy C and C++
programs thanks to null-terminated strings? Almost as bad as null itself. ;)

Having some way to delimit log events could be handy later on for
garbage-free networked streaming appenders that want to break up log events
into individual messages as opposed to how the file-type streaming
appenders work.

On 20 July 2017 at 14:53, Mikael Ståldal <mi...@apache.org> wrote:

> Now JsonLayout, XmlLayout, YamlLayout and GelfLayout has an
> includeNullDelimiter option to terminate each log event with a 0-byte.
>
> Would it be useful to have the same in any other layout?
>
> I could think of HtmlLayout, SyslogLayout and RFC5424Layout.
>



-- 
Matt Sicker <bo...@gmail.com>