You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Shamsudeen Jameer <sh...@prth.com> on 2024/03/29 02:01:42 UTC
Hit\Miss Writing to Syslog system
Hello everyone,
I'm currently running Nifi version 1.8.0, starting from last week I noticed
that I'm missing log from my syslog system. Some logs would appear and
some wouldn't write to the syslog system. I checked my syslog system and
was unable to find any issue. No other systems are experiencing issue
writing to the syslog system. I enabled DEBUG on PutSysLog and it shows it
was successful in sending the message to the syslog system but no entry
exists. At the same time other PutSysLog does record the entries and it
flows through the process group. Any suggestions on how to troubleshoot
this issue?
Thanks!
--
PROPRIETARY
/ CONFIDENTIALITY NOTICE
This message (including any
attachments) contains information
that is proprietary and confidential to
PRIORITY TECHNOLOGY HOLDINGS, INC. and its
affiliates and subsidiaries,
including the sender hereof, and is for the sole
use of the intended
recipients. If you are not an intended recipient, you may
not read, print,
retain, use, copy, distribute, forward or disclose to anyone
this message
or any information contained in this message (including any
attachments).
If you have received this message in error, please advise the
sender of
this error by reply e-mail, and please destroy all copies of this
message
(including any attachments).
Re: Hit\Miss Writing to Syslog system
Posted by David Handermann <ex...@apache.org>.
Hi Shamsudeen,
Thanks for the reply and clarifying the use of the protocol. UDP does
not offer any kind of delivery guarantees at the protocol level, so it
is even less reliable than TCP for syslog. In other words, data loss
can happen without any type of error being logged.
If the data loss is frequent, a network packet monitor such as tcpdump
[1] is one tool for evaluating network communication.
Regards,
David Handermann
[1] https://www.tcpdump.org/
On Fri, Mar 29, 2024 at 8:44 AM Shamsudeen Jameer
<sh...@prth.com> wrote:
>
> Hi David,
> Thanks for the information. I'm actually using the UDP port to transfer logs to the syslog system. Planning to upgrade to TCP in the future but just need to figure out why it's hit \ miss.
>
> Regards,
> Shamsudeen Jameer
>
>
>
> On Thu, Mar 28, 2024 at 10:37 PM David Handermann <ex...@apache.org> wrote:
>>
>> Hi Shamsudeen,
>>
>> Although TCP incorporates packet acknowledgement as a core feature of
>> the protocol, syslog does not provide any type of delivery guarantees
>> on top of TCP. This means that it is possible for a system such as
>> NiFi to send a syslog message, without logging a socket communication
>> error, but for the message to be lost. This may not be common, but it
>> is a known possibility.
>>
>> The following post is from 2008, but the explanation on the
>> unreliability of syslog still applies:
>>
>> https://rainer.gerhards.net/2008/04/on-unreliability-of-plain-tcp-syslog.html
>>
>> There are several modern alternatives to syslog, with OpenTelemetry
>> Logging [1] as one example of a transport protocol that includes
>> delivery acknowledgement, providing much better reliability.
>>
>> Recent versions of NiFi have incorporated both improvements to syslog
>> processing and support for OpenTelemetry [2]. Upgrading to a more
>> recent version of NiFi would be a useful step, regardless of the
>> protocol used.
>>
>> Regards,
>> David Handermann
>>
>> [1] https://opentelemetry.io/docs/specs/otel/logs/
>> [2] https://exceptionfactory.com/posts/2024/02/26/building-opentelemetry-collection-in-apache-nifi-with-netty/
>>
>> On Thu, Mar 28, 2024 at 9:03 PM Shamsudeen Jameer
>> <sh...@prth.com> wrote:
>> >
>> > Hello everyone,
>> > I'm currently running Nifi version 1.8.0, starting from last week I noticed that I'm missing log from my syslog system. Some logs would appear and some wouldn't write to the syslog system. I checked my syslog system and was unable to find any issue. No other systems are experiencing issue writing to the syslog system. I enabled DEBUG on PutSysLog and it shows it was successful in sending the message to the syslog system but no entry exists. At the same time other PutSysLog does record the entries and it flows through the process group. Any suggestions on how to troubleshoot this issue?
>> > Thanks!
>> >
>> > PROPRIETARY / CONFIDENTIALITY NOTICE
>> >
>> > This message (including any attachments) contains information that is proprietary and confidential to PRIORITY TECHNOLOGY HOLDINGS, INC. and its affiliates and subsidiaries, including the sender hereof, and is for the sole use of the intended recipients. If you are not an intended recipient, you may not read, print, retain, use, copy, distribute, forward or disclose to anyone this message or any information contained in this message (including any attachments). If you have received this message in error, please advise the sender of this error by reply e-mail, and please destroy all copies of this message (including any attachments).
>
>
> PROPRIETARY / CONFIDENTIALITY NOTICE
>
> This message (including any attachments) contains information that is proprietary and confidential to PRIORITY TECHNOLOGY HOLDINGS, INC. and its affiliates and subsidiaries, including the sender hereof, and is for the sole use of the intended recipients. If you are not an intended recipient, you may not read, print, retain, use, copy, distribute, forward or disclose to anyone this message or any information contained in this message (including any attachments). If you have received this message in error, please advise the sender of this error by reply e-mail, and please destroy all copies of this message (including any attachments).
Re: Hit\Miss Writing to Syslog system
Posted by Shamsudeen Jameer <sh...@prth.com>.
Hi David,
Thanks for the information. I'm actually using the UDP port to transfer
logs to the syslog system. Planning to upgrade to TCP in the future but
just need to figure out why it's hit \ miss.
Regards,
*Shamsudeen Jameer*
On Thu, Mar 28, 2024 at 10:37 PM David Handermann <
exceptionfactory@apache.org> wrote:
> Hi Shamsudeen,
>
> Although TCP incorporates packet acknowledgement as a core feature of
> the protocol, syslog does not provide any type of delivery guarantees
> on top of TCP. This means that it is possible for a system such as
> NiFi to send a syslog message, without logging a socket communication
> error, but for the message to be lost. This may not be common, but it
> is a known possibility.
>
> The following post is from 2008, but the explanation on the
> unreliability of syslog still applies:
>
>
> https://rainer.gerhards.net/2008/04/on-unreliability-of-plain-tcp-syslog.html
>
> There are several modern alternatives to syslog, with OpenTelemetry
> Logging [1] as one example of a transport protocol that includes
> delivery acknowledgement, providing much better reliability.
>
> Recent versions of NiFi have incorporated both improvements to syslog
> processing and support for OpenTelemetry [2]. Upgrading to a more
> recent version of NiFi would be a useful step, regardless of the
> protocol used.
>
> Regards,
> David Handermann
>
> [1] https://opentelemetry.io/docs/specs/otel/logs/
> [2]
> https://exceptionfactory.com/posts/2024/02/26/building-opentelemetry-collection-in-apache-nifi-with-netty/
>
> On Thu, Mar 28, 2024 at 9:03 PM Shamsudeen Jameer
> <sh...@prth.com> wrote:
> >
> > Hello everyone,
> > I'm currently running Nifi version 1.8.0, starting from last week I
> noticed that I'm missing log from my syslog system. Some logs would appear
> and some wouldn't write to the syslog system. I checked my syslog system
> and was unable to find any issue. No other systems are experiencing issue
> writing to the syslog system. I enabled DEBUG on PutSysLog and it shows it
> was successful in sending the message to the syslog system but no entry
> exists. At the same time other PutSysLog does record the entries and it
> flows through the process group. Any suggestions on how to troubleshoot
> this issue?
> > Thanks!
> >
> > PROPRIETARY / CONFIDENTIALITY NOTICE
> >
> > This message (including any attachments) contains information that is
> proprietary and confidential to PRIORITY TECHNOLOGY HOLDINGS, INC. and its
> affiliates and subsidiaries, including the sender hereof, and is for the
> sole use of the intended recipients. If you are not an intended recipient,
> you may not read, print, retain, use, copy, distribute, forward or disclose
> to anyone this message or any information contained in this message
> (including any attachments). If you have received this message in error,
> please advise the sender of this error by reply e-mail, and please destroy
> all copies of this message (including any attachments).
>
--
PROPRIETARY
/ CONFIDENTIALITY NOTICE
This message (including any
attachments) contains information
that is proprietary and confidential to
PRIORITY TECHNOLOGY HOLDINGS, INC. and its
affiliates and subsidiaries,
including the sender hereof, and is for the sole
use of the intended
recipients. If you are not an intended recipient, you may
not read, print,
retain, use, copy, distribute, forward or disclose to anyone
this message
or any information contained in this message (including any
attachments).
If you have received this message in error, please advise the
sender of
this error by reply e-mail, and please destroy all copies of this
message
(including any attachments).
Re: Hit\Miss Writing to Syslog system
Posted by David Handermann <ex...@apache.org>.
Hi Shamsudeen,
Although TCP incorporates packet acknowledgement as a core feature of
the protocol, syslog does not provide any type of delivery guarantees
on top of TCP. This means that it is possible for a system such as
NiFi to send a syslog message, without logging a socket communication
error, but for the message to be lost. This may not be common, but it
is a known possibility.
The following post is from 2008, but the explanation on the
unreliability of syslog still applies:
https://rainer.gerhards.net/2008/04/on-unreliability-of-plain-tcp-syslog.html
There are several modern alternatives to syslog, with OpenTelemetry
Logging [1] as one example of a transport protocol that includes
delivery acknowledgement, providing much better reliability.
Recent versions of NiFi have incorporated both improvements to syslog
processing and support for OpenTelemetry [2]. Upgrading to a more
recent version of NiFi would be a useful step, regardless of the
protocol used.
Regards,
David Handermann
[1] https://opentelemetry.io/docs/specs/otel/logs/
[2] https://exceptionfactory.com/posts/2024/02/26/building-opentelemetry-collection-in-apache-nifi-with-netty/
On Thu, Mar 28, 2024 at 9:03 PM Shamsudeen Jameer
<sh...@prth.com> wrote:
>
> Hello everyone,
> I'm currently running Nifi version 1.8.0, starting from last week I noticed that I'm missing log from my syslog system. Some logs would appear and some wouldn't write to the syslog system. I checked my syslog system and was unable to find any issue. No other systems are experiencing issue writing to the syslog system. I enabled DEBUG on PutSysLog and it shows it was successful in sending the message to the syslog system but no entry exists. At the same time other PutSysLog does record the entries and it flows through the process group. Any suggestions on how to troubleshoot this issue?
> Thanks!
>
> PROPRIETARY / CONFIDENTIALITY NOTICE
>
> This message (including any attachments) contains information that is proprietary and confidential to PRIORITY TECHNOLOGY HOLDINGS, INC. and its affiliates and subsidiaries, including the sender hereof, and is for the sole use of the intended recipients. If you are not an intended recipient, you may not read, print, retain, use, copy, distribute, forward or disclose to anyone this message or any information contained in this message (including any attachments). If you have received this message in error, please advise the sender of this error by reply e-mail, and please destroy all copies of this message (including any attachments).