You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Nick Carenza <ni...@thecontrolgroup.com> on 2017/08/22 23:11:20 UTC
Problems with ListenBeats 1.3.0
I am trying to cut Logstash out of my stack because i have moved all of its
functionality into Nifi but I haven't been able to get Filebeat to play
nicely with the ListenBeats processor so I have to keep Logstash around
just to forward messages to Nifi.
I am using latest stable 5.5.2 but the ListenBeats processor throws these
errors and gets yielded for 30s in an endless loop.
2017-08-22 22:26:53,183 ERROR [pool-812781-thread-29]
> o.a.nifi.processors.beats.ListenBeats
> ListenBeats[id=9e14179f-101f-1159-4cf1-738e9a425a6e] Error reading Beats
> frames due to Error decoding Beats frame: Error decompressing frame:
> Unexpected end of ZLIB input stream:
> org.apache.nifi.processors.beats.frame.BeatsFrameException: Error decoding
> Beats frame: Error decompressing frame: Unexpected end of ZLIB input stream
>
> org.apache.nifi.processors.beats.frame.BeatsFrameException: Error decoding
> Beats frame: Error decompressing frame: Unexpected end of ZLIB input stream
>
> at
> org.apache.nifi.processors.beats.frame.BeatsDecoder.process(BeatsDecoder.java:123)
>
> at
> org.apache.nifi.processors.beats.handler.BeatsSocketChannelHandler.processBuffer(BeatsSocketChannelHandler.java:71)
>
> Caused by: org.apache.nifi.processors.beats.frame.BeatsFrameException:
> Error decompressing frame: Unexpected end of ZLIB input stream
>
> at
> org.apache.nifi.processors.beats.frame.BeatsDecoder.processPAYLOAD(BeatsDecoder.java:292)
>
> at
> org.apache.nifi.processors.beats.frame.BeatsDecoder.process(BeatsDecoder.java:103)
>
> at
> org.apache.nifi.processors.beats.frame.BeatsDecoder.processPAYLOAD(BeatsDecoder.java:277)
>
while filebeat is logging
Aug 22 22:28:57 app-server /usr/share/filebeat/bin/filebeat[15883]:
> sync.go:85: Failed to publish events caused by: read tcp
> localhost:45416->remotehost:5044: i/o timeout
Aug 22 22:28:57 app-server /usr/share/filebeat/bin/filebeat[15883]:
> single.go:91: Error publishing events (retrying): read tcp
> localhost:45416-> remotehost:5044: i/o timeout
Aug 22 22:29:26 app-server /usr/share/filebeat/bin/filebeat[15883]:
> metrics.go:39: Non-zero metrics in the last 30s:
> libbeat.logstash.call_count.PublishEvents=1
> libbeat.logstash.publish.read_bytes=186
> libbeat.logstash.publish.read_errors=1
> libbeat.logstash.publish.write_bytes=31399
> libbeat.logstash.published_and_acked_events=98
> libbeat.logstash.published_but_not_acked_events=1950
Aug 22 22:29:28 app-server /usr/share/filebeat/bin/filebeat[15883]:
> sync.go:85: Failed to publish events caused by: read tcp
> localhost:49790-> remotehost:5044: i/o timeout
Aug 22 22:29:28 app-server /usr/share/filebeat/bin/filebeat[15883]:
> single.go:91: Error publishing events (retrying): read tcp
> localhost:49790-> remotehost:5044: i/o timeout
Aug 22 22:29:56 app-server /usr/share/filebeat/bin/filebeat[15883]:
> metrics.go:39: Non-zero metrics in the last 30s:
> libbeat.logstash.call_count.PublishEvents=1
> libbeat.logstash.publish.read_bytes=138
> libbeat.logstash.publish.read_errors=1
> libbeat.logstash.publish.write_bytes=27456
> libbeat.logstash.published_and_acked_events=31
> libbeat.logstash.published_but_not_acked_events=1919
Aug 22 22:29:59 app-server /usr/share/filebeat/bin/filebeat[15883]:
> sync.go:85: Failed to publish events caused by: read tcp
> localhost:54060-> remotehost:5044: i/o timeout
Aug 22 22:29:59 app-server /usr/share/filebeat/bin/filebeat[15883]:
> single.go:91: Error publishing events (retrying): read tcp
> localhost:54060-> remotehost:5044: i/o timeout
Aug 22 22:30:26 app-server /usr/share/filebeat/bin/filebeat[15883]:
> metrics.go:39: Non-zero metrics in the last 30s:
> libbeat.logstash.call_count.PublishEvents=1
> libbeat.logstash.publish.read_bytes=492
> libbeat.logstash.publish.read_errors=1
> libbeat.logstash.publish.write_bytes=63110
> libbeat.logstash.published_and_acked_events=23
> libbeat.logstash.published_but_not_acked_events=1896
Aug 22 22:30:30 app-server /usr/share/filebeat/bin/filebeat[15883]:
> sync.go:85: Failed to publish events caused by: read tcp
> localhost:58359-> remotehost:5044: i/o timeout
Aug 22 22:30:30 app-server /usr/share/filebeat/bin/filebeat[15883]:
> single.go:91: Error publishing events (retrying): read tcp
> localhost:58359-> remotehost:5044: i/o timeout
has anyone seen something similar or have any idea what might be going on
here?
my processor params are:
Port
5044
Receive Buffer Size
10 MB
Max Size of Message Queue
10000
Max Size of Socket Buffer
8 MB
Character Set
UTF-8
Max Batch Size
1
Batching Message Delimiter
\n
Max Number of TCP Connections
100
SSL Context Service
No value set
Thanks,
Nick
Re: Problems with ListenBeats 1.3.0
Posted by Andre <an...@fucs.org>.
Nick,
I use it with:
filebeat:
spool_size: 2048
output.logstash:
compression_level: 2
bulk_max_size: 100
timeout: 30
On the NiFi side I have:
Receive Buffer Size: 128KB
Max Size of Message Queue: 10000
Max Size of Socket Buffer: 2MB
Can you try and let me know? Perhaps something changed on the filebeat size
that accentuated the issue. If that's the case I am happy to try to
prioritise fixing it.
Cheers
On Fri, Aug 25, 2017 at 5:08 AM, Nick Carenza <
nick.carenza@thecontrolgroup.com> wrote:
> It seemed to hold up a little bit better but I ended up getting the same
> errors.
>
> On Thu, Aug 24, 2017 at 11:35 AM, Nick Carenza <
> nick.carenza@thecontrolgroup.com> wrote:
>
>> Okay, thank you Andre!
>>
>> - Nick
>>
>> On Wed, Aug 23, 2017 at 4:03 PM, Andre <an...@fucs.org> wrote:
>>
>>> Nick,
>>>
>>> This is a bug (NIFI-3753). I am also able to confirm there is a
>>> workaround: just change the compression settings to a lower compression
>>> level (I believe anything between none and 3 should work) and/or reduce the
>>> number of messages batched on a single beats payload.
>>>
>>> Cheers
>>>
>>> On 23 Aug 2017 09:11, "Nick Carenza" <ni...@thecontrolgroup.com>
>>> wrote:
>>>
>>>> I am trying to cut Logstash out of my stack because i have moved all of
>>>> its functionality into Nifi but I haven't been able to get Filebeat to play
>>>> nicely with the ListenBeats processor so I have to keep Logstash around
>>>> just to forward messages to Nifi.
>>>>
>>>> I am using latest stable 5.5.2 but the ListenBeats processor throws
>>>> these errors and gets yielded for 30s in an endless loop.
>>>>
>>>> 2017-08-22 22:26:53,183 ERROR [pool-812781-thread-29]
>>>>> o.a.nifi.processors.beats.ListenBeats ListenBeats[id=9e14179f-101f-1159-4cf1-738e9a425a6e]
>>>>> Error reading Beats frames due to Error decoding Beats frame: Error
>>>>> decompressing frame: Unexpected end of ZLIB input stream:
>>>>> org.apache.nifi.processors.beats.frame.BeatsFrameException: Error
>>>>> decoding Beats frame: Error decompressing frame: Unexpected end of ZLIB
>>>>> input stream
>>>>>
>>>>> org.apache.nifi.processors.beats.frame.BeatsFrameException: Error
>>>>> decoding Beats frame: Error decompressing frame: Unexpected end of ZLIB
>>>>> input stream
>>>>>
>>>>> at org.apache.nifi.processors.beats.frame.BeatsDecoder.process(
>>>>> BeatsDecoder.java:123)
>>>>>
>>>>> at org.apache.nifi.processors.beats.handler.BeatsSocketChannelH
>>>>> andler.processBuffer(BeatsSocketChannelHandler.java:71)
>>>>>
>>>>> Caused by: org.apache.nifi.processors.beats.frame.BeatsFrameException:
>>>>> Error decompressing frame: Unexpected end of ZLIB input stream
>>>>>
>>>>> at org.apache.nifi.processors.beats.frame.BeatsDecoder.processP
>>>>> AYLOAD(BeatsDecoder.java:292)
>>>>>
>>>>> at org.apache.nifi.processors.beats.frame.BeatsDecoder.process(
>>>>> BeatsDecoder.java:103)
>>>>>
>>>>> at org.apache.nifi.processors.beats.frame.BeatsDecoder.processP
>>>>> AYLOAD(BeatsDecoder.java:277)
>>>>>
>>>> while filebeat is logging
>>>>
>>>>
>>>>
>>>> Aug 22 22:28:57 app-server /usr/share/filebeat/bin/filebeat[15883]:
>>>>> sync.go:85: Failed to publish events caused by: read tcp
>>>>> localhost:45416->remotehost:5044: i/o timeout
>>>>
>>>> Aug 22 22:28:57 app-server /usr/share/filebeat/bin/filebeat[15883]:
>>>>> single.go:91: Error publishing events (retrying): read tcp
>>>>> localhost:45416-> remotehost:5044: i/o timeout
>>>>
>>>> Aug 22 22:29:26 app-server /usr/share/filebeat/bin/filebeat[15883]:
>>>>> metrics.go:39: Non-zero metrics in the last 30s:
>>>>> libbeat.logstash.call_count.PublishEvents=1
>>>>> libbeat.logstash.publish.read_bytes=186 libbeat.logstash.publish.read_errors=1
>>>>> libbeat.logstash.publish.write_bytes=31399
>>>>> libbeat.logstash.published_and_acked_events=98
>>>>> libbeat.logstash.published_but_not_acked_events=1950
>>>>
>>>> Aug 22 22:29:28 app-server /usr/share/filebeat/bin/filebeat[15883]:
>>>>> sync.go:85: Failed to publish events caused by: read tcp
>>>>> localhost:49790-> remotehost:5044: i/o timeout
>>>>
>>>> Aug 22 22:29:28 app-server /usr/share/filebeat/bin/filebeat[15883]:
>>>>> single.go:91: Error publishing events (retrying): read tcp
>>>>> localhost:49790-> remotehost:5044: i/o timeout
>>>>
>>>> Aug 22 22:29:56 app-server /usr/share/filebeat/bin/filebeat[15883]:
>>>>> metrics.go:39: Non-zero metrics in the last 30s:
>>>>> libbeat.logstash.call_count.PublishEvents=1
>>>>> libbeat.logstash.publish.read_bytes=138 libbeat.logstash.publish.read_errors=1
>>>>> libbeat.logstash.publish.write_bytes=27456
>>>>> libbeat.logstash.published_and_acked_events=31
>>>>> libbeat.logstash.published_but_not_acked_events=1919
>>>>
>>>> Aug 22 22:29:59 app-server /usr/share/filebeat/bin/filebeat[15883]:
>>>>> sync.go:85: Failed to publish events caused by: read tcp
>>>>> localhost:54060-> remotehost:5044: i/o timeout
>>>>
>>>> Aug 22 22:29:59 app-server /usr/share/filebeat/bin/filebeat[15883]:
>>>>> single.go:91: Error publishing events (retrying): read tcp
>>>>> localhost:54060-> remotehost:5044: i/o timeout
>>>>
>>>> Aug 22 22:30:26 app-server /usr/share/filebeat/bin/filebeat[15883]:
>>>>> metrics.go:39: Non-zero metrics in the last 30s:
>>>>> libbeat.logstash.call_count.PublishEvents=1
>>>>> libbeat.logstash.publish.read_bytes=492 libbeat.logstash.publish.read_errors=1
>>>>> libbeat.logstash.publish.write_bytes=63110
>>>>> libbeat.logstash.published_and_acked_events=23
>>>>> libbeat.logstash.published_but_not_acked_events=1896
>>>>
>>>> Aug 22 22:30:30 app-server /usr/share/filebeat/bin/filebeat[15883]:
>>>>> sync.go:85: Failed to publish events caused by: read tcp
>>>>> localhost:58359-> remotehost:5044: i/o timeout
>>>>
>>>> Aug 22 22:30:30 app-server /usr/share/filebeat/bin/filebeat[15883]:
>>>>> single.go:91: Error publishing events (retrying): read tcp
>>>>> localhost:58359-> remotehost:5044: i/o timeout
>>>>
>>>>
>>>> has anyone seen something similar or have any idea what might be going
>>>> on here?
>>>>
>>>> my processor params are:
>>>>
>>>>
>>>> Port
>>>> 5044
>>>>
>>>> Receive Buffer Size
>>>> 10 MB
>>>>
>>>> Max Size of Message Queue
>>>> 10000
>>>>
>>>> Max Size of Socket Buffer
>>>> 8 MB
>>>>
>>>> Character Set
>>>> UTF-8
>>>>
>>>> Max Batch Size
>>>> 1
>>>>
>>>> Batching Message Delimiter
>>>> \n
>>>>
>>>> Max Number of TCP Connections
>>>> 100
>>>>
>>>> SSL Context Service
>>>> No value set
>>>>
>>>> Thanks,
>>>> Nick
>>>>
>>>
>>
>
Re: Problems with ListenBeats 1.3.0
Posted by Nick Carenza <ni...@thecontrolgroup.com>.
It seemed to hold up a little bit better but I ended up getting the same
errors.
On Thu, Aug 24, 2017 at 11:35 AM, Nick Carenza <
nick.carenza@thecontrolgroup.com> wrote:
> Okay, thank you Andre!
>
> - Nick
>
> On Wed, Aug 23, 2017 at 4:03 PM, Andre <an...@fucs.org> wrote:
>
>> Nick,
>>
>> This is a bug (NIFI-3753). I am also able to confirm there is a
>> workaround: just change the compression settings to a lower compression
>> level (I believe anything between none and 3 should work) and/or reduce the
>> number of messages batched on a single beats payload.
>>
>> Cheers
>>
>> On 23 Aug 2017 09:11, "Nick Carenza" <ni...@thecontrolgroup.com>
>> wrote:
>>
>>> I am trying to cut Logstash out of my stack because i have moved all of
>>> its functionality into Nifi but I haven't been able to get Filebeat to play
>>> nicely with the ListenBeats processor so I have to keep Logstash around
>>> just to forward messages to Nifi.
>>>
>>> I am using latest stable 5.5.2 but the ListenBeats processor throws
>>> these errors and gets yielded for 30s in an endless loop.
>>>
>>> 2017-08-22 22:26:53,183 ERROR [pool-812781-thread-29]
>>>> o.a.nifi.processors.beats.ListenBeats ListenBeats[id=9e14179f-101f-1159-4cf1-738e9a425a6e]
>>>> Error reading Beats frames due to Error decoding Beats frame: Error
>>>> decompressing frame: Unexpected end of ZLIB input stream:
>>>> org.apache.nifi.processors.beats.frame.BeatsFrameException: Error
>>>> decoding Beats frame: Error decompressing frame: Unexpected end of ZLIB
>>>> input stream
>>>>
>>>> org.apache.nifi.processors.beats.frame.BeatsFrameException: Error
>>>> decoding Beats frame: Error decompressing frame: Unexpected end of ZLIB
>>>> input stream
>>>>
>>>> at org.apache.nifi.processors.beats.frame.BeatsDecoder.process(
>>>> BeatsDecoder.java:123)
>>>>
>>>> at org.apache.nifi.processors.beats.handler.BeatsSocketChannelH
>>>> andler.processBuffer(BeatsSocketChannelHandler.java:71)
>>>>
>>>> Caused by: org.apache.nifi.processors.beats.frame.BeatsFrameException:
>>>> Error decompressing frame: Unexpected end of ZLIB input stream
>>>>
>>>> at org.apache.nifi.processors.beats.frame.BeatsDecoder.processP
>>>> AYLOAD(BeatsDecoder.java:292)
>>>>
>>>> at org.apache.nifi.processors.beats.frame.BeatsDecoder.process(
>>>> BeatsDecoder.java:103)
>>>>
>>>> at org.apache.nifi.processors.beats.frame.BeatsDecoder.processP
>>>> AYLOAD(BeatsDecoder.java:277)
>>>>
>>> while filebeat is logging
>>>
>>>
>>>
>>> Aug 22 22:28:57 app-server /usr/share/filebeat/bin/filebeat[15883]:
>>>> sync.go:85: Failed to publish events caused by: read tcp
>>>> localhost:45416->remotehost:5044: i/o timeout
>>>
>>> Aug 22 22:28:57 app-server /usr/share/filebeat/bin/filebeat[15883]:
>>>> single.go:91: Error publishing events (retrying): read tcp
>>>> localhost:45416-> remotehost:5044: i/o timeout
>>>
>>> Aug 22 22:29:26 app-server /usr/share/filebeat/bin/filebeat[15883]:
>>>> metrics.go:39: Non-zero metrics in the last 30s:
>>>> libbeat.logstash.call_count.PublishEvents=1
>>>> libbeat.logstash.publish.read_bytes=186 libbeat.logstash.publish.read_errors=1
>>>> libbeat.logstash.publish.write_bytes=31399
>>>> libbeat.logstash.published_and_acked_events=98
>>>> libbeat.logstash.published_but_not_acked_events=1950
>>>
>>> Aug 22 22:29:28 app-server /usr/share/filebeat/bin/filebeat[15883]:
>>>> sync.go:85: Failed to publish events caused by: read tcp
>>>> localhost:49790-> remotehost:5044: i/o timeout
>>>
>>> Aug 22 22:29:28 app-server /usr/share/filebeat/bin/filebeat[15883]:
>>>> single.go:91: Error publishing events (retrying): read tcp
>>>> localhost:49790-> remotehost:5044: i/o timeout
>>>
>>> Aug 22 22:29:56 app-server /usr/share/filebeat/bin/filebeat[15883]:
>>>> metrics.go:39: Non-zero metrics in the last 30s:
>>>> libbeat.logstash.call_count.PublishEvents=1
>>>> libbeat.logstash.publish.read_bytes=138 libbeat.logstash.publish.read_errors=1
>>>> libbeat.logstash.publish.write_bytes=27456
>>>> libbeat.logstash.published_and_acked_events=31
>>>> libbeat.logstash.published_but_not_acked_events=1919
>>>
>>> Aug 22 22:29:59 app-server /usr/share/filebeat/bin/filebeat[15883]:
>>>> sync.go:85: Failed to publish events caused by: read tcp
>>>> localhost:54060-> remotehost:5044: i/o timeout
>>>
>>> Aug 22 22:29:59 app-server /usr/share/filebeat/bin/filebeat[15883]:
>>>> single.go:91: Error publishing events (retrying): read tcp
>>>> localhost:54060-> remotehost:5044: i/o timeout
>>>
>>> Aug 22 22:30:26 app-server /usr/share/filebeat/bin/filebeat[15883]:
>>>> metrics.go:39: Non-zero metrics in the last 30s:
>>>> libbeat.logstash.call_count.PublishEvents=1
>>>> libbeat.logstash.publish.read_bytes=492 libbeat.logstash.publish.read_errors=1
>>>> libbeat.logstash.publish.write_bytes=63110
>>>> libbeat.logstash.published_and_acked_events=23
>>>> libbeat.logstash.published_but_not_acked_events=1896
>>>
>>> Aug 22 22:30:30 app-server /usr/share/filebeat/bin/filebeat[15883]:
>>>> sync.go:85: Failed to publish events caused by: read tcp
>>>> localhost:58359-> remotehost:5044: i/o timeout
>>>
>>> Aug 22 22:30:30 app-server /usr/share/filebeat/bin/filebeat[15883]:
>>>> single.go:91: Error publishing events (retrying): read tcp
>>>> localhost:58359-> remotehost:5044: i/o timeout
>>>
>>>
>>> has anyone seen something similar or have any idea what might be going
>>> on here?
>>>
>>> my processor params are:
>>>
>>>
>>> Port
>>> 5044
>>>
>>> Receive Buffer Size
>>> 10 MB
>>>
>>> Max Size of Message Queue
>>> 10000
>>>
>>> Max Size of Socket Buffer
>>> 8 MB
>>>
>>> Character Set
>>> UTF-8
>>>
>>> Max Batch Size
>>> 1
>>>
>>> Batching Message Delimiter
>>> \n
>>>
>>> Max Number of TCP Connections
>>> 100
>>>
>>> SSL Context Service
>>> No value set
>>>
>>> Thanks,
>>> Nick
>>>
>>
>
Re: Problems with ListenBeats 1.3.0
Posted by Nick Carenza <ni...@thecontrolgroup.com>.
Okay, thank you Andre!
- Nick
On Wed, Aug 23, 2017 at 4:03 PM, Andre <an...@fucs.org> wrote:
> Nick,
>
> This is a bug (NIFI-3753). I am also able to confirm there is a
> workaround: just change the compression settings to a lower compression
> level (I believe anything between none and 3 should work) and/or reduce the
> number of messages batched on a single beats payload.
>
> Cheers
>
> On 23 Aug 2017 09:11, "Nick Carenza" <ni...@thecontrolgroup.com>
> wrote:
>
>> I am trying to cut Logstash out of my stack because i have moved all of
>> its functionality into Nifi but I haven't been able to get Filebeat to play
>> nicely with the ListenBeats processor so I have to keep Logstash around
>> just to forward messages to Nifi.
>>
>> I am using latest stable 5.5.2 but the ListenBeats processor throws these
>> errors and gets yielded for 30s in an endless loop.
>>
>> 2017-08-22 22:26:53,183 ERROR [pool-812781-thread-29]
>>> o.a.nifi.processors.beats.ListenBeats ListenBeats[id=9e14179f-101f-1159-4cf1-738e9a425a6e]
>>> Error reading Beats frames due to Error decoding Beats frame: Error
>>> decompressing frame: Unexpected end of ZLIB input stream:
>>> org.apache.nifi.processors.beats.frame.BeatsFrameException: Error
>>> decoding Beats frame: Error decompressing frame: Unexpected end of ZLIB
>>> input stream
>>>
>>> org.apache.nifi.processors.beats.frame.BeatsFrameException: Error
>>> decoding Beats frame: Error decompressing frame: Unexpected end of ZLIB
>>> input stream
>>>
>>> at org.apache.nifi.processors.beats.frame.BeatsDecoder.process(
>>> BeatsDecoder.java:123)
>>>
>>> at org.apache.nifi.processors.beats.handler.BeatsSocketChannelH
>>> andler.processBuffer(BeatsSocketChannelHandler.java:71)
>>>
>>> Caused by: org.apache.nifi.processors.beats.frame.BeatsFrameException:
>>> Error decompressing frame: Unexpected end of ZLIB input stream
>>>
>>> at org.apache.nifi.processors.beats.frame.BeatsDecoder.processP
>>> AYLOAD(BeatsDecoder.java:292)
>>>
>>> at org.apache.nifi.processors.beats.frame.BeatsDecoder.process(
>>> BeatsDecoder.java:103)
>>>
>>> at org.apache.nifi.processors.beats.frame.BeatsDecoder.processP
>>> AYLOAD(BeatsDecoder.java:277)
>>>
>> while filebeat is logging
>>
>>
>>
>> Aug 22 22:28:57 app-server /usr/share/filebeat/bin/filebeat[15883]:
>>> sync.go:85: Failed to publish events caused by: read tcp
>>> localhost:45416->remotehost:5044: i/o timeout
>>
>> Aug 22 22:28:57 app-server /usr/share/filebeat/bin/filebeat[15883]:
>>> single.go:91: Error publishing events (retrying): read tcp
>>> localhost:45416-> remotehost:5044: i/o timeout
>>
>> Aug 22 22:29:26 app-server /usr/share/filebeat/bin/filebeat[15883]:
>>> metrics.go:39: Non-zero metrics in the last 30s:
>>> libbeat.logstash.call_count.PublishEvents=1
>>> libbeat.logstash.publish.read_bytes=186 libbeat.logstash.publish.read_errors=1
>>> libbeat.logstash.publish.write_bytes=31399
>>> libbeat.logstash.published_and_acked_events=98
>>> libbeat.logstash.published_but_not_acked_events=1950
>>
>> Aug 22 22:29:28 app-server /usr/share/filebeat/bin/filebeat[15883]:
>>> sync.go:85: Failed to publish events caused by: read tcp
>>> localhost:49790-> remotehost:5044: i/o timeout
>>
>> Aug 22 22:29:28 app-server /usr/share/filebeat/bin/filebeat[15883]:
>>> single.go:91: Error publishing events (retrying): read tcp
>>> localhost:49790-> remotehost:5044: i/o timeout
>>
>> Aug 22 22:29:56 app-server /usr/share/filebeat/bin/filebeat[15883]:
>>> metrics.go:39: Non-zero metrics in the last 30s:
>>> libbeat.logstash.call_count.PublishEvents=1
>>> libbeat.logstash.publish.read_bytes=138 libbeat.logstash.publish.read_errors=1
>>> libbeat.logstash.publish.write_bytes=27456
>>> libbeat.logstash.published_and_acked_events=31
>>> libbeat.logstash.published_but_not_acked_events=1919
>>
>> Aug 22 22:29:59 app-server /usr/share/filebeat/bin/filebeat[15883]:
>>> sync.go:85: Failed to publish events caused by: read tcp
>>> localhost:54060-> remotehost:5044: i/o timeout
>>
>> Aug 22 22:29:59 app-server /usr/share/filebeat/bin/filebeat[15883]:
>>> single.go:91: Error publishing events (retrying): read tcp
>>> localhost:54060-> remotehost:5044: i/o timeout
>>
>> Aug 22 22:30:26 app-server /usr/share/filebeat/bin/filebeat[15883]:
>>> metrics.go:39: Non-zero metrics in the last 30s:
>>> libbeat.logstash.call_count.PublishEvents=1
>>> libbeat.logstash.publish.read_bytes=492 libbeat.logstash.publish.read_errors=1
>>> libbeat.logstash.publish.write_bytes=63110
>>> libbeat.logstash.published_and_acked_events=23
>>> libbeat.logstash.published_but_not_acked_events=1896
>>
>> Aug 22 22:30:30 app-server /usr/share/filebeat/bin/filebeat[15883]:
>>> sync.go:85: Failed to publish events caused by: read tcp
>>> localhost:58359-> remotehost:5044: i/o timeout
>>
>> Aug 22 22:30:30 app-server /usr/share/filebeat/bin/filebeat[15883]:
>>> single.go:91: Error publishing events (retrying): read tcp
>>> localhost:58359-> remotehost:5044: i/o timeout
>>
>>
>> has anyone seen something similar or have any idea what might be going
>> on here?
>>
>> my processor params are:
>>
>>
>> Port
>> 5044
>>
>> Receive Buffer Size
>> 10 MB
>>
>> Max Size of Message Queue
>> 10000
>>
>> Max Size of Socket Buffer
>> 8 MB
>>
>> Character Set
>> UTF-8
>>
>> Max Batch Size
>> 1
>>
>> Batching Message Delimiter
>> \n
>>
>> Max Number of TCP Connections
>> 100
>>
>> SSL Context Service
>> No value set
>>
>> Thanks,
>> Nick
>>
>
Re: Problems with ListenBeats 1.3.0
Posted by Andre <an...@fucs.org>.
Nick,
This is a bug (NIFI-3753). I am also able to confirm there is a workaround:
just change the compression settings to a lower compression level (I
believe anything between none and 3 should work) and/or reduce the number
of messages batched on a single beats payload.
Cheers
On 23 Aug 2017 09:11, "Nick Carenza" <ni...@thecontrolgroup.com>
wrote:
> I am trying to cut Logstash out of my stack because i have moved all of
> its functionality into Nifi but I haven't been able to get Filebeat to play
> nicely with the ListenBeats processor so I have to keep Logstash around
> just to forward messages to Nifi.
>
> I am using latest stable 5.5.2 but the ListenBeats processor throws these
> errors and gets yielded for 30s in an endless loop.
>
> 2017-08-22 22:26:53,183 ERROR [pool-812781-thread-29]
>> o.a.nifi.processors.beats.ListenBeats ListenBeats[id=9e14179f-101f-1159-4cf1-738e9a425a6e]
>> Error reading Beats frames due to Error decoding Beats frame: Error
>> decompressing frame: Unexpected end of ZLIB input stream:
>> org.apache.nifi.processors.beats.frame.BeatsFrameException: Error
>> decoding Beats frame: Error decompressing frame: Unexpected end of ZLIB
>> input stream
>>
>> org.apache.nifi.processors.beats.frame.BeatsFrameException: Error
>> decoding Beats frame: Error decompressing frame: Unexpected end of ZLIB
>> input stream
>>
>> at org.apache.nifi.processors.beats.frame.BeatsDecoder.
>> process(BeatsDecoder.java:123)
>>
>> at org.apache.nifi.processors.beats.handler.BeatsSocketChannelHandler.
>> processBuffer(BeatsSocketChannelHandler.java:71)
>>
>> Caused by: org.apache.nifi.processors.beats.frame.BeatsFrameException:
>> Error decompressing frame: Unexpected end of ZLIB input stream
>>
>> at org.apache.nifi.processors.beats.frame.BeatsDecoder.
>> processPAYLOAD(BeatsDecoder.java:292)
>>
>> at org.apache.nifi.processors.beats.frame.BeatsDecoder.
>> process(BeatsDecoder.java:103)
>>
>> at org.apache.nifi.processors.beats.frame.BeatsDecoder.
>> processPAYLOAD(BeatsDecoder.java:277)
>>
> while filebeat is logging
>
>
>
> Aug 22 22:28:57 app-server /usr/share/filebeat/bin/filebeat[15883]:
>> sync.go:85: Failed to publish events caused by: read tcp
>> localhost:45416->remotehost:5044: i/o timeout
>
> Aug 22 22:28:57 app-server /usr/share/filebeat/bin/filebeat[15883]:
>> single.go:91: Error publishing events (retrying): read tcp
>> localhost:45416-> remotehost:5044: i/o timeout
>
> Aug 22 22:29:26 app-server /usr/share/filebeat/bin/filebeat[15883]:
>> metrics.go:39: Non-zero metrics in the last 30s:
>> libbeat.logstash.call_count.PublishEvents=1
>> libbeat.logstash.publish.read_bytes=186 libbeat.logstash.publish.read_errors=1
>> libbeat.logstash.publish.write_bytes=31399 libbeat.logstash.published_and_acked_events=98
>> libbeat.logstash.published_but_not_acked_events=1950
>
> Aug 22 22:29:28 app-server /usr/share/filebeat/bin/filebeat[15883]:
>> sync.go:85: Failed to publish events caused by: read tcp
>> localhost:49790-> remotehost:5044: i/o timeout
>
> Aug 22 22:29:28 app-server /usr/share/filebeat/bin/filebeat[15883]:
>> single.go:91: Error publishing events (retrying): read tcp
>> localhost:49790-> remotehost:5044: i/o timeout
>
> Aug 22 22:29:56 app-server /usr/share/filebeat/bin/filebeat[15883]:
>> metrics.go:39: Non-zero metrics in the last 30s:
>> libbeat.logstash.call_count.PublishEvents=1
>> libbeat.logstash.publish.read_bytes=138 libbeat.logstash.publish.read_errors=1
>> libbeat.logstash.publish.write_bytes=27456 libbeat.logstash.published_and_acked_events=31
>> libbeat.logstash.published_but_not_acked_events=1919
>
> Aug 22 22:29:59 app-server /usr/share/filebeat/bin/filebeat[15883]:
>> sync.go:85: Failed to publish events caused by: read tcp
>> localhost:54060-> remotehost:5044: i/o timeout
>
> Aug 22 22:29:59 app-server /usr/share/filebeat/bin/filebeat[15883]:
>> single.go:91: Error publishing events (retrying): read tcp
>> localhost:54060-> remotehost:5044: i/o timeout
>
> Aug 22 22:30:26 app-server /usr/share/filebeat/bin/filebeat[15883]:
>> metrics.go:39: Non-zero metrics in the last 30s:
>> libbeat.logstash.call_count.PublishEvents=1
>> libbeat.logstash.publish.read_bytes=492 libbeat.logstash.publish.read_errors=1
>> libbeat.logstash.publish.write_bytes=63110 libbeat.logstash.published_and_acked_events=23
>> libbeat.logstash.published_but_not_acked_events=1896
>
> Aug 22 22:30:30 app-server /usr/share/filebeat/bin/filebeat[15883]:
>> sync.go:85: Failed to publish events caused by: read tcp
>> localhost:58359-> remotehost:5044: i/o timeout
>
> Aug 22 22:30:30 app-server /usr/share/filebeat/bin/filebeat[15883]:
>> single.go:91: Error publishing events (retrying): read tcp
>> localhost:58359-> remotehost:5044: i/o timeout
>
>
> has anyone seen something similar or have any idea what might be going on
> here?
>
> my processor params are:
>
>
> Port
> 5044
>
> Receive Buffer Size
> 10 MB
>
> Max Size of Message Queue
> 10000
>
> Max Size of Socket Buffer
> 8 MB
>
> Character Set
> UTF-8
>
> Max Batch Size
> 1
>
> Batching Message Delimiter
> \n
>
> Max Number of TCP Connections
> 100
>
> SSL Context Service
> No value set
>
> Thanks,
> Nick
>