You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Breno Cesar <br...@gmail.com> on 2022/02/10 01:17:01 UTC
ListenTCP not generating flowfiles
i'm tring to receive a tcp stream using the listentcp module, but when the
connection start, i see the traffic reaching my server (usgin tcpdump), but
the nifi does note generate a flow file.
Ther is no error ow warning in the module or in the nifi cluster.
Changing in the processor the bulleting level to "debug" , i can see the
following message for every package that i receive:
SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000-0000-000040efa66e]
Accepted incoming connection from /192.0.2.12:55209
SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000-0000-000040efa66e] bytes
read 46
SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000-0000-000040efa66e] No more
data available, returning for selection
SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000-0000-000040efa66e] Reached
EOF, closing connection
bellow is the same message, but in the tcpdump format.
Internet Protocol Version 4, Src: 192.0.2.12, Dst: 192.0.2.128
Transmission Control Protocol, Src Port: 55209 , Dst Port: 4737, Seq: 1,
Ack: 1, Len: 46
Source Port: 55209
Destination Port: 4737
<Source or Destination Port: 55209 >
<Source or Destination Port: 4737>
[Stream index: 0]
[TCP Segment Len: 46]
Sequence number: 1 (relative sequence number)
Sequence number (raw): 3765145522
[Next sequence number: 47 (relative sequence number)]
Acknowledgment number: 1 (relative ack number)
Acknowledgment number (raw): 274707205
0101 .... = Header Length: 20 bytes (5)
Flags: 0x018 (PSH, ACK)
Window size value: 10000
[Calculated window size: 10000]
[Window size scaling factor: 1]
Checksum: 0x5fd7 [unverified]
[Checksum Status: Unverified]
Urgent pointer: 0
[SEQ/ACK analysis]
[Timestamps]
TCP payload (46 bytes)
[PDU Size: 46]
IPDR
Version: 2
Message id: CONNECT (5)
Session id: 0
Message flags: 0x00
Message length: 46
Initiator id: 0.0.0.0
Initiator port: 0
Capabilities: 0x00000006
Keep-alive interval: 90
Vendor id: ARRIS E6000 Exporter
Here somethings that i tried:
- Increase the number of simultaneos connections and batch size, but it
does not change at all.
- Use listentcprecord but the behavior is the same, it only change when i
select the avroreader, then i got an error informing that is a "wrong data
format".
So i belive that nifi is receving the messages that come from tcp , but for
some reason is not generating flowfiles.
Does anyone got same behavior ?
There is anything else that i can do ?
i'm using nifi 1.15.3 with a 4 nodes cluster.
Breno
Re: ListenTCP not generating flowfiles
Posted by Michael Loftis <ml...@wgops.com>.
On Thu, Feb 10, 2022 at 4:00 PM Breno Cesar <br...@gmail.com> wrote:
>
> Michael, Natan and Joe,
>
> Thank you very much for your replies.
> Here some stuffs that i done today:
>
> Changing the Batching Message Delimiter had no effect, according to the documentation it only works when a package in TCP is already identified, and apparently how is done and their delimiter is "hardcoded" in nifi.
> Searching a bit I found this project, i tested it here, and put "0" as the delimiter of the income packages, and apparently it worked, i received some connection flowfiles, I'll study the packages a little better and see which delimiter (or sequence of delimiters) works best for my situation.
A fixed delimiter will not work correctly with IDPR. It's a variable
length binary protocol, which means it'll need it's own RecordReader
to ingest correctly into NiFi....and....honestly likely to do any real
work with it anyway... So you either have to build a record reader,
or put something in front that'll digest the IDPR messages into
something NiFi can work with.
> I'm not a developer (I just know how to google well), and making an IDPRRecordReader from scratch knowing the basics, it might take a few years.
> i'm going to open a request in jira to develop the IDPRRecordReader, as well to check if they could add the functionality to change the delimiter of income packages in listentcp, and pray that someone cares about =p.
>
> Again guys, thank you very much for your help.
>
> Breno
>
>
>
> Em qui., 10 de fev. de 2022 às 00:13, Joe Witt <jo...@gmail.com> escreveu:
>>
>> Looking at
>> https://datatracker.ietf.org/doc/html/draft-ietf-idpr-specv1#section-2.4
>>
>> I think youll want to write an IDPRRecordReader
>>
>> Thanks
>>
>> On Wed, Feb 9, 2022 at 8:05 PM Nathan Gough <th...@apache.org> wrote:
>>>
>>> You should be able to set the message delimiter in ListenTCP with the "Message Delimiter" property. The default is \n. I don't believe there's a great way to see how data is being interpreted other than using interactive debug mode with NiFi set through the bootstrap.conf.
>>>
>>> On Wed, Feb 9, 2022 at 9:29 PM Breno Cesar <br...@gmail.com> wrote:
>>>>
>>>> Hi guys,
>>>>
>>>> First of all i apologize for my english, and thank you very much for your answers.
>>>> The data type i need is a data stream, which starts when the ipdr protocol connect process ends.
>>>> Is there any way to change the line terminator? or somehow see how nifi is interpreting the data when It its received ?
>>>>
>>>> Em qua., 9 de fev. de 2022 22:31, Joe Witt <jo...@gmail.com> escreveu:
>>>>>
>>>>> Correct. With both ListenTCP and ListenTCPRecord it listens for data and needs some way of framing them to generate records/bundles of data as flowfiles.
>>>>>
>>>>> What kind of data do you want to listen for?
>>>>>
>>>>> Thanks
>>>>>
>>>>> On Wed, Feb 9, 2022 at 6:26 PM Michael Loftis <ml...@wgops.com> wrote:
>>>>>>
>>>>>> ListenTCP isn’t fit for ingesting arbitrary data. It’s record oriented, default separator is Unix newline “\n”. I don’t think it will generate a flow if it never receives a newline before the connection closes.
>>>>>>
>>>>>> On Wed, Feb 9, 2022 at 18:17 Breno Cesar <br...@gmail.com> wrote:
>>>>>>>
>>>>>>> i'm tring to receive a tcp stream using the listentcp module, but when the connection start, i see the traffic reaching my server (usgin tcpdump), but the nifi does note generate a flow file.
>>>>>>> Ther is no error ow warning in the module or in the nifi cluster.
>>>>>>> Changing in the processor the bulleting level to "debug" , i can see the following message for every package that i receive:
>>>>>>>
>>>>>>> SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000-0000-000040efa66e] Accepted incoming connection from /192.0.2.12:55209
>>>>>>> SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000-0000-000040efa66e] bytes read 46
>>>>>>> SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000-0000-000040efa66e] No more data available, returning for selection
>>>>>>> SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000-0000-000040efa66e] Reached EOF, closing connection
>>>>>>>
>>>>>>> bellow is the same message, but in the tcpdump format.
>>>>>>>
>>>>>>> Internet Protocol Version 4, Src: 192.0.2.12, Dst: 192.0.2.128
>>>>>>> Transmission Control Protocol, Src Port: 55209 , Dst Port: 4737, Seq: 1, Ack: 1, Len: 46
>>>>>>> Source Port: 55209
>>>>>>> Destination Port: 4737
>>>>>>> <Source or Destination Port: 55209 >
>>>>>>> <Source or Destination Port: 4737>
>>>>>>> [Stream index: 0]
>>>>>>> [TCP Segment Len: 46]
>>>>>>> Sequence number: 1 (relative sequence number)
>>>>>>> Sequence number (raw): 3765145522
>>>>>>> [Next sequence number: 47 (relative sequence number)]
>>>>>>> Acknowledgment number: 1 (relative ack number)
>>>>>>> Acknowledgment number (raw): 274707205
>>>>>>> 0101 .... = Header Length: 20 bytes (5)
>>>>>>> Flags: 0x018 (PSH, ACK)
>>>>>>> Window size value: 10000
>>>>>>> [Calculated window size: 10000]
>>>>>>> [Window size scaling factor: 1]
>>>>>>> Checksum: 0x5fd7 [unverified]
>>>>>>> [Checksum Status: Unverified]
>>>>>>> Urgent pointer: 0
>>>>>>> [SEQ/ACK analysis]
>>>>>>> [Timestamps]
>>>>>>> TCP payload (46 bytes)
>>>>>>> [PDU Size: 46]
>>>>>>> IPDR
>>>>>>> Version: 2
>>>>>>> Message id: CONNECT (5)
>>>>>>> Session id: 0
>>>>>>> Message flags: 0x00
>>>>>>> Message length: 46
>>>>>>> Initiator id: 0.0.0.0
>>>>>>> Initiator port: 0
>>>>>>> Capabilities: 0x00000006
>>>>>>> Keep-alive interval: 90
>>>>>>> Vendor id: ARRIS E6000 Exporter
>>>>>>>
>>>>>>> Here somethings that i tried:
>>>>>>>
>>>>>>> - Increase the number of simultaneos connections and batch size, but it does not change at all.
>>>>>>> - Use listentcprecord but the behavior is the same, it only change when i select the avroreader, then i got an error informing that is a "wrong data format".
>>>>>>>
>>>>>>> So i belive that nifi is receving the messages that come from tcp , but for some reason is not generating flowfiles.
>>>>>>>
>>>>>>> Does anyone got same behavior ?
>>>>>>> There is anything else that i can do ?
>>>>>>>
>>>>>>> i'm using nifi 1.15.3 with a 4 nodes cluster.
>>>>>>>
>>>>>>>
>>>>>>> Breno
>>>>>>>
>>>>>>>
>>>>>> --
>>>>>>
>>>>>> "Genius might be described as a supreme capacity for getting its possessors
>>>>>> into trouble of all kinds."
>>>>>> -- Samuel Butler
--
"Genius might be described as a supreme capacity for getting its possessors
into trouble of all kinds."
-- Samuel Butler
Re: ListenTCP not generating flowfiles
Posted by Michael Loftis <ml...@wgops.com>.
On Thu, Feb 10, 2022 at 4:00 PM Breno Cesar <br...@gmail.com> wrote:
>
> Michael, Natan and Joe,
>
> Thank you very much for your replies.
> Here some stuffs that i done today:
>
> Changing the Batching Message Delimiter had no effect, according to the documentation it only works when a package in TCP is already identified, and apparently how is done and their delimiter is "hardcoded" in nifi.
> Searching a bit I found this project, i tested it here, and put "0" as the delimiter of the income packages, and apparently it worked, i received some connection flowfiles, I'll study the packages a little better and see which delimiter (or sequence of delimiters) works best for my situation.
A fixed delimiter will not work correctly with IDPR. It's a variable
length binary protocol, which means it'll need it's own RecordReader
to ingest correctly into NiFi....and....honestly likely to do any real
work with it anyway... So you either have to build a record reader,
or put something in front that'll digest the IDPR messages into
something NiFi can work with.
> I'm not a developer (I just know how to google well), and making an IDPRRecordReader from scratch knowing the basics, it might take a few years.
> i'm going to open a request in jira to develop the IDPRRecordReader, as well to check if they could add the functionality to change the delimiter of income packages in listentcp, and pray that someone cares about =p.
>
> Again guys, thank you very much for your help.
>
> Breno
>
>
>
> Em qui., 10 de fev. de 2022 às 00:13, Joe Witt <jo...@gmail.com> escreveu:
>>
>> Looking at
>> https://datatracker.ietf.org/doc/html/draft-ietf-idpr-specv1#section-2.4
>>
>> I think youll want to write an IDPRRecordReader
>>
>> Thanks
>>
>> On Wed, Feb 9, 2022 at 8:05 PM Nathan Gough <th...@apache.org> wrote:
>>>
>>> You should be able to set the message delimiter in ListenTCP with the "Message Delimiter" property. The default is \n. I don't believe there's a great way to see how data is being interpreted other than using interactive debug mode with NiFi set through the bootstrap.conf.
>>>
>>> On Wed, Feb 9, 2022 at 9:29 PM Breno Cesar <br...@gmail.com> wrote:
>>>>
>>>> Hi guys,
>>>>
>>>> First of all i apologize for my english, and thank you very much for your answers.
>>>> The data type i need is a data stream, which starts when the ipdr protocol connect process ends.
>>>> Is there any way to change the line terminator? or somehow see how nifi is interpreting the data when It its received ?
>>>>
>>>> Em qua., 9 de fev. de 2022 22:31, Joe Witt <jo...@gmail.com> escreveu:
>>>>>
>>>>> Correct. With both ListenTCP and ListenTCPRecord it listens for data and needs some way of framing them to generate records/bundles of data as flowfiles.
>>>>>
>>>>> What kind of data do you want to listen for?
>>>>>
>>>>> Thanks
>>>>>
>>>>> On Wed, Feb 9, 2022 at 6:26 PM Michael Loftis <ml...@wgops.com> wrote:
>>>>>>
>>>>>> ListenTCP isn’t fit for ingesting arbitrary data. It’s record oriented, default separator is Unix newline “\n”. I don’t think it will generate a flow if it never receives a newline before the connection closes.
>>>>>>
>>>>>> On Wed, Feb 9, 2022 at 18:17 Breno Cesar <br...@gmail.com> wrote:
>>>>>>>
>>>>>>> i'm tring to receive a tcp stream using the listentcp module, but when the connection start, i see the traffic reaching my server (usgin tcpdump), but the nifi does note generate a flow file.
>>>>>>> Ther is no error ow warning in the module or in the nifi cluster.
>>>>>>> Changing in the processor the bulleting level to "debug" , i can see the following message for every package that i receive:
>>>>>>>
>>>>>>> SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000-0000-000040efa66e] Accepted incoming connection from /192.0.2.12:55209
>>>>>>> SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000-0000-000040efa66e] bytes read 46
>>>>>>> SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000-0000-000040efa66e] No more data available, returning for selection
>>>>>>> SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000-0000-000040efa66e] Reached EOF, closing connection
>>>>>>>
>>>>>>> bellow is the same message, but in the tcpdump format.
>>>>>>>
>>>>>>> Internet Protocol Version 4, Src: 192.0.2.12, Dst: 192.0.2.128
>>>>>>> Transmission Control Protocol, Src Port: 55209 , Dst Port: 4737, Seq: 1, Ack: 1, Len: 46
>>>>>>> Source Port: 55209
>>>>>>> Destination Port: 4737
>>>>>>> <Source or Destination Port: 55209 >
>>>>>>> <Source or Destination Port: 4737>
>>>>>>> [Stream index: 0]
>>>>>>> [TCP Segment Len: 46]
>>>>>>> Sequence number: 1 (relative sequence number)
>>>>>>> Sequence number (raw): 3765145522
>>>>>>> [Next sequence number: 47 (relative sequence number)]
>>>>>>> Acknowledgment number: 1 (relative ack number)
>>>>>>> Acknowledgment number (raw): 274707205
>>>>>>> 0101 .... = Header Length: 20 bytes (5)
>>>>>>> Flags: 0x018 (PSH, ACK)
>>>>>>> Window size value: 10000
>>>>>>> [Calculated window size: 10000]
>>>>>>> [Window size scaling factor: 1]
>>>>>>> Checksum: 0x5fd7 [unverified]
>>>>>>> [Checksum Status: Unverified]
>>>>>>> Urgent pointer: 0
>>>>>>> [SEQ/ACK analysis]
>>>>>>> [Timestamps]
>>>>>>> TCP payload (46 bytes)
>>>>>>> [PDU Size: 46]
>>>>>>> IPDR
>>>>>>> Version: 2
>>>>>>> Message id: CONNECT (5)
>>>>>>> Session id: 0
>>>>>>> Message flags: 0x00
>>>>>>> Message length: 46
>>>>>>> Initiator id: 0.0.0.0
>>>>>>> Initiator port: 0
>>>>>>> Capabilities: 0x00000006
>>>>>>> Keep-alive interval: 90
>>>>>>> Vendor id: ARRIS E6000 Exporter
>>>>>>>
>>>>>>> Here somethings that i tried:
>>>>>>>
>>>>>>> - Increase the number of simultaneos connections and batch size, but it does not change at all.
>>>>>>> - Use listentcprecord but the behavior is the same, it only change when i select the avroreader, then i got an error informing that is a "wrong data format".
>>>>>>>
>>>>>>> So i belive that nifi is receving the messages that come from tcp , but for some reason is not generating flowfiles.
>>>>>>>
>>>>>>> Does anyone got same behavior ?
>>>>>>> There is anything else that i can do ?
>>>>>>>
>>>>>>> i'm using nifi 1.15.3 with a 4 nodes cluster.
>>>>>>>
>>>>>>>
>>>>>>> Breno
>>>>>>>
>>>>>>>
>>>>>> --
>>>>>>
>>>>>> "Genius might be described as a supreme capacity for getting its possessors
>>>>>> into trouble of all kinds."
>>>>>> -- Samuel Butler
--
"Genius might be described as a supreme capacity for getting its possessors
into trouble of all kinds."
-- Samuel Butler
Re: ListenTCP not generating flowfiles
Posted by Breno Cesar <br...@gmail.com>.
Michael, Natan and Joe,
Thank you very much for your replies.
Here some stuffs that i done today:
- Changing the Batching Message Delimiter had no effect, according to
the documentation it only works when a package in TCP is already
identified, and apparently how is done and their delimiter is "hardcoded"
in nifi.
- Searching a bit I found this project
<https://github.com/linksmart/nifi-customlistentcp-processor>, i tested
it here, and put "0" as the delimiter of the income packages, and
apparently it worked, i received some connection flowfiles, I'll study the
packages a little better and see which delimiter (or sequence of
delimiters) works best for my situation.
- I'm not a developer (I just know how to google well), and making an
IDPRRecordReader from scratch knowing the basics, it might take a few
years.
- i'm going to open a request in jira to develop the IDPRRecordReader,
as well to check if they could add the functionality to change the
delimiter of income packages in listentcp, and pray that someone cares
about =p.
Again guys, thank you very much for your help.
*Breno *
Em qui., 10 de fev. de 2022 às 00:13, Joe Witt <jo...@gmail.com>
escreveu:
> Looking at
> https://datatracker.ietf.org/doc/html/draft-ietf-idpr-specv1#section-2.4
>
> I think youll want to write an IDPRRecordReader
>
> Thanks
>
> On Wed, Feb 9, 2022 at 8:05 PM Nathan Gough <th...@apache.org> wrote:
>
>> You should be able to set the message delimiter in ListenTCP with the
>> "Message Delimiter" property. The default is \n. I don't believe there's a
>> great way to see how data is being interpreted other than using interactive
>> debug mode with NiFi set through the bootstrap.conf.
>>
>> On Wed, Feb 9, 2022 at 9:29 PM Breno Cesar <br...@gmail.com>
>> wrote:
>>
>>> Hi guys,
>>>
>>> First of all i apologize for my english, and thank you very much for
>>> your answers.
>>> The data type i need is a data stream, which starts when the ipdr
>>> protocol connect process ends.
>>> Is there any way to change the line terminator? or somehow see how nifi
>>> is interpreting the data when It its received ?
>>>
>>> Em qua., 9 de fev. de 2022 22:31, Joe Witt <jo...@gmail.com>
>>> escreveu:
>>>
>>>> Correct. With both ListenTCP and ListenTCPRecord it listens for data
>>>> and needs some way of framing them to generate records/bundles of data as
>>>> flowfiles.
>>>>
>>>> What kind of data do you want to listen for?
>>>>
>>>> Thanks
>>>>
>>>> On Wed, Feb 9, 2022 at 6:26 PM Michael Loftis <ml...@wgops.com>
>>>> wrote:
>>>>
>>>>> ListenTCP isn’t fit for ingesting arbitrary data. It’s record
>>>>> oriented, default separator is Unix newline “\n”. I don’t think it will
>>>>> generate a flow if it never receives a newline before the connection closes.
>>>>>
>>>>> On Wed, Feb 9, 2022 at 18:17 Breno Cesar <br...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> i'm tring to receive a tcp stream using the listentcp module, but
>>>>>> when the connection start, i see the traffic reaching my server (usgin
>>>>>> tcpdump), but the nifi does note generate a flow file.
>>>>>> Ther is no error ow warning in the module or in the nifi cluster.
>>>>>> Changing in the processor the bulleting level to "debug" , i can see
>>>>>> the following message for every package that i receive:
>>>>>>
>>>>>> SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000-0000-000040efa66e]
>>>>>> Accepted incoming connection from /192.0.2.12:55209
>>>>>> SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000-0000-000040efa66e]
>>>>>> bytes read 46
>>>>>> SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000-0000-000040efa66e]
>>>>>> No more data available, returning for selection
>>>>>> SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000-0000-000040efa66e]
>>>>>> Reached EOF, closing connection
>>>>>>
>>>>>> bellow is the same message, but in the tcpdump format.
>>>>>>
>>>>>> Internet Protocol Version 4, Src: 192.0.2.12, Dst: 192.0.2.128
>>>>>> Transmission Control Protocol, Src Port: 55209 , Dst Port: 4737,
>>>>>> Seq: 1, Ack: 1, Len: 46
>>>>>> Source Port: 55209
>>>>>> Destination Port: 4737
>>>>>> <Source or Destination Port: 55209 >
>>>>>> <Source or Destination Port: 4737>
>>>>>> [Stream index: 0]
>>>>>> [TCP Segment Len: 46]
>>>>>> Sequence number: 1 (relative sequence number)
>>>>>> Sequence number (raw): 3765145522
>>>>>> [Next sequence number: 47 (relative sequence number)]
>>>>>> Acknowledgment number: 1 (relative ack number)
>>>>>> Acknowledgment number (raw): 274707205
>>>>>> 0101 .... = Header Length: 20 bytes (5)
>>>>>> Flags: 0x018 (PSH, ACK)
>>>>>> Window size value: 10000
>>>>>> [Calculated window size: 10000]
>>>>>> [Window size scaling factor: 1]
>>>>>> Checksum: 0x5fd7 [unverified]
>>>>>> [Checksum Status: Unverified]
>>>>>> Urgent pointer: 0
>>>>>> [SEQ/ACK analysis]
>>>>>> [Timestamps]
>>>>>> TCP payload (46 bytes)
>>>>>> [PDU Size: 46]
>>>>>> IPDR
>>>>>> Version: 2
>>>>>> Message id: CONNECT (5)
>>>>>> Session id: 0
>>>>>> Message flags: 0x00
>>>>>> Message length: 46
>>>>>> Initiator id: 0.0.0.0
>>>>>> Initiator port: 0
>>>>>> Capabilities: 0x00000006
>>>>>> Keep-alive interval: 90
>>>>>> Vendor id: ARRIS E6000 Exporter
>>>>>>
>>>>>> Here somethings that i tried:
>>>>>>
>>>>>> - Increase the number of simultaneos connections and batch size, but
>>>>>> it does not change at all.
>>>>>> - Use listentcprecord but the behavior is the same, it only change
>>>>>> when i select the avroreader, then i got an error informing that is a
>>>>>> "wrong data format".
>>>>>>
>>>>>> So i belive that nifi is receving the messages that come from tcp ,
>>>>>> but for some reason is not generating flowfiles.
>>>>>>
>>>>>> Does anyone got same behavior ?
>>>>>> There is anything else that i can do ?
>>>>>>
>>>>>> i'm using nifi 1.15.3 with a 4 nodes cluster.
>>>>>>
>>>>>>
>>>>>> Breno
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>
>>>>> "Genius might be described as a supreme capacity for getting its
>>>>> possessors
>>>>> into trouble of all kinds."
>>>>> -- Samuel Butler
>>>>>
>>>>
Re: ListenTCP not generating flowfiles
Posted by Breno Cesar <br...@gmail.com>.
Michael, Natan and Joe,
Thank you very much for your replies.
Here some stuffs that i done today:
- Changing the Batching Message Delimiter had no effect, according to
the documentation it only works when a package in TCP is already
identified, and apparently how is done and their delimiter is "hardcoded"
in nifi.
- Searching a bit I found this project
<https://github.com/linksmart/nifi-customlistentcp-processor>, i tested
it here, and put "0" as the delimiter of the income packages, and
apparently it worked, i received some connection flowfiles, I'll study the
packages a little better and see which delimiter (or sequence of
delimiters) works best for my situation.
- I'm not a developer (I just know how to google well), and making an
IDPRRecordReader from scratch knowing the basics, it might take a few
years.
- i'm going to open a request in jira to develop the IDPRRecordReader,
as well to check if they could add the functionality to change the
delimiter of income packages in listentcp, and pray that someone cares
about =p.
Again guys, thank you very much for your help.
*Breno *
Em qui., 10 de fev. de 2022 às 00:13, Joe Witt <jo...@gmail.com>
escreveu:
> Looking at
> https://datatracker.ietf.org/doc/html/draft-ietf-idpr-specv1#section-2.4
>
> I think youll want to write an IDPRRecordReader
>
> Thanks
>
> On Wed, Feb 9, 2022 at 8:05 PM Nathan Gough <th...@apache.org> wrote:
>
>> You should be able to set the message delimiter in ListenTCP with the
>> "Message Delimiter" property. The default is \n. I don't believe there's a
>> great way to see how data is being interpreted other than using interactive
>> debug mode with NiFi set through the bootstrap.conf.
>>
>> On Wed, Feb 9, 2022 at 9:29 PM Breno Cesar <br...@gmail.com>
>> wrote:
>>
>>> Hi guys,
>>>
>>> First of all i apologize for my english, and thank you very much for
>>> your answers.
>>> The data type i need is a data stream, which starts when the ipdr
>>> protocol connect process ends.
>>> Is there any way to change the line terminator? or somehow see how nifi
>>> is interpreting the data when It its received ?
>>>
>>> Em qua., 9 de fev. de 2022 22:31, Joe Witt <jo...@gmail.com>
>>> escreveu:
>>>
>>>> Correct. With both ListenTCP and ListenTCPRecord it listens for data
>>>> and needs some way of framing them to generate records/bundles of data as
>>>> flowfiles.
>>>>
>>>> What kind of data do you want to listen for?
>>>>
>>>> Thanks
>>>>
>>>> On Wed, Feb 9, 2022 at 6:26 PM Michael Loftis <ml...@wgops.com>
>>>> wrote:
>>>>
>>>>> ListenTCP isn’t fit for ingesting arbitrary data. It’s record
>>>>> oriented, default separator is Unix newline “\n”. I don’t think it will
>>>>> generate a flow if it never receives a newline before the connection closes.
>>>>>
>>>>> On Wed, Feb 9, 2022 at 18:17 Breno Cesar <br...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> i'm tring to receive a tcp stream using the listentcp module, but
>>>>>> when the connection start, i see the traffic reaching my server (usgin
>>>>>> tcpdump), but the nifi does note generate a flow file.
>>>>>> Ther is no error ow warning in the module or in the nifi cluster.
>>>>>> Changing in the processor the bulleting level to "debug" , i can see
>>>>>> the following message for every package that i receive:
>>>>>>
>>>>>> SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000-0000-000040efa66e]
>>>>>> Accepted incoming connection from /192.0.2.12:55209
>>>>>> SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000-0000-000040efa66e]
>>>>>> bytes read 46
>>>>>> SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000-0000-000040efa66e]
>>>>>> No more data available, returning for selection
>>>>>> SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000-0000-000040efa66e]
>>>>>> Reached EOF, closing connection
>>>>>>
>>>>>> bellow is the same message, but in the tcpdump format.
>>>>>>
>>>>>> Internet Protocol Version 4, Src: 192.0.2.12, Dst: 192.0.2.128
>>>>>> Transmission Control Protocol, Src Port: 55209 , Dst Port: 4737,
>>>>>> Seq: 1, Ack: 1, Len: 46
>>>>>> Source Port: 55209
>>>>>> Destination Port: 4737
>>>>>> <Source or Destination Port: 55209 >
>>>>>> <Source or Destination Port: 4737>
>>>>>> [Stream index: 0]
>>>>>> [TCP Segment Len: 46]
>>>>>> Sequence number: 1 (relative sequence number)
>>>>>> Sequence number (raw): 3765145522
>>>>>> [Next sequence number: 47 (relative sequence number)]
>>>>>> Acknowledgment number: 1 (relative ack number)
>>>>>> Acknowledgment number (raw): 274707205
>>>>>> 0101 .... = Header Length: 20 bytes (5)
>>>>>> Flags: 0x018 (PSH, ACK)
>>>>>> Window size value: 10000
>>>>>> [Calculated window size: 10000]
>>>>>> [Window size scaling factor: 1]
>>>>>> Checksum: 0x5fd7 [unverified]
>>>>>> [Checksum Status: Unverified]
>>>>>> Urgent pointer: 0
>>>>>> [SEQ/ACK analysis]
>>>>>> [Timestamps]
>>>>>> TCP payload (46 bytes)
>>>>>> [PDU Size: 46]
>>>>>> IPDR
>>>>>> Version: 2
>>>>>> Message id: CONNECT (5)
>>>>>> Session id: 0
>>>>>> Message flags: 0x00
>>>>>> Message length: 46
>>>>>> Initiator id: 0.0.0.0
>>>>>> Initiator port: 0
>>>>>> Capabilities: 0x00000006
>>>>>> Keep-alive interval: 90
>>>>>> Vendor id: ARRIS E6000 Exporter
>>>>>>
>>>>>> Here somethings that i tried:
>>>>>>
>>>>>> - Increase the number of simultaneos connections and batch size, but
>>>>>> it does not change at all.
>>>>>> - Use listentcprecord but the behavior is the same, it only change
>>>>>> when i select the avroreader, then i got an error informing that is a
>>>>>> "wrong data format".
>>>>>>
>>>>>> So i belive that nifi is receving the messages that come from tcp ,
>>>>>> but for some reason is not generating flowfiles.
>>>>>>
>>>>>> Does anyone got same behavior ?
>>>>>> There is anything else that i can do ?
>>>>>>
>>>>>> i'm using nifi 1.15.3 with a 4 nodes cluster.
>>>>>>
>>>>>>
>>>>>> Breno
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>
>>>>> "Genius might be described as a supreme capacity for getting its
>>>>> possessors
>>>>> into trouble of all kinds."
>>>>> -- Samuel Butler
>>>>>
>>>>
Re: ListenTCP not generating flowfiles
Posted by Joe Witt <jo...@gmail.com>.
Looking at
https://datatracker.ietf.org/doc/html/draft-ietf-idpr-specv1#section-2.4
I think youll want to write an IDPRRecordReader
Thanks
On Wed, Feb 9, 2022 at 8:05 PM Nathan Gough <th...@apache.org> wrote:
> You should be able to set the message delimiter in ListenTCP with the
> "Message Delimiter" property. The default is \n. I don't believe there's a
> great way to see how data is being interpreted other than using interactive
> debug mode with NiFi set through the bootstrap.conf.
>
> On Wed, Feb 9, 2022 at 9:29 PM Breno Cesar <br...@gmail.com>
> wrote:
>
>> Hi guys,
>>
>> First of all i apologize for my english, and thank you very much for your
>> answers.
>> The data type i need is a data stream, which starts when the ipdr
>> protocol connect process ends.
>> Is there any way to change the line terminator? or somehow see how nifi
>> is interpreting the data when It its received ?
>>
>> Em qua., 9 de fev. de 2022 22:31, Joe Witt <jo...@gmail.com> escreveu:
>>
>>> Correct. With both ListenTCP and ListenTCPRecord it listens for data
>>> and needs some way of framing them to generate records/bundles of data as
>>> flowfiles.
>>>
>>> What kind of data do you want to listen for?
>>>
>>> Thanks
>>>
>>> On Wed, Feb 9, 2022 at 6:26 PM Michael Loftis <ml...@wgops.com> wrote:
>>>
>>>> ListenTCP isn’t fit for ingesting arbitrary data. It’s record oriented,
>>>> default separator is Unix newline “\n”. I don’t think it will generate a
>>>> flow if it never receives a newline before the connection closes.
>>>>
>>>> On Wed, Feb 9, 2022 at 18:17 Breno Cesar <br...@gmail.com>
>>>> wrote:
>>>>
>>>>> i'm tring to receive a tcp stream using the listentcp module, but when
>>>>> the connection start, i see the traffic reaching my server (usgin tcpdump),
>>>>> but the nifi does note generate a flow file.
>>>>> Ther is no error ow warning in the module or in the nifi cluster.
>>>>> Changing in the processor the bulleting level to "debug" , i can see
>>>>> the following message for every package that i receive:
>>>>>
>>>>> SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000-0000-000040efa66e]
>>>>> Accepted incoming connection from /192.0.2.12:55209
>>>>> SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000-0000-000040efa66e]
>>>>> bytes read 46
>>>>> SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000-0000-000040efa66e] No
>>>>> more data available, returning for selection
>>>>> SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000-0000-000040efa66e]
>>>>> Reached EOF, closing connection
>>>>>
>>>>> bellow is the same message, but in the tcpdump format.
>>>>>
>>>>> Internet Protocol Version 4, Src: 192.0.2.12, Dst: 192.0.2.128
>>>>> Transmission Control Protocol, Src Port: 55209 , Dst Port: 4737,
>>>>> Seq: 1, Ack: 1, Len: 46
>>>>> Source Port: 55209
>>>>> Destination Port: 4737
>>>>> <Source or Destination Port: 55209 >
>>>>> <Source or Destination Port: 4737>
>>>>> [Stream index: 0]
>>>>> [TCP Segment Len: 46]
>>>>> Sequence number: 1 (relative sequence number)
>>>>> Sequence number (raw): 3765145522
>>>>> [Next sequence number: 47 (relative sequence number)]
>>>>> Acknowledgment number: 1 (relative ack number)
>>>>> Acknowledgment number (raw): 274707205
>>>>> 0101 .... = Header Length: 20 bytes (5)
>>>>> Flags: 0x018 (PSH, ACK)
>>>>> Window size value: 10000
>>>>> [Calculated window size: 10000]
>>>>> [Window size scaling factor: 1]
>>>>> Checksum: 0x5fd7 [unverified]
>>>>> [Checksum Status: Unverified]
>>>>> Urgent pointer: 0
>>>>> [SEQ/ACK analysis]
>>>>> [Timestamps]
>>>>> TCP payload (46 bytes)
>>>>> [PDU Size: 46]
>>>>> IPDR
>>>>> Version: 2
>>>>> Message id: CONNECT (5)
>>>>> Session id: 0
>>>>> Message flags: 0x00
>>>>> Message length: 46
>>>>> Initiator id: 0.0.0.0
>>>>> Initiator port: 0
>>>>> Capabilities: 0x00000006
>>>>> Keep-alive interval: 90
>>>>> Vendor id: ARRIS E6000 Exporter
>>>>>
>>>>> Here somethings that i tried:
>>>>>
>>>>> - Increase the number of simultaneos connections and batch size, but
>>>>> it does not change at all.
>>>>> - Use listentcprecord but the behavior is the same, it only change
>>>>> when i select the avroreader, then i got an error informing that is a
>>>>> "wrong data format".
>>>>>
>>>>> So i belive that nifi is receving the messages that come from tcp ,
>>>>> but for some reason is not generating flowfiles.
>>>>>
>>>>> Does anyone got same behavior ?
>>>>> There is anything else that i can do ?
>>>>>
>>>>> i'm using nifi 1.15.3 with a 4 nodes cluster.
>>>>>
>>>>>
>>>>> Breno
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>
>>>> "Genius might be described as a supreme capacity for getting its
>>>> possessors
>>>> into trouble of all kinds."
>>>> -- Samuel Butler
>>>>
>>>
Re: ListenTCP not generating flowfiles
Posted by Joe Witt <jo...@gmail.com>.
Looking at
https://datatracker.ietf.org/doc/html/draft-ietf-idpr-specv1#section-2.4
I think youll want to write an IDPRRecordReader
Thanks
On Wed, Feb 9, 2022 at 8:05 PM Nathan Gough <th...@apache.org> wrote:
> You should be able to set the message delimiter in ListenTCP with the
> "Message Delimiter" property. The default is \n. I don't believe there's a
> great way to see how data is being interpreted other than using interactive
> debug mode with NiFi set through the bootstrap.conf.
>
> On Wed, Feb 9, 2022 at 9:29 PM Breno Cesar <br...@gmail.com>
> wrote:
>
>> Hi guys,
>>
>> First of all i apologize for my english, and thank you very much for your
>> answers.
>> The data type i need is a data stream, which starts when the ipdr
>> protocol connect process ends.
>> Is there any way to change the line terminator? or somehow see how nifi
>> is interpreting the data when It its received ?
>>
>> Em qua., 9 de fev. de 2022 22:31, Joe Witt <jo...@gmail.com> escreveu:
>>
>>> Correct. With both ListenTCP and ListenTCPRecord it listens for data
>>> and needs some way of framing them to generate records/bundles of data as
>>> flowfiles.
>>>
>>> What kind of data do you want to listen for?
>>>
>>> Thanks
>>>
>>> On Wed, Feb 9, 2022 at 6:26 PM Michael Loftis <ml...@wgops.com> wrote:
>>>
>>>> ListenTCP isn’t fit for ingesting arbitrary data. It’s record oriented,
>>>> default separator is Unix newline “\n”. I don’t think it will generate a
>>>> flow if it never receives a newline before the connection closes.
>>>>
>>>> On Wed, Feb 9, 2022 at 18:17 Breno Cesar <br...@gmail.com>
>>>> wrote:
>>>>
>>>>> i'm tring to receive a tcp stream using the listentcp module, but when
>>>>> the connection start, i see the traffic reaching my server (usgin tcpdump),
>>>>> but the nifi does note generate a flow file.
>>>>> Ther is no error ow warning in the module or in the nifi cluster.
>>>>> Changing in the processor the bulleting level to "debug" , i can see
>>>>> the following message for every package that i receive:
>>>>>
>>>>> SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000-0000-000040efa66e]
>>>>> Accepted incoming connection from /192.0.2.12:55209
>>>>> SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000-0000-000040efa66e]
>>>>> bytes read 46
>>>>> SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000-0000-000040efa66e] No
>>>>> more data available, returning for selection
>>>>> SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000-0000-000040efa66e]
>>>>> Reached EOF, closing connection
>>>>>
>>>>> bellow is the same message, but in the tcpdump format.
>>>>>
>>>>> Internet Protocol Version 4, Src: 192.0.2.12, Dst: 192.0.2.128
>>>>> Transmission Control Protocol, Src Port: 55209 , Dst Port: 4737,
>>>>> Seq: 1, Ack: 1, Len: 46
>>>>> Source Port: 55209
>>>>> Destination Port: 4737
>>>>> <Source or Destination Port: 55209 >
>>>>> <Source or Destination Port: 4737>
>>>>> [Stream index: 0]
>>>>> [TCP Segment Len: 46]
>>>>> Sequence number: 1 (relative sequence number)
>>>>> Sequence number (raw): 3765145522
>>>>> [Next sequence number: 47 (relative sequence number)]
>>>>> Acknowledgment number: 1 (relative ack number)
>>>>> Acknowledgment number (raw): 274707205
>>>>> 0101 .... = Header Length: 20 bytes (5)
>>>>> Flags: 0x018 (PSH, ACK)
>>>>> Window size value: 10000
>>>>> [Calculated window size: 10000]
>>>>> [Window size scaling factor: 1]
>>>>> Checksum: 0x5fd7 [unverified]
>>>>> [Checksum Status: Unverified]
>>>>> Urgent pointer: 0
>>>>> [SEQ/ACK analysis]
>>>>> [Timestamps]
>>>>> TCP payload (46 bytes)
>>>>> [PDU Size: 46]
>>>>> IPDR
>>>>> Version: 2
>>>>> Message id: CONNECT (5)
>>>>> Session id: 0
>>>>> Message flags: 0x00
>>>>> Message length: 46
>>>>> Initiator id: 0.0.0.0
>>>>> Initiator port: 0
>>>>> Capabilities: 0x00000006
>>>>> Keep-alive interval: 90
>>>>> Vendor id: ARRIS E6000 Exporter
>>>>>
>>>>> Here somethings that i tried:
>>>>>
>>>>> - Increase the number of simultaneos connections and batch size, but
>>>>> it does not change at all.
>>>>> - Use listentcprecord but the behavior is the same, it only change
>>>>> when i select the avroreader, then i got an error informing that is a
>>>>> "wrong data format".
>>>>>
>>>>> So i belive that nifi is receving the messages that come from tcp ,
>>>>> but for some reason is not generating flowfiles.
>>>>>
>>>>> Does anyone got same behavior ?
>>>>> There is anything else that i can do ?
>>>>>
>>>>> i'm using nifi 1.15.3 with a 4 nodes cluster.
>>>>>
>>>>>
>>>>> Breno
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>
>>>> "Genius might be described as a supreme capacity for getting its
>>>> possessors
>>>> into trouble of all kinds."
>>>> -- Samuel Butler
>>>>
>>>
Re: ListenTCP not generating flowfiles
Posted by Nathan Gough <th...@apache.org>.
You should be able to set the message delimiter in ListenTCP with the
"Message Delimiter" property. The default is \n. I don't believe there's a
great way to see how data is being interpreted other than using interactive
debug mode with NiFi set through the bootstrap.conf.
On Wed, Feb 9, 2022 at 9:29 PM Breno Cesar <br...@gmail.com> wrote:
> Hi guys,
>
> First of all i apologize for my english, and thank you very much for your
> answers.
> The data type i need is a data stream, which starts when the ipdr protocol
> connect process ends.
> Is there any way to change the line terminator? or somehow see how nifi
> is interpreting the data when It its received ?
>
> Em qua., 9 de fev. de 2022 22:31, Joe Witt <jo...@gmail.com> escreveu:
>
>> Correct. With both ListenTCP and ListenTCPRecord it listens for data and
>> needs some way of framing them to generate records/bundles of data as
>> flowfiles.
>>
>> What kind of data do you want to listen for?
>>
>> Thanks
>>
>> On Wed, Feb 9, 2022 at 6:26 PM Michael Loftis <ml...@wgops.com> wrote:
>>
>>> ListenTCP isn’t fit for ingesting arbitrary data. It’s record oriented,
>>> default separator is Unix newline “\n”. I don’t think it will generate a
>>> flow if it never receives a newline before the connection closes.
>>>
>>> On Wed, Feb 9, 2022 at 18:17 Breno Cesar <br...@gmail.com>
>>> wrote:
>>>
>>>> i'm tring to receive a tcp stream using the listentcp module, but when
>>>> the connection start, i see the traffic reaching my server (usgin tcpdump),
>>>> but the nifi does note generate a flow file.
>>>> Ther is no error ow warning in the module or in the nifi cluster.
>>>> Changing in the processor the bulleting level to "debug" , i can see
>>>> the following message for every package that i receive:
>>>>
>>>> SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000-0000-000040efa66e]
>>>> Accepted incoming connection from /192.0.2.12:55209
>>>> SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000-0000-000040efa66e]
>>>> bytes read 46
>>>> SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000-0000-000040efa66e] No
>>>> more data available, returning for selection
>>>> SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000-0000-000040efa66e]
>>>> Reached EOF, closing connection
>>>>
>>>> bellow is the same message, but in the tcpdump format.
>>>>
>>>> Internet Protocol Version 4, Src: 192.0.2.12, Dst: 192.0.2.128
>>>> Transmission Control Protocol, Src Port: 55209 , Dst Port: 4737, Seq:
>>>> 1, Ack: 1, Len: 46
>>>> Source Port: 55209
>>>> Destination Port: 4737
>>>> <Source or Destination Port: 55209 >
>>>> <Source or Destination Port: 4737>
>>>> [Stream index: 0]
>>>> [TCP Segment Len: 46]
>>>> Sequence number: 1 (relative sequence number)
>>>> Sequence number (raw): 3765145522
>>>> [Next sequence number: 47 (relative sequence number)]
>>>> Acknowledgment number: 1 (relative ack number)
>>>> Acknowledgment number (raw): 274707205
>>>> 0101 .... = Header Length: 20 bytes (5)
>>>> Flags: 0x018 (PSH, ACK)
>>>> Window size value: 10000
>>>> [Calculated window size: 10000]
>>>> [Window size scaling factor: 1]
>>>> Checksum: 0x5fd7 [unverified]
>>>> [Checksum Status: Unverified]
>>>> Urgent pointer: 0
>>>> [SEQ/ACK analysis]
>>>> [Timestamps]
>>>> TCP payload (46 bytes)
>>>> [PDU Size: 46]
>>>> IPDR
>>>> Version: 2
>>>> Message id: CONNECT (5)
>>>> Session id: 0
>>>> Message flags: 0x00
>>>> Message length: 46
>>>> Initiator id: 0.0.0.0
>>>> Initiator port: 0
>>>> Capabilities: 0x00000006
>>>> Keep-alive interval: 90
>>>> Vendor id: ARRIS E6000 Exporter
>>>>
>>>> Here somethings that i tried:
>>>>
>>>> - Increase the number of simultaneos connections and batch size, but
>>>> it does not change at all.
>>>> - Use listentcprecord but the behavior is the same, it only change
>>>> when i select the avroreader, then i got an error informing that is a
>>>> "wrong data format".
>>>>
>>>> So i belive that nifi is receving the messages that come from tcp , but
>>>> for some reason is not generating flowfiles.
>>>>
>>>> Does anyone got same behavior ?
>>>> There is anything else that i can do ?
>>>>
>>>> i'm using nifi 1.15.3 with a 4 nodes cluster.
>>>>
>>>>
>>>> Breno
>>>>
>>>>
>>>>
>>>> --
>>>
>>> "Genius might be described as a supreme capacity for getting its
>>> possessors
>>> into trouble of all kinds."
>>> -- Samuel Butler
>>>
>>
Re: ListenTCP not generating flowfiles
Posted by Breno Cesar <br...@gmail.com>.
Hi guys,
First of all i apologize for my english, and thank you very much for your
answers.
The data type i need is a data stream, which starts when the ipdr protocol
connect process ends.
Is there any way to change the line terminator? or somehow see how nifi is
interpreting the data when It its received ?
Em qua., 9 de fev. de 2022 22:31, Joe Witt <jo...@gmail.com> escreveu:
> Correct. With both ListenTCP and ListenTCPRecord it listens for data and
> needs some way of framing them to generate records/bundles of data as
> flowfiles.
>
> What kind of data do you want to listen for?
>
> Thanks
>
> On Wed, Feb 9, 2022 at 6:26 PM Michael Loftis <ml...@wgops.com> wrote:
>
>> ListenTCP isn’t fit for ingesting arbitrary data. It’s record oriented,
>> default separator is Unix newline “\n”. I don’t think it will generate a
>> flow if it never receives a newline before the connection closes.
>>
>> On Wed, Feb 9, 2022 at 18:17 Breno Cesar <br...@gmail.com>
>> wrote:
>>
>>> i'm tring to receive a tcp stream using the listentcp module, but when
>>> the connection start, i see the traffic reaching my server (usgin tcpdump),
>>> but the nifi does note generate a flow file.
>>> Ther is no error ow warning in the module or in the nifi cluster.
>>> Changing in the processor the bulleting level to "debug" , i can see the
>>> following message for every package that i receive:
>>>
>>> SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000-0000-000040efa66e]
>>> Accepted incoming connection from /192.0.2.12:55209
>>> SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000-0000-000040efa66e]
>>> bytes read 46
>>> SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000-0000-000040efa66e] No
>>> more data available, returning for selection
>>> SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000-0000-000040efa66e]
>>> Reached EOF, closing connection
>>>
>>> bellow is the same message, but in the tcpdump format.
>>>
>>> Internet Protocol Version 4, Src: 192.0.2.12, Dst: 192.0.2.128
>>> Transmission Control Protocol, Src Port: 55209 , Dst Port: 4737, Seq:
>>> 1, Ack: 1, Len: 46
>>> Source Port: 55209
>>> Destination Port: 4737
>>> <Source or Destination Port: 55209 >
>>> <Source or Destination Port: 4737>
>>> [Stream index: 0]
>>> [TCP Segment Len: 46]
>>> Sequence number: 1 (relative sequence number)
>>> Sequence number (raw): 3765145522
>>> [Next sequence number: 47 (relative sequence number)]
>>> Acknowledgment number: 1 (relative ack number)
>>> Acknowledgment number (raw): 274707205
>>> 0101 .... = Header Length: 20 bytes (5)
>>> Flags: 0x018 (PSH, ACK)
>>> Window size value: 10000
>>> [Calculated window size: 10000]
>>> [Window size scaling factor: 1]
>>> Checksum: 0x5fd7 [unverified]
>>> [Checksum Status: Unverified]
>>> Urgent pointer: 0
>>> [SEQ/ACK analysis]
>>> [Timestamps]
>>> TCP payload (46 bytes)
>>> [PDU Size: 46]
>>> IPDR
>>> Version: 2
>>> Message id: CONNECT (5)
>>> Session id: 0
>>> Message flags: 0x00
>>> Message length: 46
>>> Initiator id: 0.0.0.0
>>> Initiator port: 0
>>> Capabilities: 0x00000006
>>> Keep-alive interval: 90
>>> Vendor id: ARRIS E6000 Exporter
>>>
>>> Here somethings that i tried:
>>>
>>> - Increase the number of simultaneos connections and batch size, but it
>>> does not change at all.
>>> - Use listentcprecord but the behavior is the same, it only change when
>>> i select the avroreader, then i got an error informing that is a "wrong
>>> data format".
>>>
>>> So i belive that nifi is receving the messages that come from tcp , but
>>> for some reason is not generating flowfiles.
>>>
>>> Does anyone got same behavior ?
>>> There is anything else that i can do ?
>>>
>>> i'm using nifi 1.15.3 with a 4 nodes cluster.
>>>
>>>
>>> Breno
>>>
>>>
>>>
>>> --
>>
>> "Genius might be described as a supreme capacity for getting its
>> possessors
>> into trouble of all kinds."
>> -- Samuel Butler
>>
>
Re: ListenTCP not generating flowfiles
Posted by Joe Witt <jo...@gmail.com>.
Correct. With both ListenTCP and ListenTCPRecord it listens for data and
needs some way of framing them to generate records/bundles of data as
flowfiles.
What kind of data do you want to listen for?
Thanks
On Wed, Feb 9, 2022 at 6:26 PM Michael Loftis <ml...@wgops.com> wrote:
> ListenTCP isn’t fit for ingesting arbitrary data. It’s record oriented,
> default separator is Unix newline “\n”. I don’t think it will generate a
> flow if it never receives a newline before the connection closes.
>
> On Wed, Feb 9, 2022 at 18:17 Breno Cesar <br...@gmail.com> wrote:
>
>> i'm tring to receive a tcp stream using the listentcp module, but when
>> the connection start, i see the traffic reaching my server (usgin tcpdump),
>> but the nifi does note generate a flow file.
>> Ther is no error ow warning in the module or in the nifi cluster.
>> Changing in the processor the bulleting level to "debug" , i can see the
>> following message for every package that i receive:
>>
>> SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000-0000-000040efa66e]
>> Accepted incoming connection from /192.0.2.12:55209
>> SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000-0000-000040efa66e] bytes
>> read 46
>> SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000-0000-000040efa66e] No
>> more data available, returning for selection
>> SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000-0000-000040efa66e]
>> Reached EOF, closing connection
>>
>> bellow is the same message, but in the tcpdump format.
>>
>> Internet Protocol Version 4, Src: 192.0.2.12, Dst: 192.0.2.128
>> Transmission Control Protocol, Src Port: 55209 , Dst Port: 4737, Seq:
>> 1, Ack: 1, Len: 46
>> Source Port: 55209
>> Destination Port: 4737
>> <Source or Destination Port: 55209 >
>> <Source or Destination Port: 4737>
>> [Stream index: 0]
>> [TCP Segment Len: 46]
>> Sequence number: 1 (relative sequence number)
>> Sequence number (raw): 3765145522
>> [Next sequence number: 47 (relative sequence number)]
>> Acknowledgment number: 1 (relative ack number)
>> Acknowledgment number (raw): 274707205
>> 0101 .... = Header Length: 20 bytes (5)
>> Flags: 0x018 (PSH, ACK)
>> Window size value: 10000
>> [Calculated window size: 10000]
>> [Window size scaling factor: 1]
>> Checksum: 0x5fd7 [unverified]
>> [Checksum Status: Unverified]
>> Urgent pointer: 0
>> [SEQ/ACK analysis]
>> [Timestamps]
>> TCP payload (46 bytes)
>> [PDU Size: 46]
>> IPDR
>> Version: 2
>> Message id: CONNECT (5)
>> Session id: 0
>> Message flags: 0x00
>> Message length: 46
>> Initiator id: 0.0.0.0
>> Initiator port: 0
>> Capabilities: 0x00000006
>> Keep-alive interval: 90
>> Vendor id: ARRIS E6000 Exporter
>>
>> Here somethings that i tried:
>>
>> - Increase the number of simultaneos connections and batch size, but it
>> does not change at all.
>> - Use listentcprecord but the behavior is the same, it only change when
>> i select the avroreader, then i got an error informing that is a "wrong
>> data format".
>>
>> So i belive that nifi is receving the messages that come from tcp , but
>> for some reason is not generating flowfiles.
>>
>> Does anyone got same behavior ?
>> There is anything else that i can do ?
>>
>> i'm using nifi 1.15.3 with a 4 nodes cluster.
>>
>>
>> Breno
>>
>>
>>
>> --
>
> "Genius might be described as a supreme capacity for getting its possessors
> into trouble of all kinds."
> -- Samuel Butler
>
Re: ListenTCP not generating flowfiles
Posted by Michael Loftis <ml...@wgops.com>.
ListenTCP isn’t fit for ingesting arbitrary data. It’s record oriented,
default separator is Unix newline “\n”. I don’t think it will generate a
flow if it never receives a newline before the connection closes.
On Wed, Feb 9, 2022 at 18:17 Breno Cesar <br...@gmail.com> wrote:
> i'm tring to receive a tcp stream using the listentcp module, but when the
> connection start, i see the traffic reaching my server (usgin tcpdump), but
> the nifi does note generate a flow file.
> Ther is no error ow warning in the module or in the nifi cluster.
> Changing in the processor the bulleting level to "debug" , i can see the
> following message for every package that i receive:
>
> SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000-0000-000040efa66e]
> Accepted incoming connection from /192.0.2.12:55209
> SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000-0000-000040efa66e] bytes
> read 46
> SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000-0000-000040efa66e] No
> more data available, returning for selection
> SPOLPAVMW131:8080ListenTCP[id=e083186d-017e-1000-0000-000040efa66e]
> Reached EOF, closing connection
>
> bellow is the same message, but in the tcpdump format.
>
> Internet Protocol Version 4, Src: 192.0.2.12, Dst: 192.0.2.128
> Transmission Control Protocol, Src Port: 55209 , Dst Port: 4737, Seq: 1,
> Ack: 1, Len: 46
> Source Port: 55209
> Destination Port: 4737
> <Source or Destination Port: 55209 >
> <Source or Destination Port: 4737>
> [Stream index: 0]
> [TCP Segment Len: 46]
> Sequence number: 1 (relative sequence number)
> Sequence number (raw): 3765145522
> [Next sequence number: 47 (relative sequence number)]
> Acknowledgment number: 1 (relative ack number)
> Acknowledgment number (raw): 274707205
> 0101 .... = Header Length: 20 bytes (5)
> Flags: 0x018 (PSH, ACK)
> Window size value: 10000
> [Calculated window size: 10000]
> [Window size scaling factor: 1]
> Checksum: 0x5fd7 [unverified]
> [Checksum Status: Unverified]
> Urgent pointer: 0
> [SEQ/ACK analysis]
> [Timestamps]
> TCP payload (46 bytes)
> [PDU Size: 46]
> IPDR
> Version: 2
> Message id: CONNECT (5)
> Session id: 0
> Message flags: 0x00
> Message length: 46
> Initiator id: 0.0.0.0
> Initiator port: 0
> Capabilities: 0x00000006
> Keep-alive interval: 90
> Vendor id: ARRIS E6000 Exporter
>
> Here somethings that i tried:
>
> - Increase the number of simultaneos connections and batch size, but it
> does not change at all.
> - Use listentcprecord but the behavior is the same, it only change when i
> select the avroreader, then i got an error informing that is a "wrong data
> format".
>
> So i belive that nifi is receving the messages that come from tcp , but
> for some reason is not generating flowfiles.
>
> Does anyone got same behavior ?
> There is anything else that i can do ?
>
> i'm using nifi 1.15.3 with a 4 nodes cluster.
>
>
> Breno
>
>
>
> --
"Genius might be described as a supreme capacity for getting its possessors
into trouble of all kinds."
-- Samuel Butler