You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Rajesh Biswas <ra...@bridgera.com> on 2018/10/08 17:52:29 UTC
Error log in ListenTCP
Hello Team,
I am facing one issue to get binary data into Nifi.
As of now I am using ListenTCP processor, when I am sending character string
data I am able to see the data in flow file.
But when I am trying to send Binary data, I am not able to see any data thru
Data province.
Would you please guide me:
1. Which log files to see if there are any error (I checked
nifi-app.log and nifi-user.log, did not see anything)
2. Which processor to use to read binary data thru TCP protocol (I am
not able to change Character Set other than UTF-8)
Would you please help me with your experience?
Thanks and Regards,
Rajesh Biswas | +91 9886433461 | <http://www.bridgera.com/>
www.bridgera.com
---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus
Re: Error log in ListenTCP
Posted by Bryan Bende <bb...@gmail.com>.
ListenTCP only uses the Character Set property for writing the
demarcator to the output stream of the flow file, it takes the String
value of Character Set and calls value.getBytes(charset).
For the actual data it is never converted from bytes to characters, so
it takes byte[] from the incoming stream and writes byte[] directly to
the flow file output stream.
On Tue, Oct 9, 2018 at 11:17 AM Rajesh Biswas
<ra...@bridgera.com> wrote:
>
> Hello Bryan,
>
> We need to read binary stream data thru TCP.
> As the default chatacter encoding is UTF-8 for ListenTCP, the binary data is getting altered after passing through the processors.
> Would you please suggest any other alternative?
> Below is the sample data:
> "00010001990010000015B00100019900101001"
> The processor group we are using
> ListenTCP -> PutFile
>
> Configuration for ListenTCP
> Local Network InterfaceNo value set No Value
> Port 9111
> Receive Buffer Size 65507 B
> Max Size of Message Queue 10000
> Max Size of Socket Buffer 1 MB
> Character Set UTF-8
> Max Batch Size 1
> Batching Message Delimiter \n
> Max Number of TCP Connections 2
> SSL Context ServiceNo value set
> Client Auth NONE
>
>
> Thank you very much:
>
>
> Thanks and Regards,
> Rajesh Biswas | +91 9886433461 | www.bridgera.com
>
> -----Original Message-----
> From: Bryan Bende [mailto:bbende@gmail.com]
> Sent: Tuesday, October 9, 2018 1:20 AM
> To: dev@nifi.apache.org
> Subject: Re: Error log in ListenTCP
>
> ListenTCP uses new-line characters to determine logical message boundaries,
> so the data itself does not have to be characters, but there has to be
> new-lines indicating the end of a logical piece of data.
>
> Imagine streaming the content of several images over the same connection,
> the client would have to send something like the following... bytes of
> image 1, new line, bytes of image 2, new line, bytes of image 3, new line,
> etc.
>
>
> On Mon, Oct 8, 2018 at 1:52 PM Rajesh Biswas <ra...@bridgera.com>
> wrote:
>
> > Hello Team,
> >
> > I am facing one issue to get binary data into Nifi.
> >
> > As of now I am using ListenTCP processor, when I am sending character
> > string data I am able to see the data in flow file.
> >
> > But when I am trying to send Binary data, I am not able to see any data
> > thru Data province.
> >
> > Would you please guide me:
> >
> > 1. Which log files to see if there are any error (I checked
> > nifi-app.log and nifi-user.log, did not see anything)
> >
> > 2. Which processor to use to read binary data thru TCP protocol (I
> > am not able to change Character Set other than UTF-8)
> >
> >
> >
> > Would you please help me with your experience?
> >
> >
> >
> > Thanks and Regards,
> >
> > *Rajesh Biswas* | +91 9886433461 | www.bridgera.com
> >
> >
> >
> >
> > <https://www.avast.com/en-in/recommend?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient&utm_term=default3&tag=8220120d-9b09-4fc1-8af8-da3fc2cdea29> I’m
> > protected online with Avast Free Antivirus. Get it here — it’s free
> > forever.
> > <https://www.avast.com/en-in/recommend?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient&utm_term=default3&tag=8220120d-9b09-4fc1-8af8-da3fc2cdea29>
> > <#m_-3931440512116331171_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
> >
>
>
> ---
> This email has been checked for viruses by Avast antivirus software.
> https://www.avast.com/antivirus
>
RE: Error log in ListenTCP
Posted by Rajesh Biswas <ra...@bridgera.com>.
Hello Bryan,
We need to read binary stream data thru TCP.
As the default chatacter encoding is UTF-8 for ListenTCP, the binary data is getting altered after passing through the processors.
Would you please suggest any other alternative?
Below is the sample data:
"00010001990010000015B00100019900101001"
The processor group we are using
ListenTCP -> PutFile
Configuration for ListenTCP
Local Network InterfaceNo value set No Value
Port 9111
Receive Buffer Size 65507 B
Max Size of Message Queue 10000
Max Size of Socket Buffer 1 MB
Character Set UTF-8
Max Batch Size 1
Batching Message Delimiter \n
Max Number of TCP Connections 2
SSL Context ServiceNo value set
Client Auth NONE
Thank you very much:
Thanks and Regards,
Rajesh Biswas | +91 9886433461 | www.bridgera.com
-----Original Message-----
From: Bryan Bende [mailto:bbende@gmail.com]
Sent: Tuesday, October 9, 2018 1:20 AM
To: dev@nifi.apache.org
Subject: Re: Error log in ListenTCP
ListenTCP uses new-line characters to determine logical message boundaries,
so the data itself does not have to be characters, but there has to be
new-lines indicating the end of a logical piece of data.
Imagine streaming the content of several images over the same connection,
the client would have to send something like the following... bytes of
image 1, new line, bytes of image 2, new line, bytes of image 3, new line,
etc.
On Mon, Oct 8, 2018 at 1:52 PM Rajesh Biswas <ra...@bridgera.com>
wrote:
> Hello Team,
>
> I am facing one issue to get binary data into Nifi.
>
> As of now I am using ListenTCP processor, when I am sending character
> string data I am able to see the data in flow file.
>
> But when I am trying to send Binary data, I am not able to see any data
> thru Data province.
>
> Would you please guide me:
>
> 1. Which log files to see if there are any error (I checked
> nifi-app.log and nifi-user.log, did not see anything)
>
> 2. Which processor to use to read binary data thru TCP protocol (I
> am not able to change Character Set other than UTF-8)
>
>
>
> Would you please help me with your experience?
>
>
>
> Thanks and Regards,
>
> *Rajesh Biswas* | +91 9886433461 | www.bridgera.com
>
>
>
>
> <https://www.avast.com/en-in/recommend?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient&utm_term=default3&tag=8220120d-9b09-4fc1-8af8-da3fc2cdea29> I’m
> protected online with Avast Free Antivirus. Get it here — it’s free
> forever.
> <https://www.avast.com/en-in/recommend?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient&utm_term=default3&tag=8220120d-9b09-4fc1-8af8-da3fc2cdea29>
> <#m_-3931440512116331171_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>
---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus
Re: Error log in ListenTCP
Posted by Bryan Bende <bb...@gmail.com>.
ListenTCP uses new-line characters to determine logical message boundaries,
so the data itself does not have to be characters, but there has to be
new-lines indicating the end of a logical piece of data.
Imagine streaming the content of several images over the same connection,
the client would have to send something like the following... bytes of
image 1, new line, bytes of image 2, new line, bytes of image 3, new line,
etc.
On Mon, Oct 8, 2018 at 1:52 PM Rajesh Biswas <ra...@bridgera.com>
wrote:
> Hello Team,
>
> I am facing one issue to get binary data into Nifi.
>
> As of now I am using ListenTCP processor, when I am sending character
> string data I am able to see the data in flow file.
>
> But when I am trying to send Binary data, I am not able to see any data
> thru Data province.
>
> Would you please guide me:
>
> 1. Which log files to see if there are any error (I checked
> nifi-app.log and nifi-user.log, did not see anything)
>
> 2. Which processor to use to read binary data thru TCP protocol (I
> am not able to change Character Set other than UTF-8)
>
>
>
> Would you please help me with your experience?
>
>
>
> Thanks and Regards,
>
> *Rajesh Biswas* | +91 9886433461 | www.bridgera.com
>
>
>
>
> <https://www.avast.com/en-in/recommend?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient&utm_term=default3&tag=8220120d-9b09-4fc1-8af8-da3fc2cdea29> I’m
> protected online with Avast Free Antivirus. Get it here — it’s free
> forever.
> <https://www.avast.com/en-in/recommend?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient&utm_term=default3&tag=8220120d-9b09-4fc1-8af8-da3fc2cdea29>
> <#m_-3931440512116331171_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>