You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Ryan Ward <ry...@gmail.com> on 2018/01/26 14:56:31 UTC
putTCP odd behavior with EL
I'm seeing odd behavior trying to use attributes for the hostname and port
fields.
using ${endpoint_port} (9003) + hardcoded IP results in flowfile yielding
failed to process session due to java.lang.NumberFormatException: For input
string: "": {}
java.lang.NumberFormatException: For input string: ""
at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:592)
at java.lang.Integer.parseInt(Integer.java:615)
at
org.apache.nifi.attribute.expression.language.StandardPropertyValue.asInteger(StandardPropertyValue.java:78)
at org.apache.nifi.processors.standard.PutTCP.createSender
(PutTCP.java:111)
...
at org.apache.nifi.processors.standard.PutTCP.createSender
(PutTCP.java:179)
using hardcoded 9003 + ${endpoint} results in flowfile failing due to
connection refused
DEBUG ...No available connections, creating a new one...
ERROR ...No available connections, and unable to create a new one ....to
failure: java.net.ConnectException: Connection refused
Adding listenTCP to the cluster on 9003 leaving ${endpoint} and hardcoded
port
DEBUG...Connected to local port 23250
DEBUG....Relinquishing sender
Flow files transferred to success, its unclear where the data went or why I
needed to have the nodes listening on this port. Is the attribute value
being ignored and defaulting to localhost? Watching this behavior via
netstat I could see 127.0.0.1 was indeed connected to itself on 9003. Odd
thing is no data came in on the ListenTCP either.
Thanks,
Ryan
Re: putTCP odd behavior with EL
Posted by Ryan Ward <ry...@gmail.com>.
So I understand, is the concern with multiple flow files with different
destinations in the same queue would require the processor close the
connection for each flow file? Could you use the same connection pool
approach similar to fetchSFTP with keeping connections open for a period of
time.
In the variable approach it would require a processor per host:port pair? I
was trying to get around multiple processors and having to create a
spiderweb with distribute load to handle failures by using update attribute
with state and automatically assigning the endpoint based on a counter. We
do something similar with putSFTP postHTTP. It would be great if the
post/put processors could accept a list or array of hosts with some basic
load balance options.
On Fri, Jan 26, 2018 at 12:29 PM, Bryan Bende <bb...@gmail.com> wrote:
> In the default case, "Connection Per Flow File" is false, which means
> there is one connection created and used across many flow files, which
> will perform best.
>
> Setting "Connection Per Flow File" to true, means it will close the
> connection at the end of every on trigger call.
>
> We could potentially evaluate against flows when "Connection Per Flow
> File" is true b/c then we can create a new one every time, but when it
> is false it means you would have to potentially keep open many
> connections.
>
> It also may be confusing though to have differing evaluation based on
> another property.
>
> On Fri, Jan 26, 2018 at 12:16 PM, Ryan Ward <ry...@gmail.com> wrote:
> > Thanks, that's a good idea, as I would automatically assume if a property
> > supports EL it would work against flowfiles. I see the ticket 3231 was
> > specific to variable registry and explicitly says not flow files. Any
> > particular reason?
> >
> > On Fri, Jan 26, 2018 at 12:02 PM, Pierre Villard <
> > pierre.villard.fr@gmail.com> wrote:
> >
> >> And to add a bit of info on the last remark from Joe, I'm working on
> >> NIFI-4149 to make the scope of EL clearer to users. Didn't have time to
> >> work on it lately but will definitely try to get back to it very soon.
> >>
> >> Pierre
> >>
> >> 2018-01-26 17:14 GMT+01:00 Joe Witt <jo...@gmail.com>:
> >>
> >> > Stated differently - it considers what it can glean from variables
> >> > made available other than the flowfile itself. With Apache NiFi 1.5.0
> >> > that means process group variables, variables set in nifi.properites,
> >> > and environment variables.
> >> >
> >> > We need to make sure that we call this out when we indicate a given
> >> > property supports expression language so the user knows the scope at
> >> > which it will help them.
> >> >
> >> > On Fri, Jan 26, 2018 at 10:44 AM, Marco Gaido <marcogaido91@gmail.com
> >
> >> > wrote:
> >> > > Hi Ryan,
> >> > >
> >> > > probably the reason of the behavior is that EL on PutTCP is enabled
> but
> >> > it
> >> > > is not run on the incoming flowfile. So it doesn't care the
> attributes
> >> of
> >> > > your flowfile. It considers only environment variables.
> >> > >
> >> > > Thanks,
> >> > > Marco
> >> > >
> >> > > 2018-01-26 15:56 GMT+01:00 Ryan Ward <ry...@gmail.com>:
> >> > >
> >> > >> I'm seeing odd behavior trying to use attributes for the hostname
> and
> >> > port
> >> > >> fields.
> >> > >>
> >> > >> using ${endpoint_port} (9003) + hardcoded IP results in flowfile
> >> > yielding
> >> > >> failed to process session due to java.lang.NumberFormatException:
> For
> >> > >> input
> >> > >> string: "": {}
> >> > >> java.lang.NumberFormatException: For input string: ""
> >> > >> at
> >> > >> java.lang.NumberFormatException.forInputString(
> >> > >> NumberFormatException.java:65)
> >> > >> at java.lang.Integer.parseInt(Integer.java:592)
> >> > >> at java.lang.Integer.parseInt(Integer.java:615)
> >> > >> at
> >> > >> org.apache.nifi.attribute.expression.language.
> StandardPropertyValue.
> >> > >> asInteger(StandardPropertyValue.java:78)
> >> > >>
> >> > >> at org.apache.nifi.processors.standard.PutTCP.createSender
> >> > >> (PutTCP.java:111)
> >> > >> ...
> >> > >> at org.apache.nifi.processors.standard.PutTCP.createSender
> >> > >> (PutTCP.java:179)
> >> > >>
> >> > >> using hardcoded 9003 + ${endpoint} results in flowfile failing due
> to
> >> > >> connection refused
> >> > >> DEBUG ...No available connections, creating a new one...
> >> > >> ERROR ...No available connections, and unable to create a new one
> >> ....to
> >> > >> failure: java.net.ConnectException: Connection refused
> >> > >>
> >> > >> Adding listenTCP to the cluster on 9003 leaving ${endpoint} and
> >> > hardcoded
> >> > >> port
> >> > >> DEBUG...Connected to local port 23250
> >> > >> DEBUG....Relinquishing sender
> >> > >> Flow files transferred to success, its unclear where the data went
> or
> >> > why I
> >> > >> needed to have the nodes listening on this port. Is the attribute
> >> value
> >> > >> being ignored and defaulting to localhost? Watching this behavior
> via
> >> > >> netstat I could see 127.0.0.1 was indeed connected to itself on
> 9003.
> >> > Odd
> >> > >> thing is no data came in on the ListenTCP either.
> >> > >>
> >> > >> Thanks,
> >> > >> Ryan
> >> > >>
> >> >
> >>
>
Re: putTCP odd behavior with EL
Posted by Bryan Bende <bb...@gmail.com>.
In the default case, "Connection Per Flow File" is false, which means
there is one connection created and used across many flow files, which
will perform best.
Setting "Connection Per Flow File" to true, means it will close the
connection at the end of every on trigger call.
We could potentially evaluate against flows when "Connection Per Flow
File" is true b/c then we can create a new one every time, but when it
is false it means you would have to potentially keep open many
connections.
It also may be confusing though to have differing evaluation based on
another property.
On Fri, Jan 26, 2018 at 12:16 PM, Ryan Ward <ry...@gmail.com> wrote:
> Thanks, that's a good idea, as I would automatically assume if a property
> supports EL it would work against flowfiles. I see the ticket 3231 was
> specific to variable registry and explicitly says not flow files. Any
> particular reason?
>
> On Fri, Jan 26, 2018 at 12:02 PM, Pierre Villard <
> pierre.villard.fr@gmail.com> wrote:
>
>> And to add a bit of info on the last remark from Joe, I'm working on
>> NIFI-4149 to make the scope of EL clearer to users. Didn't have time to
>> work on it lately but will definitely try to get back to it very soon.
>>
>> Pierre
>>
>> 2018-01-26 17:14 GMT+01:00 Joe Witt <jo...@gmail.com>:
>>
>> > Stated differently - it considers what it can glean from variables
>> > made available other than the flowfile itself. With Apache NiFi 1.5.0
>> > that means process group variables, variables set in nifi.properites,
>> > and environment variables.
>> >
>> > We need to make sure that we call this out when we indicate a given
>> > property supports expression language so the user knows the scope at
>> > which it will help them.
>> >
>> > On Fri, Jan 26, 2018 at 10:44 AM, Marco Gaido <ma...@gmail.com>
>> > wrote:
>> > > Hi Ryan,
>> > >
>> > > probably the reason of the behavior is that EL on PutTCP is enabled but
>> > it
>> > > is not run on the incoming flowfile. So it doesn't care the attributes
>> of
>> > > your flowfile. It considers only environment variables.
>> > >
>> > > Thanks,
>> > > Marco
>> > >
>> > > 2018-01-26 15:56 GMT+01:00 Ryan Ward <ry...@gmail.com>:
>> > >
>> > >> I'm seeing odd behavior trying to use attributes for the hostname and
>> > port
>> > >> fields.
>> > >>
>> > >> using ${endpoint_port} (9003) + hardcoded IP results in flowfile
>> > yielding
>> > >> failed to process session due to java.lang.NumberFormatException: For
>> > >> input
>> > >> string: "": {}
>> > >> java.lang.NumberFormatException: For input string: ""
>> > >> at
>> > >> java.lang.NumberFormatException.forInputString(
>> > >> NumberFormatException.java:65)
>> > >> at java.lang.Integer.parseInt(Integer.java:592)
>> > >> at java.lang.Integer.parseInt(Integer.java:615)
>> > >> at
>> > >> org.apache.nifi.attribute.expression.language.StandardPropertyValue.
>> > >> asInteger(StandardPropertyValue.java:78)
>> > >>
>> > >> at org.apache.nifi.processors.standard.PutTCP.createSender
>> > >> (PutTCP.java:111)
>> > >> ...
>> > >> at org.apache.nifi.processors.standard.PutTCP.createSender
>> > >> (PutTCP.java:179)
>> > >>
>> > >> using hardcoded 9003 + ${endpoint} results in flowfile failing due to
>> > >> connection refused
>> > >> DEBUG ...No available connections, creating a new one...
>> > >> ERROR ...No available connections, and unable to create a new one
>> ....to
>> > >> failure: java.net.ConnectException: Connection refused
>> > >>
>> > >> Adding listenTCP to the cluster on 9003 leaving ${endpoint} and
>> > hardcoded
>> > >> port
>> > >> DEBUG...Connected to local port 23250
>> > >> DEBUG....Relinquishing sender
>> > >> Flow files transferred to success, its unclear where the data went or
>> > why I
>> > >> needed to have the nodes listening on this port. Is the attribute
>> value
>> > >> being ignored and defaulting to localhost? Watching this behavior via
>> > >> netstat I could see 127.0.0.1 was indeed connected to itself on 9003.
>> > Odd
>> > >> thing is no data came in on the ListenTCP either.
>> > >>
>> > >> Thanks,
>> > >> Ryan
>> > >>
>> >
>>
Re: putTCP odd behavior with EL
Posted by Ryan Ward <ry...@gmail.com>.
Thanks, that's a good idea, as I would automatically assume if a property
supports EL it would work against flowfiles. I see the ticket 3231 was
specific to variable registry and explicitly says not flow files. Any
particular reason?
On Fri, Jan 26, 2018 at 12:02 PM, Pierre Villard <
pierre.villard.fr@gmail.com> wrote:
> And to add a bit of info on the last remark from Joe, I'm working on
> NIFI-4149 to make the scope of EL clearer to users. Didn't have time to
> work on it lately but will definitely try to get back to it very soon.
>
> Pierre
>
> 2018-01-26 17:14 GMT+01:00 Joe Witt <jo...@gmail.com>:
>
> > Stated differently - it considers what it can glean from variables
> > made available other than the flowfile itself. With Apache NiFi 1.5.0
> > that means process group variables, variables set in nifi.properites,
> > and environment variables.
> >
> > We need to make sure that we call this out when we indicate a given
> > property supports expression language so the user knows the scope at
> > which it will help them.
> >
> > On Fri, Jan 26, 2018 at 10:44 AM, Marco Gaido <ma...@gmail.com>
> > wrote:
> > > Hi Ryan,
> > >
> > > probably the reason of the behavior is that EL on PutTCP is enabled but
> > it
> > > is not run on the incoming flowfile. So it doesn't care the attributes
> of
> > > your flowfile. It considers only environment variables.
> > >
> > > Thanks,
> > > Marco
> > >
> > > 2018-01-26 15:56 GMT+01:00 Ryan Ward <ry...@gmail.com>:
> > >
> > >> I'm seeing odd behavior trying to use attributes for the hostname and
> > port
> > >> fields.
> > >>
> > >> using ${endpoint_port} (9003) + hardcoded IP results in flowfile
> > yielding
> > >> failed to process session due to java.lang.NumberFormatException: For
> > >> input
> > >> string: "": {}
> > >> java.lang.NumberFormatException: For input string: ""
> > >> at
> > >> java.lang.NumberFormatException.forInputString(
> > >> NumberFormatException.java:65)
> > >> at java.lang.Integer.parseInt(Integer.java:592)
> > >> at java.lang.Integer.parseInt(Integer.java:615)
> > >> at
> > >> org.apache.nifi.attribute.expression.language.StandardPropertyValue.
> > >> asInteger(StandardPropertyValue.java:78)
> > >>
> > >> at org.apache.nifi.processors.standard.PutTCP.createSender
> > >> (PutTCP.java:111)
> > >> ...
> > >> at org.apache.nifi.processors.standard.PutTCP.createSender
> > >> (PutTCP.java:179)
> > >>
> > >> using hardcoded 9003 + ${endpoint} results in flowfile failing due to
> > >> connection refused
> > >> DEBUG ...No available connections, creating a new one...
> > >> ERROR ...No available connections, and unable to create a new one
> ....to
> > >> failure: java.net.ConnectException: Connection refused
> > >>
> > >> Adding listenTCP to the cluster on 9003 leaving ${endpoint} and
> > hardcoded
> > >> port
> > >> DEBUG...Connected to local port 23250
> > >> DEBUG....Relinquishing sender
> > >> Flow files transferred to success, its unclear where the data went or
> > why I
> > >> needed to have the nodes listening on this port. Is the attribute
> value
> > >> being ignored and defaulting to localhost? Watching this behavior via
> > >> netstat I could see 127.0.0.1 was indeed connected to itself on 9003.
> > Odd
> > >> thing is no data came in on the ListenTCP either.
> > >>
> > >> Thanks,
> > >> Ryan
> > >>
> >
>
Re: putTCP odd behavior with EL
Posted by Pierre Villard <pi...@gmail.com>.
And to add a bit of info on the last remark from Joe, I'm working on
NIFI-4149 to make the scope of EL clearer to users. Didn't have time to
work on it lately but will definitely try to get back to it very soon.
Pierre
2018-01-26 17:14 GMT+01:00 Joe Witt <jo...@gmail.com>:
> Stated differently - it considers what it can glean from variables
> made available other than the flowfile itself. With Apache NiFi 1.5.0
> that means process group variables, variables set in nifi.properites,
> and environment variables.
>
> We need to make sure that we call this out when we indicate a given
> property supports expression language so the user knows the scope at
> which it will help them.
>
> On Fri, Jan 26, 2018 at 10:44 AM, Marco Gaido <ma...@gmail.com>
> wrote:
> > Hi Ryan,
> >
> > probably the reason of the behavior is that EL on PutTCP is enabled but
> it
> > is not run on the incoming flowfile. So it doesn't care the attributes of
> > your flowfile. It considers only environment variables.
> >
> > Thanks,
> > Marco
> >
> > 2018-01-26 15:56 GMT+01:00 Ryan Ward <ry...@gmail.com>:
> >
> >> I'm seeing odd behavior trying to use attributes for the hostname and
> port
> >> fields.
> >>
> >> using ${endpoint_port} (9003) + hardcoded IP results in flowfile
> yielding
> >> failed to process session due to java.lang.NumberFormatException: For
> >> input
> >> string: "": {}
> >> java.lang.NumberFormatException: For input string: ""
> >> at
> >> java.lang.NumberFormatException.forInputString(
> >> NumberFormatException.java:65)
> >> at java.lang.Integer.parseInt(Integer.java:592)
> >> at java.lang.Integer.parseInt(Integer.java:615)
> >> at
> >> org.apache.nifi.attribute.expression.language.StandardPropertyValue.
> >> asInteger(StandardPropertyValue.java:78)
> >>
> >> at org.apache.nifi.processors.standard.PutTCP.createSender
> >> (PutTCP.java:111)
> >> ...
> >> at org.apache.nifi.processors.standard.PutTCP.createSender
> >> (PutTCP.java:179)
> >>
> >> using hardcoded 9003 + ${endpoint} results in flowfile failing due to
> >> connection refused
> >> DEBUG ...No available connections, creating a new one...
> >> ERROR ...No available connections, and unable to create a new one ....to
> >> failure: java.net.ConnectException: Connection refused
> >>
> >> Adding listenTCP to the cluster on 9003 leaving ${endpoint} and
> hardcoded
> >> port
> >> DEBUG...Connected to local port 23250
> >> DEBUG....Relinquishing sender
> >> Flow files transferred to success, its unclear where the data went or
> why I
> >> needed to have the nodes listening on this port. Is the attribute value
> >> being ignored and defaulting to localhost? Watching this behavior via
> >> netstat I could see 127.0.0.1 was indeed connected to itself on 9003.
> Odd
> >> thing is no data came in on the ListenTCP either.
> >>
> >> Thanks,
> >> Ryan
> >>
>
Re: putTCP odd behavior with EL
Posted by Joe Witt <jo...@gmail.com>.
Stated differently - it considers what it can glean from variables
made available other than the flowfile itself. With Apache NiFi 1.5.0
that means process group variables, variables set in nifi.properites,
and environment variables.
We need to make sure that we call this out when we indicate a given
property supports expression language so the user knows the scope at
which it will help them.
On Fri, Jan 26, 2018 at 10:44 AM, Marco Gaido <ma...@gmail.com> wrote:
> Hi Ryan,
>
> probably the reason of the behavior is that EL on PutTCP is enabled but it
> is not run on the incoming flowfile. So it doesn't care the attributes of
> your flowfile. It considers only environment variables.
>
> Thanks,
> Marco
>
> 2018-01-26 15:56 GMT+01:00 Ryan Ward <ry...@gmail.com>:
>
>> I'm seeing odd behavior trying to use attributes for the hostname and port
>> fields.
>>
>> using ${endpoint_port} (9003) + hardcoded IP results in flowfile yielding
>> failed to process session due to java.lang.NumberFormatException: For
>> input
>> string: "": {}
>> java.lang.NumberFormatException: For input string: ""
>> at
>> java.lang.NumberFormatException.forInputString(
>> NumberFormatException.java:65)
>> at java.lang.Integer.parseInt(Integer.java:592)
>> at java.lang.Integer.parseInt(Integer.java:615)
>> at
>> org.apache.nifi.attribute.expression.language.StandardPropertyValue.
>> asInteger(StandardPropertyValue.java:78)
>>
>> at org.apache.nifi.processors.standard.PutTCP.createSender
>> (PutTCP.java:111)
>> ...
>> at org.apache.nifi.processors.standard.PutTCP.createSender
>> (PutTCP.java:179)
>>
>> using hardcoded 9003 + ${endpoint} results in flowfile failing due to
>> connection refused
>> DEBUG ...No available connections, creating a new one...
>> ERROR ...No available connections, and unable to create a new one ....to
>> failure: java.net.ConnectException: Connection refused
>>
>> Adding listenTCP to the cluster on 9003 leaving ${endpoint} and hardcoded
>> port
>> DEBUG...Connected to local port 23250
>> DEBUG....Relinquishing sender
>> Flow files transferred to success, its unclear where the data went or why I
>> needed to have the nodes listening on this port. Is the attribute value
>> being ignored and defaulting to localhost? Watching this behavior via
>> netstat I could see 127.0.0.1 was indeed connected to itself on 9003. Odd
>> thing is no data came in on the ListenTCP either.
>>
>> Thanks,
>> Ryan
>>
Re: putTCP odd behavior with EL
Posted by Marco Gaido <ma...@gmail.com>.
Hi Ryan,
probably the reason of the behavior is that EL on PutTCP is enabled but it
is not run on the incoming flowfile. So it doesn't care the attributes of
your flowfile. It considers only environment variables.
Thanks,
Marco
2018-01-26 15:56 GMT+01:00 Ryan Ward <ry...@gmail.com>:
> I'm seeing odd behavior trying to use attributes for the hostname and port
> fields.
>
> using ${endpoint_port} (9003) + hardcoded IP results in flowfile yielding
> failed to process session due to java.lang.NumberFormatException: For
> input
> string: "": {}
> java.lang.NumberFormatException: For input string: ""
> at
> java.lang.NumberFormatException.forInputString(
> NumberFormatException.java:65)
> at java.lang.Integer.parseInt(Integer.java:592)
> at java.lang.Integer.parseInt(Integer.java:615)
> at
> org.apache.nifi.attribute.expression.language.StandardPropertyValue.
> asInteger(StandardPropertyValue.java:78)
>
> at org.apache.nifi.processors.standard.PutTCP.createSender
> (PutTCP.java:111)
> ...
> at org.apache.nifi.processors.standard.PutTCP.createSender
> (PutTCP.java:179)
>
> using hardcoded 9003 + ${endpoint} results in flowfile failing due to
> connection refused
> DEBUG ...No available connections, creating a new one...
> ERROR ...No available connections, and unable to create a new one ....to
> failure: java.net.ConnectException: Connection refused
>
> Adding listenTCP to the cluster on 9003 leaving ${endpoint} and hardcoded
> port
> DEBUG...Connected to local port 23250
> DEBUG....Relinquishing sender
> Flow files transferred to success, its unclear where the data went or why I
> needed to have the nodes listening on this port. Is the attribute value
> being ignored and defaulting to localhost? Watching this behavior via
> netstat I could see 127.0.0.1 was indeed connected to itself on 9003. Odd
> thing is no data came in on the ListenTCP either.
>
> Thanks,
> Ryan
>