You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Tony Kurc <tr...@gmail.com> on 2016/03/04 20:57:03 UTC
heisenbug causing "lost" content claims
All,
I wanted to describe an issue on a nifi instance we've been using 0.4.1 on,
and why diagnosing it and reproducing it may be difficult. This is on a
linux server, where we have a reasonably high load, and the error happens
infrequently, but when it does, it really gums up operations.
At some point we get an IOException for too many open files. (with an
awfully high limit of open files in ulimit, so not sure why that is
happening).
Some time later, when trying to read a flowfile in a processor, we get a
ContentNotFoundException because presumably a flowfile is pointing to
content that was never written. When this happens, we basically have to
remove the flowfile manually (and if no one is watching at the moment or
the processor that reads isn't configured to handle this, or if you're not
using 0.5.x where you can selectively remove flowfiles from a queue this
can cause operational challenges).
Because this happens so infrequently, I'm not sure if others have seen
this. I'm not sure if something in the framework may need to adjustment if
a content claim goes wrong, but I really didn't expect that a flowfile with
no actual content should be able to be created, which seems to be what
happened (rather than the content being deleted or corrupted).
Anyone else experience this, or know maybe if something in 0.5.X may have
addressed this (looking through the release notes, nothing jumped out).
Tony
Re: heisenbug causing "lost" content claims
Posted by Michael Moser <mo...@gmail.com>.
MergeContent exists in the flow in several places. I haven't noticed any
UnknownHostExceptions, though my grep of the logs is still running ...
-- Mike
On Fri, Mar 4, 2016 at 5:13 PM, Joe Gresock <jg...@gmail.com> wrote:
> Are there any UnknownHostExceptions in the logs prior to entering this
> state? We had a similar problem where NiFi kept opening sockets to connect
> to an RPG and eventually ran into "too many open files" after several
> days. We had to add the fqdn of the RPG host to our /etc/hosts file before
> NiFi would resolve it, and the problem went away.
>
> On Fri, Mar 4, 2016 at 5:04 PM, Joe Witt <jo...@gmail.com> wrote:
>
> > Mike,
> >
> > Does this flow have MergeContent processor on it?
> >
> > Thanks
> > Joe
> >
> > On Fri, Mar 4, 2016 at 4:59 PM, Michael Moser <mo...@gmail.com>
> wrote:
> > > Thanks for the reply, Mark.
> > >
> > > NIFI-1577 isn't the cause because I don't think we were using any
> > processor
> > > that does ProcessSession.append().
> > > NIFI-1527 mentions a problem that occurs when NiFi starts, and our NiFi
> > had
> > > been running for several days.
> > >
> > > Setting aside the "Too many open files" cause for the moment. Here's
> > what
> > > we saw when the NiFi JVM encountered Too many open files:
> > >
> > > ERROR [Site-to-Site Worker Thread]
> > o.a.nifi.remote.SocketRemoteSiteListener
> > > Unable to communicate with remote instance due to
> > > o.a.nifi.processor.exception.ProcessException:
> > > o.a.nifi.processor.exception.FlowFileAccessException: Failed to import
> > data
> > > from org.apache.nifi.stream.io.MinimumLengthInputStream@1234 for
> > > StandardFlowFileRecord[uuid=foo,claim=,offset=0,name=filename,size=0]
> due
> > > to org.apache.nifi.processor.exception.FlowFileAccessException: Unable
> to
> > > create ContentClaim due to java.io.FileNotFoundException:
> > > content_repository/1/1-1 (Too many open files); closing connection
> > >
> > > This NiFi instance was using a remote process group Input Port to
> accept
> > > new files. It appears after the exception that a flowfile exists in
> the
> > > flowfile_repository but the ContentClaim doesn't get a chance to exist
> in
> > > the content_repository.
> > >
> > > -- Mike
> > >
> > >
> > >
> > > On Fri, Mar 4, 2016 at 3:03 PM, Mark Payne <ma...@hotmail.com>
> wrote:
> > >
> > >> Tony,
> > >>
> > >> The two tickets that come to mind are:
> > >> https://issues.apache.org/jira/browse/NIFI-1577 <
> > >> https://issues.apache.org/jira/browse/NIFI-1577> (Too many open
> files)
> > >> https://issues.apache.org/jira/browse/NIFI-1527 <
> > >> https://issues.apache.org/jira/browse/NIFI-1527> (ContentNotFound)
> > >>
> > >> Do these sound like they may be what is causing your issues?
> > >>
> > >> Thanks
> > >> -Mark
> > >>
> > >>
> > >> > On Mar 4, 2016, at 2:57 PM, Tony Kurc <tr...@gmail.com> wrote:
> > >> >
> > >> > All,
> > >> > I wanted to describe an issue on a nifi instance we've been using
> > 0.4.1
> > >> on,
> > >> > and why diagnosing it and reproducing it may be difficult. This is
> on
> > a
> > >> > linux server, where we have a reasonably high load, and the error
> > happens
> > >> > infrequently, but when it does, it really gums up operations.
> > >> >
> > >> > At some point we get an IOException for too many open files. (with
> an
> > >> > awfully high limit of open files in ulimit, so not sure why that is
> > >> > happening).
> > >> >
> > >> > Some time later, when trying to read a flowfile in a processor, we
> > get a
> > >> > ContentNotFoundException because presumably a flowfile is pointing
> to
> > >> > content that was never written. When this happens, we basically have
> > to
> > >> > remove the flowfile manually (and if no one is watching at the
> moment
> > or
> > >> > the processor that reads isn't configured to handle this, or if
> you're
> > >> not
> > >> > using 0.5.x where you can selectively remove flowfiles from a queue
> > this
> > >> > can cause operational challenges).
> > >> >
> > >> > Because this happens so infrequently, I'm not sure if others have
> seen
> > >> > this. I'm not sure if something in the framework may need to
> > adjustment
> > >> if
> > >> > a content claim goes wrong, but I really didn't expect that a
> flowfile
> > >> with
> > >> > no actual content should be able to be created, which seems to be
> what
> > >> > happened (rather than the content being deleted or corrupted).
> > >> >
> > >> > Anyone else experience this, or know maybe if something in 0.5.X may
> > have
> > >> > addressed this (looking through the release notes, nothing jumped
> > out).
> > >> >
> > >> > Tony
> > >>
> > >>
> >
>
>
>
> --
> I know what it is to be in need, and I know what it is to have plenty. I
> have learned the secret of being content in any and every situation,
> whether well fed or hungry, whether living in plenty or in want. I can do
> all this through him who gives me strength. *-Philippians 4:12-13*
>
> This email has been sent from a virus-free computer protected by Avast.
> www.avast.com
> <
> https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail
> >
> <#DDB4FAA8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>
Re: heisenbug causing "lost" content claims
Posted by Joe Gresock <jg...@gmail.com>.
Are there any UnknownHostExceptions in the logs prior to entering this
state? We had a similar problem where NiFi kept opening sockets to connect
to an RPG and eventually ran into "too many open files" after several
days. We had to add the fqdn of the RPG host to our /etc/hosts file before
NiFi would resolve it, and the problem went away.
On Fri, Mar 4, 2016 at 5:04 PM, Joe Witt <jo...@gmail.com> wrote:
> Mike,
>
> Does this flow have MergeContent processor on it?
>
> Thanks
> Joe
>
> On Fri, Mar 4, 2016 at 4:59 PM, Michael Moser <mo...@gmail.com> wrote:
> > Thanks for the reply, Mark.
> >
> > NIFI-1577 isn't the cause because I don't think we were using any
> processor
> > that does ProcessSession.append().
> > NIFI-1527 mentions a problem that occurs when NiFi starts, and our NiFi
> had
> > been running for several days.
> >
> > Setting aside the "Too many open files" cause for the moment. Here's
> what
> > we saw when the NiFi JVM encountered Too many open files:
> >
> > ERROR [Site-to-Site Worker Thread]
> o.a.nifi.remote.SocketRemoteSiteListener
> > Unable to communicate with remote instance due to
> > o.a.nifi.processor.exception.ProcessException:
> > o.a.nifi.processor.exception.FlowFileAccessException: Failed to import
> data
> > from org.apache.nifi.stream.io.MinimumLengthInputStream@1234 for
> > StandardFlowFileRecord[uuid=foo,claim=,offset=0,name=filename,size=0] due
> > to org.apache.nifi.processor.exception.FlowFileAccessException: Unable to
> > create ContentClaim due to java.io.FileNotFoundException:
> > content_repository/1/1-1 (Too many open files); closing connection
> >
> > This NiFi instance was using a remote process group Input Port to accept
> > new files. It appears after the exception that a flowfile exists in the
> > flowfile_repository but the ContentClaim doesn't get a chance to exist in
> > the content_repository.
> >
> > -- Mike
> >
> >
> >
> > On Fri, Mar 4, 2016 at 3:03 PM, Mark Payne <ma...@hotmail.com> wrote:
> >
> >> Tony,
> >>
> >> The two tickets that come to mind are:
> >> https://issues.apache.org/jira/browse/NIFI-1577 <
> >> https://issues.apache.org/jira/browse/NIFI-1577> (Too many open files)
> >> https://issues.apache.org/jira/browse/NIFI-1527 <
> >> https://issues.apache.org/jira/browse/NIFI-1527> (ContentNotFound)
> >>
> >> Do these sound like they may be what is causing your issues?
> >>
> >> Thanks
> >> -Mark
> >>
> >>
> >> > On Mar 4, 2016, at 2:57 PM, Tony Kurc <tr...@gmail.com> wrote:
> >> >
> >> > All,
> >> > I wanted to describe an issue on a nifi instance we've been using
> 0.4.1
> >> on,
> >> > and why diagnosing it and reproducing it may be difficult. This is on
> a
> >> > linux server, where we have a reasonably high load, and the error
> happens
> >> > infrequently, but when it does, it really gums up operations.
> >> >
> >> > At some point we get an IOException for too many open files. (with an
> >> > awfully high limit of open files in ulimit, so not sure why that is
> >> > happening).
> >> >
> >> > Some time later, when trying to read a flowfile in a processor, we
> get a
> >> > ContentNotFoundException because presumably a flowfile is pointing to
> >> > content that was never written. When this happens, we basically have
> to
> >> > remove the flowfile manually (and if no one is watching at the moment
> or
> >> > the processor that reads isn't configured to handle this, or if you're
> >> not
> >> > using 0.5.x where you can selectively remove flowfiles from a queue
> this
> >> > can cause operational challenges).
> >> >
> >> > Because this happens so infrequently, I'm not sure if others have seen
> >> > this. I'm not sure if something in the framework may need to
> adjustment
> >> if
> >> > a content claim goes wrong, but I really didn't expect that a flowfile
> >> with
> >> > no actual content should be able to be created, which seems to be what
> >> > happened (rather than the content being deleted or corrupted).
> >> >
> >> > Anyone else experience this, or know maybe if something in 0.5.X may
> have
> >> > addressed this (looking through the release notes, nothing jumped
> out).
> >> >
> >> > Tony
> >>
> >>
>
--
I know what it is to be in need, and I know what it is to have plenty. I
have learned the secret of being content in any and every situation,
whether well fed or hungry, whether living in plenty or in want. I can do
all this through him who gives me strength. *-Philippians 4:12-13*
This email has been sent from a virus-free computer protected by Avast.
www.avast.com
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
<#DDB4FAA8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
Re: heisenbug causing "lost" content claims
Posted by Joe Witt <jo...@gmail.com>.
Mike,
Does this flow have MergeContent processor on it?
Thanks
Joe
On Fri, Mar 4, 2016 at 4:59 PM, Michael Moser <mo...@gmail.com> wrote:
> Thanks for the reply, Mark.
>
> NIFI-1577 isn't the cause because I don't think we were using any processor
> that does ProcessSession.append().
> NIFI-1527 mentions a problem that occurs when NiFi starts, and our NiFi had
> been running for several days.
>
> Setting aside the "Too many open files" cause for the moment. Here's what
> we saw when the NiFi JVM encountered Too many open files:
>
> ERROR [Site-to-Site Worker Thread] o.a.nifi.remote.SocketRemoteSiteListener
> Unable to communicate with remote instance due to
> o.a.nifi.processor.exception.ProcessException:
> o.a.nifi.processor.exception.FlowFileAccessException: Failed to import data
> from org.apache.nifi.stream.io.MinimumLengthInputStream@1234 for
> StandardFlowFileRecord[uuid=foo,claim=,offset=0,name=filename,size=0] due
> to org.apache.nifi.processor.exception.FlowFileAccessException: Unable to
> create ContentClaim due to java.io.FileNotFoundException:
> content_repository/1/1-1 (Too many open files); closing connection
>
> This NiFi instance was using a remote process group Input Port to accept
> new files. It appears after the exception that a flowfile exists in the
> flowfile_repository but the ContentClaim doesn't get a chance to exist in
> the content_repository.
>
> -- Mike
>
>
>
> On Fri, Mar 4, 2016 at 3:03 PM, Mark Payne <ma...@hotmail.com> wrote:
>
>> Tony,
>>
>> The two tickets that come to mind are:
>> https://issues.apache.org/jira/browse/NIFI-1577 <
>> https://issues.apache.org/jira/browse/NIFI-1577> (Too many open files)
>> https://issues.apache.org/jira/browse/NIFI-1527 <
>> https://issues.apache.org/jira/browse/NIFI-1527> (ContentNotFound)
>>
>> Do these sound like they may be what is causing your issues?
>>
>> Thanks
>> -Mark
>>
>>
>> > On Mar 4, 2016, at 2:57 PM, Tony Kurc <tr...@gmail.com> wrote:
>> >
>> > All,
>> > I wanted to describe an issue on a nifi instance we've been using 0.4.1
>> on,
>> > and why diagnosing it and reproducing it may be difficult. This is on a
>> > linux server, where we have a reasonably high load, and the error happens
>> > infrequently, but when it does, it really gums up operations.
>> >
>> > At some point we get an IOException for too many open files. (with an
>> > awfully high limit of open files in ulimit, so not sure why that is
>> > happening).
>> >
>> > Some time later, when trying to read a flowfile in a processor, we get a
>> > ContentNotFoundException because presumably a flowfile is pointing to
>> > content that was never written. When this happens, we basically have to
>> > remove the flowfile manually (and if no one is watching at the moment or
>> > the processor that reads isn't configured to handle this, or if you're
>> not
>> > using 0.5.x where you can selectively remove flowfiles from a queue this
>> > can cause operational challenges).
>> >
>> > Because this happens so infrequently, I'm not sure if others have seen
>> > this. I'm not sure if something in the framework may need to adjustment
>> if
>> > a content claim goes wrong, but I really didn't expect that a flowfile
>> with
>> > no actual content should be able to be created, which seems to be what
>> > happened (rather than the content being deleted or corrupted).
>> >
>> > Anyone else experience this, or know maybe if something in 0.5.X may have
>> > addressed this (looking through the release notes, nothing jumped out).
>> >
>> > Tony
>>
>>
Re: heisenbug causing "lost" content claims
Posted by Michael Moser <mo...@gmail.com>.
Thanks for the reply, Mark.
NIFI-1577 isn't the cause because I don't think we were using any processor
that does ProcessSession.append().
NIFI-1527 mentions a problem that occurs when NiFi starts, and our NiFi had
been running for several days.
Setting aside the "Too many open files" cause for the moment. Here's what
we saw when the NiFi JVM encountered Too many open files:
ERROR [Site-to-Site Worker Thread] o.a.nifi.remote.SocketRemoteSiteListener
Unable to communicate with remote instance due to
o.a.nifi.processor.exception.ProcessException:
o.a.nifi.processor.exception.FlowFileAccessException: Failed to import data
from org.apache.nifi.stream.io.MinimumLengthInputStream@1234 for
StandardFlowFileRecord[uuid=foo,claim=,offset=0,name=filename,size=0] due
to org.apache.nifi.processor.exception.FlowFileAccessException: Unable to
create ContentClaim due to java.io.FileNotFoundException:
content_repository/1/1-1 (Too many open files); closing connection
This NiFi instance was using a remote process group Input Port to accept
new files. It appears after the exception that a flowfile exists in the
flowfile_repository but the ContentClaim doesn't get a chance to exist in
the content_repository.
-- Mike
On Fri, Mar 4, 2016 at 3:03 PM, Mark Payne <ma...@hotmail.com> wrote:
> Tony,
>
> The two tickets that come to mind are:
> https://issues.apache.org/jira/browse/NIFI-1577 <
> https://issues.apache.org/jira/browse/NIFI-1577> (Too many open files)
> https://issues.apache.org/jira/browse/NIFI-1527 <
> https://issues.apache.org/jira/browse/NIFI-1527> (ContentNotFound)
>
> Do these sound like they may be what is causing your issues?
>
> Thanks
> -Mark
>
>
> > On Mar 4, 2016, at 2:57 PM, Tony Kurc <tr...@gmail.com> wrote:
> >
> > All,
> > I wanted to describe an issue on a nifi instance we've been using 0.4.1
> on,
> > and why diagnosing it and reproducing it may be difficult. This is on a
> > linux server, where we have a reasonably high load, and the error happens
> > infrequently, but when it does, it really gums up operations.
> >
> > At some point we get an IOException for too many open files. (with an
> > awfully high limit of open files in ulimit, so not sure why that is
> > happening).
> >
> > Some time later, when trying to read a flowfile in a processor, we get a
> > ContentNotFoundException because presumably a flowfile is pointing to
> > content that was never written. When this happens, we basically have to
> > remove the flowfile manually (and if no one is watching at the moment or
> > the processor that reads isn't configured to handle this, or if you're
> not
> > using 0.5.x where you can selectively remove flowfiles from a queue this
> > can cause operational challenges).
> >
> > Because this happens so infrequently, I'm not sure if others have seen
> > this. I'm not sure if something in the framework may need to adjustment
> if
> > a content claim goes wrong, but I really didn't expect that a flowfile
> with
> > no actual content should be able to be created, which seems to be what
> > happened (rather than the content being deleted or corrupted).
> >
> > Anyone else experience this, or know maybe if something in 0.5.X may have
> > addressed this (looking through the release notes, nothing jumped out).
> >
> > Tony
>
>
Re: heisenbug causing "lost" content claims
Posted by Mark Payne <ma...@hotmail.com>.
Tony,
The two tickets that come to mind are:
https://issues.apache.org/jira/browse/NIFI-1577 <https://issues.apache.org/jira/browse/NIFI-1577> (Too many open files)
https://issues.apache.org/jira/browse/NIFI-1527 <https://issues.apache.org/jira/browse/NIFI-1527> (ContentNotFound)
Do these sound like they may be what is causing your issues?
Thanks
-Mark
> On Mar 4, 2016, at 2:57 PM, Tony Kurc <tr...@gmail.com> wrote:
>
> All,
> I wanted to describe an issue on a nifi instance we've been using 0.4.1 on,
> and why diagnosing it and reproducing it may be difficult. This is on a
> linux server, where we have a reasonably high load, and the error happens
> infrequently, but when it does, it really gums up operations.
>
> At some point we get an IOException for too many open files. (with an
> awfully high limit of open files in ulimit, so not sure why that is
> happening).
>
> Some time later, when trying to read a flowfile in a processor, we get a
> ContentNotFoundException because presumably a flowfile is pointing to
> content that was never written. When this happens, we basically have to
> remove the flowfile manually (and if no one is watching at the moment or
> the processor that reads isn't configured to handle this, or if you're not
> using 0.5.x where you can selectively remove flowfiles from a queue this
> can cause operational challenges).
>
> Because this happens so infrequently, I'm not sure if others have seen
> this. I'm not sure if something in the framework may need to adjustment if
> a content claim goes wrong, but I really didn't expect that a flowfile with
> no actual content should be able to be created, which seems to be what
> happened (rather than the content being deleted or corrupted).
>
> Anyone else experience this, or know maybe if something in 0.5.X may have
> addressed this (looking through the release notes, nothing jumped out).
>
> Tony