You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Gino Lisignoli <gl...@gmail.com> on 2017/09/13 20:06:49 UTC
New ListFTP (and possibly ListXXX) processors fail to pickup new files
In 1.4.0-SNAPSHOT, the ListFTP processor fails to add new files after
several hours.
At the moment I'm testing this on a cluster of 2. But I'm looking to
replicate my issue on a single instance as well.
I'm not sure if this is a cluster state problem, a processor state problem
or the new ListXXX commits (
https://github.com/apache/nifi/commit/e68ff153e81ddb82d1136d44a96bdb7a70da86d1
and
https://github.com/apache/nifi/commit/28ee70222b892fb799f5f74a31a9de678d9fb629
).
I'll be looking at those commits first around the ListFTP process. Unless
theres any advice can be offered.
Should I submit this to Jira issues, or wait until I can confirm/replicate
the behaviour?
Re: New ListFTP (and possibly ListXXX) processors fail to pickup new
files
Posted by Bryan Bende <bb...@gmail.com>.
A single node still needs a state manager to survive restarts of the application, but there is no more distributed state at that point, so the local state manager is backed by a write-ahead-log store in ./state under NiFi’s home directory.
- Bryan
> On Sep 20, 2017, at 8:06 PM, Gino Lisignoli <gl...@gmail.com> wrote:
>
> I can confirm the sate is being managed by zookeeper. I've been doing some more testing today and I haven't been able to make the ListXXX processes fail by manually causing the primary node to change (just using tcpdump to drop zookeeper traffic, causing a node change). So I'm not sure at the moment if it's because I'm in a cluster or not. I'm setting up a seperate nifi node that will only perform ListXXX operations without being in a cluster.
>
> Though even if this is a single node, will it need a state manager configured?
>
> On Thu, Sep 21, 2017 at 10:58 AM, Bryan Bende <bbende@gmail.com <ma...@gmail.com>> wrote:
> This should be correct behavior... the cache approach was used before the state manager got introduced, and the cache was left there to migrate old state, but we do need to pick a release and remove those lingering cache properties.
>
> Do you have the cluster state manager configured in state managers xml?
>
> It should be using zoo keeper when in a cluster.
>
> On Wed, Sep 20, 2017 at 6:23 PM Gino Lisignoli <glisignoli@gmail.com <ma...@gmail.com>> wrote:
> Hi Joe
>
> The distributed cache service property of ListXXX doesn't seem to matter anymore. Which was quite a surprise to me. It just gets handled by the state manager: https://github.com/apache/nifi/blob/e68ff153e81ddb82d1136d44a96bdb7a70da86d1/nifi-nar-bundles/nifi-extension-utils/nifi-processor-utils/src/main/java/org/apache/nifi/processor/util/list/AbstractListProcessor.java#L264 <https://github.com/apache/nifi/blob/e68ff153e81ddb82d1136d44a96bdb7a70da86d1/nifi-nar-bundles/nifi-extension-utils/nifi-processor-utils/src/main/java/org/apache/nifi/processor/util/list/AbstractListProcessor.java#L264>
>
> You can test this via setting the Distributed Cache Service to a redis server and watch no connections ever being made, which was how I found out. Not sure if this is intended or not.
>
>
> On Wed, Sep 20, 2017 at 12:42 PM, Joe Witt <joe.witt@gmail.com <ma...@gmail.com>> wrote:
> Gino
>
> When the primary node shifts the new primary node should take over the
> function of doing the listings for you. You'll want to be using the
> distributed cache property of the ListFTP/etc.. processors to help
> keep state in such cases.
>
> Thanks
>
> On Tue, Sep 19, 2017 at 4:50 PM, Gino Lisignoli <glisignoli@gmail.com <ma...@gmail.com>> wrote:
> > Hi
> >
> > I can confirm that the cause of my problem is the primary node shifting to a
> > new node in my cluster. This causes all of the ListXXX processors to stop
> > working (as they are set to only run on my primary node). I'm not sure what
> > is causing my primary node to shift, or why the processors are unable to
> > recover when the node shifts. If I find any more info I'll raise a ticket.
> >
> > On Thu, Sep 14, 2017 at 8:08 AM, Joe Witt <joe.witt@gmail.com <ma...@gmail.com>> wrote:
> >>
> >> Thanks Gino. If you can confirm/replicate the behavior please do file a
> >> jira.
> >>
> >> Thanks
> >>
> >> On Wed, Sep 13, 2017 at 4:06 PM, Gino Lisignoli <glisignoli@gmail.com <ma...@gmail.com>>
> >> wrote:
> >> > In 1.4.0-SNAPSHOT, the ListFTP processor fails to add new files after
> >> > several hours.
> >> >
> >> > At the moment I'm testing this on a cluster of 2. But I'm looking to
> >> > replicate my issue on a single instance as well.
> >> >
> >> > I'm not sure if this is a cluster state problem, a processor state
> >> > problem
> >> > or the new ListXXX commits
> >> >
> >> > (https://github.com/apache/nifi/commit/e68ff153e81ddb82d1136d44a96bdb7a70da86d1 <https://github.com/apache/nifi/commit/e68ff153e81ddb82d1136d44a96bdb7a70da86d1>
> >> > and
> >> >
> >> > https://github.com/apache/nifi/commit/28ee70222b892fb799f5f74a31a9de678d9fb629 <https://github.com/apache/nifi/commit/28ee70222b892fb799f5f74a31a9de678d9fb629>).
> >> >
> >> > I'll be looking at those commits first around the ListFTP process.
> >> > Unless
> >> > theres any advice can be offered.
> >> >
> >> > Should I submit this to Jira issues, or wait until I can
> >> > confirm/replicate
> >> > the behaviour?
> >
> >
>
> --
> Sent from Gmail Mobile
>
Re: New ListFTP (and possibly ListXXX) processors fail to pickup new files
Posted by Gino Lisignoli <gl...@gmail.com>.
I can confirm the sate is being managed by zookeeper. I've been doing some
more testing today and I haven't been able to make the ListXXX processes
fail by manually causing the primary node to change (just using tcpdump to
drop zookeeper traffic, causing a node change). So I'm not sure at the
moment if it's because I'm in a cluster or not. I'm setting up a seperate
nifi node that will only perform ListXXX operations without being in a
cluster.
Though even if this is a single node, will it need a state manager
configured?
On Thu, Sep 21, 2017 at 10:58 AM, Bryan Bende <bb...@gmail.com> wrote:
> This should be correct behavior... the cache approach was used before the
> state manager got introduced, and the cache was left there to migrate old
> state, but we do need to pick a release and remove those lingering cache
> properties.
>
> Do you have the cluster state manager configured in state managers xml?
>
> It should be using zoo keeper when in a cluster.
>
> On Wed, Sep 20, 2017 at 6:23 PM Gino Lisignoli <gl...@gmail.com>
> wrote:
>
>> Hi Joe
>>
>> The distributed cache service property of ListXXX doesn't seem to matter
>> anymore. Which was quite a surprise to me. It just gets handled by the
>> state manager: https://github.com/apache/nifi/blob/
>> e68ff153e81ddb82d1136d44a96bdb7a70da86d1/nifi-nar-bundles/
>> nifi-extension-utils/nifi-processor-utils/src/main/java/
>> org/apache/nifi/processor/util/list/AbstractListProcessor.java#L264
>>
>> You can test this via setting the Distributed Cache Service to a redis
>> server and watch no connections ever being made, which was how I found out.
>> Not sure if this is intended or not.
>>
>>
>> On Wed, Sep 20, 2017 at 12:42 PM, Joe Witt <jo...@gmail.com> wrote:
>>
>>> Gino
>>>
>>> When the primary node shifts the new primary node should take over the
>>> function of doing the listings for you. You'll want to be using the
>>> distributed cache property of the ListFTP/etc.. processors to help
>>> keep state in such cases.
>>>
>>> Thanks
>>>
>>> On Tue, Sep 19, 2017 at 4:50 PM, Gino Lisignoli <gl...@gmail.com>
>>> wrote:
>>> > Hi
>>> >
>>> > I can confirm that the cause of my problem is the primary node
>>> shifting to a
>>> > new node in my cluster. This causes all of the ListXXX processors to
>>> stop
>>> > working (as they are set to only run on my primary node). I'm not sure
>>> what
>>> > is causing my primary node to shift, or why the processors are unable
>>> to
>>> > recover when the node shifts. If I find any more info I'll raise a
>>> ticket.
>>> >
>>> > On Thu, Sep 14, 2017 at 8:08 AM, Joe Witt <jo...@gmail.com> wrote:
>>> >>
>>> >> Thanks Gino. If you can confirm/replicate the behavior please do
>>> file a
>>> >> jira.
>>> >>
>>> >> Thanks
>>> >>
>>> >> On Wed, Sep 13, 2017 at 4:06 PM, Gino Lisignoli <glisignoli@gmail.com
>>> >
>>> >> wrote:
>>> >> > In 1.4.0-SNAPSHOT, the ListFTP processor fails to add new files
>>> after
>>> >> > several hours.
>>> >> >
>>> >> > At the moment I'm testing this on a cluster of 2. But I'm looking to
>>> >> > replicate my issue on a single instance as well.
>>> >> >
>>> >> > I'm not sure if this is a cluster state problem, a processor state
>>> >> > problem
>>> >> > or the new ListXXX commits
>>> >> >
>>> >> > (https://github.com/apache/nifi/commit/
>>> e68ff153e81ddb82d1136d44a96bdb7a70da86d1
>>> >> > and
>>> >> >
>>> >> > https://github.com/apache/nifi/commit/
>>> 28ee70222b892fb799f5f74a31a9de678d9fb629).
>>> >> >
>>> >> > I'll be looking at those commits first around the ListFTP process.
>>> >> > Unless
>>> >> > theres any advice can be offered.
>>> >> >
>>> >> > Should I submit this to Jira issues, or wait until I can
>>> >> > confirm/replicate
>>> >> > the behaviour?
>>> >
>>> >
>>>
>>
>> --
> Sent from Gmail Mobile
>
Re: New ListFTP (and possibly ListXXX) processors fail to pickup new files
Posted by Bryan Bende <bb...@gmail.com>.
This should be correct behavior... the cache approach was used before the
state manager got introduced, and the cache was left there to migrate old
state, but we do need to pick a release and remove those lingering cache
properties.
Do you have the cluster state manager configured in state managers xml?
It should be using zoo keeper when in a cluster.
On Wed, Sep 20, 2017 at 6:23 PM Gino Lisignoli <gl...@gmail.com> wrote:
> Hi Joe
>
> The distributed cache service property of ListXXX doesn't seem to matter
> anymore. Which was quite a surprise to me. It just gets handled by the
> state manager:
> https://github.com/apache/nifi/blob/e68ff153e81ddb82d1136d44a96bdb7a70da86d1/nifi-nar-bundles/nifi-extension-utils/nifi-processor-utils/src/main/java/org/apache/nifi/processor/util/list/AbstractListProcessor.java#L264
>
> You can test this via setting the Distributed Cache Service to a redis
> server and watch no connections ever being made, which was how I found out.
> Not sure if this is intended or not.
>
>
> On Wed, Sep 20, 2017 at 12:42 PM, Joe Witt <jo...@gmail.com> wrote:
>
>> Gino
>>
>> When the primary node shifts the new primary node should take over the
>> function of doing the listings for you. You'll want to be using the
>> distributed cache property of the ListFTP/etc.. processors to help
>> keep state in such cases.
>>
>> Thanks
>>
>> On Tue, Sep 19, 2017 at 4:50 PM, Gino Lisignoli <gl...@gmail.com>
>> wrote:
>> > Hi
>> >
>> > I can confirm that the cause of my problem is the primary node shifting
>> to a
>> > new node in my cluster. This causes all of the ListXXX processors to
>> stop
>> > working (as they are set to only run on my primary node). I'm not sure
>> what
>> > is causing my primary node to shift, or why the processors are unable to
>> > recover when the node shifts. If I find any more info I'll raise a
>> ticket.
>> >
>> > On Thu, Sep 14, 2017 at 8:08 AM, Joe Witt <jo...@gmail.com> wrote:
>> >>
>> >> Thanks Gino. If you can confirm/replicate the behavior please do file
>> a
>> >> jira.
>> >>
>> >> Thanks
>> >>
>> >> On Wed, Sep 13, 2017 at 4:06 PM, Gino Lisignoli <gl...@gmail.com>
>> >> wrote:
>> >> > In 1.4.0-SNAPSHOT, the ListFTP processor fails to add new files after
>> >> > several hours.
>> >> >
>> >> > At the moment I'm testing this on a cluster of 2. But I'm looking to
>> >> > replicate my issue on a single instance as well.
>> >> >
>> >> > I'm not sure if this is a cluster state problem, a processor state
>> >> > problem
>> >> > or the new ListXXX commits
>> >> >
>> >> > (
>> https://github.com/apache/nifi/commit/e68ff153e81ddb82d1136d44a96bdb7a70da86d1
>> >> > and
>> >> >
>> >> >
>> https://github.com/apache/nifi/commit/28ee70222b892fb799f5f74a31a9de678d9fb629
>> ).
>> >> >
>> >> > I'll be looking at those commits first around the ListFTP process.
>> >> > Unless
>> >> > theres any advice can be offered.
>> >> >
>> >> > Should I submit this to Jira issues, or wait until I can
>> >> > confirm/replicate
>> >> > the behaviour?
>> >
>> >
>>
>
> --
Sent from Gmail Mobile
Re: New ListFTP (and possibly ListXXX) processors fail to pickup new files
Posted by Gino Lisignoli <gl...@gmail.com>.
Hi Joe
The distributed cache service property of ListXXX doesn't seem to matter
anymore. Which was quite a surprise to me. It just gets handled by the
state manager:
https://github.com/apache/nifi/blob/e68ff153e81ddb82d1136d44a96bdb7a70da86d1/nifi-nar-bundles/nifi-extension-utils/nifi-processor-utils/src/main/java/org/apache/nifi/processor/util/list/AbstractListProcessor.java#L264
You can test this via setting the Distributed Cache Service to a redis
server and watch no connections ever being made, which was how I found out.
Not sure if this is intended or not.
On Wed, Sep 20, 2017 at 12:42 PM, Joe Witt <jo...@gmail.com> wrote:
> Gino
>
> When the primary node shifts the new primary node should take over the
> function of doing the listings for you. You'll want to be using the
> distributed cache property of the ListFTP/etc.. processors to help
> keep state in such cases.
>
> Thanks
>
> On Tue, Sep 19, 2017 at 4:50 PM, Gino Lisignoli <gl...@gmail.com>
> wrote:
> > Hi
> >
> > I can confirm that the cause of my problem is the primary node shifting
> to a
> > new node in my cluster. This causes all of the ListXXX processors to stop
> > working (as they are set to only run on my primary node). I'm not sure
> what
> > is causing my primary node to shift, or why the processors are unable to
> > recover when the node shifts. If I find any more info I'll raise a
> ticket.
> >
> > On Thu, Sep 14, 2017 at 8:08 AM, Joe Witt <jo...@gmail.com> wrote:
> >>
> >> Thanks Gino. If you can confirm/replicate the behavior please do file a
> >> jira.
> >>
> >> Thanks
> >>
> >> On Wed, Sep 13, 2017 at 4:06 PM, Gino Lisignoli <gl...@gmail.com>
> >> wrote:
> >> > In 1.4.0-SNAPSHOT, the ListFTP processor fails to add new files after
> >> > several hours.
> >> >
> >> > At the moment I'm testing this on a cluster of 2. But I'm looking to
> >> > replicate my issue on a single instance as well.
> >> >
> >> > I'm not sure if this is a cluster state problem, a processor state
> >> > problem
> >> > or the new ListXXX commits
> >> >
> >> > (https://github.com/apache/nifi/commit/e68ff153e81ddb82d1136d44a96bdb
> 7a70da86d1
> >> > and
> >> >
> >> > https://github.com/apache/nifi/commit/28ee70222b892fb799f5f74a31a9de
> 678d9fb629).
> >> >
> >> > I'll be looking at those commits first around the ListFTP process.
> >> > Unless
> >> > theres any advice can be offered.
> >> >
> >> > Should I submit this to Jira issues, or wait until I can
> >> > confirm/replicate
> >> > the behaviour?
> >
> >
>
Re: New ListFTP (and possibly ListXXX) processors fail to pickup new files
Posted by Joe Witt <jo...@gmail.com>.
Gino
When the primary node shifts the new primary node should take over the
function of doing the listings for you. You'll want to be using the
distributed cache property of the ListFTP/etc.. processors to help
keep state in such cases.
Thanks
On Tue, Sep 19, 2017 at 4:50 PM, Gino Lisignoli <gl...@gmail.com> wrote:
> Hi
>
> I can confirm that the cause of my problem is the primary node shifting to a
> new node in my cluster. This causes all of the ListXXX processors to stop
> working (as they are set to only run on my primary node). I'm not sure what
> is causing my primary node to shift, or why the processors are unable to
> recover when the node shifts. If I find any more info I'll raise a ticket.
>
> On Thu, Sep 14, 2017 at 8:08 AM, Joe Witt <jo...@gmail.com> wrote:
>>
>> Thanks Gino. If you can confirm/replicate the behavior please do file a
>> jira.
>>
>> Thanks
>>
>> On Wed, Sep 13, 2017 at 4:06 PM, Gino Lisignoli <gl...@gmail.com>
>> wrote:
>> > In 1.4.0-SNAPSHOT, the ListFTP processor fails to add new files after
>> > several hours.
>> >
>> > At the moment I'm testing this on a cluster of 2. But I'm looking to
>> > replicate my issue on a single instance as well.
>> >
>> > I'm not sure if this is a cluster state problem, a processor state
>> > problem
>> > or the new ListXXX commits
>> >
>> > (https://github.com/apache/nifi/commit/e68ff153e81ddb82d1136d44a96bdb7a70da86d1
>> > and
>> >
>> > https://github.com/apache/nifi/commit/28ee70222b892fb799f5f74a31a9de678d9fb629).
>> >
>> > I'll be looking at those commits first around the ListFTP process.
>> > Unless
>> > theres any advice can be offered.
>> >
>> > Should I submit this to Jira issues, or wait until I can
>> > confirm/replicate
>> > the behaviour?
>
>
Re: New ListFTP (and possibly ListXXX) processors fail to pickup new files
Posted by Gino Lisignoli <gl...@gmail.com>.
Hi
I can confirm that the cause of my problem is the primary node shifting to
a new node in my cluster. This causes all of the ListXXX processors to stop
working (as they are set to only run on my primary node). I'm not sure what
is causing my primary node to shift, or why the processors are unable to
recover when the node shifts. If I find any more info I'll raise a ticket.
On Thu, Sep 14, 2017 at 8:08 AM, Joe Witt <jo...@gmail.com> wrote:
> Thanks Gino. If you can confirm/replicate the behavior please do file a
> jira.
>
> Thanks
>
> On Wed, Sep 13, 2017 at 4:06 PM, Gino Lisignoli <gl...@gmail.com>
> wrote:
> > In 1.4.0-SNAPSHOT, the ListFTP processor fails to add new files after
> > several hours.
> >
> > At the moment I'm testing this on a cluster of 2. But I'm looking to
> > replicate my issue on a single instance as well.
> >
> > I'm not sure if this is a cluster state problem, a processor state
> problem
> > or the new ListXXX commits
> > (https://github.com/apache/nifi/commit/e68ff153e81ddb82d1136d44a96bdb
> 7a70da86d1
> > and
> > https://github.com/apache/nifi/commit/28ee70222b892fb799f5f74a31a9de
> 678d9fb629).
> >
> > I'll be looking at those commits first around the ListFTP process. Unless
> > theres any advice can be offered.
> >
> > Should I submit this to Jira issues, or wait until I can
> confirm/replicate
> > the behaviour?
>
Re: New ListFTP (and possibly ListXXX) processors fail to pickup new files
Posted by Joe Witt <jo...@gmail.com>.
Thanks Gino. If you can confirm/replicate the behavior please do file a jira.
Thanks
On Wed, Sep 13, 2017 at 4:06 PM, Gino Lisignoli <gl...@gmail.com> wrote:
> In 1.4.0-SNAPSHOT, the ListFTP processor fails to add new files after
> several hours.
>
> At the moment I'm testing this on a cluster of 2. But I'm looking to
> replicate my issue on a single instance as well.
>
> I'm not sure if this is a cluster state problem, a processor state problem
> or the new ListXXX commits
> (https://github.com/apache/nifi/commit/e68ff153e81ddb82d1136d44a96bdb7a70da86d1
> and
> https://github.com/apache/nifi/commit/28ee70222b892fb799f5f74a31a9de678d9fb629).
>
> I'll be looking at those commits first around the ListFTP process. Unless
> theres any advice can be offered.
>
> Should I submit this to Jira issues, or wait until I can confirm/replicate
> the behaviour?