You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Selvam Raman <se...@gmail.com> on 2016/09/28 11:14:59 UTC
State management not captured in cluster mode
Hi,
This is my state-management.xml attribute
<local-provider>
<id>local-provider</id>
<class>org.apache.nifi.controller.state.providers.local.WriteAheadLocalStateProvider</class>
<property name="Directory">./state/local</property>
</local-provider>
<cluster-provider>
<id>zk-provider</id>
<class>org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider</class>
<property name="Connect
String">hostname:2181,hostname:2181</property>
<property name="Root Node">/opt/nifiroot</property>
<property name="Session Timeout">10 seconds</property>
<property name="Access Control">Open</property>
</cluster-provider>
This is my nifi.poperties file attributes
####################
# State Management #
####################
nifi.state.management.configuration.file=./conf/state-management.xml
# The ID of the local state provider
nifi.state.management.provider.local=local-provider
# The ID of the cluster-wide state provider. This will be ignored if NiFi
is not clustered but must be populated if running in a cluster.
nifi.state.management.provider.cluster=zk-provider
# Specifies whether or not this instance of NiFi should run an embedded
ZooKeeper server
nifi.state.management.embedded.zookeeper.start=false
# Properties file that provides the ZooKeeper properties to use if
<nifi.state.management.embedded.zookeeper.start> is set to true
nifi.state.management.embedded.zookeeper.properties=./conf/zookeeper.properties
# zookeeper properties, used for cluster management #
nifi.zookeeper.connect.string=hostname:2181,hostname:2181
nifi.zookeeper.connect.timeout=3 secs
nifi.zookeeper.session.timeout=3 secs
nifi.zookeeper.root.node=/opt/nifiroot
the question here is, i am running nifi in cluster mode and i am expecting
state should be stored in zk-provider. But the state stored in
local-provider.
local-state provider:
/home/nifi/nifi-1.0.0/state/local/partition-*
zk-provider:(empty directory)
/opt/nifiroot
any help on this.
--
Selvam Raman
"லஞ்சம் தவிர்த்து நெஞ்சம் நிமிர்த்து"
Re: State management not captured in cluster mode
Posted by Pierre Villard <pi...@gmail.com>.
At the moment the sate management description (if any) of a processor is
available by right clicking on it and going into "View state".
Pierre
2016-09-28 16:15 GMT+02:00 Bryan Bende <bb...@gmail.com>:
> What I was referring to is in the code of each processor, it is annotated
> with something describing the state.
>
> For example:
> https://github.com/apache/nifi/blob/master/nifi-nar-
> bundles/nifi-standard-bundle/nifi-standard-processors/src/
> main/java/org/apache/nifi/processors/standard/ListFile.java#L102-L105
>
> Currently this is not visible anywhere in the NiFi UI, but what Joe
> referenced was an improvement that Pierre submitted so that we can display
> this information in the documentation for the processor when clicking
> "usage".
>
>
> On Wed, Sep 28, 2016 at 10:09 AM, Selvam Raman <se...@gmail.com> wrote:
>
>> Hi Bryan,
>>
>> Thanks for the information. Can you please share a picture where i can
>> able to see the state (local or cluster). I could not see anywhere.
>>
>> On Wed, Sep 28, 2016 at 1:22 PM, Bryan Bende <bb...@gmail.com> wrote:
>>
>>> Hi Selvam,
>>>
>>> It depends what processor you are using. For example, ListFile using a
>>> local file path will always store state locally even when clustered because
>>> no other node can take over that state since the directory to list only
>>> exists on that node. Each processor has an annotation at the top of it
>>> which specifies what type of state it stores, local or clustered. We should
>>> consider adding this to the docs if it's not included already.
>>>
>>> -Bryan
>>>
>>>
>>> On Wednesday, September 28, 2016, Selvam Raman <se...@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> This is my state-management.xml attribute
>>>>
>>>> <local-provider>
>>>> <id>local-provider</id>
>>>> <class>org.apache.nifi.controller.state.providers.local.Writ
>>>> eAheadLocalStateProvider</class>
>>>> <property name="Directory">./state/local</property>
>>>> </local-provider>
>>>>
>>>>
>>>> <cluster-provider>
>>>> <id>zk-provider</id>
>>>> <class>org.apache.nifi.controller.state.providers.zookeeper.
>>>> ZooKeeperStateProvider</class>
>>>> <property name="Connect String">hostname:2181,hostname
>>>> :2181</property>
>>>> <property name="Root Node">/opt/nifiroot</property>
>>>> <property name="Session Timeout">10 seconds</property>
>>>> <property name="Access Control">Open</property>
>>>> </cluster-provider>
>>>>
>>>>
>>>> This is my nifi.poperties file attributes
>>>>
>>>> ####################
>>>> # State Management #
>>>> ####################
>>>> nifi.state.management.configuration.file=./conf/state-management.xml
>>>> # The ID of the local state provider
>>>> nifi.state.management.provider.local=local-provider
>>>> # The ID of the cluster-wide state provider. This will be ignored if
>>>> NiFi is not clustered but must be populated if running in a cluster.
>>>> nifi.state.management.provider.cluster=zk-provider
>>>> # Specifies whether or not this instance of NiFi should run an embedded
>>>> ZooKeeper server
>>>> nifi.state.management.embedded.zookeeper.start=false
>>>> # Properties file that provides the ZooKeeper properties to use if
>>>> <nifi.state.management.embedded.zookeeper.start> is set to true
>>>> nifi.state.management.embedded.zookeeper.properties=./conf/z
>>>> ookeeper.properties
>>>>
>>>> # zookeeper properties, used for cluster management #
>>>> nifi.zookeeper.connect.string=hostname:2181,hostname:2181
>>>> nifi.zookeeper.connect.timeout=3 secs
>>>> nifi.zookeeper.session.timeout=3 secs
>>>> nifi.zookeeper.root.node=/opt/nifiroot
>>>>
>>>> the question here is, i am running nifi in cluster mode and i am
>>>> expecting state should be stored in zk-provider. But the state stored in
>>>> local-provider.
>>>>
>>>> local-state provider:
>>>> /home/nifi/nifi-1.0.0/state/local/partition-*
>>>>
>>>> zk-provider:(empty directory)
>>>> /opt/nifiroot
>>>>
>>>> any help on this.
>>>>
>>>>
>>>>
>>>> --
>>>> Selvam Raman
>>>> "லஞ்சம் தவிர்த்து நெஞ்சம் நிமிர்த்து"
>>>>
>>>
>>>
>>> --
>>> Sent from Gmail Mobile
>>>
>>
>>
>>
>> --
>> Selvam Raman
>> "லஞ்சம் தவிர்த்து நெஞ்சம் நிமிர்த்து"
>>
>
>
Re: State management not captured in cluster mode
Posted by Bryan Bende <bb...@gmail.com>.
What I was referring to is in the code of each processor, it is annotated
with something describing the state.
For example:
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ListFile.java#L102-L105
Currently this is not visible anywhere in the NiFi UI, but what Joe
referenced was an improvement that Pierre submitted so that we can display
this information in the documentation for the processor when clicking
"usage".
On Wed, Sep 28, 2016 at 10:09 AM, Selvam Raman <se...@gmail.com> wrote:
> Hi Bryan,
>
> Thanks for the information. Can you please share a picture where i can
> able to see the state (local or cluster). I could not see anywhere.
>
> On Wed, Sep 28, 2016 at 1:22 PM, Bryan Bende <bb...@gmail.com> wrote:
>
>> Hi Selvam,
>>
>> It depends what processor you are using. For example, ListFile using a
>> local file path will always store state locally even when clustered because
>> no other node can take over that state since the directory to list only
>> exists on that node. Each processor has an annotation at the top of it
>> which specifies what type of state it stores, local or clustered. We should
>> consider adding this to the docs if it's not included already.
>>
>> -Bryan
>>
>>
>> On Wednesday, September 28, 2016, Selvam Raman <se...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> This is my state-management.xml attribute
>>>
>>> <local-provider>
>>> <id>local-provider</id>
>>> <class>org.apache.nifi.controller.state.providers.local.Writ
>>> eAheadLocalStateProvider</class>
>>> <property name="Directory">./state/local</property>
>>> </local-provider>
>>>
>>>
>>> <cluster-provider>
>>> <id>zk-provider</id>
>>> <class>org.apache.nifi.controller.state.providers.zookeeper.
>>> ZooKeeperStateProvider</class>
>>> <property name="Connect String">hostname:2181,hostname
>>> :2181</property>
>>> <property name="Root Node">/opt/nifiroot</property>
>>> <property name="Session Timeout">10 seconds</property>
>>> <property name="Access Control">Open</property>
>>> </cluster-provider>
>>>
>>>
>>> This is my nifi.poperties file attributes
>>>
>>> ####################
>>> # State Management #
>>> ####################
>>> nifi.state.management.configuration.file=./conf/state-management.xml
>>> # The ID of the local state provider
>>> nifi.state.management.provider.local=local-provider
>>> # The ID of the cluster-wide state provider. This will be ignored if
>>> NiFi is not clustered but must be populated if running in a cluster.
>>> nifi.state.management.provider.cluster=zk-provider
>>> # Specifies whether or not this instance of NiFi should run an embedded
>>> ZooKeeper server
>>> nifi.state.management.embedded.zookeeper.start=false
>>> # Properties file that provides the ZooKeeper properties to use if
>>> <nifi.state.management.embedded.zookeeper.start> is set to true
>>> nifi.state.management.embedded.zookeeper.properties=./conf/z
>>> ookeeper.properties
>>>
>>> # zookeeper properties, used for cluster management #
>>> nifi.zookeeper.connect.string=hostname:2181,hostname:2181
>>> nifi.zookeeper.connect.timeout=3 secs
>>> nifi.zookeeper.session.timeout=3 secs
>>> nifi.zookeeper.root.node=/opt/nifiroot
>>>
>>> the question here is, i am running nifi in cluster mode and i am
>>> expecting state should be stored in zk-provider. But the state stored in
>>> local-provider.
>>>
>>> local-state provider:
>>> /home/nifi/nifi-1.0.0/state/local/partition-*
>>>
>>> zk-provider:(empty directory)
>>> /opt/nifiroot
>>>
>>> any help on this.
>>>
>>>
>>>
>>> --
>>> Selvam Raman
>>> "லஞ்சம் தவிர்த்து நெஞ்சம் நிமிர்த்து"
>>>
>>
>>
>> --
>> Sent from Gmail Mobile
>>
>
>
>
> --
> Selvam Raman
> "லஞ்சம் தவிர்த்து நெஞ்சம் நிமிர்த்து"
>
Re: State management not captured in cluster mode
Posted by Selvam Raman <se...@gmail.com>.
Hi Bryan,
Thanks for the information. Can you please share a picture where i can able
to see the state (local or cluster). I could not see anywhere.
On Wed, Sep 28, 2016 at 1:22 PM, Bryan Bende <bb...@gmail.com> wrote:
> Hi Selvam,
>
> It depends what processor you are using. For example, ListFile using a
> local file path will always store state locally even when clustered because
> no other node can take over that state since the directory to list only
> exists on that node. Each processor has an annotation at the top of it
> which specifies what type of state it stores, local or clustered. We should
> consider adding this to the docs if it's not included already.
>
> -Bryan
>
>
> On Wednesday, September 28, 2016, Selvam Raman <se...@gmail.com> wrote:
>
>> Hi,
>>
>> This is my state-management.xml attribute
>>
>> <local-provider>
>> <id>local-provider</id>
>> <class>org.apache.nifi.controller.state.providers.local.Writ
>> eAheadLocalStateProvider</class>
>> <property name="Directory">./state/local</property>
>> </local-provider>
>>
>>
>> <cluster-provider>
>> <id>zk-provider</id>
>> <class>org.apache.nifi.controller.state.providers.zookeeper.
>> ZooKeeperStateProvider</class>
>> <property name="Connect String">hostname:2181,hostname
>> :2181</property>
>> <property name="Root Node">/opt/nifiroot</property>
>> <property name="Session Timeout">10 seconds</property>
>> <property name="Access Control">Open</property>
>> </cluster-provider>
>>
>>
>> This is my nifi.poperties file attributes
>>
>> ####################
>> # State Management #
>> ####################
>> nifi.state.management.configuration.file=./conf/state-management.xml
>> # The ID of the local state provider
>> nifi.state.management.provider.local=local-provider
>> # The ID of the cluster-wide state provider. This will be ignored if NiFi
>> is not clustered but must be populated if running in a cluster.
>> nifi.state.management.provider.cluster=zk-provider
>> # Specifies whether or not this instance of NiFi should run an embedded
>> ZooKeeper server
>> nifi.state.management.embedded.zookeeper.start=false
>> # Properties file that provides the ZooKeeper properties to use if
>> <nifi.state.management.embedded.zookeeper.start> is set to true
>> nifi.state.management.embedded.zookeeper.properties=./conf/
>> zookeeper.properties
>>
>> # zookeeper properties, used for cluster management #
>> nifi.zookeeper.connect.string=hostname:2181,hostname:2181
>> nifi.zookeeper.connect.timeout=3 secs
>> nifi.zookeeper.session.timeout=3 secs
>> nifi.zookeeper.root.node=/opt/nifiroot
>>
>> the question here is, i am running nifi in cluster mode and i am
>> expecting state should be stored in zk-provider. But the state stored in
>> local-provider.
>>
>> local-state provider:
>> /home/nifi/nifi-1.0.0/state/local/partition-*
>>
>> zk-provider:(empty directory)
>> /opt/nifiroot
>>
>> any help on this.
>>
>>
>>
>> --
>> Selvam Raman
>> "லஞ்சம் தவிர்த்து நெஞ்சம் நிமிர்த்து"
>>
>
>
> --
> Sent from Gmail Mobile
>
--
Selvam Raman
"லஞ்சம் தவிர்த்து நெஞ்சம் நிமிர்த்து"
Re: State management not captured in cluster mode
Posted by Joe Witt <jo...@gmail.com>.
Great timing. Pierre put in a JIRA/PR for this yesterday.
https://issues.apache.org/jira/browse/NIFI-2832
Thanks
Joe
On Wed, Sep 28, 2016 at 8:22 AM, Bryan Bende <bb...@gmail.com> wrote:
> Hi Selvam,
>
> It depends what processor you are using. For example, ListFile using a local
> file path will always store state locally even when clustered because no
> other node can take over that state since the directory to list only exists
> on that node. Each processor has an annotation at the top of it which
> specifies what type of state it stores, local or clustered. We should
> consider adding this to the docs if it's not included already.
>
> -Bryan
>
>
> On Wednesday, September 28, 2016, Selvam Raman <se...@gmail.com> wrote:
>>
>> Hi,
>>
>> This is my state-management.xml attribute
>>
>> <local-provider>
>> <id>local-provider</id>
>>
>> <class>org.apache.nifi.controller.state.providers.local.WriteAheadLocalStateProvider</class>
>> <property name="Directory">./state/local</property>
>> </local-provider>
>>
>>
>> <cluster-provider>
>> <id>zk-provider</id>
>>
>> <class>org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider</class>
>> <property name="Connect
>> String">hostname:2181,hostname:2181</property>
>> <property name="Root Node">/opt/nifiroot</property>
>> <property name="Session Timeout">10 seconds</property>
>> <property name="Access Control">Open</property>
>> </cluster-provider>
>>
>>
>> This is my nifi.poperties file attributes
>>
>> ####################
>> # State Management #
>> ####################
>> nifi.state.management.configuration.file=./conf/state-management.xml
>> # The ID of the local state provider
>> nifi.state.management.provider.local=local-provider
>> # The ID of the cluster-wide state provider. This will be ignored if NiFi
>> is not clustered but must be populated if running in a cluster.
>> nifi.state.management.provider.cluster=zk-provider
>> # Specifies whether or not this instance of NiFi should run an embedded
>> ZooKeeper server
>> nifi.state.management.embedded.zookeeper.start=false
>> # Properties file that provides the ZooKeeper properties to use if
>> <nifi.state.management.embedded.zookeeper.start> is set to true
>>
>> nifi.state.management.embedded.zookeeper.properties=./conf/zookeeper.properties
>>
>> # zookeeper properties, used for cluster management #
>> nifi.zookeeper.connect.string=hostname:2181,hostname:2181
>> nifi.zookeeper.connect.timeout=3 secs
>> nifi.zookeeper.session.timeout=3 secs
>> nifi.zookeeper.root.node=/opt/nifiroot
>>
>> the question here is, i am running nifi in cluster mode and i am expecting
>> state should be stored in zk-provider. But the state stored in
>> local-provider.
>>
>> local-state provider:
>> /home/nifi/nifi-1.0.0/state/local/partition-*
>>
>> zk-provider:(empty directory)
>> /opt/nifiroot
>>
>> any help on this.
>>
>>
>>
>> --
>> Selvam Raman
>> "லஞ்சம் தவிர்த்து நெஞ்சம் நிமிர்த்து"
>
>
>
> --
> Sent from Gmail Mobile
Re: State management not captured in cluster mode
Posted by Bryan Bende <bb...@gmail.com>.
Hi Selvam,
It depends what processor you are using. For example, ListFile using a
local file path will always store state locally even when clustered because
no other node can take over that state since the directory to list only
exists on that node. Each processor has an annotation at the top of it
which specifies what type of state it stores, local or clustered. We should
consider adding this to the docs if it's not included already.
-Bryan
On Wednesday, September 28, 2016, Selvam Raman <se...@gmail.com> wrote:
> Hi,
>
> This is my state-management.xml attribute
>
> <local-provider>
> <id>local-provider</id>
> <class>org.apache.nifi.controller.state.providers.local.
> WriteAheadLocalStateProvider</class>
> <property name="Directory">./state/local</property>
> </local-provider>
>
>
> <cluster-provider>
> <id>zk-provider</id>
> <class>org.apache.nifi.controller.state.providers.zookeeper.
> ZooKeeperStateProvider</class>
> <property name="Connect String">hostname:2181,
> hostname:2181</property>
> <property name="Root Node">/opt/nifiroot</property>
> <property name="Session Timeout">10 seconds</property>
> <property name="Access Control">Open</property>
> </cluster-provider>
>
>
> This is my nifi.poperties file attributes
>
> ####################
> # State Management #
> ####################
> nifi.state.management.configuration.file=./conf/state-management.xml
> # The ID of the local state provider
> nifi.state.management.provider.local=local-provider
> # The ID of the cluster-wide state provider. This will be ignored if NiFi
> is not clustered but must be populated if running in a cluster.
> nifi.state.management.provider.cluster=zk-provider
> # Specifies whether or not this instance of NiFi should run an embedded
> ZooKeeper server
> nifi.state.management.embedded.zookeeper.start=false
> # Properties file that provides the ZooKeeper properties to use if
> <nifi.state.management.embedded.zookeeper.start> is set to true
> nifi.state.management.embedded.zookeeper.properties=
> ./conf/zookeeper.properties
>
> # zookeeper properties, used for cluster management #
> nifi.zookeeper.connect.string=hostname:2181,hostname:2181
> nifi.zookeeper.connect.timeout=3 secs
> nifi.zookeeper.session.timeout=3 secs
> nifi.zookeeper.root.node=/opt/nifiroot
>
> the question here is, i am running nifi in cluster mode and i am
> expecting state should be stored in zk-provider. But the state stored in
> local-provider.
>
> local-state provider:
> /home/nifi/nifi-1.0.0/state/local/partition-*
>
> zk-provider:(empty directory)
> /opt/nifiroot
>
> any help on this.
>
>
>
> --
> Selvam Raman
> "லஞ்சம் தவிர்த்து நெஞ்சம் நிமிர்த்து"
>
--
Sent from Gmail Mobile