You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Etienne Jouvin <la...@gmail.com> on 2020/11/23 12:58:27 UTC
Ldap Cluster and Node Identity
Hello all.
I am currently setting up a NiFi, 1.12.1, Cluster with LDAP authentication.
For now the accessPolicyProvider is the default one with the configuration
template :
<accessPolicyProvider>
<identifier>file-access-policy-provider</identifier>
<class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
<property name="User Group
Provider">file-user-group-provider</property>
<property name="Authorizations
File">./conf/authorizations.xml</property>
<property name="Initial Admin Identity"></property>
<property name="Legacy Authorized Users File"></property>
<property name="Node Identity 1"></property>
<property name="Node Group"></property>
</accessPolicyProvider>
But I do not really understand the purpose of the Node Identity X property.
If I well understood, all nodes should have the same configuration file,
and I should register all nodes identity.
But what about if I want to add a new node in the cluster on the fly ?
Should I register a new node identity, and then I should change all nodes
configurations ?
The comment, in the configuration file, mentions the configuration Node
Group, The name of a group containing NiFi cluster nodes. The typical use
for this is when nodes are dynamically added/removed from the cluster.
Should I just put a Node group name and this will do the trick ?
What should I put ? At the following link,
https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.0.3/bk_administration/content/cluster-node-identities.html,
it is said something like : cn=nifi-1,ou=people,dc=example,dc=com
In that case, what should be the obejct class for the node cn=nifi-1 in the
LDAP ?
Any documentation links will be appreciated.
Regards.
Etienne Jouvin
Re: Ldap Cluster and Node Identity
Posted by David Handermann <ex...@gmail.com>.
Etienne,
No problem, I understand, it sounds like you are close to getting it
working. Feel free to follow up if you run into additional issues.
Regards,
David Handermann
On Wed, Nov 25, 2020 at 8:28 AM Etienne Jouvin <la...@gmail.com>
wrote:
> David.
>
> Did not have time this morning to test.
> But it may be something really "stupid", my fault. It seems I made a
> mistake while generating certificates on nodes, regarding the CA....
>
> Hope to have time this afternoon and I will return.
>
> Etienne
>
>
>
> Le mer. 25 nov. 2020 à 14:18, David Handermann <ex...@gmail.com>
> a écrit :
>
>> I am not as familiar with the LDAP user group provider, but based on the
>> "Untrusted proxy" message you are seeing, it sounds like the nodes are not
>> being identified properly as members of the "nodes" group from LDAP. Just
>> for testing purposes, you could try specifying the node distinguished names
>> in the "Node Identity N" properties of the access policy provider, using
>> "Node Identity 1", "Node Identity 2" and "Node Identity 3" to specify each
>> node DN. If that works, then it sounds like a configuration issue with the
>> Node Group, either on the LDAP server, or in the way NiFi is attempting to
>> query LDAP.
>>
>> Regards,
>> David Handermann
>>
>> On Wed, Nov 25, 2020 at 5:19 AM Etienne Jouvin <la...@gmail.com>
>> wrote:
>>
>>> Just for information, did not have time to test it from now.
>>> I was not able to get this Walk Throughs documentation.
>>> https://nifi.apache.org/docs/nifi-docs/html/walkthroughs.html
>>>
>>> Hope I will find the error I have about certificate (I have a little
>>> idea)
>>>
>>> Etienne
>>>
>>>
>>>
>>>
>>> Le mer. 25 nov. 2020 à 08:36, Etienne Jouvin <la...@gmail.com>
>>> a écrit :
>>>
>>>> Hello.
>>>>
>>>> I made some progress yesterday.
>>>> I did setup in LDAP groups and person
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Groups :
>>>> cn=administrators,ou=groups,ou=nifi,dc=amexio,dc=ch : for administrators
>>>> cn=supervisors,ou=groups,ou=nifi,dc=amexio,dc=ch : for supervisors
>>>> cn=nodes,ou=groups,ou=nifi,dc=amexio,dc=ch : group where I put all
>>>> "person" representing NiFi nodes.
>>>>
>>>> Users :
>>>> cn=migX.amexio.ch,ou=users,ou=nifi,dc=amexio,dc=ch : for each node,
>>>> replacing X by the index, and with object class person
>>>> uid=mig-admin,ou=users,ou=nifi,dc=amexio,dc=ch : example of "real" user
>>>> used to connect on the platform, with object class inetOrgperson
>>>>
>>>> In NiFi configuration.
>>>> I did activate a userGroupProvider linked to the LDAP
>>>> <userGroupProvider>
>>>> <identifier>amexio-ldap-user-group-provider</identifier>
>>>>
>>>> <class>org.apache.nifi.ldap.tenants.LdapUserGroupProvider</class>
>>>> <property name="Authentication Strategy">SIMPLE</property>
>>>>
>>>> <property name="Manager DN">uid=admin,ou=system</property>
>>>> <property name="Manager Password">secret</property>
>>>>
>>>> <property name="TLS - Keystore"></property>
>>>> <property name="TLS - Keystore Password"></property>
>>>> <property name="TLS - Keystore Type"></property>
>>>> <property name="TLS - Truststore"></property>
>>>> <property name="TLS - Truststore Password"></property>
>>>> <property name="TLS - Truststore Type"></property>
>>>> <property name="TLS - Client Auth"></property>
>>>> <property name="TLS - Protocol"></property>
>>>> <property name="TLS - Shutdown Gracefully"></property>
>>>>
>>>> <property name="Referral Strategy">FOLLOW</property>
>>>> <property name="Connect Timeout">10 secs</property>
>>>> <property name="Read Timeout">10 secs</property>
>>>>
>>>> <property name="Url">ldap://localhost:10389</property>
>>>> <property name="Page Size">50</property>
>>>> <!-- <property name="Sync Interval">30 mins</property> -->
>>>> <property name="Sync Interval">30 seconds</property>
>>>> <property name="Group Membership - Enforce Case
>>>> Sensitivity">false</property>
>>>>
>>>> <property name="User Search
>>>> Base">ou=users,ou=nifi,dc=amexio,dc=ch</property>
>>>> <property name="User Object Class">person</property>
>>>> <property name="User Search Scope">ONE_LEVEL</property>
>>>> <property name="User Search Filter"></property>
>>>> <property name="User Identity Attribute"></property>
>>>> <property name="User Group Name Attribute"></property>
>>>> <property name="User Group Name Attribute - Referenced Group
>>>> Attribute"></property>
>>>>
>>>> <property name="Group Search
>>>> Base">ou=groups,ou=nifi,dc=amexio,dc=ch</property>
>>>> <property name="Group Object Class">groupOfNames</property>
>>>> <property name="Group Search Scope">ONE_LEVEL</property>
>>>> <property name="Group Search Filter"></property>
>>>> <property name="Group Name Attribute">cn</property>
>>>> <property name="Group Member Attribute">member</property>
>>>> <property name="Group Member Attribute - Referenced User
>>>> Attribute"></property>
>>>> </userGroupProvider>
>>>>
>>>> Of course, register it inside the accessPolicyProvider
>>>> <accessPolicyProvider>
>>>> <identifier>file-access-policy-provider</identifier>
>>>>
>>>> <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
>>>> <!-- <property name="User Group
>>>> Provider">file-user-group-provider</property> -->
>>>> <property name="User Group
>>>> Provider">amexio-ldap-user-group-provider</property>
>>>> <property name="Authorizations
>>>> File">./conf/authorizations.xml</property>
>>>> <!-- <property name="Initial Admin Identity"></property> -->
>>>> <property name="Initial Admin
>>>> Identity">uid=mig-admin,ou=users,ou=nifi,dc=amexio,dc=ch</property>
>>>> <property name="Legacy Authorized Users File"></property>
>>>> <property name="Node Identity 1"></property>
>>>> <property name="Node Group">nodes</property>
>>>> </accessPolicyProvider>
>>>>
>>>> I am able to connect with the initial administrator account, when the
>>>> first node is started.
>>>> And all nodes are synchronized in the NiFi instance.
>>>>
>>>>
>>>>
>>>>
>>>> As soon as I start an additional node, I can not connect to the first
>>>> node
>>>> Erreur message
>>>> Untrusted proxy CN=mig1.amexio.ch, OU=users, OU=nifi, DC=amexio, DC=ch
>>>>
>>>> But I can connect on the second node.
>>>>
>>>>
>>>> So all this is about the certificate I guess.
>>>> for reminder, I use nls-toolkit to generate certificate on all nodes
>>>> with something like :
>>>> tls-toolkit.bat standalone -f
>>>> "C:\nifi-1.12.1\node1\conf\nifi.properties" -o
>>>> "C:\nifi-1.12.1\node1\conf\secured" -n mig1.amexio.ch --nifiDnPrefix
>>>> cn= --nifiDnSuffix ou=users,ou=nifi,dc=amexio,dc=ch
>>>>
>>>> Proxy is untrusted, ok fine. So may be I should not use the standalone
>>>> function of toolkit, but using server and client. In that case, I have to
>>>> stay alive the server from toolkit ?
>>>> Also, it seems I did not add certificate from node1 inside node2
>>>> trutstore, and node2 certificate inside node1 truststore ?
>>>> But in this case, if I have to add a new node, let's say node4, I would
>>>> have to push the certificate from node4 inside all existing nodes ?
>>>>
>>>> I continue to search, but any idea / input will be appreciated.
>>>>
>>>> Etienne
>>>>
>>>>
>>>> Le lun. 23 nov. 2020 à 18:39, Bryan Bende <bb...@gmail.com> a écrit :
>>>>
>>>>> Yes it will be the DN of the server's certificate which comes from the
>>>>> keystore.
>>>>>
>>>>> NiFi will get an incoming request, see that there is an X509 cert,
>>>>> take the DN and go to the user group provider and ask for the user
>>>>> with this identity.
>>>>>
>>>>> On Mon, Nov 23, 2020 at 12:01 PM Etienne Jouvin <
>>>>> lapinoujoujou@gmail.com> wrote:
>>>>> >
>>>>> > Hum OK,
>>>>> >
>>>>> > I will give it a try.
>>>>> > But one more thing...
>>>>> >
>>>>> > If I only set the group node;
>>>>> > How NiFi will connect the node with the nodeId in the LDAP ?
>>>>> > Where does it take the nodeid value ?
>>>>> > Is it the value we set in the keystore / truststore, by default
>>>>> cn=localhost, dc=NIFI (something like this) ?
>>>>> >
>>>>> > Etienne
>>>>> >
>>>>> >
>>>>> > Le lun. 23 nov. 2020 à 17:54, Bryan Bende <bb...@gmail.com> a
>>>>> écrit :
>>>>> >>
>>>>> >> I don't really know the LDAP specifics too well, so I'm not
>>>>> actually sure.
>>>>> >>
>>>>> >> You just need the nodes to come back from the LDAP UserGroupProvider
>>>>> >> as if they were regular users and members of some group "foo", which
>>>>> >> you then put "foo" into the "Node Group".
>>>>> >>
>>>>> >> On Mon, Nov 23, 2020 at 11:50 AM Etienne Jouvin <
>>>>> lapinoujoujou@gmail.com> wrote:
>>>>> >> >
>>>>> >> > Thanks Bryan.
>>>>> >> >
>>>>> >> > With your answer.... I will go to the Node Group and assign node
>>>>> identities.
>>>>> >> > Better for deployment and setup on the fly, I guess.
>>>>> >> >
>>>>> >> > One more point, you said "creating ldap entries for your nodes
>>>>> and assigning them group membership in ldap". What type of objectClass
>>>>> would you assign to the node in LDAP ?
>>>>> >> > This is not inetOrgPerson. The node should not have password.
>>>>> >> > If I create groupOfMembers for each node, is it correct ?
>>>>> >> >
>>>>> >> >
>>>>> >> > Thanks
>>>>> >> >
>>>>> >> > Etienne
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> > Le lun. 23 nov. 2020 à 17:27, Bryan Bende <bb...@gmail.com> a
>>>>> écrit :
>>>>> >> >>
>>>>> >> >> Hello,
>>>>> >> >>
>>>>> >> >> "Node Identity" is similar to the "Initial Admin" concept, in
>>>>> that it
>>>>> >> >> sets up the policies for the initial nodes to have permissions to
>>>>> >> >> proxy.
>>>>> >> >>
>>>>> >> >> If you are creating ldap entries for your nodes and assigning
>>>>> them
>>>>> >> >> group membership in ldap, then yes you could put that group name
>>>>> as
>>>>> >> >> the "Node Group" and then you don't need to specify the "Node
>>>>> >> >> Identities".
>>>>> >> >>
>>>>> >> >> If you are creating the node users in NiFi's file-based user
>>>>> group
>>>>> >> >> provider then you need to use node identities, and when adding a
>>>>> new
>>>>> >> >> node to the cluster you'd have to add the user first through the
>>>>> >> >> UI/REST API and grant it proxy, then actually connect it to the
>>>>> >> >> cluster.
>>>>> >> >>
>>>>> >> >> Thanks,
>>>>> >> >>
>>>>> >> >> Bryan
>>>>> >> >>
>>>>> >> >>
>>>>> >> >> On Mon, Nov 23, 2020 at 7:58 AM Etienne Jouvin <
>>>>> lapinoujoujou@gmail.com> wrote:
>>>>> >> >> >
>>>>> >> >> > Hello all.
>>>>> >> >> >
>>>>> >> >> >
>>>>> >> >> > I am currently setting up a NiFi, 1.12.1, Cluster with LDAP
>>>>> authentication.
>>>>> >> >> > For now the accessPolicyProvider is the default one with the
>>>>> configuration template :
>>>>> >> >> > <accessPolicyProvider>
>>>>> >> >> > <identifier>file-access-policy-provider</identifier>
>>>>> >> >> >
>>>>> <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
>>>>> >> >> > <property name="User Group
>>>>> Provider">file-user-group-provider</property>
>>>>> >> >> > <property name="Authorizations
>>>>> File">./conf/authorizations.xml</property>
>>>>> >> >> > <property name="Initial Admin Identity"></property>
>>>>> >> >> > <property name="Legacy Authorized Users
>>>>> File"></property>
>>>>> >> >> > <property name="Node Identity 1"></property>
>>>>> >> >> > <property name="Node Group"></property>
>>>>> >> >> > </accessPolicyProvider>
>>>>> >> >> >
>>>>> >> >> > But I do not really understand the purpose of the Node
>>>>> Identity X property.
>>>>> >> >> > If I well understood, all nodes should have the same
>>>>> configuration file, and I should register all nodes identity.
>>>>> >> >> >
>>>>> >> >> > But what about if I want to add a new node in the cluster on
>>>>> the fly ?
>>>>> >> >> > Should I register a new node identity, and then I should
>>>>> change all nodes configurations ?
>>>>> >> >> > The comment, in the configuration file, mentions the
>>>>> configuration Node Group, The name of a group containing NiFi cluster
>>>>> nodes. The typical use for this is when nodes are dynamically added/removed
>>>>> from the cluster.
>>>>> >> >> > Should I just put a Node group name and this will do the trick
>>>>> ?
>>>>> >> >> >
>>>>> >> >> > What should I put ? At the following link,
>>>>> https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.0.3/bk_administration/content/cluster-node-identities.html,
>>>>> it is said something like : cn=nifi-1,ou=people,dc=example,dc=com
>>>>> >> >> > In that case, what should be the obejct class for the node
>>>>> cn=nifi-1 in the LDAP ?
>>>>> >> >> >
>>>>> >> >> > Any documentation links will be appreciated.
>>>>> >> >> >
>>>>> >> >> > Regards.
>>>>> >> >> >
>>>>> >> >> > Etienne Jouvin
>>>>>
>>>>
Re: Ldap Cluster and Node Identity
Posted by Etienne Jouvin <la...@gmail.com>.
David.
Did not have time this morning to test.
But it may be something really "stupid", my fault. It seems I made a
mistake while generating certificates on nodes, regarding the CA....
Hope to have time this afternoon and I will return.
Etienne
Le mer. 25 nov. 2020 à 14:18, David Handermann <ex...@gmail.com>
a écrit :
> I am not as familiar with the LDAP user group provider, but based on the
> "Untrusted proxy" message you are seeing, it sounds like the nodes are not
> being identified properly as members of the "nodes" group from LDAP. Just
> for testing purposes, you could try specifying the node distinguished names
> in the "Node Identity N" properties of the access policy provider, using
> "Node Identity 1", "Node Identity 2" and "Node Identity 3" to specify each
> node DN. If that works, then it sounds like a configuration issue with the
> Node Group, either on the LDAP server, or in the way NiFi is attempting to
> query LDAP.
>
> Regards,
> David Handermann
>
> On Wed, Nov 25, 2020 at 5:19 AM Etienne Jouvin <la...@gmail.com>
> wrote:
>
>> Just for information, did not have time to test it from now.
>> I was not able to get this Walk Throughs documentation.
>> https://nifi.apache.org/docs/nifi-docs/html/walkthroughs.html
>>
>> Hope I will find the error I have about certificate (I have a little idea)
>>
>> Etienne
>>
>>
>>
>>
>> Le mer. 25 nov. 2020 à 08:36, Etienne Jouvin <la...@gmail.com> a
>> écrit :
>>
>>> Hello.
>>>
>>> I made some progress yesterday.
>>> I did setup in LDAP groups and person
>>>
>>>
>>>
>>>
>>>
>>> Groups :
>>> cn=administrators,ou=groups,ou=nifi,dc=amexio,dc=ch : for administrators
>>> cn=supervisors,ou=groups,ou=nifi,dc=amexio,dc=ch : for supervisors
>>> cn=nodes,ou=groups,ou=nifi,dc=amexio,dc=ch : group where I put all
>>> "person" representing NiFi nodes.
>>>
>>> Users :
>>> cn=migX.amexio.ch,ou=users,ou=nifi,dc=amexio,dc=ch : for each node,
>>> replacing X by the index, and with object class person
>>> uid=mig-admin,ou=users,ou=nifi,dc=amexio,dc=ch : example of "real" user
>>> used to connect on the platform, with object class inetOrgperson
>>>
>>> In NiFi configuration.
>>> I did activate a userGroupProvider linked to the LDAP
>>> <userGroupProvider>
>>> <identifier>amexio-ldap-user-group-provider</identifier>
>>> <class>org.apache.nifi.ldap.tenants.LdapUserGroupProvider</class>
>>> <property name="Authentication Strategy">SIMPLE</property>
>>>
>>> <property name="Manager DN">uid=admin,ou=system</property>
>>> <property name="Manager Password">secret</property>
>>>
>>> <property name="TLS - Keystore"></property>
>>> <property name="TLS - Keystore Password"></property>
>>> <property name="TLS - Keystore Type"></property>
>>> <property name="TLS - Truststore"></property>
>>> <property name="TLS - Truststore Password"></property>
>>> <property name="TLS - Truststore Type"></property>
>>> <property name="TLS - Client Auth"></property>
>>> <property name="TLS - Protocol"></property>
>>> <property name="TLS - Shutdown Gracefully"></property>
>>>
>>> <property name="Referral Strategy">FOLLOW</property>
>>> <property name="Connect Timeout">10 secs</property>
>>> <property name="Read Timeout">10 secs</property>
>>>
>>> <property name="Url">ldap://localhost:10389</property>
>>> <property name="Page Size">50</property>
>>> <!-- <property name="Sync Interval">30 mins</property> -->
>>> <property name="Sync Interval">30 seconds</property>
>>> <property name="Group Membership - Enforce Case
>>> Sensitivity">false</property>
>>>
>>> <property name="User Search
>>> Base">ou=users,ou=nifi,dc=amexio,dc=ch</property>
>>> <property name="User Object Class">person</property>
>>> <property name="User Search Scope">ONE_LEVEL</property>
>>> <property name="User Search Filter"></property>
>>> <property name="User Identity Attribute"></property>
>>> <property name="User Group Name Attribute"></property>
>>> <property name="User Group Name Attribute - Referenced Group
>>> Attribute"></property>
>>>
>>> <property name="Group Search
>>> Base">ou=groups,ou=nifi,dc=amexio,dc=ch</property>
>>> <property name="Group Object Class">groupOfNames</property>
>>> <property name="Group Search Scope">ONE_LEVEL</property>
>>> <property name="Group Search Filter"></property>
>>> <property name="Group Name Attribute">cn</property>
>>> <property name="Group Member Attribute">member</property>
>>> <property name="Group Member Attribute - Referenced User
>>> Attribute"></property>
>>> </userGroupProvider>
>>>
>>> Of course, register it inside the accessPolicyProvider
>>> <accessPolicyProvider>
>>> <identifier>file-access-policy-provider</identifier>
>>>
>>> <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
>>> <!-- <property name="User Group
>>> Provider">file-user-group-provider</property> -->
>>> <property name="User Group
>>> Provider">amexio-ldap-user-group-provider</property>
>>> <property name="Authorizations
>>> File">./conf/authorizations.xml</property>
>>> <!-- <property name="Initial Admin Identity"></property> -->
>>> <property name="Initial Admin
>>> Identity">uid=mig-admin,ou=users,ou=nifi,dc=amexio,dc=ch</property>
>>> <property name="Legacy Authorized Users File"></property>
>>> <property name="Node Identity 1"></property>
>>> <property name="Node Group">nodes</property>
>>> </accessPolicyProvider>
>>>
>>> I am able to connect with the initial administrator account, when the
>>> first node is started.
>>> And all nodes are synchronized in the NiFi instance.
>>>
>>>
>>>
>>>
>>> As soon as I start an additional node, I can not connect to the first
>>> node
>>> Erreur message
>>> Untrusted proxy CN=mig1.amexio.ch, OU=users, OU=nifi, DC=amexio, DC=ch
>>>
>>> But I can connect on the second node.
>>>
>>>
>>> So all this is about the certificate I guess.
>>> for reminder, I use nls-toolkit to generate certificate on all nodes
>>> with something like :
>>> tls-toolkit.bat standalone -f
>>> "C:\nifi-1.12.1\node1\conf\nifi.properties" -o
>>> "C:\nifi-1.12.1\node1\conf\secured" -n mig1.amexio.ch --nifiDnPrefix
>>> cn= --nifiDnSuffix ou=users,ou=nifi,dc=amexio,dc=ch
>>>
>>> Proxy is untrusted, ok fine. So may be I should not use the standalone
>>> function of toolkit, but using server and client. In that case, I have to
>>> stay alive the server from toolkit ?
>>> Also, it seems I did not add certificate from node1 inside node2
>>> trutstore, and node2 certificate inside node1 truststore ?
>>> But in this case, if I have to add a new node, let's say node4, I would
>>> have to push the certificate from node4 inside all existing nodes ?
>>>
>>> I continue to search, but any idea / input will be appreciated.
>>>
>>> Etienne
>>>
>>>
>>> Le lun. 23 nov. 2020 à 18:39, Bryan Bende <bb...@gmail.com> a écrit :
>>>
>>>> Yes it will be the DN of the server's certificate which comes from the
>>>> keystore.
>>>>
>>>> NiFi will get an incoming request, see that there is an X509 cert,
>>>> take the DN and go to the user group provider and ask for the user
>>>> with this identity.
>>>>
>>>> On Mon, Nov 23, 2020 at 12:01 PM Etienne Jouvin <
>>>> lapinoujoujou@gmail.com> wrote:
>>>> >
>>>> > Hum OK,
>>>> >
>>>> > I will give it a try.
>>>> > But one more thing...
>>>> >
>>>> > If I only set the group node;
>>>> > How NiFi will connect the node with the nodeId in the LDAP ?
>>>> > Where does it take the nodeid value ?
>>>> > Is it the value we set in the keystore / truststore, by default
>>>> cn=localhost, dc=NIFI (something like this) ?
>>>> >
>>>> > Etienne
>>>> >
>>>> >
>>>> > Le lun. 23 nov. 2020 à 17:54, Bryan Bende <bb...@gmail.com> a écrit
>>>> :
>>>> >>
>>>> >> I don't really know the LDAP specifics too well, so I'm not actually
>>>> sure.
>>>> >>
>>>> >> You just need the nodes to come back from the LDAP UserGroupProvider
>>>> >> as if they were regular users and members of some group "foo", which
>>>> >> you then put "foo" into the "Node Group".
>>>> >>
>>>> >> On Mon, Nov 23, 2020 at 11:50 AM Etienne Jouvin <
>>>> lapinoujoujou@gmail.com> wrote:
>>>> >> >
>>>> >> > Thanks Bryan.
>>>> >> >
>>>> >> > With your answer.... I will go to the Node Group and assign node
>>>> identities.
>>>> >> > Better for deployment and setup on the fly, I guess.
>>>> >> >
>>>> >> > One more point, you said "creating ldap entries for your nodes and
>>>> assigning them group membership in ldap". What type of objectClass would
>>>> you assign to the node in LDAP ?
>>>> >> > This is not inetOrgPerson. The node should not have password.
>>>> >> > If I create groupOfMembers for each node, is it correct ?
>>>> >> >
>>>> >> >
>>>> >> > Thanks
>>>> >> >
>>>> >> > Etienne
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > Le lun. 23 nov. 2020 à 17:27, Bryan Bende <bb...@gmail.com> a
>>>> écrit :
>>>> >> >>
>>>> >> >> Hello,
>>>> >> >>
>>>> >> >> "Node Identity" is similar to the "Initial Admin" concept, in
>>>> that it
>>>> >> >> sets up the policies for the initial nodes to have permissions to
>>>> >> >> proxy.
>>>> >> >>
>>>> >> >> If you are creating ldap entries for your nodes and assigning them
>>>> >> >> group membership in ldap, then yes you could put that group name
>>>> as
>>>> >> >> the "Node Group" and then you don't need to specify the "Node
>>>> >> >> Identities".
>>>> >> >>
>>>> >> >> If you are creating the node users in NiFi's file-based user group
>>>> >> >> provider then you need to use node identities, and when adding a
>>>> new
>>>> >> >> node to the cluster you'd have to add the user first through the
>>>> >> >> UI/REST API and grant it proxy, then actually connect it to the
>>>> >> >> cluster.
>>>> >> >>
>>>> >> >> Thanks,
>>>> >> >>
>>>> >> >> Bryan
>>>> >> >>
>>>> >> >>
>>>> >> >> On Mon, Nov 23, 2020 at 7:58 AM Etienne Jouvin <
>>>> lapinoujoujou@gmail.com> wrote:
>>>> >> >> >
>>>> >> >> > Hello all.
>>>> >> >> >
>>>> >> >> >
>>>> >> >> > I am currently setting up a NiFi, 1.12.1, Cluster with LDAP
>>>> authentication.
>>>> >> >> > For now the accessPolicyProvider is the default one with the
>>>> configuration template :
>>>> >> >> > <accessPolicyProvider>
>>>> >> >> > <identifier>file-access-policy-provider</identifier>
>>>> >> >> >
>>>> <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
>>>> >> >> > <property name="User Group
>>>> Provider">file-user-group-provider</property>
>>>> >> >> > <property name="Authorizations
>>>> File">./conf/authorizations.xml</property>
>>>> >> >> > <property name="Initial Admin Identity"></property>
>>>> >> >> > <property name="Legacy Authorized Users
>>>> File"></property>
>>>> >> >> > <property name="Node Identity 1"></property>
>>>> >> >> > <property name="Node Group"></property>
>>>> >> >> > </accessPolicyProvider>
>>>> >> >> >
>>>> >> >> > But I do not really understand the purpose of the Node Identity
>>>> X property.
>>>> >> >> > If I well understood, all nodes should have the same
>>>> configuration file, and I should register all nodes identity.
>>>> >> >> >
>>>> >> >> > But what about if I want to add a new node in the cluster on
>>>> the fly ?
>>>> >> >> > Should I register a new node identity, and then I should change
>>>> all nodes configurations ?
>>>> >> >> > The comment, in the configuration file, mentions the
>>>> configuration Node Group, The name of a group containing NiFi cluster
>>>> nodes. The typical use for this is when nodes are dynamically added/removed
>>>> from the cluster.
>>>> >> >> > Should I just put a Node group name and this will do the trick ?
>>>> >> >> >
>>>> >> >> > What should I put ? At the following link,
>>>> https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.0.3/bk_administration/content/cluster-node-identities.html,
>>>> it is said something like : cn=nifi-1,ou=people,dc=example,dc=com
>>>> >> >> > In that case, what should be the obejct class for the node
>>>> cn=nifi-1 in the LDAP ?
>>>> >> >> >
>>>> >> >> > Any documentation links will be appreciated.
>>>> >> >> >
>>>> >> >> > Regards.
>>>> >> >> >
>>>> >> >> > Etienne Jouvin
>>>>
>>>
Re: NIFI and Out of Memory Error
Posted by jgunvaldson <jg...@cox.net>.
I passed this along to the primary developers - Thanks Mike!
Best Regards
John
> On Dec 3, 2020, at 1:11 PM, Mike Thomsen <mi...@gmail.com> wrote:
>
> One of my colleagues ran into a similar situation, and all that was
> required to fix it was to make ReplaceText work line by line. When you
> do that, you shouldn't run into any issues.
>
Re: NIFI and Out of Memory Error
Posted by Mike Thomsen <mi...@gmail.com>.
One of my colleagues ran into a similar situation, and all that was
required to fix it was to make ReplaceText work line by line. When you
do that, you shouldn't run into any issues.
On Thu, Dec 3, 2020 at 1:04 PM jgunvaldson <jg...@cox.net> wrote:
>
> Just looking for an opinion
>
> Knowing (for one example) that ReplaceText Processor can be very memory intensive with large files - we are finding it more and more common to wake up to an Out of Memory error like the following
>
> 2020-12-03 15:07:21,748ZUTC ERROR [Timer-Driven Process Thread-31] o.a.nifi.processors.standard.ReplaceText ReplaceText[id=352afe80-4195-3f56-8798-aaf8be160581] ReplaceText[id=352afe80-4195-3f56-8798-aaf8be160581] failed to process session due to java.lang.OutOfMemoryError: Java heap space; Processor Administratively Yielded for 1 sec: java.lang.OutOfMemoryError: Java heap space
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.nifi.processors.standard.ReplaceText.onTrigger(ReplaceText.java:255)
> at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162)
>
>
> My question is this. Knowing that "When an OOME occurs in a JVM this can cause the JVM to skip instructions. Skipping instructions can compromise the integrity of the JVM memory without displaying errors. You can't always tell from the outside if a JVM has compromised memory, the only safe thing to do is restart the JVM.”
>
> And in this case “Restart NIFI”
>
> Is that “our collective” understanding also, that a Restart of NIFI is mandatory - or optional?
>
> Thanks
>
> John
>
Re: NIFI and Out of Memory Error
Posted by Joe Witt <jo...@gmail.com>.
I am honestly not sure if it is required - but it is probably a good idea.
Please let us know what you find using those.
Also, definitely we should change that flow to avoid large memory
consumption. Want to share more details on the input data and config that
results in this?
On Thu, Dec 3, 2020 at 11:21 AM jgunvaldson <jg...@cox.net> wrote:
> Thanks Joe,
>
> I am getting the general opinion that on OOM restart is not optional, must
> be done. In that case I am going to also look at some of the following
>
> -XX:+ExitOnOutOfMemoryError
> -XX:+CrashOnOutOfMemoryError
>
> *ExitOnOutOfMemoryError*
> When you enable this option, the JVM exits on the first occurrence of an
> out-of-memory error. It can be used if you prefer restarting an instance of
> the JVM rather than handling out of memory errors.
>
> *CrashOnOutOfMemoryError*
> If this option is enabled, when an out-of-memory error occurs, the JVM
> crashes and produces text and binary crash files.
>
> Best Regards
> John
>
>
>
> On Dec 3, 2020, at 10:09 AM, Joe Witt <jo...@gmail.com> wrote:
>
> John,
>
> First, as a general rule it is usually very doable to build flows which
> are very stream oriented rather than entire file oriented. That processor
> by its nature isn't friendly in this way if configured to work with large
> memory chunks. Alternatives often exist.
>
> Second, I do think it is wise to restart the JVM in the event of an OOME.
> There are ways to configure your JVM to do this automatically. Googling
> 'restart JVM on oome' for instance could be helpful there.
>
> Thanks
>
>>
>>
>
Re: NIFI and Out of Memory Error
Posted by jgunvaldson <jg...@cox.net>.
Thanks Joe,
I am getting the general opinion that on OOM restart is not optional, must be done. In that case I am going to also look at some of the following
-XX:+ExitOnOutOfMemoryError
-XX:+CrashOnOutOfMemoryError
ExitOnOutOfMemoryError
When you enable this option, the JVM exits on the first occurrence of an out-of-memory error. It can be used if you prefer restarting an instance of the JVM rather than handling out of memory errors.
CrashOnOutOfMemoryError
If this option is enabled, when an out-of-memory error occurs, the JVM crashes and produces text and binary crash files.
Best Regards
John
> On Dec 3, 2020, at 10:09 AM, Joe Witt <jo...@gmail.com> wrote:
>
> John,
>
> First, as a general rule it is usually very doable to build flows which are very stream oriented rather than entire file oriented. That processor by its nature isn't friendly in this way if configured to work with large memory chunks. Alternatives often exist.
>
> Second, I do think it is wise to restart the JVM in the event of an OOME. There are ways to configure your JVM to do this automatically. Googling 'restart JVM on oome' for instance could be helpful there.
>
> Thanks
>
Re: NIFI and Out of Memory Error
Posted by Joe Witt <jo...@gmail.com>.
John,
First, as a general rule it is usually very doable to build flows which are
very stream oriented rather than entire file oriented. That processor by
its nature isn't friendly in this way if configured to work with large
memory chunks. Alternatives often exist.
Second, I do think it is wise to restart the JVM in the event of an OOME.
There are ways to configure your JVM to do this automatically. Googling
'restart JVM on oome' for instance could be helpful there.
Thanks
On Thu, Dec 3, 2020 at 11:04 AM jgunvaldson <jg...@cox.net> wrote:
> Just looking for an opinion
>
> Knowing (for one example) that ReplaceText Processor can be very memory
> intensive with large files - we are finding it more and more common to wake
> up to an Out of Memory error like the following
>
> 2020-12-03 15:07:21,748ZUTC ERROR [Timer-Driven Process Thread-31]
> o.a.nifi.processors.standard.ReplaceText
> ReplaceText[id=352afe80-4195-3f56-8798-aaf8be160581]
> ReplaceText[id=352afe80-4195-3f56-8798-aaf8be160581] failed to process
> session due to java.lang.OutOfMemoryError: Java heap space; Processor
> Administratively Yielded for 1 sec: java.lang.OutOfMemoryError: Java heap
> space
> java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.nifi.processors.standard.ReplaceText.onTrigger(ReplaceText.java:255)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162)
>
>
> My question is this. Knowing that "When an OOME occurs in a JVM this can
> cause the JVM to skip instructions. Skipping instructions can compromise
> the integrity of the JVM memory without displaying errors. You can't always
> tell from the outside if a JVM has compromised memory, the only safe thing
> to do is restart the JVM.”
>
> And in this case “Restart NIFI”
>
> Is that “our collective” understanding also, that a Restart of NIFI is
> mandatory - or optional?
>
> Thanks
>
> John
>
>
NIFI and Out of Memory Error
Posted by jgunvaldson <jg...@cox.net>.
Just looking for an opinion
Knowing (for one example) that ReplaceText Processor can be very memory intensive with large files - we are finding it more and more common to wake up to an Out of Memory error like the following
2020-12-03 15:07:21,748ZUTC ERROR [Timer-Driven Process Thread-31] o.a.nifi.processors.standard.ReplaceText ReplaceText[id=352afe80-4195-3f56-8798-aaf8be160581] ReplaceText[id=352afe80-4195-3f56-8798-aaf8be160581] failed to process session due to java.lang.OutOfMemoryError: Java heap space; Processor Administratively Yielded for 1 sec: java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
at org.apache.nifi.processors.standard.ReplaceText.onTrigger(ReplaceText.java:255)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162)
My question is this. Knowing that "When an OOME occurs in a JVM this can cause the JVM to skip instructions. Skipping instructions can compromise the integrity of the JVM memory without displaying errors. You can't always tell from the outside if a JVM has compromised memory, the only safe thing to do is restart the JVM.”
And in this case “Restart NIFI”
Is that “our collective” understanding also, that a Restart of NIFI is mandatory - or optional?
Thanks
John
Re: Ldap Cluster and Node Identity
Posted by Etienne Jouvin <la...@gmail.com>.
Pierre.
Ok, I keep this in mind.
For now, I have an other "issue" configuring secure cluster and registry.
I study this and will give a try to push documentation. May be throw a Jira
to just expose my difficulties. Writing the documentation by my side may
not be well formed ;)
Regards
Etienne
Le jeu. 26 nov. 2020 à 12:14, Pierre Villard <pi...@gmail.com>
a écrit :
> Etienne,
>
> If you feel like there would be nice additions to have in the
> documentation, please feel free to share your suggestions to improve it
> (though a JIRA / pull request). We don't want NiFi users/administrators to
> have a painful process when configuring all of this.
>
> Thanks,
> Pierre
>
> Le jeu. 26 nov. 2020 à 12:01, Etienne Jouvin <la...@gmail.com> a
> écrit :
>
>> Hello all.
>>
>> And finally I got it.
>> When I am not able to find in the documentation, because I was not
>> correctly searching offcourse, I read the source code and do remote debug.
>>
>> And the conclusion that it is all about identity mapping.
>>
>> For reminder, the objective was to have a secured cluster and everything
>> managed in the LDAP.
>> I follow the walk throughs documentation. But in it, it is only matter of
>> nodes managed in userGroupIdentity, based on files.
>>
>> In my case, users and groups are managed in the LDAP, even nodes for
>> cluster.
>>
>> With remote debugging, I touched the class X509AuthenticationProvider and
>> specially the function authenticate. By the way, this is the only class
>> that fires an exception with message Untrusted proxy ...
>> In it, identities are extracted from the request and linked certificates.
>> Ids extracted were :
>>
>> - uid=mig-admin,ou=users,ou=nifi,dc=amexio,dc=ch , for the "real" user
>> - CN=mig1.amexio.ch <http://mig1.assura.ch/>, OU=users, OU=nifi,
>> DC=amexio, DC=ch, for the "proxy" node
>>
>>
>> But in the LDAP, the node as the entry DN : cn=mig1.amexio.ch
>> ,ou=users,ou=nifi;dc=amexio,dc=ch
>>
>> So when the provider validate identities, the NiFi identity "CN=
>> mig1.amexio.ch <http://mig1.assura.ch/>, OU=users, OU=nifi, DC=amexio,
>> DC=ch" is not found. No user/group, and does not belongs to any group. Then
>> the validation on policy "proxy", with "write" access, failed and the
>> connection was rejected.
>>
>> What I found also in this function, there is a identity mapping,
>> function mapIdentity from class NiFiAuthenticationProvider.
>> This will use function mapIdentity in class IdentityMappingUtil.
>> With this, the identity is normalized and this is the trick to transform
>> the idnetity retrieved from the certificate, with space and upper case, to
>> an identity matching the one retrieved from LDAP.
>>
>>
>> That was the point. Si in the file nifi.properties, for all nodes, here
>> is modifications :
>> nifi.security.identity.mapping.pattern.ldapCertNode=^CN=(.*?), OU=(.*?),
>> OU=(.*?), DC=(.*?), DC=(.*?)$
>>
>> nifi.security.identity.mapping.value.ldapCertNode=cn=$1,ou=$2,ou=$3,dc=$4,dc=$5
>> nifi.security.identity.mapping.transform.ldapCertNode=NONE
>>
>> Like this, the identity retrieved from certificate, for the proxy node,
>> is CN=mig1.amexio.ch <http://mig1.assura.ch/>, OU=users, OU=nifi,
>> DC=amexio, DC=ch
>> After identity mapping it is cn=mig1.amexio.ch <http://mig1.assura.ch/>
>> ,ou=users,ou=nifi,dc=amexio,dc=ch
>> And it is exactly matching identity retrieved from the LDAP for the node.
>>
>> Et voilà.
>>
>> Hard work (for me) and happy to find the way to manage this.
>>
>> Thanks all for your help.
>>
>>
>> Etienne
>>
>>
>>
>> Le mer. 25 nov. 2020 à 19:50, Etienne Jouvin <la...@gmail.com> a
>> écrit :
>>
>>> That's what I suspect for now.
>>>
>>> But this is strange to have a transformation with space introduced.
>>> In the certificate (I use KeyStore Explorer to visualize the
>>> certificate) there is no space.
>>>
>>> But I will give a try to use the identity mapping from here
>>> # Identity Mapping Properties #
>>> # These properties allow normalizing user identities such that
>>> identities coming from different identity providers
>>> # (certificates, LDAP, Kerberos) can be treated the same internally in
>>> NiFi. The following example demonstrates normalizing
>>> # DNs from certificates and principals from Kerberos into a common
>>> identity string:
>>> #
>>> # nifi.security.identity.mapping.pattern.dn=^CN=(.*?), OU=(.*?),
>>> O=(.*?), L=(.*?), ST=(.*?), C=(.*?)$
>>> # nifi.security.identity.mapping.value.dn=$1@$2
>>> # nifi.security.identity.mapping.transform.dn=NONE
>>> # nifi.security.identity.mapping.pattern.kerb=^(.*?)/instance@(.*?)$
>>> # nifi.security.identity.mapping.value.kerb=$1@$2
>>> # nifi.security.identity.mapping.transform.kerb=UPPER
>>>
>>> # Group Mapping Properties #
>>> # These properties allow normalizing group names coming from external
>>> sources like LDAP. The following example
>>> # lowercases any group name.
>>> #
>>> # nifi.security.group.mapping.pattern.anygroup=^(.*)$
>>> # nifi.security.group.mapping.value.anygroup=$1
>>> # nifi.security.group.mapping.transform.anygroup=LOWER
>>>
>>> for now, I do not know what to put ;)
>>>
>>> Etienne
>>>
>>>
>>>
>>> Le mer. 25 nov. 2020 à 19:45, Bryan Bende <bb...@gmail.com> a écrit :
>>>
>>>> The values have to match exactly, it’s case and white space sensitive.
>>>> There are identity transforms in Nifi properties that possibly help.
>>>>
>>>> The value from where it says “Attempting request for...” is the value
>>>> nifi is getting from the certificate. You can also use key tool to list the
>>>> contents of the keystore jks and it will probably be the same as the log.
>>>>
>>>> On Wed, Nov 25, 2020 at 1:30 PM Etienne Jouvin <la...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hello all.
>>>>>
>>>>> Still no luck.
>>>>> I follow steps from here :
>>>>> https://nifi.apache.org/docs/nifi-docs/html/walkthroughs.html#creating-and-securing-a-nifi-cluster-with-the-tls-toolkit
>>>>>
>>>>> Reminder, everything is currently in LDAP, I do not have any file
>>>>> user/group provider.
>>>>> When only one node, I can connect, this is correct.
>>>>>
>>>>> I can see the group having nodes "users" from the LDAP and the group
>>>>> has the proxy policy.
>>>>>
>>>>> But one thing strange, may be not the cause.
>>>>> In the LDAP, the DN is something like :
>>>>> cn=mig1.amexio.ch,ou=users,ou=nifi,dc=amexio,dc=ch
>>>>>
>>>>> And when I try to connect, I can see thoses errors :
>>>>> In nifi-user.log for the node where I connect :
>>>>> INFO [NiFi Web Server-22] o.a.n.w.s.NiFiAuthenticationFilter
>>>>> Authentication success for uid=mig-admin,ou=users,ou=nifi,dc=amexio,dc=ch
>>>>>
>>>>> In nifi-user.log on the second node :
>>>>> INFO [NiFi Web Server-29] o.a.n.w.s.NiFiAuthenticationFilter
>>>>> Attempting request for (<uid=mig-admin,ou=users,ou=nifi,dc=amexio,dc=ch><CN=
>>>>> mig1.amexio.ch <http://mig1.assura.ch>, OU=users, OU=nifi, DC=amexio,
>>>>> DC=ch>) GET https://mig2.amexio.ch:9442/nifi-api/flow/current-user
>>>>> (source ip: 127.0.0.1)
>>>>> WARN [NiFi Web Server-29] o.a.n.w.s.NiFiAuthenticationFilter Rejecting
>>>>> access to web api: Untrusted proxy CN=mig1.amexio.ch, OU=users,
>>>>> OU=nifi, DC=amexio, DC=ch
>>>>>
>>>>> The doubt is on the message, why the node DN has spaces between each
>>>>> level ?
>>>>> That does not match the value in the certificate and not the DN in
>>>>> LDAP.
>>>>>
>>>>> Maybe this is not the good route to investigate. Still searching. I
>>>>> have to find a way to "debug" the NiFiAuthenticationFilter and check the
>>>>> certificate comparison.
>>>>>
>>>>> Etienne
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Le mer. 25 nov. 2020 à 14:18, David Handermann <
>>>>> exceptionfactory@gmail.com> a écrit :
>>>>>
>>>>>> I am not as familiar with the LDAP user group provider, but based on
>>>>>> the "Untrusted proxy" message you are seeing, it sounds like the nodes are
>>>>>> not being identified properly as members of the "nodes" group from LDAP.
>>>>>> Just for testing purposes, you could try specifying the node distinguished
>>>>>> names in the "Node Identity N" properties of the access policy provider,
>>>>>> using "Node Identity 1", "Node Identity 2" and "Node Identity 3" to specify
>>>>>> each node DN. If that works, then it sounds like a configuration issue
>>>>>> with the Node Group, either on the LDAP server, or in the way NiFi is
>>>>>> attempting to query LDAP.
>>>>>>
>>>>>> Regards,
>>>>>> David Handermann
>>>>>>
>>>>>> On Wed, Nov 25, 2020 at 5:19 AM Etienne Jouvin <
>>>>>> lapinoujoujou@gmail.com> wrote:
>>>>>>
>>>>>>> Just for information, did not have time to test it from now.
>>>>>>> I was not able to get this Walk Throughs documentation.
>>>>>>> https://nifi.apache.org/docs/nifi-docs/html/walkthroughs.html
>>>>>>>
>>>>>>> Hope I will find the error I have about certificate (I have a little
>>>>>>> idea)
>>>>>>>
>>>>>>> Etienne
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Le mer. 25 nov. 2020 à 08:36, Etienne Jouvin <
>>>>>>> lapinoujoujou@gmail.com> a écrit :
>>>>>>>
>>>>>>>> Hello.
>>>>>>>>
>>>>>>>> I made some progress yesterday.
>>>>>>>> I did setup in LDAP groups and person
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Groups :
>>>>>>>> cn=administrators,ou=groups,ou=nifi,dc=amexio,dc=ch : for
>>>>>>>> administrators
>>>>>>>> cn=supervisors,ou=groups,ou=nifi,dc=amexio,dc=ch : for supervisors
>>>>>>>> cn=nodes,ou=groups,ou=nifi,dc=amexio,dc=ch : group where I put all
>>>>>>>> "person" representing NiFi nodes.
>>>>>>>>
>>>>>>>> Users :
>>>>>>>> cn=migX.amexio.ch,ou=users,ou=nifi,dc=amexio,dc=ch : for each
>>>>>>>> node, replacing X by the index, and with object class person
>>>>>>>> uid=mig-admin,ou=users,ou=nifi,dc=amexio,dc=ch : example of "real"
>>>>>>>> user used to connect on the platform, with object class inetOrgperson
>>>>>>>>
>>>>>>>> In NiFi configuration.
>>>>>>>> I did activate a userGroupProvider linked to the LDAP
>>>>>>>> <userGroupProvider>
>>>>>>>> <identifier>amexio-ldap-user-group-provider</identifier>
>>>>>>>>
>>>>>>>> <class>org.apache.nifi.ldap.tenants.LdapUserGroupProvider</class>
>>>>>>>> <property name="Authentication Strategy">SIMPLE</property>
>>>>>>>>
>>>>>>>> <property name="Manager DN">uid=admin,ou=system</property>
>>>>>>>> <property name="Manager Password">secret</property>
>>>>>>>>
>>>>>>>> <property name="TLS - Keystore"></property>
>>>>>>>> <property name="TLS - Keystore Password"></property>
>>>>>>>> <property name="TLS - Keystore Type"></property>
>>>>>>>> <property name="TLS - Truststore"></property>
>>>>>>>> <property name="TLS - Truststore Password"></property>
>>>>>>>> <property name="TLS - Truststore Type"></property>
>>>>>>>> <property name="TLS - Client Auth"></property>
>>>>>>>> <property name="TLS - Protocol"></property>
>>>>>>>> <property name="TLS - Shutdown Gracefully"></property>
>>>>>>>>
>>>>>>>> <property name="Referral Strategy">FOLLOW</property>
>>>>>>>> <property name="Connect Timeout">10 secs</property>
>>>>>>>> <property name="Read Timeout">10 secs</property>
>>>>>>>>
>>>>>>>> <property name="Url">ldap://localhost:10389</property>
>>>>>>>> <property name="Page Size">50</property>
>>>>>>>> <!-- <property name="Sync Interval">30 mins</property> -->
>>>>>>>> <property name="Sync Interval">30 seconds</property>
>>>>>>>> <property name="Group Membership - Enforce Case
>>>>>>>> Sensitivity">false</property>
>>>>>>>>
>>>>>>>> <property name="User Search
>>>>>>>> Base">ou=users,ou=nifi,dc=amexio,dc=ch</property>
>>>>>>>> <property name="User Object Class">person</property>
>>>>>>>> <property name="User Search Scope">ONE_LEVEL</property>
>>>>>>>> <property name="User Search Filter"></property>
>>>>>>>> <property name="User Identity Attribute"></property>
>>>>>>>> <property name="User Group Name Attribute"></property>
>>>>>>>> <property name="User Group Name Attribute - Referenced
>>>>>>>> Group Attribute"></property>
>>>>>>>>
>>>>>>>> <property name="Group Search
>>>>>>>> Base">ou=groups,ou=nifi,dc=amexio,dc=ch</property>
>>>>>>>> <property name="Group Object Class">groupOfNames</property>
>>>>>>>> <property name="Group Search Scope">ONE_LEVEL</property>
>>>>>>>> <property name="Group Search Filter"></property>
>>>>>>>> <property name="Group Name Attribute">cn</property>
>>>>>>>> <property name="Group Member Attribute">member</property>
>>>>>>>> <property name="Group Member Attribute - Referenced User
>>>>>>>> Attribute"></property>
>>>>>>>> </userGroupProvider>
>>>>>>>>
>>>>>>>> Of course, register it inside the accessPolicyProvider
>>>>>>>> <accessPolicyProvider>
>>>>>>>> <identifier>file-access-policy-provider</identifier>
>>>>>>>>
>>>>>>>> <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
>>>>>>>> <!-- <property name="User Group
>>>>>>>> Provider">file-user-group-provider</property> -->
>>>>>>>> <property name="User Group
>>>>>>>> Provider">amexio-ldap-user-group-provider</property>
>>>>>>>> <property name="Authorizations
>>>>>>>> File">./conf/authorizations.xml</property>
>>>>>>>> <!-- <property name="Initial Admin Identity"></property> -->
>>>>>>>> <property name="Initial Admin
>>>>>>>> Identity">uid=mig-admin,ou=users,ou=nifi,dc=amexio,dc=ch</property>
>>>>>>>> <property name="Legacy Authorized Users File"></property>
>>>>>>>> <property name="Node Identity 1"></property>
>>>>>>>> <property name="Node Group">nodes</property>
>>>>>>>> </accessPolicyProvider>
>>>>>>>>
>>>>>>>> I am able to connect with the initial administrator account, when
>>>>>>>> the first node is started.
>>>>>>>> And all nodes are synchronized in the NiFi instance.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> As soon as I start an additional node, I can not connect to the
>>>>>>>> first node
>>>>>>>> Erreur message
>>>>>>>> Untrusted proxy CN=mig1.amexio.ch, OU=users, OU=nifi, DC=amexio,
>>>>>>>> DC=ch
>>>>>>>>
>>>>>>>> But I can connect on the second node.
>>>>>>>>
>>>>>>>>
>>>>>>>> So all this is about the certificate I guess.
>>>>>>>> for reminder, I use nls-toolkit to generate certificate on all
>>>>>>>> nodes with something like :
>>>>>>>> tls-toolkit.bat standalone -f
>>>>>>>> "C:\nifi-1.12.1\node1\conf\nifi.properties" -o
>>>>>>>> "C:\nifi-1.12.1\node1\conf\secured" -n mig1.amexio.ch
>>>>>>>> --nifiDnPrefix cn= --nifiDnSuffix ou=users,ou=nifi,dc=amexio,dc=ch
>>>>>>>>
>>>>>>>> Proxy is untrusted, ok fine. So may be I should not use the
>>>>>>>> standalone function of toolkit, but using server and client. In that case,
>>>>>>>> I have to stay alive the server from toolkit ?
>>>>>>>> Also, it seems I did not add certificate from node1 inside node2
>>>>>>>> trutstore, and node2 certificate inside node1 truststore ?
>>>>>>>> But in this case, if I have to add a new node, let's say node4, I
>>>>>>>> would have to push the certificate from node4 inside all existing nodes ?
>>>>>>>>
>>>>>>>> I continue to search, but any idea / input will be appreciated.
>>>>>>>>
>>>>>>>> Etienne
>>>>>>>>
>>>>>>>>
>>>>>>>> Le lun. 23 nov. 2020 à 18:39, Bryan Bende <bb...@gmail.com> a
>>>>>>>> écrit :
>>>>>>>>
>>>>>>>>> Yes it will be the DN of the server's certificate which comes from
>>>>>>>>> the keystore.
>>>>>>>>>
>>>>>>>>> NiFi will get an incoming request, see that there is an X509 cert,
>>>>>>>>> take the DN and go to the user group provider and ask for the user
>>>>>>>>> with this identity.
>>>>>>>>>
>>>>>>>>> On Mon, Nov 23, 2020 at 12:01 PM Etienne Jouvin <
>>>>>>>>> lapinoujoujou@gmail.com> wrote:
>>>>>>>>> >
>>>>>>>>> > Hum OK,
>>>>>>>>> >
>>>>>>>>> > I will give it a try.
>>>>>>>>> > But one more thing...
>>>>>>>>> >
>>>>>>>>> > If I only set the group node;
>>>>>>>>> > How NiFi will connect the node with the nodeId in the LDAP ?
>>>>>>>>> > Where does it take the nodeid value ?
>>>>>>>>> > Is it the value we set in the keystore / truststore, by default
>>>>>>>>> cn=localhost, dc=NIFI (something like this) ?
>>>>>>>>> >
>>>>>>>>> > Etienne
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > Le lun. 23 nov. 2020 à 17:54, Bryan Bende <bb...@gmail.com> a
>>>>>>>>> écrit :
>>>>>>>>> >>
>>>>>>>>> >> I don't really know the LDAP specifics too well, so I'm not
>>>>>>>>> actually sure.
>>>>>>>>> >>
>>>>>>>>> >> You just need the nodes to come back from the LDAP
>>>>>>>>> UserGroupProvider
>>>>>>>>> >> as if they were regular users and members of some group "foo",
>>>>>>>>> which
>>>>>>>>> >> you then put "foo" into the "Node Group".
>>>>>>>>> >>
>>>>>>>>> >> On Mon, Nov 23, 2020 at 11:50 AM Etienne Jouvin <
>>>>>>>>> lapinoujoujou@gmail.com> wrote:
>>>>>>>>> >> >
>>>>>>>>> >> > Thanks Bryan.
>>>>>>>>> >> >
>>>>>>>>> >> > With your answer.... I will go to the Node Group and assign
>>>>>>>>> node identities.
>>>>>>>>> >> > Better for deployment and setup on the fly, I guess.
>>>>>>>>> >> >
>>>>>>>>> >> > One more point, you said "creating ldap entries for your
>>>>>>>>> nodes and assigning them group membership in ldap". What type of
>>>>>>>>> objectClass would you assign to the node in LDAP ?
>>>>>>>>> >> > This is not inetOrgPerson. The node should not have password.
>>>>>>>>> >> > If I create groupOfMembers for each node, is it correct ?
>>>>>>>>> >> >
>>>>>>>>> >> >
>>>>>>>>> >> > Thanks
>>>>>>>>> >> >
>>>>>>>>> >> > Etienne
>>>>>>>>> >> >
>>>>>>>>> >> >
>>>>>>>>> >> >
>>>>>>>>> >> > Le lun. 23 nov. 2020 à 17:27, Bryan Bende <bb...@gmail.com>
>>>>>>>>> a écrit :
>>>>>>>>> >> >>
>>>>>>>>> >> >> Hello,
>>>>>>>>> >> >>
>>>>>>>>> >> >> "Node Identity" is similar to the "Initial Admin" concept,
>>>>>>>>> in that it
>>>>>>>>> >> >> sets up the policies for the initial nodes to have
>>>>>>>>> permissions to
>>>>>>>>> >> >> proxy.
>>>>>>>>> >> >>
>>>>>>>>> >> >> If you are creating ldap entries for your nodes and
>>>>>>>>> assigning them
>>>>>>>>> >> >> group membership in ldap, then yes you could put that group
>>>>>>>>> name as
>>>>>>>>> >> >> the "Node Group" and then you don't need to specify the "Node
>>>>>>>>> >> >> Identities".
>>>>>>>>> >> >>
>>>>>>>>> >> >> If you are creating the node users in NiFi's file-based user
>>>>>>>>> group
>>>>>>>>> >> >> provider then you need to use node identities, and when
>>>>>>>>> adding a new
>>>>>>>>> >> >> node to the cluster you'd have to add the user first through
>>>>>>>>> the
>>>>>>>>> >> >> UI/REST API and grant it proxy, then actually connect it to
>>>>>>>>> the
>>>>>>>>> >> >> cluster.
>>>>>>>>> >> >>
>>>>>>>>> >> >> Thanks,
>>>>>>>>> >> >>
>>>>>>>>> >> >> Bryan
>>>>>>>>> >> >>
>>>>>>>>> >> >>
>>>>>>>>> >> >> On Mon, Nov 23, 2020 at 7:58 AM Etienne Jouvin <
>>>>>>>>> lapinoujoujou@gmail.com> wrote:
>>>>>>>>> >> >> >
>>>>>>>>> >> >> > Hello all.
>>>>>>>>> >> >> >
>>>>>>>>> >> >> >
>>>>>>>>> >> >> > I am currently setting up a NiFi, 1.12.1, Cluster with
>>>>>>>>> LDAP authentication.
>>>>>>>>> >> >> > For now the accessPolicyProvider is the default one with
>>>>>>>>> the configuration template :
>>>>>>>>> >> >> > <accessPolicyProvider>
>>>>>>>>> >> >> >
>>>>>>>>> <identifier>file-access-policy-provider</identifier>
>>>>>>>>> >> >> >
>>>>>>>>> <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
>>>>>>>>> >> >> > <property name="User Group
>>>>>>>>> Provider">file-user-group-provider</property>
>>>>>>>>> >> >> > <property name="Authorizations
>>>>>>>>> File">./conf/authorizations.xml</property>
>>>>>>>>> >> >> > <property name="Initial Admin Identity"></property>
>>>>>>>>> >> >> > <property name="Legacy Authorized Users
>>>>>>>>> File"></property>
>>>>>>>>> >> >> > <property name="Node Identity 1"></property>
>>>>>>>>> >> >> > <property name="Node Group"></property>
>>>>>>>>> >> >> > </accessPolicyProvider>
>>>>>>>>> >> >> >
>>>>>>>>> >> >> > But I do not really understand the purpose of the Node
>>>>>>>>> Identity X property.
>>>>>>>>> >> >> > If I well understood, all nodes should have the same
>>>>>>>>> configuration file, and I should register all nodes identity.
>>>>>>>>> >> >> >
>>>>>>>>> >> >> > But what about if I want to add a new node in the cluster
>>>>>>>>> on the fly ?
>>>>>>>>> >> >> > Should I register a new node identity, and then I should
>>>>>>>>> change all nodes configurations ?
>>>>>>>>> >> >> > The comment, in the configuration file, mentions the
>>>>>>>>> configuration Node Group, The name of a group containing NiFi cluster
>>>>>>>>> nodes. The typical use for this is when nodes are dynamically added/removed
>>>>>>>>> from the cluster.
>>>>>>>>> >> >> > Should I just put a Node group name and this will do the
>>>>>>>>> trick ?
>>>>>>>>> >> >> >
>>>>>>>>> >> >> > What should I put ? At the following link,
>>>>>>>>> https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.0.3/bk_administration/content/cluster-node-identities.html,
>>>>>>>>> it is said something like : cn=nifi-1,ou=people,dc=example,dc=com
>>>>>>>>> >> >> > In that case, what should be the obejct class for the node
>>>>>>>>> cn=nifi-1 in the LDAP ?
>>>>>>>>> >> >> >
>>>>>>>>> >> >> > Any documentation links will be appreciated.
>>>>>>>>> >> >> >
>>>>>>>>> >> >> > Regards.
>>>>>>>>> >> >> >
>>>>>>>>> >> >> > Etienne Jouvin
>>>>>>>>>
>>>>>>>> --
>>>> Sent from Gmail Mobile
>>>>
>>>
Re: Ldap Cluster and Node Identity
Posted by Pierre Villard <pi...@gmail.com>.
Etienne,
If you feel like there would be nice additions to have in the
documentation, please feel free to share your suggestions to improve it
(though a JIRA / pull request). We don't want NiFi users/administrators to
have a painful process when configuring all of this.
Thanks,
Pierre
Le jeu. 26 nov. 2020 à 12:01, Etienne Jouvin <la...@gmail.com> a
écrit :
> Hello all.
>
> And finally I got it.
> When I am not able to find in the documentation, because I was not
> correctly searching offcourse, I read the source code and do remote debug.
>
> And the conclusion that it is all about identity mapping.
>
> For reminder, the objective was to have a secured cluster and everything
> managed in the LDAP.
> I follow the walk throughs documentation. But in it, it is only matter of
> nodes managed in userGroupIdentity, based on files.
>
> In my case, users and groups are managed in the LDAP, even nodes for
> cluster.
>
> With remote debugging, I touched the class X509AuthenticationProvider and
> specially the function authenticate. By the way, this is the only class
> that fires an exception with message Untrusted proxy ...
> In it, identities are extracted from the request and linked certificates.
> Ids extracted were :
>
> - uid=mig-admin,ou=users,ou=nifi,dc=amexio,dc=ch , for the "real" user
> - CN=mig1.amexio.ch <http://mig1.assura.ch/>, OU=users, OU=nifi,
> DC=amexio, DC=ch, for the "proxy" node
>
>
> But in the LDAP, the node as the entry DN : cn=mig1.amexio.ch
> ,ou=users,ou=nifi;dc=amexio,dc=ch
>
> So when the provider validate identities, the NiFi identity "CN=
> mig1.amexio.ch <http://mig1.assura.ch/>, OU=users, OU=nifi, DC=amexio,
> DC=ch" is not found. No user/group, and does not belongs to any group. Then
> the validation on policy "proxy", with "write" access, failed and the
> connection was rejected.
>
> What I found also in this function, there is a identity mapping,
> function mapIdentity from class NiFiAuthenticationProvider.
> This will use function mapIdentity in class IdentityMappingUtil.
> With this, the identity is normalized and this is the trick to transform
> the idnetity retrieved from the certificate, with space and upper case, to
> an identity matching the one retrieved from LDAP.
>
>
> That was the point. Si in the file nifi.properties, for all nodes, here is
> modifications :
> nifi.security.identity.mapping.pattern.ldapCertNode=^CN=(.*?), OU=(.*?),
> OU=(.*?), DC=(.*?), DC=(.*?)$
>
> nifi.security.identity.mapping.value.ldapCertNode=cn=$1,ou=$2,ou=$3,dc=$4,dc=$5
> nifi.security.identity.mapping.transform.ldapCertNode=NONE
>
> Like this, the identity retrieved from certificate, for the proxy node, is
> CN=mig1.amexio.ch <http://mig1.assura.ch/>, OU=users, OU=nifi, DC=amexio,
> DC=ch
> After identity mapping it is cn=mig1.amexio.ch <http://mig1.assura.ch/>
> ,ou=users,ou=nifi,dc=amexio,dc=ch
> And it is exactly matching identity retrieved from the LDAP for the node.
>
> Et voilà.
>
> Hard work (for me) and happy to find the way to manage this.
>
> Thanks all for your help.
>
>
> Etienne
>
>
>
> Le mer. 25 nov. 2020 à 19:50, Etienne Jouvin <la...@gmail.com> a
> écrit :
>
>> That's what I suspect for now.
>>
>> But this is strange to have a transformation with space introduced.
>> In the certificate (I use KeyStore Explorer to visualize the certificate)
>> there is no space.
>>
>> But I will give a try to use the identity mapping from here
>> # Identity Mapping Properties #
>> # These properties allow normalizing user identities such that identities
>> coming from different identity providers
>> # (certificates, LDAP, Kerberos) can be treated the same internally in
>> NiFi. The following example demonstrates normalizing
>> # DNs from certificates and principals from Kerberos into a common
>> identity string:
>> #
>> # nifi.security.identity.mapping.pattern.dn=^CN=(.*?), OU=(.*?), O=(.*?),
>> L=(.*?), ST=(.*?), C=(.*?)$
>> # nifi.security.identity.mapping.value.dn=$1@$2
>> # nifi.security.identity.mapping.transform.dn=NONE
>> # nifi.security.identity.mapping.pattern.kerb=^(.*?)/instance@(.*?)$
>> # nifi.security.identity.mapping.value.kerb=$1@$2
>> # nifi.security.identity.mapping.transform.kerb=UPPER
>>
>> # Group Mapping Properties #
>> # These properties allow normalizing group names coming from external
>> sources like LDAP. The following example
>> # lowercases any group name.
>> #
>> # nifi.security.group.mapping.pattern.anygroup=^(.*)$
>> # nifi.security.group.mapping.value.anygroup=$1
>> # nifi.security.group.mapping.transform.anygroup=LOWER
>>
>> for now, I do not know what to put ;)
>>
>> Etienne
>>
>>
>>
>> Le mer. 25 nov. 2020 à 19:45, Bryan Bende <bb...@gmail.com> a écrit :
>>
>>> The values have to match exactly, it’s case and white space sensitive.
>>> There are identity transforms in Nifi properties that possibly help.
>>>
>>> The value from where it says “Attempting request for...” is the value
>>> nifi is getting from the certificate. You can also use key tool to list the
>>> contents of the keystore jks and it will probably be the same as the log.
>>>
>>> On Wed, Nov 25, 2020 at 1:30 PM Etienne Jouvin <la...@gmail.com>
>>> wrote:
>>>
>>>> Hello all.
>>>>
>>>> Still no luck.
>>>> I follow steps from here :
>>>> https://nifi.apache.org/docs/nifi-docs/html/walkthroughs.html#creating-and-securing-a-nifi-cluster-with-the-tls-toolkit
>>>>
>>>> Reminder, everything is currently in LDAP, I do not have any file
>>>> user/group provider.
>>>> When only one node, I can connect, this is correct.
>>>>
>>>> I can see the group having nodes "users" from the LDAP and the group
>>>> has the proxy policy.
>>>>
>>>> But one thing strange, may be not the cause.
>>>> In the LDAP, the DN is something like :
>>>> cn=mig1.amexio.ch,ou=users,ou=nifi,dc=amexio,dc=ch
>>>>
>>>> And when I try to connect, I can see thoses errors :
>>>> In nifi-user.log for the node where I connect :
>>>> INFO [NiFi Web Server-22] o.a.n.w.s.NiFiAuthenticationFilter
>>>> Authentication success for uid=mig-admin,ou=users,ou=nifi,dc=amexio,dc=ch
>>>>
>>>> In nifi-user.log on the second node :
>>>> INFO [NiFi Web Server-29] o.a.n.w.s.NiFiAuthenticationFilter Attempting
>>>> request for (<uid=mig-admin,ou=users,ou=nifi,dc=amexio,dc=ch><CN=
>>>> mig1.amexio.ch <http://mig1.assura.ch>, OU=users, OU=nifi, DC=amexio,
>>>> DC=ch>) GET https://mig2.amexio.ch:9442/nifi-api/flow/current-user
>>>> (source ip: 127.0.0.1)
>>>> WARN [NiFi Web Server-29] o.a.n.w.s.NiFiAuthenticationFilter Rejecting
>>>> access to web api: Untrusted proxy CN=mig1.amexio.ch, OU=users,
>>>> OU=nifi, DC=amexio, DC=ch
>>>>
>>>> The doubt is on the message, why the node DN has spaces between each
>>>> level ?
>>>> That does not match the value in the certificate and not the DN in LDAP.
>>>>
>>>> Maybe this is not the good route to investigate. Still searching. I
>>>> have to find a way to "debug" the NiFiAuthenticationFilter and check the
>>>> certificate comparison.
>>>>
>>>> Etienne
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Le mer. 25 nov. 2020 à 14:18, David Handermann <
>>>> exceptionfactory@gmail.com> a écrit :
>>>>
>>>>> I am not as familiar with the LDAP user group provider, but based on
>>>>> the "Untrusted proxy" message you are seeing, it sounds like the nodes are
>>>>> not being identified properly as members of the "nodes" group from LDAP.
>>>>> Just for testing purposes, you could try specifying the node distinguished
>>>>> names in the "Node Identity N" properties of the access policy provider,
>>>>> using "Node Identity 1", "Node Identity 2" and "Node Identity 3" to specify
>>>>> each node DN. If that works, then it sounds like a configuration issue
>>>>> with the Node Group, either on the LDAP server, or in the way NiFi is
>>>>> attempting to query LDAP.
>>>>>
>>>>> Regards,
>>>>> David Handermann
>>>>>
>>>>> On Wed, Nov 25, 2020 at 5:19 AM Etienne Jouvin <
>>>>> lapinoujoujou@gmail.com> wrote:
>>>>>
>>>>>> Just for information, did not have time to test it from now.
>>>>>> I was not able to get this Walk Throughs documentation.
>>>>>> https://nifi.apache.org/docs/nifi-docs/html/walkthroughs.html
>>>>>>
>>>>>> Hope I will find the error I have about certificate (I have a little
>>>>>> idea)
>>>>>>
>>>>>> Etienne
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Le mer. 25 nov. 2020 à 08:36, Etienne Jouvin <la...@gmail.com>
>>>>>> a écrit :
>>>>>>
>>>>>>> Hello.
>>>>>>>
>>>>>>> I made some progress yesterday.
>>>>>>> I did setup in LDAP groups and person
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Groups :
>>>>>>> cn=administrators,ou=groups,ou=nifi,dc=amexio,dc=ch : for
>>>>>>> administrators
>>>>>>> cn=supervisors,ou=groups,ou=nifi,dc=amexio,dc=ch : for supervisors
>>>>>>> cn=nodes,ou=groups,ou=nifi,dc=amexio,dc=ch : group where I put all
>>>>>>> "person" representing NiFi nodes.
>>>>>>>
>>>>>>> Users :
>>>>>>> cn=migX.amexio.ch,ou=users,ou=nifi,dc=amexio,dc=ch : for each node,
>>>>>>> replacing X by the index, and with object class person
>>>>>>> uid=mig-admin,ou=users,ou=nifi,dc=amexio,dc=ch : example of "real"
>>>>>>> user used to connect on the platform, with object class inetOrgperson
>>>>>>>
>>>>>>> In NiFi configuration.
>>>>>>> I did activate a userGroupProvider linked to the LDAP
>>>>>>> <userGroupProvider>
>>>>>>> <identifier>amexio-ldap-user-group-provider</identifier>
>>>>>>>
>>>>>>> <class>org.apache.nifi.ldap.tenants.LdapUserGroupProvider</class>
>>>>>>> <property name="Authentication Strategy">SIMPLE</property>
>>>>>>>
>>>>>>> <property name="Manager DN">uid=admin,ou=system</property>
>>>>>>> <property name="Manager Password">secret</property>
>>>>>>>
>>>>>>> <property name="TLS - Keystore"></property>
>>>>>>> <property name="TLS - Keystore Password"></property>
>>>>>>> <property name="TLS - Keystore Type"></property>
>>>>>>> <property name="TLS - Truststore"></property>
>>>>>>> <property name="TLS - Truststore Password"></property>
>>>>>>> <property name="TLS - Truststore Type"></property>
>>>>>>> <property name="TLS - Client Auth"></property>
>>>>>>> <property name="TLS - Protocol"></property>
>>>>>>> <property name="TLS - Shutdown Gracefully"></property>
>>>>>>>
>>>>>>> <property name="Referral Strategy">FOLLOW</property>
>>>>>>> <property name="Connect Timeout">10 secs</property>
>>>>>>> <property name="Read Timeout">10 secs</property>
>>>>>>>
>>>>>>> <property name="Url">ldap://localhost:10389</property>
>>>>>>> <property name="Page Size">50</property>
>>>>>>> <!-- <property name="Sync Interval">30 mins</property> -->
>>>>>>> <property name="Sync Interval">30 seconds</property>
>>>>>>> <property name="Group Membership - Enforce Case
>>>>>>> Sensitivity">false</property>
>>>>>>>
>>>>>>> <property name="User Search
>>>>>>> Base">ou=users,ou=nifi,dc=amexio,dc=ch</property>
>>>>>>> <property name="User Object Class">person</property>
>>>>>>> <property name="User Search Scope">ONE_LEVEL</property>
>>>>>>> <property name="User Search Filter"></property>
>>>>>>> <property name="User Identity Attribute"></property>
>>>>>>> <property name="User Group Name Attribute"></property>
>>>>>>> <property name="User Group Name Attribute - Referenced Group
>>>>>>> Attribute"></property>
>>>>>>>
>>>>>>> <property name="Group Search
>>>>>>> Base">ou=groups,ou=nifi,dc=amexio,dc=ch</property>
>>>>>>> <property name="Group Object Class">groupOfNames</property>
>>>>>>> <property name="Group Search Scope">ONE_LEVEL</property>
>>>>>>> <property name="Group Search Filter"></property>
>>>>>>> <property name="Group Name Attribute">cn</property>
>>>>>>> <property name="Group Member Attribute">member</property>
>>>>>>> <property name="Group Member Attribute - Referenced User
>>>>>>> Attribute"></property>
>>>>>>> </userGroupProvider>
>>>>>>>
>>>>>>> Of course, register it inside the accessPolicyProvider
>>>>>>> <accessPolicyProvider>
>>>>>>> <identifier>file-access-policy-provider</identifier>
>>>>>>>
>>>>>>> <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
>>>>>>> <!-- <property name="User Group
>>>>>>> Provider">file-user-group-provider</property> -->
>>>>>>> <property name="User Group
>>>>>>> Provider">amexio-ldap-user-group-provider</property>
>>>>>>> <property name="Authorizations
>>>>>>> File">./conf/authorizations.xml</property>
>>>>>>> <!-- <property name="Initial Admin Identity"></property> -->
>>>>>>> <property name="Initial Admin
>>>>>>> Identity">uid=mig-admin,ou=users,ou=nifi,dc=amexio,dc=ch</property>
>>>>>>> <property name="Legacy Authorized Users File"></property>
>>>>>>> <property name="Node Identity 1"></property>
>>>>>>> <property name="Node Group">nodes</property>
>>>>>>> </accessPolicyProvider>
>>>>>>>
>>>>>>> I am able to connect with the initial administrator account, when
>>>>>>> the first node is started.
>>>>>>> And all nodes are synchronized in the NiFi instance.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> As soon as I start an additional node, I can not connect to the
>>>>>>> first node
>>>>>>> Erreur message
>>>>>>> Untrusted proxy CN=mig1.amexio.ch, OU=users, OU=nifi, DC=amexio,
>>>>>>> DC=ch
>>>>>>>
>>>>>>> But I can connect on the second node.
>>>>>>>
>>>>>>>
>>>>>>> So all this is about the certificate I guess.
>>>>>>> for reminder, I use nls-toolkit to generate certificate on all nodes
>>>>>>> with something like :
>>>>>>> tls-toolkit.bat standalone -f
>>>>>>> "C:\nifi-1.12.1\node1\conf\nifi.properties" -o
>>>>>>> "C:\nifi-1.12.1\node1\conf\secured" -n mig1.amexio.ch
>>>>>>> --nifiDnPrefix cn= --nifiDnSuffix ou=users,ou=nifi,dc=amexio,dc=ch
>>>>>>>
>>>>>>> Proxy is untrusted, ok fine. So may be I should not use the
>>>>>>> standalone function of toolkit, but using server and client. In that case,
>>>>>>> I have to stay alive the server from toolkit ?
>>>>>>> Also, it seems I did not add certificate from node1 inside node2
>>>>>>> trutstore, and node2 certificate inside node1 truststore ?
>>>>>>> But in this case, if I have to add a new node, let's say node4, I
>>>>>>> would have to push the certificate from node4 inside all existing nodes ?
>>>>>>>
>>>>>>> I continue to search, but any idea / input will be appreciated.
>>>>>>>
>>>>>>> Etienne
>>>>>>>
>>>>>>>
>>>>>>> Le lun. 23 nov. 2020 à 18:39, Bryan Bende <bb...@gmail.com> a
>>>>>>> écrit :
>>>>>>>
>>>>>>>> Yes it will be the DN of the server's certificate which comes from
>>>>>>>> the keystore.
>>>>>>>>
>>>>>>>> NiFi will get an incoming request, see that there is an X509 cert,
>>>>>>>> take the DN and go to the user group provider and ask for the user
>>>>>>>> with this identity.
>>>>>>>>
>>>>>>>> On Mon, Nov 23, 2020 at 12:01 PM Etienne Jouvin <
>>>>>>>> lapinoujoujou@gmail.com> wrote:
>>>>>>>> >
>>>>>>>> > Hum OK,
>>>>>>>> >
>>>>>>>> > I will give it a try.
>>>>>>>> > But one more thing...
>>>>>>>> >
>>>>>>>> > If I only set the group node;
>>>>>>>> > How NiFi will connect the node with the nodeId in the LDAP ?
>>>>>>>> > Where does it take the nodeid value ?
>>>>>>>> > Is it the value we set in the keystore / truststore, by default
>>>>>>>> cn=localhost, dc=NIFI (something like this) ?
>>>>>>>> >
>>>>>>>> > Etienne
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > Le lun. 23 nov. 2020 à 17:54, Bryan Bende <bb...@gmail.com> a
>>>>>>>> écrit :
>>>>>>>> >>
>>>>>>>> >> I don't really know the LDAP specifics too well, so I'm not
>>>>>>>> actually sure.
>>>>>>>> >>
>>>>>>>> >> You just need the nodes to come back from the LDAP
>>>>>>>> UserGroupProvider
>>>>>>>> >> as if they were regular users and members of some group "foo",
>>>>>>>> which
>>>>>>>> >> you then put "foo" into the "Node Group".
>>>>>>>> >>
>>>>>>>> >> On Mon, Nov 23, 2020 at 11:50 AM Etienne Jouvin <
>>>>>>>> lapinoujoujou@gmail.com> wrote:
>>>>>>>> >> >
>>>>>>>> >> > Thanks Bryan.
>>>>>>>> >> >
>>>>>>>> >> > With your answer.... I will go to the Node Group and assign
>>>>>>>> node identities.
>>>>>>>> >> > Better for deployment and setup on the fly, I guess.
>>>>>>>> >> >
>>>>>>>> >> > One more point, you said "creating ldap entries for your nodes
>>>>>>>> and assigning them group membership in ldap". What type of objectClass
>>>>>>>> would you assign to the node in LDAP ?
>>>>>>>> >> > This is not inetOrgPerson. The node should not have password.
>>>>>>>> >> > If I create groupOfMembers for each node, is it correct ?
>>>>>>>> >> >
>>>>>>>> >> >
>>>>>>>> >> > Thanks
>>>>>>>> >> >
>>>>>>>> >> > Etienne
>>>>>>>> >> >
>>>>>>>> >> >
>>>>>>>> >> >
>>>>>>>> >> > Le lun. 23 nov. 2020 à 17:27, Bryan Bende <bb...@gmail.com>
>>>>>>>> a écrit :
>>>>>>>> >> >>
>>>>>>>> >> >> Hello,
>>>>>>>> >> >>
>>>>>>>> >> >> "Node Identity" is similar to the "Initial Admin" concept, in
>>>>>>>> that it
>>>>>>>> >> >> sets up the policies for the initial nodes to have
>>>>>>>> permissions to
>>>>>>>> >> >> proxy.
>>>>>>>> >> >>
>>>>>>>> >> >> If you are creating ldap entries for your nodes and assigning
>>>>>>>> them
>>>>>>>> >> >> group membership in ldap, then yes you could put that group
>>>>>>>> name as
>>>>>>>> >> >> the "Node Group" and then you don't need to specify the "Node
>>>>>>>> >> >> Identities".
>>>>>>>> >> >>
>>>>>>>> >> >> If you are creating the node users in NiFi's file-based user
>>>>>>>> group
>>>>>>>> >> >> provider then you need to use node identities, and when
>>>>>>>> adding a new
>>>>>>>> >> >> node to the cluster you'd have to add the user first through
>>>>>>>> the
>>>>>>>> >> >> UI/REST API and grant it proxy, then actually connect it to
>>>>>>>> the
>>>>>>>> >> >> cluster.
>>>>>>>> >> >>
>>>>>>>> >> >> Thanks,
>>>>>>>> >> >>
>>>>>>>> >> >> Bryan
>>>>>>>> >> >>
>>>>>>>> >> >>
>>>>>>>> >> >> On Mon, Nov 23, 2020 at 7:58 AM Etienne Jouvin <
>>>>>>>> lapinoujoujou@gmail.com> wrote:
>>>>>>>> >> >> >
>>>>>>>> >> >> > Hello all.
>>>>>>>> >> >> >
>>>>>>>> >> >> >
>>>>>>>> >> >> > I am currently setting up a NiFi, 1.12.1, Cluster with LDAP
>>>>>>>> authentication.
>>>>>>>> >> >> > For now the accessPolicyProvider is the default one with
>>>>>>>> the configuration template :
>>>>>>>> >> >> > <accessPolicyProvider>
>>>>>>>> >> >> > <identifier>file-access-policy-provider</identifier>
>>>>>>>> >> >> >
>>>>>>>> <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
>>>>>>>> >> >> > <property name="User Group
>>>>>>>> Provider">file-user-group-provider</property>
>>>>>>>> >> >> > <property name="Authorizations
>>>>>>>> File">./conf/authorizations.xml</property>
>>>>>>>> >> >> > <property name="Initial Admin Identity"></property>
>>>>>>>> >> >> > <property name="Legacy Authorized Users
>>>>>>>> File"></property>
>>>>>>>> >> >> > <property name="Node Identity 1"></property>
>>>>>>>> >> >> > <property name="Node Group"></property>
>>>>>>>> >> >> > </accessPolicyProvider>
>>>>>>>> >> >> >
>>>>>>>> >> >> > But I do not really understand the purpose of the Node
>>>>>>>> Identity X property.
>>>>>>>> >> >> > If I well understood, all nodes should have the same
>>>>>>>> configuration file, and I should register all nodes identity.
>>>>>>>> >> >> >
>>>>>>>> >> >> > But what about if I want to add a new node in the cluster
>>>>>>>> on the fly ?
>>>>>>>> >> >> > Should I register a new node identity, and then I should
>>>>>>>> change all nodes configurations ?
>>>>>>>> >> >> > The comment, in the configuration file, mentions the
>>>>>>>> configuration Node Group, The name of a group containing NiFi cluster
>>>>>>>> nodes. The typical use for this is when nodes are dynamically added/removed
>>>>>>>> from the cluster.
>>>>>>>> >> >> > Should I just put a Node group name and this will do the
>>>>>>>> trick ?
>>>>>>>> >> >> >
>>>>>>>> >> >> > What should I put ? At the following link,
>>>>>>>> https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.0.3/bk_administration/content/cluster-node-identities.html,
>>>>>>>> it is said something like : cn=nifi-1,ou=people,dc=example,dc=com
>>>>>>>> >> >> > In that case, what should be the obejct class for the node
>>>>>>>> cn=nifi-1 in the LDAP ?
>>>>>>>> >> >> >
>>>>>>>> >> >> > Any documentation links will be appreciated.
>>>>>>>> >> >> >
>>>>>>>> >> >> > Regards.
>>>>>>>> >> >> >
>>>>>>>> >> >> > Etienne Jouvin
>>>>>>>>
>>>>>>> --
>>> Sent from Gmail Mobile
>>>
>>
Re: Ldap Cluster and Node Identity
Posted by Etienne Jouvin <la...@gmail.com>.
Hello all.
And finally I got it.
When I am not able to find in the documentation, because I was not
correctly searching offcourse, I read the source code and do remote debug.
And the conclusion that it is all about identity mapping.
For reminder, the objective was to have a secured cluster and everything
managed in the LDAP.
I follow the walk throughs documentation. But in it, it is only matter of
nodes managed in userGroupIdentity, based on files.
In my case, users and groups are managed in the LDAP, even nodes for
cluster.
With remote debugging, I touched the class X509AuthenticationProvider and
specially the function authenticate. By the way, this is the only class
that fires an exception with message Untrusted proxy ...
In it, identities are extracted from the request and linked certificates.
Ids extracted were :
- uid=mig-admin,ou=users,ou=nifi,dc=amexio,dc=ch , for the "real" user
- CN=mig1.amexio.ch <http://mig1.assura.ch/>, OU=users, OU=nifi,
DC=amexio, DC=ch, for the "proxy" node
But in the LDAP, the node as the entry DN : cn=mig1.amexio.ch
,ou=users,ou=nifi;dc=amexio,dc=ch
So when the provider validate identities, the NiFi identity "CN=
mig1.amexio.ch <http://mig1.assura.ch/>, OU=users, OU=nifi, DC=amexio,
DC=ch" is not found. No user/group, and does not belongs to any group. Then
the validation on policy "proxy", with "write" access, failed and the
connection was rejected.
What I found also in this function, there is a identity mapping,
function mapIdentity from class NiFiAuthenticationProvider.
This will use function mapIdentity in class IdentityMappingUtil.
With this, the identity is normalized and this is the trick to transform
the idnetity retrieved from the certificate, with space and upper case, to
an identity matching the one retrieved from LDAP.
That was the point. Si in the file nifi.properties, for all nodes, here is
modifications :
nifi.security.identity.mapping.pattern.ldapCertNode=^CN=(.*?), OU=(.*?),
OU=(.*?), DC=(.*?), DC=(.*?)$
nifi.security.identity.mapping.value.ldapCertNode=cn=$1,ou=$2,ou=$3,dc=$4,dc=$5
nifi.security.identity.mapping.transform.ldapCertNode=NONE
Like this, the identity retrieved from certificate, for the proxy node, is
CN=mig1.amexio.ch <http://mig1.assura.ch/>, OU=users, OU=nifi, DC=amexio,
DC=ch
After identity mapping it is cn=mig1.amexio.ch <http://mig1.assura.ch/>
,ou=users,ou=nifi,dc=amexio,dc=ch
And it is exactly matching identity retrieved from the LDAP for the node.
Et voilà.
Hard work (for me) and happy to find the way to manage this.
Thanks all for your help.
Etienne
Le mer. 25 nov. 2020 à 19:50, Etienne Jouvin <la...@gmail.com> a
écrit :
> That's what I suspect for now.
>
> But this is strange to have a transformation with space introduced.
> In the certificate (I use KeyStore Explorer to visualize the certificate)
> there is no space.
>
> But I will give a try to use the identity mapping from here
> # Identity Mapping Properties #
> # These properties allow normalizing user identities such that identities
> coming from different identity providers
> # (certificates, LDAP, Kerberos) can be treated the same internally in
> NiFi. The following example demonstrates normalizing
> # DNs from certificates and principals from Kerberos into a common
> identity string:
> #
> # nifi.security.identity.mapping.pattern.dn=^CN=(.*?), OU=(.*?), O=(.*?),
> L=(.*?), ST=(.*?), C=(.*?)$
> # nifi.security.identity.mapping.value.dn=$1@$2
> # nifi.security.identity.mapping.transform.dn=NONE
> # nifi.security.identity.mapping.pattern.kerb=^(.*?)/instance@(.*?)$
> # nifi.security.identity.mapping.value.kerb=$1@$2
> # nifi.security.identity.mapping.transform.kerb=UPPER
>
> # Group Mapping Properties #
> # These properties allow normalizing group names coming from external
> sources like LDAP. The following example
> # lowercases any group name.
> #
> # nifi.security.group.mapping.pattern.anygroup=^(.*)$
> # nifi.security.group.mapping.value.anygroup=$1
> # nifi.security.group.mapping.transform.anygroup=LOWER
>
> for now, I do not know what to put ;)
>
> Etienne
>
>
>
> Le mer. 25 nov. 2020 à 19:45, Bryan Bende <bb...@gmail.com> a écrit :
>
>> The values have to match exactly, it’s case and white space sensitive.
>> There are identity transforms in Nifi properties that possibly help.
>>
>> The value from where it says “Attempting request for...” is the value
>> nifi is getting from the certificate. You can also use key tool to list the
>> contents of the keystore jks and it will probably be the same as the log.
>>
>> On Wed, Nov 25, 2020 at 1:30 PM Etienne Jouvin <la...@gmail.com>
>> wrote:
>>
>>> Hello all.
>>>
>>> Still no luck.
>>> I follow steps from here :
>>> https://nifi.apache.org/docs/nifi-docs/html/walkthroughs.html#creating-and-securing-a-nifi-cluster-with-the-tls-toolkit
>>>
>>> Reminder, everything is currently in LDAP, I do not have any file
>>> user/group provider.
>>> When only one node, I can connect, this is correct.
>>>
>>> I can see the group having nodes "users" from the LDAP and the group has
>>> the proxy policy.
>>>
>>> But one thing strange, may be not the cause.
>>> In the LDAP, the DN is something like :
>>> cn=mig1.amexio.ch,ou=users,ou=nifi,dc=amexio,dc=ch
>>>
>>> And when I try to connect, I can see thoses errors :
>>> In nifi-user.log for the node where I connect :
>>> INFO [NiFi Web Server-22] o.a.n.w.s.NiFiAuthenticationFilter
>>> Authentication success for uid=mig-admin,ou=users,ou=nifi,dc=amexio,dc=ch
>>>
>>> In nifi-user.log on the second node :
>>> INFO [NiFi Web Server-29] o.a.n.w.s.NiFiAuthenticationFilter Attempting
>>> request for (<uid=mig-admin,ou=users,ou=nifi,dc=amexio,dc=ch><CN=
>>> mig1.amexio.ch <http://mig1.assura.ch>, OU=users, OU=nifi, DC=amexio,
>>> DC=ch>) GET https://mig2.amexio.ch:9442/nifi-api/flow/current-user
>>> (source ip: 127.0.0.1)
>>> WARN [NiFi Web Server-29] o.a.n.w.s.NiFiAuthenticationFilter Rejecting
>>> access to web api: Untrusted proxy CN=mig1.amexio.ch, OU=users,
>>> OU=nifi, DC=amexio, DC=ch
>>>
>>> The doubt is on the message, why the node DN has spaces between each
>>> level ?
>>> That does not match the value in the certificate and not the DN in LDAP.
>>>
>>> Maybe this is not the good route to investigate. Still searching. I have
>>> to find a way to "debug" the NiFiAuthenticationFilter and check the
>>> certificate comparison.
>>>
>>> Etienne
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Le mer. 25 nov. 2020 à 14:18, David Handermann <
>>> exceptionfactory@gmail.com> a écrit :
>>>
>>>> I am not as familiar with the LDAP user group provider, but based on
>>>> the "Untrusted proxy" message you are seeing, it sounds like the nodes are
>>>> not being identified properly as members of the "nodes" group from LDAP.
>>>> Just for testing purposes, you could try specifying the node distinguished
>>>> names in the "Node Identity N" properties of the access policy provider,
>>>> using "Node Identity 1", "Node Identity 2" and "Node Identity 3" to specify
>>>> each node DN. If that works, then it sounds like a configuration issue
>>>> with the Node Group, either on the LDAP server, or in the way NiFi is
>>>> attempting to query LDAP.
>>>>
>>>> Regards,
>>>> David Handermann
>>>>
>>>> On Wed, Nov 25, 2020 at 5:19 AM Etienne Jouvin <la...@gmail.com>
>>>> wrote:
>>>>
>>>>> Just for information, did not have time to test it from now.
>>>>> I was not able to get this Walk Throughs documentation.
>>>>> https://nifi.apache.org/docs/nifi-docs/html/walkthroughs.html
>>>>>
>>>>> Hope I will find the error I have about certificate (I have a little
>>>>> idea)
>>>>>
>>>>> Etienne
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Le mer. 25 nov. 2020 à 08:36, Etienne Jouvin <la...@gmail.com>
>>>>> a écrit :
>>>>>
>>>>>> Hello.
>>>>>>
>>>>>> I made some progress yesterday.
>>>>>> I did setup in LDAP groups and person
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Groups :
>>>>>> cn=administrators,ou=groups,ou=nifi,dc=amexio,dc=ch : for
>>>>>> administrators
>>>>>> cn=supervisors,ou=groups,ou=nifi,dc=amexio,dc=ch : for supervisors
>>>>>> cn=nodes,ou=groups,ou=nifi,dc=amexio,dc=ch : group where I put all
>>>>>> "person" representing NiFi nodes.
>>>>>>
>>>>>> Users :
>>>>>> cn=migX.amexio.ch,ou=users,ou=nifi,dc=amexio,dc=ch : for each node,
>>>>>> replacing X by the index, and with object class person
>>>>>> uid=mig-admin,ou=users,ou=nifi,dc=amexio,dc=ch : example of "real"
>>>>>> user used to connect on the platform, with object class inetOrgperson
>>>>>>
>>>>>> In NiFi configuration.
>>>>>> I did activate a userGroupProvider linked to the LDAP
>>>>>> <userGroupProvider>
>>>>>> <identifier>amexio-ldap-user-group-provider</identifier>
>>>>>>
>>>>>> <class>org.apache.nifi.ldap.tenants.LdapUserGroupProvider</class>
>>>>>> <property name="Authentication Strategy">SIMPLE</property>
>>>>>>
>>>>>> <property name="Manager DN">uid=admin,ou=system</property>
>>>>>> <property name="Manager Password">secret</property>
>>>>>>
>>>>>> <property name="TLS - Keystore"></property>
>>>>>> <property name="TLS - Keystore Password"></property>
>>>>>> <property name="TLS - Keystore Type"></property>
>>>>>> <property name="TLS - Truststore"></property>
>>>>>> <property name="TLS - Truststore Password"></property>
>>>>>> <property name="TLS - Truststore Type"></property>
>>>>>> <property name="TLS - Client Auth"></property>
>>>>>> <property name="TLS - Protocol"></property>
>>>>>> <property name="TLS - Shutdown Gracefully"></property>
>>>>>>
>>>>>> <property name="Referral Strategy">FOLLOW</property>
>>>>>> <property name="Connect Timeout">10 secs</property>
>>>>>> <property name="Read Timeout">10 secs</property>
>>>>>>
>>>>>> <property name="Url">ldap://localhost:10389</property>
>>>>>> <property name="Page Size">50</property>
>>>>>> <!-- <property name="Sync Interval">30 mins</property> -->
>>>>>> <property name="Sync Interval">30 seconds</property>
>>>>>> <property name="Group Membership - Enforce Case
>>>>>> Sensitivity">false</property>
>>>>>>
>>>>>> <property name="User Search
>>>>>> Base">ou=users,ou=nifi,dc=amexio,dc=ch</property>
>>>>>> <property name="User Object Class">person</property>
>>>>>> <property name="User Search Scope">ONE_LEVEL</property>
>>>>>> <property name="User Search Filter"></property>
>>>>>> <property name="User Identity Attribute"></property>
>>>>>> <property name="User Group Name Attribute"></property>
>>>>>> <property name="User Group Name Attribute - Referenced Group
>>>>>> Attribute"></property>
>>>>>>
>>>>>> <property name="Group Search
>>>>>> Base">ou=groups,ou=nifi,dc=amexio,dc=ch</property>
>>>>>> <property name="Group Object Class">groupOfNames</property>
>>>>>> <property name="Group Search Scope">ONE_LEVEL</property>
>>>>>> <property name="Group Search Filter"></property>
>>>>>> <property name="Group Name Attribute">cn</property>
>>>>>> <property name="Group Member Attribute">member</property>
>>>>>> <property name="Group Member Attribute - Referenced User
>>>>>> Attribute"></property>
>>>>>> </userGroupProvider>
>>>>>>
>>>>>> Of course, register it inside the accessPolicyProvider
>>>>>> <accessPolicyProvider>
>>>>>> <identifier>file-access-policy-provider</identifier>
>>>>>>
>>>>>> <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
>>>>>> <!-- <property name="User Group
>>>>>> Provider">file-user-group-provider</property> -->
>>>>>> <property name="User Group
>>>>>> Provider">amexio-ldap-user-group-provider</property>
>>>>>> <property name="Authorizations
>>>>>> File">./conf/authorizations.xml</property>
>>>>>> <!-- <property name="Initial Admin Identity"></property> -->
>>>>>> <property name="Initial Admin
>>>>>> Identity">uid=mig-admin,ou=users,ou=nifi,dc=amexio,dc=ch</property>
>>>>>> <property name="Legacy Authorized Users File"></property>
>>>>>> <property name="Node Identity 1"></property>
>>>>>> <property name="Node Group">nodes</property>
>>>>>> </accessPolicyProvider>
>>>>>>
>>>>>> I am able to connect with the initial administrator account, when the
>>>>>> first node is started.
>>>>>> And all nodes are synchronized in the NiFi instance.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> As soon as I start an additional node, I can not connect to the first
>>>>>> node
>>>>>> Erreur message
>>>>>> Untrusted proxy CN=mig1.amexio.ch, OU=users, OU=nifi, DC=amexio,
>>>>>> DC=ch
>>>>>>
>>>>>> But I can connect on the second node.
>>>>>>
>>>>>>
>>>>>> So all this is about the certificate I guess.
>>>>>> for reminder, I use nls-toolkit to generate certificate on all nodes
>>>>>> with something like :
>>>>>> tls-toolkit.bat standalone -f
>>>>>> "C:\nifi-1.12.1\node1\conf\nifi.properties" -o
>>>>>> "C:\nifi-1.12.1\node1\conf\secured" -n mig1.amexio.ch --nifiDnPrefix
>>>>>> cn= --nifiDnSuffix ou=users,ou=nifi,dc=amexio,dc=ch
>>>>>>
>>>>>> Proxy is untrusted, ok fine. So may be I should not use the
>>>>>> standalone function of toolkit, but using server and client. In that case,
>>>>>> I have to stay alive the server from toolkit ?
>>>>>> Also, it seems I did not add certificate from node1 inside node2
>>>>>> trutstore, and node2 certificate inside node1 truststore ?
>>>>>> But in this case, if I have to add a new node, let's say node4, I
>>>>>> would have to push the certificate from node4 inside all existing nodes ?
>>>>>>
>>>>>> I continue to search, but any idea / input will be appreciated.
>>>>>>
>>>>>> Etienne
>>>>>>
>>>>>>
>>>>>> Le lun. 23 nov. 2020 à 18:39, Bryan Bende <bb...@gmail.com> a
>>>>>> écrit :
>>>>>>
>>>>>>> Yes it will be the DN of the server's certificate which comes from
>>>>>>> the keystore.
>>>>>>>
>>>>>>> NiFi will get an incoming request, see that there is an X509 cert,
>>>>>>> take the DN and go to the user group provider and ask for the user
>>>>>>> with this identity.
>>>>>>>
>>>>>>> On Mon, Nov 23, 2020 at 12:01 PM Etienne Jouvin <
>>>>>>> lapinoujoujou@gmail.com> wrote:
>>>>>>> >
>>>>>>> > Hum OK,
>>>>>>> >
>>>>>>> > I will give it a try.
>>>>>>> > But one more thing...
>>>>>>> >
>>>>>>> > If I only set the group node;
>>>>>>> > How NiFi will connect the node with the nodeId in the LDAP ?
>>>>>>> > Where does it take the nodeid value ?
>>>>>>> > Is it the value we set in the keystore / truststore, by default
>>>>>>> cn=localhost, dc=NIFI (something like this) ?
>>>>>>> >
>>>>>>> > Etienne
>>>>>>> >
>>>>>>> >
>>>>>>> > Le lun. 23 nov. 2020 à 17:54, Bryan Bende <bb...@gmail.com> a
>>>>>>> écrit :
>>>>>>> >>
>>>>>>> >> I don't really know the LDAP specifics too well, so I'm not
>>>>>>> actually sure.
>>>>>>> >>
>>>>>>> >> You just need the nodes to come back from the LDAP
>>>>>>> UserGroupProvider
>>>>>>> >> as if they were regular users and members of some group "foo",
>>>>>>> which
>>>>>>> >> you then put "foo" into the "Node Group".
>>>>>>> >>
>>>>>>> >> On Mon, Nov 23, 2020 at 11:50 AM Etienne Jouvin <
>>>>>>> lapinoujoujou@gmail.com> wrote:
>>>>>>> >> >
>>>>>>> >> > Thanks Bryan.
>>>>>>> >> >
>>>>>>> >> > With your answer.... I will go to the Node Group and assign
>>>>>>> node identities.
>>>>>>> >> > Better for deployment and setup on the fly, I guess.
>>>>>>> >> >
>>>>>>> >> > One more point, you said "creating ldap entries for your nodes
>>>>>>> and assigning them group membership in ldap". What type of objectClass
>>>>>>> would you assign to the node in LDAP ?
>>>>>>> >> > This is not inetOrgPerson. The node should not have password.
>>>>>>> >> > If I create groupOfMembers for each node, is it correct ?
>>>>>>> >> >
>>>>>>> >> >
>>>>>>> >> > Thanks
>>>>>>> >> >
>>>>>>> >> > Etienne
>>>>>>> >> >
>>>>>>> >> >
>>>>>>> >> >
>>>>>>> >> > Le lun. 23 nov. 2020 à 17:27, Bryan Bende <bb...@gmail.com> a
>>>>>>> écrit :
>>>>>>> >> >>
>>>>>>> >> >> Hello,
>>>>>>> >> >>
>>>>>>> >> >> "Node Identity" is similar to the "Initial Admin" concept, in
>>>>>>> that it
>>>>>>> >> >> sets up the policies for the initial nodes to have permissions
>>>>>>> to
>>>>>>> >> >> proxy.
>>>>>>> >> >>
>>>>>>> >> >> If you are creating ldap entries for your nodes and assigning
>>>>>>> them
>>>>>>> >> >> group membership in ldap, then yes you could put that group
>>>>>>> name as
>>>>>>> >> >> the "Node Group" and then you don't need to specify the "Node
>>>>>>> >> >> Identities".
>>>>>>> >> >>
>>>>>>> >> >> If you are creating the node users in NiFi's file-based user
>>>>>>> group
>>>>>>> >> >> provider then you need to use node identities, and when adding
>>>>>>> a new
>>>>>>> >> >> node to the cluster you'd have to add the user first through
>>>>>>> the
>>>>>>> >> >> UI/REST API and grant it proxy, then actually connect it to the
>>>>>>> >> >> cluster.
>>>>>>> >> >>
>>>>>>> >> >> Thanks,
>>>>>>> >> >>
>>>>>>> >> >> Bryan
>>>>>>> >> >>
>>>>>>> >> >>
>>>>>>> >> >> On Mon, Nov 23, 2020 at 7:58 AM Etienne Jouvin <
>>>>>>> lapinoujoujou@gmail.com> wrote:
>>>>>>> >> >> >
>>>>>>> >> >> > Hello all.
>>>>>>> >> >> >
>>>>>>> >> >> >
>>>>>>> >> >> > I am currently setting up a NiFi, 1.12.1, Cluster with LDAP
>>>>>>> authentication.
>>>>>>> >> >> > For now the accessPolicyProvider is the default one with the
>>>>>>> configuration template :
>>>>>>> >> >> > <accessPolicyProvider>
>>>>>>> >> >> > <identifier>file-access-policy-provider</identifier>
>>>>>>> >> >> >
>>>>>>> <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
>>>>>>> >> >> > <property name="User Group
>>>>>>> Provider">file-user-group-provider</property>
>>>>>>> >> >> > <property name="Authorizations
>>>>>>> File">./conf/authorizations.xml</property>
>>>>>>> >> >> > <property name="Initial Admin Identity"></property>
>>>>>>> >> >> > <property name="Legacy Authorized Users
>>>>>>> File"></property>
>>>>>>> >> >> > <property name="Node Identity 1"></property>
>>>>>>> >> >> > <property name="Node Group"></property>
>>>>>>> >> >> > </accessPolicyProvider>
>>>>>>> >> >> >
>>>>>>> >> >> > But I do not really understand the purpose of the Node
>>>>>>> Identity X property.
>>>>>>> >> >> > If I well understood, all nodes should have the same
>>>>>>> configuration file, and I should register all nodes identity.
>>>>>>> >> >> >
>>>>>>> >> >> > But what about if I want to add a new node in the cluster on
>>>>>>> the fly ?
>>>>>>> >> >> > Should I register a new node identity, and then I should
>>>>>>> change all nodes configurations ?
>>>>>>> >> >> > The comment, in the configuration file, mentions the
>>>>>>> configuration Node Group, The name of a group containing NiFi cluster
>>>>>>> nodes. The typical use for this is when nodes are dynamically added/removed
>>>>>>> from the cluster.
>>>>>>> >> >> > Should I just put a Node group name and this will do the
>>>>>>> trick ?
>>>>>>> >> >> >
>>>>>>> >> >> > What should I put ? At the following link,
>>>>>>> https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.0.3/bk_administration/content/cluster-node-identities.html,
>>>>>>> it is said something like : cn=nifi-1,ou=people,dc=example,dc=com
>>>>>>> >> >> > In that case, what should be the obejct class for the node
>>>>>>> cn=nifi-1 in the LDAP ?
>>>>>>> >> >> >
>>>>>>> >> >> > Any documentation links will be appreciated.
>>>>>>> >> >> >
>>>>>>> >> >> > Regards.
>>>>>>> >> >> >
>>>>>>> >> >> > Etienne Jouvin
>>>>>>>
>>>>>> --
>> Sent from Gmail Mobile
>>
>
Re: Ldap Cluster and Node Identity
Posted by Etienne Jouvin <la...@gmail.com>.
That's what I suspect for now.
But this is strange to have a transformation with space introduced.
In the certificate (I use KeyStore Explorer to visualize the certificate)
there is no space.
But I will give a try to use the identity mapping from here
# Identity Mapping Properties #
# These properties allow normalizing user identities such that identities
coming from different identity providers
# (certificates, LDAP, Kerberos) can be treated the same internally in
NiFi. The following example demonstrates normalizing
# DNs from certificates and principals from Kerberos into a common identity
string:
#
# nifi.security.identity.mapping.pattern.dn=^CN=(.*?), OU=(.*?), O=(.*?),
L=(.*?), ST=(.*?), C=(.*?)$
# nifi.security.identity.mapping.value.dn=$1@$2
# nifi.security.identity.mapping.transform.dn=NONE
# nifi.security.identity.mapping.pattern.kerb=^(.*?)/instance@(.*?)$
# nifi.security.identity.mapping.value.kerb=$1@$2
# nifi.security.identity.mapping.transform.kerb=UPPER
# Group Mapping Properties #
# These properties allow normalizing group names coming from external
sources like LDAP. The following example
# lowercases any group name.
#
# nifi.security.group.mapping.pattern.anygroup=^(.*)$
# nifi.security.group.mapping.value.anygroup=$1
# nifi.security.group.mapping.transform.anygroup=LOWER
for now, I do not know what to put ;)
Etienne
Le mer. 25 nov. 2020 à 19:45, Bryan Bende <bb...@gmail.com> a écrit :
> The values have to match exactly, it’s case and white space sensitive.
> There are identity transforms in Nifi properties that possibly help.
>
> The value from where it says “Attempting request for...” is the value nifi
> is getting from the certificate. You can also use key tool to list the
> contents of the keystore jks and it will probably be the same as the log.
>
> On Wed, Nov 25, 2020 at 1:30 PM Etienne Jouvin <la...@gmail.com>
> wrote:
>
>> Hello all.
>>
>> Still no luck.
>> I follow steps from here :
>> https://nifi.apache.org/docs/nifi-docs/html/walkthroughs.html#creating-and-securing-a-nifi-cluster-with-the-tls-toolkit
>>
>> Reminder, everything is currently in LDAP, I do not have any file
>> user/group provider.
>> When only one node, I can connect, this is correct.
>>
>> I can see the group having nodes "users" from the LDAP and the group has
>> the proxy policy.
>>
>> But one thing strange, may be not the cause.
>> In the LDAP, the DN is something like :
>> cn=mig1.amexio.ch,ou=users,ou=nifi,dc=amexio,dc=ch
>>
>> And when I try to connect, I can see thoses errors :
>> In nifi-user.log for the node where I connect :
>> INFO [NiFi Web Server-22] o.a.n.w.s.NiFiAuthenticationFilter
>> Authentication success for uid=mig-admin,ou=users,ou=nifi,dc=amexio,dc=ch
>>
>> In nifi-user.log on the second node :
>> INFO [NiFi Web Server-29] o.a.n.w.s.NiFiAuthenticationFilter Attempting
>> request for (<uid=mig-admin,ou=users,ou=nifi,dc=amexio,dc=ch><CN=
>> mig1.assura.ch, OU=users, OU=nifi, DC=amexio, DC=ch>) GET
>> https://mig2.amexio.ch:9442/nifi-api/flow/current-user (source ip:
>> 127.0.0.1)
>> WARN [NiFi Web Server-29] o.a.n.w.s.NiFiAuthenticationFilter Rejecting
>> access to web api: Untrusted proxy CN=mig1.amexio.ch, OU=users, OU=nifi,
>> DC=amexio, DC=ch
>>
>> The doubt is on the message, why the node DN has spaces between each
>> level ?
>> That does not match the value in the certificate and not the DN in LDAP.
>>
>> Maybe this is not the good route to investigate. Still searching. I have
>> to find a way to "debug" the NiFiAuthenticationFilter and check the
>> certificate comparison.
>>
>> Etienne
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> Le mer. 25 nov. 2020 à 14:18, David Handermann <
>> exceptionfactory@gmail.com> a écrit :
>>
>>> I am not as familiar with the LDAP user group provider, but based on the
>>> "Untrusted proxy" message you are seeing, it sounds like the nodes are not
>>> being identified properly as members of the "nodes" group from LDAP. Just
>>> for testing purposes, you could try specifying the node distinguished names
>>> in the "Node Identity N" properties of the access policy provider, using
>>> "Node Identity 1", "Node Identity 2" and "Node Identity 3" to specify each
>>> node DN. If that works, then it sounds like a configuration issue with the
>>> Node Group, either on the LDAP server, or in the way NiFi is attempting to
>>> query LDAP.
>>>
>>> Regards,
>>> David Handermann
>>>
>>> On Wed, Nov 25, 2020 at 5:19 AM Etienne Jouvin <la...@gmail.com>
>>> wrote:
>>>
>>>> Just for information, did not have time to test it from now.
>>>> I was not able to get this Walk Throughs documentation.
>>>> https://nifi.apache.org/docs/nifi-docs/html/walkthroughs.html
>>>>
>>>> Hope I will find the error I have about certificate (I have a little
>>>> idea)
>>>>
>>>> Etienne
>>>>
>>>>
>>>>
>>>>
>>>> Le mer. 25 nov. 2020 à 08:36, Etienne Jouvin <la...@gmail.com>
>>>> a écrit :
>>>>
>>>>> Hello.
>>>>>
>>>>> I made some progress yesterday.
>>>>> I did setup in LDAP groups and person
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Groups :
>>>>> cn=administrators,ou=groups,ou=nifi,dc=amexio,dc=ch : for
>>>>> administrators
>>>>> cn=supervisors,ou=groups,ou=nifi,dc=amexio,dc=ch : for supervisors
>>>>> cn=nodes,ou=groups,ou=nifi,dc=amexio,dc=ch : group where I put all
>>>>> "person" representing NiFi nodes.
>>>>>
>>>>> Users :
>>>>> cn=migX.amexio.ch,ou=users,ou=nifi,dc=amexio,dc=ch : for each node,
>>>>> replacing X by the index, and with object class person
>>>>> uid=mig-admin,ou=users,ou=nifi,dc=amexio,dc=ch : example of "real"
>>>>> user used to connect on the platform, with object class inetOrgperson
>>>>>
>>>>> In NiFi configuration.
>>>>> I did activate a userGroupProvider linked to the LDAP
>>>>> <userGroupProvider>
>>>>> <identifier>amexio-ldap-user-group-provider</identifier>
>>>>>
>>>>> <class>org.apache.nifi.ldap.tenants.LdapUserGroupProvider</class>
>>>>> <property name="Authentication Strategy">SIMPLE</property>
>>>>>
>>>>> <property name="Manager DN">uid=admin,ou=system</property>
>>>>> <property name="Manager Password">secret</property>
>>>>>
>>>>> <property name="TLS - Keystore"></property>
>>>>> <property name="TLS - Keystore Password"></property>
>>>>> <property name="TLS - Keystore Type"></property>
>>>>> <property name="TLS - Truststore"></property>
>>>>> <property name="TLS - Truststore Password"></property>
>>>>> <property name="TLS - Truststore Type"></property>
>>>>> <property name="TLS - Client Auth"></property>
>>>>> <property name="TLS - Protocol"></property>
>>>>> <property name="TLS - Shutdown Gracefully"></property>
>>>>>
>>>>> <property name="Referral Strategy">FOLLOW</property>
>>>>> <property name="Connect Timeout">10 secs</property>
>>>>> <property name="Read Timeout">10 secs</property>
>>>>>
>>>>> <property name="Url">ldap://localhost:10389</property>
>>>>> <property name="Page Size">50</property>
>>>>> <!-- <property name="Sync Interval">30 mins</property> -->
>>>>> <property name="Sync Interval">30 seconds</property>
>>>>> <property name="Group Membership - Enforce Case
>>>>> Sensitivity">false</property>
>>>>>
>>>>> <property name="User Search
>>>>> Base">ou=users,ou=nifi,dc=amexio,dc=ch</property>
>>>>> <property name="User Object Class">person</property>
>>>>> <property name="User Search Scope">ONE_LEVEL</property>
>>>>> <property name="User Search Filter"></property>
>>>>> <property name="User Identity Attribute"></property>
>>>>> <property name="User Group Name Attribute"></property>
>>>>> <property name="User Group Name Attribute - Referenced Group
>>>>> Attribute"></property>
>>>>>
>>>>> <property name="Group Search
>>>>> Base">ou=groups,ou=nifi,dc=amexio,dc=ch</property>
>>>>> <property name="Group Object Class">groupOfNames</property>
>>>>> <property name="Group Search Scope">ONE_LEVEL</property>
>>>>> <property name="Group Search Filter"></property>
>>>>> <property name="Group Name Attribute">cn</property>
>>>>> <property name="Group Member Attribute">member</property>
>>>>> <property name="Group Member Attribute - Referenced User
>>>>> Attribute"></property>
>>>>> </userGroupProvider>
>>>>>
>>>>> Of course, register it inside the accessPolicyProvider
>>>>> <accessPolicyProvider>
>>>>> <identifier>file-access-policy-provider</identifier>
>>>>>
>>>>> <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
>>>>> <!-- <property name="User Group
>>>>> Provider">file-user-group-provider</property> -->
>>>>> <property name="User Group
>>>>> Provider">amexio-ldap-user-group-provider</property>
>>>>> <property name="Authorizations
>>>>> File">./conf/authorizations.xml</property>
>>>>> <!-- <property name="Initial Admin Identity"></property> -->
>>>>> <property name="Initial Admin
>>>>> Identity">uid=mig-admin,ou=users,ou=nifi,dc=amexio,dc=ch</property>
>>>>> <property name="Legacy Authorized Users File"></property>
>>>>> <property name="Node Identity 1"></property>
>>>>> <property name="Node Group">nodes</property>
>>>>> </accessPolicyProvider>
>>>>>
>>>>> I am able to connect with the initial administrator account, when the
>>>>> first node is started.
>>>>> And all nodes are synchronized in the NiFi instance.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> As soon as I start an additional node, I can not connect to the first
>>>>> node
>>>>> Erreur message
>>>>> Untrusted proxy CN=mig1.amexio.ch, OU=users, OU=nifi, DC=amexio, DC=ch
>>>>>
>>>>>
>>>>> But I can connect on the second node.
>>>>>
>>>>>
>>>>> So all this is about the certificate I guess.
>>>>> for reminder, I use nls-toolkit to generate certificate on all nodes
>>>>> with something like :
>>>>> tls-toolkit.bat standalone -f
>>>>> "C:\nifi-1.12.1\node1\conf\nifi.properties" -o
>>>>> "C:\nifi-1.12.1\node1\conf\secured" -n mig1.amexio.ch --nifiDnPrefix
>>>>> cn= --nifiDnSuffix ou=users,ou=nifi,dc=amexio,dc=ch
>>>>>
>>>>> Proxy is untrusted, ok fine. So may be I should not use the standalone
>>>>> function of toolkit, but using server and client. In that case, I have to
>>>>> stay alive the server from toolkit ?
>>>>> Also, it seems I did not add certificate from node1 inside node2
>>>>> trutstore, and node2 certificate inside node1 truststore ?
>>>>> But in this case, if I have to add a new node, let's say node4, I
>>>>> would have to push the certificate from node4 inside all existing nodes ?
>>>>>
>>>>> I continue to search, but any idea / input will be appreciated.
>>>>>
>>>>> Etienne
>>>>>
>>>>>
>>>>> Le lun. 23 nov. 2020 à 18:39, Bryan Bende <bb...@gmail.com> a écrit :
>>>>>
>>>>>> Yes it will be the DN of the server's certificate which comes from
>>>>>> the keystore.
>>>>>>
>>>>>> NiFi will get an incoming request, see that there is an X509 cert,
>>>>>> take the DN and go to the user group provider and ask for the user
>>>>>> with this identity.
>>>>>>
>>>>>> On Mon, Nov 23, 2020 at 12:01 PM Etienne Jouvin <
>>>>>> lapinoujoujou@gmail.com> wrote:
>>>>>> >
>>>>>> > Hum OK,
>>>>>> >
>>>>>> > I will give it a try.
>>>>>> > But one more thing...
>>>>>> >
>>>>>> > If I only set the group node;
>>>>>> > How NiFi will connect the node with the nodeId in the LDAP ?
>>>>>> > Where does it take the nodeid value ?
>>>>>> > Is it the value we set in the keystore / truststore, by default
>>>>>> cn=localhost, dc=NIFI (something like this) ?
>>>>>> >
>>>>>> > Etienne
>>>>>> >
>>>>>> >
>>>>>> > Le lun. 23 nov. 2020 à 17:54, Bryan Bende <bb...@gmail.com> a
>>>>>> écrit :
>>>>>> >>
>>>>>> >> I don't really know the LDAP specifics too well, so I'm not
>>>>>> actually sure.
>>>>>> >>
>>>>>> >> You just need the nodes to come back from the LDAP
>>>>>> UserGroupProvider
>>>>>> >> as if they were regular users and members of some group "foo",
>>>>>> which
>>>>>> >> you then put "foo" into the "Node Group".
>>>>>> >>
>>>>>> >> On Mon, Nov 23, 2020 at 11:50 AM Etienne Jouvin <
>>>>>> lapinoujoujou@gmail.com> wrote:
>>>>>> >> >
>>>>>> >> > Thanks Bryan.
>>>>>> >> >
>>>>>> >> > With your answer.... I will go to the Node Group and assign node
>>>>>> identities.
>>>>>> >> > Better for deployment and setup on the fly, I guess.
>>>>>> >> >
>>>>>> >> > One more point, you said "creating ldap entries for your nodes
>>>>>> and assigning them group membership in ldap". What type of objectClass
>>>>>> would you assign to the node in LDAP ?
>>>>>> >> > This is not inetOrgPerson. The node should not have password.
>>>>>> >> > If I create groupOfMembers for each node, is it correct ?
>>>>>> >> >
>>>>>> >> >
>>>>>> >> > Thanks
>>>>>> >> >
>>>>>> >> > Etienne
>>>>>> >> >
>>>>>> >> >
>>>>>> >> >
>>>>>> >> > Le lun. 23 nov. 2020 à 17:27, Bryan Bende <bb...@gmail.com> a
>>>>>> écrit :
>>>>>> >> >>
>>>>>> >> >> Hello,
>>>>>> >> >>
>>>>>> >> >> "Node Identity" is similar to the "Initial Admin" concept, in
>>>>>> that it
>>>>>> >> >> sets up the policies for the initial nodes to have permissions
>>>>>> to
>>>>>> >> >> proxy.
>>>>>> >> >>
>>>>>> >> >> If you are creating ldap entries for your nodes and assigning
>>>>>> them
>>>>>> >> >> group membership in ldap, then yes you could put that group
>>>>>> name as
>>>>>> >> >> the "Node Group" and then you don't need to specify the "Node
>>>>>> >> >> Identities".
>>>>>> >> >>
>>>>>> >> >> If you are creating the node users in NiFi's file-based user
>>>>>> group
>>>>>> >> >> provider then you need to use node identities, and when adding
>>>>>> a new
>>>>>> >> >> node to the cluster you'd have to add the user first through the
>>>>>> >> >> UI/REST API and grant it proxy, then actually connect it to the
>>>>>> >> >> cluster.
>>>>>> >> >>
>>>>>> >> >> Thanks,
>>>>>> >> >>
>>>>>> >> >> Bryan
>>>>>> >> >>
>>>>>> >> >>
>>>>>> >> >> On Mon, Nov 23, 2020 at 7:58 AM Etienne Jouvin <
>>>>>> lapinoujoujou@gmail.com> wrote:
>>>>>> >> >> >
>>>>>> >> >> > Hello all.
>>>>>> >> >> >
>>>>>> >> >> >
>>>>>> >> >> > I am currently setting up a NiFi, 1.12.1, Cluster with LDAP
>>>>>> authentication.
>>>>>> >> >> > For now the accessPolicyProvider is the default one with the
>>>>>> configuration template :
>>>>>> >> >> > <accessPolicyProvider>
>>>>>> >> >> > <identifier>file-access-policy-provider</identifier>
>>>>>> >> >> >
>>>>>> <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
>>>>>> >> >> > <property name="User Group
>>>>>> Provider">file-user-group-provider</property>
>>>>>> >> >> > <property name="Authorizations
>>>>>> File">./conf/authorizations.xml</property>
>>>>>> >> >> > <property name="Initial Admin Identity"></property>
>>>>>> >> >> > <property name="Legacy Authorized Users
>>>>>> File"></property>
>>>>>> >> >> > <property name="Node Identity 1"></property>
>>>>>> >> >> > <property name="Node Group"></property>
>>>>>> >> >> > </accessPolicyProvider>
>>>>>> >> >> >
>>>>>> >> >> > But I do not really understand the purpose of the Node
>>>>>> Identity X property.
>>>>>> >> >> > If I well understood, all nodes should have the same
>>>>>> configuration file, and I should register all nodes identity.
>>>>>> >> >> >
>>>>>> >> >> > But what about if I want to add a new node in the cluster on
>>>>>> the fly ?
>>>>>> >> >> > Should I register a new node identity, and then I should
>>>>>> change all nodes configurations ?
>>>>>> >> >> > The comment, in the configuration file, mentions the
>>>>>> configuration Node Group, The name of a group containing NiFi cluster
>>>>>> nodes. The typical use for this is when nodes are dynamically added/removed
>>>>>> from the cluster.
>>>>>> >> >> > Should I just put a Node group name and this will do the
>>>>>> trick ?
>>>>>> >> >> >
>>>>>> >> >> > What should I put ? At the following link,
>>>>>> https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.0.3/bk_administration/content/cluster-node-identities.html,
>>>>>> it is said something like : cn=nifi-1,ou=people,dc=example,dc=com
>>>>>> >> >> > In that case, what should be the obejct class for the node
>>>>>> cn=nifi-1 in the LDAP ?
>>>>>> >> >> >
>>>>>> >> >> > Any documentation links will be appreciated.
>>>>>> >> >> >
>>>>>> >> >> > Regards.
>>>>>> >> >> >
>>>>>> >> >> > Etienne Jouvin
>>>>>>
>>>>> --
> Sent from Gmail Mobile
>
Re: Ldap Cluster and Node Identity
Posted by Bryan Bende <bb...@gmail.com>.
The values have to match exactly, it’s case and white space sensitive.
There are identity transforms in Nifi properties that possibly help.
The value from where it says “Attempting request for...” is the value nifi
is getting from the certificate. You can also use key tool to list the
contents of the keystore jks and it will probably be the same as the log.
On Wed, Nov 25, 2020 at 1:30 PM Etienne Jouvin <la...@gmail.com>
wrote:
> Hello all.
>
> Still no luck.
> I follow steps from here :
> https://nifi.apache.org/docs/nifi-docs/html/walkthroughs.html#creating-and-securing-a-nifi-cluster-with-the-tls-toolkit
>
> Reminder, everything is currently in LDAP, I do not have any file
> user/group provider.
> When only one node, I can connect, this is correct.
>
> I can see the group having nodes "users" from the LDAP and the group has
> the proxy policy.
>
> But one thing strange, may be not the cause.
> In the LDAP, the DN is something like :
> cn=mig1.amexio.ch,ou=users,ou=nifi,dc=amexio,dc=ch
>
> And when I try to connect, I can see thoses errors :
> In nifi-user.log for the node where I connect :
> INFO [NiFi Web Server-22] o.a.n.w.s.NiFiAuthenticationFilter
> Authentication success for uid=mig-admin,ou=users,ou=nifi,dc=amexio,dc=ch
>
> In nifi-user.log on the second node :
> INFO [NiFi Web Server-29] o.a.n.w.s.NiFiAuthenticationFilter Attempting
> request for (<uid=mig-admin,ou=users,ou=nifi,dc=amexio,dc=ch><CN=
> mig1.assura.ch, OU=users, OU=nifi, DC=amexio, DC=ch>) GET
> https://mig2.amexio.ch:9442/nifi-api/flow/current-user (source ip:
> 127.0.0.1)
> WARN [NiFi Web Server-29] o.a.n.w.s.NiFiAuthenticationFilter Rejecting
> access to web api: Untrusted proxy CN=mig1.amexio.ch, OU=users, OU=nifi,
> DC=amexio, DC=ch
>
> The doubt is on the message, why the node DN has spaces between each level
> ?
> That does not match the value in the certificate and not the DN in LDAP.
>
> Maybe this is not the good route to investigate. Still searching. I have
> to find a way to "debug" the NiFiAuthenticationFilter and check the
> certificate comparison.
>
> Etienne
>
>
>
>
>
>
>
>
>
>
> Le mer. 25 nov. 2020 à 14:18, David Handermann <ex...@gmail.com>
> a écrit :
>
>> I am not as familiar with the LDAP user group provider, but based on the
>> "Untrusted proxy" message you are seeing, it sounds like the nodes are not
>> being identified properly as members of the "nodes" group from LDAP. Just
>> for testing purposes, you could try specifying the node distinguished names
>> in the "Node Identity N" properties of the access policy provider, using
>> "Node Identity 1", "Node Identity 2" and "Node Identity 3" to specify each
>> node DN. If that works, then it sounds like a configuration issue with the
>> Node Group, either on the LDAP server, or in the way NiFi is attempting to
>> query LDAP.
>>
>> Regards,
>> David Handermann
>>
>> On Wed, Nov 25, 2020 at 5:19 AM Etienne Jouvin <la...@gmail.com>
>> wrote:
>>
>>> Just for information, did not have time to test it from now.
>>> I was not able to get this Walk Throughs documentation.
>>> https://nifi.apache.org/docs/nifi-docs/html/walkthroughs.html
>>>
>>> Hope I will find the error I have about certificate (I have a little
>>> idea)
>>>
>>> Etienne
>>>
>>>
>>>
>>>
>>> Le mer. 25 nov. 2020 à 08:36, Etienne Jouvin <la...@gmail.com>
>>> a écrit :
>>>
>>>> Hello.
>>>>
>>>> I made some progress yesterday.
>>>> I did setup in LDAP groups and person
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Groups :
>>>> cn=administrators,ou=groups,ou=nifi,dc=amexio,dc=ch : for administrators
>>>> cn=supervisors,ou=groups,ou=nifi,dc=amexio,dc=ch : for supervisors
>>>> cn=nodes,ou=groups,ou=nifi,dc=amexio,dc=ch : group where I put all
>>>> "person" representing NiFi nodes.
>>>>
>>>> Users :
>>>> cn=migX.amexio.ch,ou=users,ou=nifi,dc=amexio,dc=ch : for each node,
>>>> replacing X by the index, and with object class person
>>>> uid=mig-admin,ou=users,ou=nifi,dc=amexio,dc=ch : example of "real" user
>>>> used to connect on the platform, with object class inetOrgperson
>>>>
>>>> In NiFi configuration.
>>>> I did activate a userGroupProvider linked to the LDAP
>>>> <userGroupProvider>
>>>> <identifier>amexio-ldap-user-group-provider</identifier>
>>>>
>>>> <class>org.apache.nifi.ldap.tenants.LdapUserGroupProvider</class>
>>>> <property name="Authentication Strategy">SIMPLE</property>
>>>>
>>>> <property name="Manager DN">uid=admin,ou=system</property>
>>>> <property name="Manager Password">secret</property>
>>>>
>>>> <property name="TLS - Keystore"></property>
>>>> <property name="TLS - Keystore Password"></property>
>>>> <property name="TLS - Keystore Type"></property>
>>>> <property name="TLS - Truststore"></property>
>>>> <property name="TLS - Truststore Password"></property>
>>>> <property name="TLS - Truststore Type"></property>
>>>> <property name="TLS - Client Auth"></property>
>>>> <property name="TLS - Protocol"></property>
>>>> <property name="TLS - Shutdown Gracefully"></property>
>>>>
>>>> <property name="Referral Strategy">FOLLOW</property>
>>>> <property name="Connect Timeout">10 secs</property>
>>>> <property name="Read Timeout">10 secs</property>
>>>>
>>>> <property name="Url">ldap://localhost:10389</property>
>>>> <property name="Page Size">50</property>
>>>> <!-- <property name="Sync Interval">30 mins</property> -->
>>>> <property name="Sync Interval">30 seconds</property>
>>>> <property name="Group Membership - Enforce Case
>>>> Sensitivity">false</property>
>>>>
>>>> <property name="User Search
>>>> Base">ou=users,ou=nifi,dc=amexio,dc=ch</property>
>>>> <property name="User Object Class">person</property>
>>>> <property name="User Search Scope">ONE_LEVEL</property>
>>>> <property name="User Search Filter"></property>
>>>> <property name="User Identity Attribute"></property>
>>>> <property name="User Group Name Attribute"></property>
>>>> <property name="User Group Name Attribute - Referenced Group
>>>> Attribute"></property>
>>>>
>>>> <property name="Group Search
>>>> Base">ou=groups,ou=nifi,dc=amexio,dc=ch</property>
>>>> <property name="Group Object Class">groupOfNames</property>
>>>> <property name="Group Search Scope">ONE_LEVEL</property>
>>>> <property name="Group Search Filter"></property>
>>>> <property name="Group Name Attribute">cn</property>
>>>> <property name="Group Member Attribute">member</property>
>>>> <property name="Group Member Attribute - Referenced User
>>>> Attribute"></property>
>>>> </userGroupProvider>
>>>>
>>>> Of course, register it inside the accessPolicyProvider
>>>> <accessPolicyProvider>
>>>> <identifier>file-access-policy-provider</identifier>
>>>>
>>>> <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
>>>> <!-- <property name="User Group
>>>> Provider">file-user-group-provider</property> -->
>>>> <property name="User Group
>>>> Provider">amexio-ldap-user-group-provider</property>
>>>> <property name="Authorizations
>>>> File">./conf/authorizations.xml</property>
>>>> <!-- <property name="Initial Admin Identity"></property> -->
>>>> <property name="Initial Admin
>>>> Identity">uid=mig-admin,ou=users,ou=nifi,dc=amexio,dc=ch</property>
>>>> <property name="Legacy Authorized Users File"></property>
>>>> <property name="Node Identity 1"></property>
>>>> <property name="Node Group">nodes</property>
>>>> </accessPolicyProvider>
>>>>
>>>> I am able to connect with the initial administrator account, when the
>>>> first node is started.
>>>> And all nodes are synchronized in the NiFi instance.
>>>>
>>>>
>>>>
>>>>
>>>> As soon as I start an additional node, I can not connect to the first
>>>> node
>>>> Erreur message
>>>> Untrusted proxy CN=mig1.amexio.ch, OU=users, OU=nifi, DC=amexio, DC=ch
>>>>
>>>> But I can connect on the second node.
>>>>
>>>>
>>>> So all this is about the certificate I guess.
>>>> for reminder, I use nls-toolkit to generate certificate on all nodes
>>>> with something like :
>>>> tls-toolkit.bat standalone -f
>>>> "C:\nifi-1.12.1\node1\conf\nifi.properties" -o
>>>> "C:\nifi-1.12.1\node1\conf\secured" -n mig1.amexio.ch --nifiDnPrefix
>>>> cn= --nifiDnSuffix ou=users,ou=nifi,dc=amexio,dc=ch
>>>>
>>>> Proxy is untrusted, ok fine. So may be I should not use the standalone
>>>> function of toolkit, but using server and client. In that case, I have to
>>>> stay alive the server from toolkit ?
>>>> Also, it seems I did not add certificate from node1 inside node2
>>>> trutstore, and node2 certificate inside node1 truststore ?
>>>> But in this case, if I have to add a new node, let's say node4, I would
>>>> have to push the certificate from node4 inside all existing nodes ?
>>>>
>>>> I continue to search, but any idea / input will be appreciated.
>>>>
>>>> Etienne
>>>>
>>>>
>>>> Le lun. 23 nov. 2020 à 18:39, Bryan Bende <bb...@gmail.com> a écrit :
>>>>
>>>>> Yes it will be the DN of the server's certificate which comes from the
>>>>> keystore.
>>>>>
>>>>> NiFi will get an incoming request, see that there is an X509 cert,
>>>>> take the DN and go to the user group provider and ask for the user
>>>>> with this identity.
>>>>>
>>>>> On Mon, Nov 23, 2020 at 12:01 PM Etienne Jouvin <
>>>>> lapinoujoujou@gmail.com> wrote:
>>>>> >
>>>>> > Hum OK,
>>>>> >
>>>>> > I will give it a try.
>>>>> > But one more thing...
>>>>> >
>>>>> > If I only set the group node;
>>>>> > How NiFi will connect the node with the nodeId in the LDAP ?
>>>>> > Where does it take the nodeid value ?
>>>>> > Is it the value we set in the keystore / truststore, by default
>>>>> cn=localhost, dc=NIFI (something like this) ?
>>>>> >
>>>>> > Etienne
>>>>> >
>>>>> >
>>>>> > Le lun. 23 nov. 2020 à 17:54, Bryan Bende <bb...@gmail.com> a
>>>>> écrit :
>>>>> >>
>>>>> >> I don't really know the LDAP specifics too well, so I'm not
>>>>> actually sure.
>>>>> >>
>>>>> >> You just need the nodes to come back from the LDAP UserGroupProvider
>>>>> >> as if they were regular users and members of some group "foo", which
>>>>> >> you then put "foo" into the "Node Group".
>>>>> >>
>>>>> >> On Mon, Nov 23, 2020 at 11:50 AM Etienne Jouvin <
>>>>> lapinoujoujou@gmail.com> wrote:
>>>>> >> >
>>>>> >> > Thanks Bryan.
>>>>> >> >
>>>>> >> > With your answer.... I will go to the Node Group and assign node
>>>>> identities.
>>>>> >> > Better for deployment and setup on the fly, I guess.
>>>>> >> >
>>>>> >> > One more point, you said "creating ldap entries for your nodes
>>>>> and assigning them group membership in ldap". What type of objectClass
>>>>> would you assign to the node in LDAP ?
>>>>> >> > This is not inetOrgPerson. The node should not have password.
>>>>> >> > If I create groupOfMembers for each node, is it correct ?
>>>>> >> >
>>>>> >> >
>>>>> >> > Thanks
>>>>> >> >
>>>>> >> > Etienne
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> > Le lun. 23 nov. 2020 à 17:27, Bryan Bende <bb...@gmail.com> a
>>>>> écrit :
>>>>> >> >>
>>>>> >> >> Hello,
>>>>> >> >>
>>>>> >> >> "Node Identity" is similar to the "Initial Admin" concept, in
>>>>> that it
>>>>> >> >> sets up the policies for the initial nodes to have permissions to
>>>>> >> >> proxy.
>>>>> >> >>
>>>>> >> >> If you are creating ldap entries for your nodes and assigning
>>>>> them
>>>>> >> >> group membership in ldap, then yes you could put that group name
>>>>> as
>>>>> >> >> the "Node Group" and then you don't need to specify the "Node
>>>>> >> >> Identities".
>>>>> >> >>
>>>>> >> >> If you are creating the node users in NiFi's file-based user
>>>>> group
>>>>> >> >> provider then you need to use node identities, and when adding a
>>>>> new
>>>>> >> >> node to the cluster you'd have to add the user first through the
>>>>> >> >> UI/REST API and grant it proxy, then actually connect it to the
>>>>> >> >> cluster.
>>>>> >> >>
>>>>> >> >> Thanks,
>>>>> >> >>
>>>>> >> >> Bryan
>>>>> >> >>
>>>>> >> >>
>>>>> >> >> On Mon, Nov 23, 2020 at 7:58 AM Etienne Jouvin <
>>>>> lapinoujoujou@gmail.com> wrote:
>>>>> >> >> >
>>>>> >> >> > Hello all.
>>>>> >> >> >
>>>>> >> >> >
>>>>> >> >> > I am currently setting up a NiFi, 1.12.1, Cluster with LDAP
>>>>> authentication.
>>>>> >> >> > For now the accessPolicyProvider is the default one with the
>>>>> configuration template :
>>>>> >> >> > <accessPolicyProvider>
>>>>> >> >> > <identifier>file-access-policy-provider</identifier>
>>>>> >> >> >
>>>>> <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
>>>>> >> >> > <property name="User Group
>>>>> Provider">file-user-group-provider</property>
>>>>> >> >> > <property name="Authorizations
>>>>> File">./conf/authorizations.xml</property>
>>>>> >> >> > <property name="Initial Admin Identity"></property>
>>>>> >> >> > <property name="Legacy Authorized Users
>>>>> File"></property>
>>>>> >> >> > <property name="Node Identity 1"></property>
>>>>> >> >> > <property name="Node Group"></property>
>>>>> >> >> > </accessPolicyProvider>
>>>>> >> >> >
>>>>> >> >> > But I do not really understand the purpose of the Node
>>>>> Identity X property.
>>>>> >> >> > If I well understood, all nodes should have the same
>>>>> configuration file, and I should register all nodes identity.
>>>>> >> >> >
>>>>> >> >> > But what about if I want to add a new node in the cluster on
>>>>> the fly ?
>>>>> >> >> > Should I register a new node identity, and then I should
>>>>> change all nodes configurations ?
>>>>> >> >> > The comment, in the configuration file, mentions the
>>>>> configuration Node Group, The name of a group containing NiFi cluster
>>>>> nodes. The typical use for this is when nodes are dynamically added/removed
>>>>> from the cluster.
>>>>> >> >> > Should I just put a Node group name and this will do the trick
>>>>> ?
>>>>> >> >> >
>>>>> >> >> > What should I put ? At the following link,
>>>>> https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.0.3/bk_administration/content/cluster-node-identities.html,
>>>>> it is said something like : cn=nifi-1,ou=people,dc=example,dc=com
>>>>> >> >> > In that case, what should be the obejct class for the node
>>>>> cn=nifi-1 in the LDAP ?
>>>>> >> >> >
>>>>> >> >> > Any documentation links will be appreciated.
>>>>> >> >> >
>>>>> >> >> > Regards.
>>>>> >> >> >
>>>>> >> >> > Etienne Jouvin
>>>>>
>>>> --
Sent from Gmail Mobile
Re: Ldap Cluster and Node Identity
Posted by Etienne Jouvin <la...@gmail.com>.
Hello all.
Still no luck.
I follow steps from here :
https://nifi.apache.org/docs/nifi-docs/html/walkthroughs.html#creating-and-securing-a-nifi-cluster-with-the-tls-toolkit
Reminder, everything is currently in LDAP, I do not have any file
user/group provider.
When only one node, I can connect, this is correct.
I can see the group having nodes "users" from the LDAP and the group has
the proxy policy.
But one thing strange, may be not the cause.
In the LDAP, the DN is something like :
cn=mig1.amexio.ch,ou=users,ou=nifi,dc=amexio,dc=ch
And when I try to connect, I can see thoses errors :
In nifi-user.log for the node where I connect :
INFO [NiFi Web Server-22] o.a.n.w.s.NiFiAuthenticationFilter Authentication
success for uid=mig-admin,ou=users,ou=nifi,dc=amexio,dc=ch
In nifi-user.log on the second node :
INFO [NiFi Web Server-29] o.a.n.w.s.NiFiAuthenticationFilter Attempting
request for (<uid=mig-admin,ou=users,ou=nifi,dc=amexio,dc=ch><CN=
mig1.assura.ch, OU=users, OU=nifi, DC=amexio, DC=ch>) GET
https://mig2.amexio.ch:9442/nifi-api/flow/current-user (source ip:
127.0.0.1)
WARN [NiFi Web Server-29] o.a.n.w.s.NiFiAuthenticationFilter Rejecting
access to web api: Untrusted proxy CN=mig1.amexio.ch, OU=users, OU=nifi,
DC=amexio, DC=ch
The doubt is on the message, why the node DN has spaces between each level ?
That does not match the value in the certificate and not the DN in LDAP.
Maybe this is not the good route to investigate. Still searching. I have to
find a way to "debug" the NiFiAuthenticationFilter and check the
certificate comparison.
Etienne
Le mer. 25 nov. 2020 à 14:18, David Handermann <ex...@gmail.com>
a écrit :
> I am not as familiar with the LDAP user group provider, but based on the
> "Untrusted proxy" message you are seeing, it sounds like the nodes are not
> being identified properly as members of the "nodes" group from LDAP. Just
> for testing purposes, you could try specifying the node distinguished names
> in the "Node Identity N" properties of the access policy provider, using
> "Node Identity 1", "Node Identity 2" and "Node Identity 3" to specify each
> node DN. If that works, then it sounds like a configuration issue with the
> Node Group, either on the LDAP server, or in the way NiFi is attempting to
> query LDAP.
>
> Regards,
> David Handermann
>
> On Wed, Nov 25, 2020 at 5:19 AM Etienne Jouvin <la...@gmail.com>
> wrote:
>
>> Just for information, did not have time to test it from now.
>> I was not able to get this Walk Throughs documentation.
>> https://nifi.apache.org/docs/nifi-docs/html/walkthroughs.html
>>
>> Hope I will find the error I have about certificate (I have a little idea)
>>
>> Etienne
>>
>>
>>
>>
>> Le mer. 25 nov. 2020 à 08:36, Etienne Jouvin <la...@gmail.com> a
>> écrit :
>>
>>> Hello.
>>>
>>> I made some progress yesterday.
>>> I did setup in LDAP groups and person
>>>
>>>
>>>
>>>
>>>
>>> Groups :
>>> cn=administrators,ou=groups,ou=nifi,dc=amexio,dc=ch : for administrators
>>> cn=supervisors,ou=groups,ou=nifi,dc=amexio,dc=ch : for supervisors
>>> cn=nodes,ou=groups,ou=nifi,dc=amexio,dc=ch : group where I put all
>>> "person" representing NiFi nodes.
>>>
>>> Users :
>>> cn=migX.amexio.ch,ou=users,ou=nifi,dc=amexio,dc=ch : for each node,
>>> replacing X by the index, and with object class person
>>> uid=mig-admin,ou=users,ou=nifi,dc=amexio,dc=ch : example of "real" user
>>> used to connect on the platform, with object class inetOrgperson
>>>
>>> In NiFi configuration.
>>> I did activate a userGroupProvider linked to the LDAP
>>> <userGroupProvider>
>>> <identifier>amexio-ldap-user-group-provider</identifier>
>>> <class>org.apache.nifi.ldap.tenants.LdapUserGroupProvider</class>
>>> <property name="Authentication Strategy">SIMPLE</property>
>>>
>>> <property name="Manager DN">uid=admin,ou=system</property>
>>> <property name="Manager Password">secret</property>
>>>
>>> <property name="TLS - Keystore"></property>
>>> <property name="TLS - Keystore Password"></property>
>>> <property name="TLS - Keystore Type"></property>
>>> <property name="TLS - Truststore"></property>
>>> <property name="TLS - Truststore Password"></property>
>>> <property name="TLS - Truststore Type"></property>
>>> <property name="TLS - Client Auth"></property>
>>> <property name="TLS - Protocol"></property>
>>> <property name="TLS - Shutdown Gracefully"></property>
>>>
>>> <property name="Referral Strategy">FOLLOW</property>
>>> <property name="Connect Timeout">10 secs</property>
>>> <property name="Read Timeout">10 secs</property>
>>>
>>> <property name="Url">ldap://localhost:10389</property>
>>> <property name="Page Size">50</property>
>>> <!-- <property name="Sync Interval">30 mins</property> -->
>>> <property name="Sync Interval">30 seconds</property>
>>> <property name="Group Membership - Enforce Case
>>> Sensitivity">false</property>
>>>
>>> <property name="User Search
>>> Base">ou=users,ou=nifi,dc=amexio,dc=ch</property>
>>> <property name="User Object Class">person</property>
>>> <property name="User Search Scope">ONE_LEVEL</property>
>>> <property name="User Search Filter"></property>
>>> <property name="User Identity Attribute"></property>
>>> <property name="User Group Name Attribute"></property>
>>> <property name="User Group Name Attribute - Referenced Group
>>> Attribute"></property>
>>>
>>> <property name="Group Search
>>> Base">ou=groups,ou=nifi,dc=amexio,dc=ch</property>
>>> <property name="Group Object Class">groupOfNames</property>
>>> <property name="Group Search Scope">ONE_LEVEL</property>
>>> <property name="Group Search Filter"></property>
>>> <property name="Group Name Attribute">cn</property>
>>> <property name="Group Member Attribute">member</property>
>>> <property name="Group Member Attribute - Referenced User
>>> Attribute"></property>
>>> </userGroupProvider>
>>>
>>> Of course, register it inside the accessPolicyProvider
>>> <accessPolicyProvider>
>>> <identifier>file-access-policy-provider</identifier>
>>>
>>> <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
>>> <!-- <property name="User Group
>>> Provider">file-user-group-provider</property> -->
>>> <property name="User Group
>>> Provider">amexio-ldap-user-group-provider</property>
>>> <property name="Authorizations
>>> File">./conf/authorizations.xml</property>
>>> <!-- <property name="Initial Admin Identity"></property> -->
>>> <property name="Initial Admin
>>> Identity">uid=mig-admin,ou=users,ou=nifi,dc=amexio,dc=ch</property>
>>> <property name="Legacy Authorized Users File"></property>
>>> <property name="Node Identity 1"></property>
>>> <property name="Node Group">nodes</property>
>>> </accessPolicyProvider>
>>>
>>> I am able to connect with the initial administrator account, when the
>>> first node is started.
>>> And all nodes are synchronized in the NiFi instance.
>>>
>>>
>>>
>>>
>>> As soon as I start an additional node, I can not connect to the first
>>> node
>>> Erreur message
>>> Untrusted proxy CN=mig1.amexio.ch, OU=users, OU=nifi, DC=amexio, DC=ch
>>>
>>> But I can connect on the second node.
>>>
>>>
>>> So all this is about the certificate I guess.
>>> for reminder, I use nls-toolkit to generate certificate on all nodes
>>> with something like :
>>> tls-toolkit.bat standalone -f
>>> "C:\nifi-1.12.1\node1\conf\nifi.properties" -o
>>> "C:\nifi-1.12.1\node1\conf\secured" -n mig1.amexio.ch --nifiDnPrefix
>>> cn= --nifiDnSuffix ou=users,ou=nifi,dc=amexio,dc=ch
>>>
>>> Proxy is untrusted, ok fine. So may be I should not use the standalone
>>> function of toolkit, but using server and client. In that case, I have to
>>> stay alive the server from toolkit ?
>>> Also, it seems I did not add certificate from node1 inside node2
>>> trutstore, and node2 certificate inside node1 truststore ?
>>> But in this case, if I have to add a new node, let's say node4, I would
>>> have to push the certificate from node4 inside all existing nodes ?
>>>
>>> I continue to search, but any idea / input will be appreciated.
>>>
>>> Etienne
>>>
>>>
>>> Le lun. 23 nov. 2020 à 18:39, Bryan Bende <bb...@gmail.com> a écrit :
>>>
>>>> Yes it will be the DN of the server's certificate which comes from the
>>>> keystore.
>>>>
>>>> NiFi will get an incoming request, see that there is an X509 cert,
>>>> take the DN and go to the user group provider and ask for the user
>>>> with this identity.
>>>>
>>>> On Mon, Nov 23, 2020 at 12:01 PM Etienne Jouvin <
>>>> lapinoujoujou@gmail.com> wrote:
>>>> >
>>>> > Hum OK,
>>>> >
>>>> > I will give it a try.
>>>> > But one more thing...
>>>> >
>>>> > If I only set the group node;
>>>> > How NiFi will connect the node with the nodeId in the LDAP ?
>>>> > Where does it take the nodeid value ?
>>>> > Is it the value we set in the keystore / truststore, by default
>>>> cn=localhost, dc=NIFI (something like this) ?
>>>> >
>>>> > Etienne
>>>> >
>>>> >
>>>> > Le lun. 23 nov. 2020 à 17:54, Bryan Bende <bb...@gmail.com> a écrit
>>>> :
>>>> >>
>>>> >> I don't really know the LDAP specifics too well, so I'm not actually
>>>> sure.
>>>> >>
>>>> >> You just need the nodes to come back from the LDAP UserGroupProvider
>>>> >> as if they were regular users and members of some group "foo", which
>>>> >> you then put "foo" into the "Node Group".
>>>> >>
>>>> >> On Mon, Nov 23, 2020 at 11:50 AM Etienne Jouvin <
>>>> lapinoujoujou@gmail.com> wrote:
>>>> >> >
>>>> >> > Thanks Bryan.
>>>> >> >
>>>> >> > With your answer.... I will go to the Node Group and assign node
>>>> identities.
>>>> >> > Better for deployment and setup on the fly, I guess.
>>>> >> >
>>>> >> > One more point, you said "creating ldap entries for your nodes and
>>>> assigning them group membership in ldap". What type of objectClass would
>>>> you assign to the node in LDAP ?
>>>> >> > This is not inetOrgPerson. The node should not have password.
>>>> >> > If I create groupOfMembers for each node, is it correct ?
>>>> >> >
>>>> >> >
>>>> >> > Thanks
>>>> >> >
>>>> >> > Etienne
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > Le lun. 23 nov. 2020 à 17:27, Bryan Bende <bb...@gmail.com> a
>>>> écrit :
>>>> >> >>
>>>> >> >> Hello,
>>>> >> >>
>>>> >> >> "Node Identity" is similar to the "Initial Admin" concept, in
>>>> that it
>>>> >> >> sets up the policies for the initial nodes to have permissions to
>>>> >> >> proxy.
>>>> >> >>
>>>> >> >> If you are creating ldap entries for your nodes and assigning them
>>>> >> >> group membership in ldap, then yes you could put that group name
>>>> as
>>>> >> >> the "Node Group" and then you don't need to specify the "Node
>>>> >> >> Identities".
>>>> >> >>
>>>> >> >> If you are creating the node users in NiFi's file-based user group
>>>> >> >> provider then you need to use node identities, and when adding a
>>>> new
>>>> >> >> node to the cluster you'd have to add the user first through the
>>>> >> >> UI/REST API and grant it proxy, then actually connect it to the
>>>> >> >> cluster.
>>>> >> >>
>>>> >> >> Thanks,
>>>> >> >>
>>>> >> >> Bryan
>>>> >> >>
>>>> >> >>
>>>> >> >> On Mon, Nov 23, 2020 at 7:58 AM Etienne Jouvin <
>>>> lapinoujoujou@gmail.com> wrote:
>>>> >> >> >
>>>> >> >> > Hello all.
>>>> >> >> >
>>>> >> >> >
>>>> >> >> > I am currently setting up a NiFi, 1.12.1, Cluster with LDAP
>>>> authentication.
>>>> >> >> > For now the accessPolicyProvider is the default one with the
>>>> configuration template :
>>>> >> >> > <accessPolicyProvider>
>>>> >> >> > <identifier>file-access-policy-provider</identifier>
>>>> >> >> >
>>>> <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
>>>> >> >> > <property name="User Group
>>>> Provider">file-user-group-provider</property>
>>>> >> >> > <property name="Authorizations
>>>> File">./conf/authorizations.xml</property>
>>>> >> >> > <property name="Initial Admin Identity"></property>
>>>> >> >> > <property name="Legacy Authorized Users
>>>> File"></property>
>>>> >> >> > <property name="Node Identity 1"></property>
>>>> >> >> > <property name="Node Group"></property>
>>>> >> >> > </accessPolicyProvider>
>>>> >> >> >
>>>> >> >> > But I do not really understand the purpose of the Node Identity
>>>> X property.
>>>> >> >> > If I well understood, all nodes should have the same
>>>> configuration file, and I should register all nodes identity.
>>>> >> >> >
>>>> >> >> > But what about if I want to add a new node in the cluster on
>>>> the fly ?
>>>> >> >> > Should I register a new node identity, and then I should change
>>>> all nodes configurations ?
>>>> >> >> > The comment, in the configuration file, mentions the
>>>> configuration Node Group, The name of a group containing NiFi cluster
>>>> nodes. The typical use for this is when nodes are dynamically added/removed
>>>> from the cluster.
>>>> >> >> > Should I just put a Node group name and this will do the trick ?
>>>> >> >> >
>>>> >> >> > What should I put ? At the following link,
>>>> https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.0.3/bk_administration/content/cluster-node-identities.html,
>>>> it is said something like : cn=nifi-1,ou=people,dc=example,dc=com
>>>> >> >> > In that case, what should be the obejct class for the node
>>>> cn=nifi-1 in the LDAP ?
>>>> >> >> >
>>>> >> >> > Any documentation links will be appreciated.
>>>> >> >> >
>>>> >> >> > Regards.
>>>> >> >> >
>>>> >> >> > Etienne Jouvin
>>>>
>>>
Re: Ldap Cluster and Node Identity
Posted by David Handermann <ex...@gmail.com>.
I am not as familiar with the LDAP user group provider, but based on the
"Untrusted proxy" message you are seeing, it sounds like the nodes are not
being identified properly as members of the "nodes" group from LDAP. Just
for testing purposes, you could try specifying the node distinguished names
in the "Node Identity N" properties of the access policy provider, using
"Node Identity 1", "Node Identity 2" and "Node Identity 3" to specify each
node DN. If that works, then it sounds like a configuration issue with the
Node Group, either on the LDAP server, or in the way NiFi is attempting to
query LDAP.
Regards,
David Handermann
On Wed, Nov 25, 2020 at 5:19 AM Etienne Jouvin <la...@gmail.com>
wrote:
> Just for information, did not have time to test it from now.
> I was not able to get this Walk Throughs documentation.
> https://nifi.apache.org/docs/nifi-docs/html/walkthroughs.html
>
> Hope I will find the error I have about certificate (I have a little idea)
>
> Etienne
>
>
>
>
> Le mer. 25 nov. 2020 à 08:36, Etienne Jouvin <la...@gmail.com> a
> écrit :
>
>> Hello.
>>
>> I made some progress yesterday.
>> I did setup in LDAP groups and person
>>
>>
>>
>>
>>
>> Groups :
>> cn=administrators,ou=groups,ou=nifi,dc=amexio,dc=ch : for administrators
>> cn=supervisors,ou=groups,ou=nifi,dc=amexio,dc=ch : for supervisors
>> cn=nodes,ou=groups,ou=nifi,dc=amexio,dc=ch : group where I put all
>> "person" representing NiFi nodes.
>>
>> Users :
>> cn=migX.amexio.ch,ou=users,ou=nifi,dc=amexio,dc=ch : for each node,
>> replacing X by the index, and with object class person
>> uid=mig-admin,ou=users,ou=nifi,dc=amexio,dc=ch : example of "real" user
>> used to connect on the platform, with object class inetOrgperson
>>
>> In NiFi configuration.
>> I did activate a userGroupProvider linked to the LDAP
>> <userGroupProvider>
>> <identifier>amexio-ldap-user-group-provider</identifier>
>> <class>org.apache.nifi.ldap.tenants.LdapUserGroupProvider</class>
>> <property name="Authentication Strategy">SIMPLE</property>
>>
>> <property name="Manager DN">uid=admin,ou=system</property>
>> <property name="Manager Password">secret</property>
>>
>> <property name="TLS - Keystore"></property>
>> <property name="TLS - Keystore Password"></property>
>> <property name="TLS - Keystore Type"></property>
>> <property name="TLS - Truststore"></property>
>> <property name="TLS - Truststore Password"></property>
>> <property name="TLS - Truststore Type"></property>
>> <property name="TLS - Client Auth"></property>
>> <property name="TLS - Protocol"></property>
>> <property name="TLS - Shutdown Gracefully"></property>
>>
>> <property name="Referral Strategy">FOLLOW</property>
>> <property name="Connect Timeout">10 secs</property>
>> <property name="Read Timeout">10 secs</property>
>>
>> <property name="Url">ldap://localhost:10389</property>
>> <property name="Page Size">50</property>
>> <!-- <property name="Sync Interval">30 mins</property> -->
>> <property name="Sync Interval">30 seconds</property>
>> <property name="Group Membership - Enforce Case
>> Sensitivity">false</property>
>>
>> <property name="User Search
>> Base">ou=users,ou=nifi,dc=amexio,dc=ch</property>
>> <property name="User Object Class">person</property>
>> <property name="User Search Scope">ONE_LEVEL</property>
>> <property name="User Search Filter"></property>
>> <property name="User Identity Attribute"></property>
>> <property name="User Group Name Attribute"></property>
>> <property name="User Group Name Attribute - Referenced Group
>> Attribute"></property>
>>
>> <property name="Group Search
>> Base">ou=groups,ou=nifi,dc=amexio,dc=ch</property>
>> <property name="Group Object Class">groupOfNames</property>
>> <property name="Group Search Scope">ONE_LEVEL</property>
>> <property name="Group Search Filter"></property>
>> <property name="Group Name Attribute">cn</property>
>> <property name="Group Member Attribute">member</property>
>> <property name="Group Member Attribute - Referenced User
>> Attribute"></property>
>> </userGroupProvider>
>>
>> Of course, register it inside the accessPolicyProvider
>> <accessPolicyProvider>
>> <identifier>file-access-policy-provider</identifier>
>>
>> <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
>> <!-- <property name="User Group
>> Provider">file-user-group-provider</property> -->
>> <property name="User Group
>> Provider">amexio-ldap-user-group-provider</property>
>> <property name="Authorizations
>> File">./conf/authorizations.xml</property>
>> <!-- <property name="Initial Admin Identity"></property> -->
>> <property name="Initial Admin
>> Identity">uid=mig-admin,ou=users,ou=nifi,dc=amexio,dc=ch</property>
>> <property name="Legacy Authorized Users File"></property>
>> <property name="Node Identity 1"></property>
>> <property name="Node Group">nodes</property>
>> </accessPolicyProvider>
>>
>> I am able to connect with the initial administrator account, when the
>> first node is started.
>> And all nodes are synchronized in the NiFi instance.
>>
>>
>>
>>
>> As soon as I start an additional node, I can not connect to the first node
>> Erreur message
>> Untrusted proxy CN=mig1.amexio.ch, OU=users, OU=nifi, DC=amexio, DC=ch
>>
>> But I can connect on the second node.
>>
>>
>> So all this is about the certificate I guess.
>> for reminder, I use nls-toolkit to generate certificate on all nodes with
>> something like :
>> tls-toolkit.bat standalone -f "C:\nifi-1.12.1\node1\conf\nifi.properties"
>> -o "C:\nifi-1.12.1\node1\conf\secured" -n mig1.amexio.ch --nifiDnPrefix
>> cn= --nifiDnSuffix ou=users,ou=nifi,dc=amexio,dc=ch
>>
>> Proxy is untrusted, ok fine. So may be I should not use the standalone
>> function of toolkit, but using server and client. In that case, I have to
>> stay alive the server from toolkit ?
>> Also, it seems I did not add certificate from node1 inside node2
>> trutstore, and node2 certificate inside node1 truststore ?
>> But in this case, if I have to add a new node, let's say node4, I would
>> have to push the certificate from node4 inside all existing nodes ?
>>
>> I continue to search, but any idea / input will be appreciated.
>>
>> Etienne
>>
>>
>> Le lun. 23 nov. 2020 à 18:39, Bryan Bende <bb...@gmail.com> a écrit :
>>
>>> Yes it will be the DN of the server's certificate which comes from the
>>> keystore.
>>>
>>> NiFi will get an incoming request, see that there is an X509 cert,
>>> take the DN and go to the user group provider and ask for the user
>>> with this identity.
>>>
>>> On Mon, Nov 23, 2020 at 12:01 PM Etienne Jouvin <la...@gmail.com>
>>> wrote:
>>> >
>>> > Hum OK,
>>> >
>>> > I will give it a try.
>>> > But one more thing...
>>> >
>>> > If I only set the group node;
>>> > How NiFi will connect the node with the nodeId in the LDAP ?
>>> > Where does it take the nodeid value ?
>>> > Is it the value we set in the keystore / truststore, by default
>>> cn=localhost, dc=NIFI (something like this) ?
>>> >
>>> > Etienne
>>> >
>>> >
>>> > Le lun. 23 nov. 2020 à 17:54, Bryan Bende <bb...@gmail.com> a écrit :
>>> >>
>>> >> I don't really know the LDAP specifics too well, so I'm not actually
>>> sure.
>>> >>
>>> >> You just need the nodes to come back from the LDAP UserGroupProvider
>>> >> as if they were regular users and members of some group "foo", which
>>> >> you then put "foo" into the "Node Group".
>>> >>
>>> >> On Mon, Nov 23, 2020 at 11:50 AM Etienne Jouvin <
>>> lapinoujoujou@gmail.com> wrote:
>>> >> >
>>> >> > Thanks Bryan.
>>> >> >
>>> >> > With your answer.... I will go to the Node Group and assign node
>>> identities.
>>> >> > Better for deployment and setup on the fly, I guess.
>>> >> >
>>> >> > One more point, you said "creating ldap entries for your nodes and
>>> assigning them group membership in ldap". What type of objectClass would
>>> you assign to the node in LDAP ?
>>> >> > This is not inetOrgPerson. The node should not have password.
>>> >> > If I create groupOfMembers for each node, is it correct ?
>>> >> >
>>> >> >
>>> >> > Thanks
>>> >> >
>>> >> > Etienne
>>> >> >
>>> >> >
>>> >> >
>>> >> > Le lun. 23 nov. 2020 à 17:27, Bryan Bende <bb...@gmail.com> a
>>> écrit :
>>> >> >>
>>> >> >> Hello,
>>> >> >>
>>> >> >> "Node Identity" is similar to the "Initial Admin" concept, in that
>>> it
>>> >> >> sets up the policies for the initial nodes to have permissions to
>>> >> >> proxy.
>>> >> >>
>>> >> >> If you are creating ldap entries for your nodes and assigning them
>>> >> >> group membership in ldap, then yes you could put that group name as
>>> >> >> the "Node Group" and then you don't need to specify the "Node
>>> >> >> Identities".
>>> >> >>
>>> >> >> If you are creating the node users in NiFi's file-based user group
>>> >> >> provider then you need to use node identities, and when adding a
>>> new
>>> >> >> node to the cluster you'd have to add the user first through the
>>> >> >> UI/REST API and grant it proxy, then actually connect it to the
>>> >> >> cluster.
>>> >> >>
>>> >> >> Thanks,
>>> >> >>
>>> >> >> Bryan
>>> >> >>
>>> >> >>
>>> >> >> On Mon, Nov 23, 2020 at 7:58 AM Etienne Jouvin <
>>> lapinoujoujou@gmail.com> wrote:
>>> >> >> >
>>> >> >> > Hello all.
>>> >> >> >
>>> >> >> >
>>> >> >> > I am currently setting up a NiFi, 1.12.1, Cluster with LDAP
>>> authentication.
>>> >> >> > For now the accessPolicyProvider is the default one with the
>>> configuration template :
>>> >> >> > <accessPolicyProvider>
>>> >> >> > <identifier>file-access-policy-provider</identifier>
>>> >> >> >
>>> <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
>>> >> >> > <property name="User Group
>>> Provider">file-user-group-provider</property>
>>> >> >> > <property name="Authorizations
>>> File">./conf/authorizations.xml</property>
>>> >> >> > <property name="Initial Admin Identity"></property>
>>> >> >> > <property name="Legacy Authorized Users File"></property>
>>> >> >> > <property name="Node Identity 1"></property>
>>> >> >> > <property name="Node Group"></property>
>>> >> >> > </accessPolicyProvider>
>>> >> >> >
>>> >> >> > But I do not really understand the purpose of the Node Identity
>>> X property.
>>> >> >> > If I well understood, all nodes should have the same
>>> configuration file, and I should register all nodes identity.
>>> >> >> >
>>> >> >> > But what about if I want to add a new node in the cluster on the
>>> fly ?
>>> >> >> > Should I register a new node identity, and then I should change
>>> all nodes configurations ?
>>> >> >> > The comment, in the configuration file, mentions the
>>> configuration Node Group, The name of a group containing NiFi cluster
>>> nodes. The typical use for this is when nodes are dynamically added/removed
>>> from the cluster.
>>> >> >> > Should I just put a Node group name and this will do the trick ?
>>> >> >> >
>>> >> >> > What should I put ? At the following link,
>>> https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.0.3/bk_administration/content/cluster-node-identities.html,
>>> it is said something like : cn=nifi-1,ou=people,dc=example,dc=com
>>> >> >> > In that case, what should be the obejct class for the node
>>> cn=nifi-1 in the LDAP ?
>>> >> >> >
>>> >> >> > Any documentation links will be appreciated.
>>> >> >> >
>>> >> >> > Regards.
>>> >> >> >
>>> >> >> > Etienne Jouvin
>>>
>>
Re: Ldap Cluster and Node Identity
Posted by Etienne Jouvin <la...@gmail.com>.
Just for information, did not have time to test it from now.
I was not able to get this Walk Throughs documentation.
https://nifi.apache.org/docs/nifi-docs/html/walkthroughs.html
Hope I will find the error I have about certificate (I have a little idea)
Etienne
Le mer. 25 nov. 2020 à 08:36, Etienne Jouvin <la...@gmail.com> a
écrit :
> Hello.
>
> I made some progress yesterday.
> I did setup in LDAP groups and person
>
>
>
>
>
> Groups :
> cn=administrators,ou=groups,ou=nifi,dc=amexio,dc=ch : for administrators
> cn=supervisors,ou=groups,ou=nifi,dc=amexio,dc=ch : for supervisors
> cn=nodes,ou=groups,ou=nifi,dc=amexio,dc=ch : group where I put all
> "person" representing NiFi nodes.
>
> Users :
> cn=migX.amexio.ch,ou=users,ou=nifi,dc=amexio,dc=ch : for each node,
> replacing X by the index, and with object class person
> uid=mig-admin,ou=users,ou=nifi,dc=amexio,dc=ch : example of "real" user
> used to connect on the platform, with object class inetOrgperson
>
> In NiFi configuration.
> I did activate a userGroupProvider linked to the LDAP
> <userGroupProvider>
> <identifier>amexio-ldap-user-group-provider</identifier>
> <class>org.apache.nifi.ldap.tenants.LdapUserGroupProvider</class>
> <property name="Authentication Strategy">SIMPLE</property>
>
> <property name="Manager DN">uid=admin,ou=system</property>
> <property name="Manager Password">secret</property>
>
> <property name="TLS - Keystore"></property>
> <property name="TLS - Keystore Password"></property>
> <property name="TLS - Keystore Type"></property>
> <property name="TLS - Truststore"></property>
> <property name="TLS - Truststore Password"></property>
> <property name="TLS - Truststore Type"></property>
> <property name="TLS - Client Auth"></property>
> <property name="TLS - Protocol"></property>
> <property name="TLS - Shutdown Gracefully"></property>
>
> <property name="Referral Strategy">FOLLOW</property>
> <property name="Connect Timeout">10 secs</property>
> <property name="Read Timeout">10 secs</property>
>
> <property name="Url">ldap://localhost:10389</property>
> <property name="Page Size">50</property>
> <!-- <property name="Sync Interval">30 mins</property> -->
> <property name="Sync Interval">30 seconds</property>
> <property name="Group Membership - Enforce Case
> Sensitivity">false</property>
>
> <property name="User Search
> Base">ou=users,ou=nifi,dc=amexio,dc=ch</property>
> <property name="User Object Class">person</property>
> <property name="User Search Scope">ONE_LEVEL</property>
> <property name="User Search Filter"></property>
> <property name="User Identity Attribute"></property>
> <property name="User Group Name Attribute"></property>
> <property name="User Group Name Attribute - Referenced Group
> Attribute"></property>
>
> <property name="Group Search
> Base">ou=groups,ou=nifi,dc=amexio,dc=ch</property>
> <property name="Group Object Class">groupOfNames</property>
> <property name="Group Search Scope">ONE_LEVEL</property>
> <property name="Group Search Filter"></property>
> <property name="Group Name Attribute">cn</property>
> <property name="Group Member Attribute">member</property>
> <property name="Group Member Attribute - Referenced User
> Attribute"></property>
> </userGroupProvider>
>
> Of course, register it inside the accessPolicyProvider
> <accessPolicyProvider>
> <identifier>file-access-policy-provider</identifier>
>
> <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
> <!-- <property name="User Group
> Provider">file-user-group-provider</property> -->
> <property name="User Group
> Provider">amexio-ldap-user-group-provider</property>
> <property name="Authorizations
> File">./conf/authorizations.xml</property>
> <!-- <property name="Initial Admin Identity"></property> -->
> <property name="Initial Admin
> Identity">uid=mig-admin,ou=users,ou=nifi,dc=amexio,dc=ch</property>
> <property name="Legacy Authorized Users File"></property>
> <property name="Node Identity 1"></property>
> <property name="Node Group">nodes</property>
> </accessPolicyProvider>
>
> I am able to connect with the initial administrator account, when the
> first node is started.
> And all nodes are synchronized in the NiFi instance.
>
>
>
>
> As soon as I start an additional node, I can not connect to the first node
> Erreur message
> Untrusted proxy CN=mig1.amexio.ch, OU=users, OU=nifi, DC=amexio, DC=ch
>
> But I can connect on the second node.
>
>
> So all this is about the certificate I guess.
> for reminder, I use nls-toolkit to generate certificate on all nodes with
> something like :
> tls-toolkit.bat standalone -f "C:\nifi-1.12.1\node1\conf\nifi.properties"
> -o "C:\nifi-1.12.1\node1\conf\secured" -n mig1.amexio.ch --nifiDnPrefix
> cn= --nifiDnSuffix ou=users,ou=nifi,dc=amexio,dc=ch
>
> Proxy is untrusted, ok fine. So may be I should not use the standalone
> function of toolkit, but using server and client. In that case, I have to
> stay alive the server from toolkit ?
> Also, it seems I did not add certificate from node1 inside node2
> trutstore, and node2 certificate inside node1 truststore ?
> But in this case, if I have to add a new node, let's say node4, I would
> have to push the certificate from node4 inside all existing nodes ?
>
> I continue to search, but any idea / input will be appreciated.
>
> Etienne
>
>
> Le lun. 23 nov. 2020 à 18:39, Bryan Bende <bb...@gmail.com> a écrit :
>
>> Yes it will be the DN of the server's certificate which comes from the
>> keystore.
>>
>> NiFi will get an incoming request, see that there is an X509 cert,
>> take the DN and go to the user group provider and ask for the user
>> with this identity.
>>
>> On Mon, Nov 23, 2020 at 12:01 PM Etienne Jouvin <la...@gmail.com>
>> wrote:
>> >
>> > Hum OK,
>> >
>> > I will give it a try.
>> > But one more thing...
>> >
>> > If I only set the group node;
>> > How NiFi will connect the node with the nodeId in the LDAP ?
>> > Where does it take the nodeid value ?
>> > Is it the value we set in the keystore / truststore, by default
>> cn=localhost, dc=NIFI (something like this) ?
>> >
>> > Etienne
>> >
>> >
>> > Le lun. 23 nov. 2020 à 17:54, Bryan Bende <bb...@gmail.com> a écrit :
>> >>
>> >> I don't really know the LDAP specifics too well, so I'm not actually
>> sure.
>> >>
>> >> You just need the nodes to come back from the LDAP UserGroupProvider
>> >> as if they were regular users and members of some group "foo", which
>> >> you then put "foo" into the "Node Group".
>> >>
>> >> On Mon, Nov 23, 2020 at 11:50 AM Etienne Jouvin <
>> lapinoujoujou@gmail.com> wrote:
>> >> >
>> >> > Thanks Bryan.
>> >> >
>> >> > With your answer.... I will go to the Node Group and assign node
>> identities.
>> >> > Better for deployment and setup on the fly, I guess.
>> >> >
>> >> > One more point, you said "creating ldap entries for your nodes and
>> assigning them group membership in ldap". What type of objectClass would
>> you assign to the node in LDAP ?
>> >> > This is not inetOrgPerson. The node should not have password.
>> >> > If I create groupOfMembers for each node, is it correct ?
>> >> >
>> >> >
>> >> > Thanks
>> >> >
>> >> > Etienne
>> >> >
>> >> >
>> >> >
>> >> > Le lun. 23 nov. 2020 à 17:27, Bryan Bende <bb...@gmail.com> a
>> écrit :
>> >> >>
>> >> >> Hello,
>> >> >>
>> >> >> "Node Identity" is similar to the "Initial Admin" concept, in that
>> it
>> >> >> sets up the policies for the initial nodes to have permissions to
>> >> >> proxy.
>> >> >>
>> >> >> If you are creating ldap entries for your nodes and assigning them
>> >> >> group membership in ldap, then yes you could put that group name as
>> >> >> the "Node Group" and then you don't need to specify the "Node
>> >> >> Identities".
>> >> >>
>> >> >> If you are creating the node users in NiFi's file-based user group
>> >> >> provider then you need to use node identities, and when adding a new
>> >> >> node to the cluster you'd have to add the user first through the
>> >> >> UI/REST API and grant it proxy, then actually connect it to the
>> >> >> cluster.
>> >> >>
>> >> >> Thanks,
>> >> >>
>> >> >> Bryan
>> >> >>
>> >> >>
>> >> >> On Mon, Nov 23, 2020 at 7:58 AM Etienne Jouvin <
>> lapinoujoujou@gmail.com> wrote:
>> >> >> >
>> >> >> > Hello all.
>> >> >> >
>> >> >> >
>> >> >> > I am currently setting up a NiFi, 1.12.1, Cluster with LDAP
>> authentication.
>> >> >> > For now the accessPolicyProvider is the default one with the
>> configuration template :
>> >> >> > <accessPolicyProvider>
>> >> >> > <identifier>file-access-policy-provider</identifier>
>> >> >> >
>> <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
>> >> >> > <property name="User Group
>> Provider">file-user-group-provider</property>
>> >> >> > <property name="Authorizations
>> File">./conf/authorizations.xml</property>
>> >> >> > <property name="Initial Admin Identity"></property>
>> >> >> > <property name="Legacy Authorized Users File"></property>
>> >> >> > <property name="Node Identity 1"></property>
>> >> >> > <property name="Node Group"></property>
>> >> >> > </accessPolicyProvider>
>> >> >> >
>> >> >> > But I do not really understand the purpose of the Node Identity X
>> property.
>> >> >> > If I well understood, all nodes should have the same
>> configuration file, and I should register all nodes identity.
>> >> >> >
>> >> >> > But what about if I want to add a new node in the cluster on the
>> fly ?
>> >> >> > Should I register a new node identity, and then I should change
>> all nodes configurations ?
>> >> >> > The comment, in the configuration file, mentions the
>> configuration Node Group, The name of a group containing NiFi cluster
>> nodes. The typical use for this is when nodes are dynamically added/removed
>> from the cluster.
>> >> >> > Should I just put a Node group name and this will do the trick ?
>> >> >> >
>> >> >> > What should I put ? At the following link,
>> https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.0.3/bk_administration/content/cluster-node-identities.html,
>> it is said something like : cn=nifi-1,ou=people,dc=example,dc=com
>> >> >> > In that case, what should be the obejct class for the node
>> cn=nifi-1 in the LDAP ?
>> >> >> >
>> >> >> > Any documentation links will be appreciated.
>> >> >> >
>> >> >> > Regards.
>> >> >> >
>> >> >> > Etienne Jouvin
>>
>
Re: Ldap Cluster and Node Identity
Posted by Etienne Jouvin <la...@gmail.com>.
Hello.
I made some progress yesterday.
I did setup in LDAP groups and person
Groups :
cn=administrators,ou=groups,ou=nifi,dc=amexio,dc=ch : for administrators
cn=supervisors,ou=groups,ou=nifi,dc=amexio,dc=ch : for supervisors
cn=nodes,ou=groups,ou=nifi,dc=amexio,dc=ch : group where I put all "person"
representing NiFi nodes.
Users :
cn=migX.amexio.ch,ou=users,ou=nifi,dc=amexio,dc=ch : for each node,
replacing X by the index, and with object class person
uid=mig-admin,ou=users,ou=nifi,dc=amexio,dc=ch : example of "real" user
used to connect on the platform, with object class inetOrgperson
In NiFi configuration.
I did activate a userGroupProvider linked to the LDAP
<userGroupProvider>
<identifier>amexio-ldap-user-group-provider</identifier>
<class>org.apache.nifi.ldap.tenants.LdapUserGroupProvider</class>
<property name="Authentication Strategy">SIMPLE</property>
<property name="Manager DN">uid=admin,ou=system</property>
<property name="Manager Password">secret</property>
<property name="TLS - Keystore"></property>
<property name="TLS - Keystore Password"></property>
<property name="TLS - Keystore Type"></property>
<property name="TLS - Truststore"></property>
<property name="TLS - Truststore Password"></property>
<property name="TLS - Truststore Type"></property>
<property name="TLS - Client Auth"></property>
<property name="TLS - Protocol"></property>
<property name="TLS - Shutdown Gracefully"></property>
<property name="Referral Strategy">FOLLOW</property>
<property name="Connect Timeout">10 secs</property>
<property name="Read Timeout">10 secs</property>
<property name="Url">ldap://localhost:10389</property>
<property name="Page Size">50</property>
<!-- <property name="Sync Interval">30 mins</property> -->
<property name="Sync Interval">30 seconds</property>
<property name="Group Membership - Enforce Case
Sensitivity">false</property>
<property name="User Search
Base">ou=users,ou=nifi,dc=amexio,dc=ch</property>
<property name="User Object Class">person</property>
<property name="User Search Scope">ONE_LEVEL</property>
<property name="User Search Filter"></property>
<property name="User Identity Attribute"></property>
<property name="User Group Name Attribute"></property>
<property name="User Group Name Attribute - Referenced Group
Attribute"></property>
<property name="Group Search
Base">ou=groups,ou=nifi,dc=amexio,dc=ch</property>
<property name="Group Object Class">groupOfNames</property>
<property name="Group Search Scope">ONE_LEVEL</property>
<property name="Group Search Filter"></property>
<property name="Group Name Attribute">cn</property>
<property name="Group Member Attribute">member</property>
<property name="Group Member Attribute - Referenced User
Attribute"></property>
</userGroupProvider>
Of course, register it inside the accessPolicyProvider
<accessPolicyProvider>
<identifier>file-access-policy-provider</identifier>
<class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
<!-- <property name="User Group
Provider">file-user-group-provider</property> -->
<property name="User Group
Provider">amexio-ldap-user-group-provider</property>
<property name="Authorizations
File">./conf/authorizations.xml</property>
<!-- <property name="Initial Admin Identity"></property> -->
<property name="Initial Admin
Identity">uid=mig-admin,ou=users,ou=nifi,dc=amexio,dc=ch</property>
<property name="Legacy Authorized Users File"></property>
<property name="Node Identity 1"></property>
<property name="Node Group">nodes</property>
</accessPolicyProvider>
I am able to connect with the initial administrator account, when the first
node is started.
And all nodes are synchronized in the NiFi instance.
As soon as I start an additional node, I can not connect to the first node
Erreur message
Untrusted proxy CN=mig1.amexio.ch, OU=users, OU=nifi, DC=amexio, DC=ch
But I can connect on the second node.
So all this is about the certificate I guess.
for reminder, I use nls-toolkit to generate certificate on all nodes with
something like :
tls-toolkit.bat standalone -f "C:\nifi-1.12.1\node1\conf\nifi.properties"
-o "C:\nifi-1.12.1\node1\conf\secured" -n mig1.amexio.ch --nifiDnPrefix cn=
--nifiDnSuffix ou=users,ou=nifi,dc=amexio,dc=ch
Proxy is untrusted, ok fine. So may be I should not use the standalone
function of toolkit, but using server and client. In that case, I have to
stay alive the server from toolkit ?
Also, it seems I did not add certificate from node1 inside node2 trutstore,
and node2 certificate inside node1 truststore ?
But in this case, if I have to add a new node, let's say node4, I would
have to push the certificate from node4 inside all existing nodes ?
I continue to search, but any idea / input will be appreciated.
Etienne
Le lun. 23 nov. 2020 à 18:39, Bryan Bende <bb...@gmail.com> a écrit :
> Yes it will be the DN of the server's certificate which comes from the
> keystore.
>
> NiFi will get an incoming request, see that there is an X509 cert,
> take the DN and go to the user group provider and ask for the user
> with this identity.
>
> On Mon, Nov 23, 2020 at 12:01 PM Etienne Jouvin <la...@gmail.com>
> wrote:
> >
> > Hum OK,
> >
> > I will give it a try.
> > But one more thing...
> >
> > If I only set the group node;
> > How NiFi will connect the node with the nodeId in the LDAP ?
> > Where does it take the nodeid value ?
> > Is it the value we set in the keystore / truststore, by default
> cn=localhost, dc=NIFI (something like this) ?
> >
> > Etienne
> >
> >
> > Le lun. 23 nov. 2020 à 17:54, Bryan Bende <bb...@gmail.com> a écrit :
> >>
> >> I don't really know the LDAP specifics too well, so I'm not actually
> sure.
> >>
> >> You just need the nodes to come back from the LDAP UserGroupProvider
> >> as if they were regular users and members of some group "foo", which
> >> you then put "foo" into the "Node Group".
> >>
> >> On Mon, Nov 23, 2020 at 11:50 AM Etienne Jouvin <
> lapinoujoujou@gmail.com> wrote:
> >> >
> >> > Thanks Bryan.
> >> >
> >> > With your answer.... I will go to the Node Group and assign node
> identities.
> >> > Better for deployment and setup on the fly, I guess.
> >> >
> >> > One more point, you said "creating ldap entries for your nodes and
> assigning them group membership in ldap". What type of objectClass would
> you assign to the node in LDAP ?
> >> > This is not inetOrgPerson. The node should not have password.
> >> > If I create groupOfMembers for each node, is it correct ?
> >> >
> >> >
> >> > Thanks
> >> >
> >> > Etienne
> >> >
> >> >
> >> >
> >> > Le lun. 23 nov. 2020 à 17:27, Bryan Bende <bb...@gmail.com> a écrit
> :
> >> >>
> >> >> Hello,
> >> >>
> >> >> "Node Identity" is similar to the "Initial Admin" concept, in that it
> >> >> sets up the policies for the initial nodes to have permissions to
> >> >> proxy.
> >> >>
> >> >> If you are creating ldap entries for your nodes and assigning them
> >> >> group membership in ldap, then yes you could put that group name as
> >> >> the "Node Group" and then you don't need to specify the "Node
> >> >> Identities".
> >> >>
> >> >> If you are creating the node users in NiFi's file-based user group
> >> >> provider then you need to use node identities, and when adding a new
> >> >> node to the cluster you'd have to add the user first through the
> >> >> UI/REST API and grant it proxy, then actually connect it to the
> >> >> cluster.
> >> >>
> >> >> Thanks,
> >> >>
> >> >> Bryan
> >> >>
> >> >>
> >> >> On Mon, Nov 23, 2020 at 7:58 AM Etienne Jouvin <
> lapinoujoujou@gmail.com> wrote:
> >> >> >
> >> >> > Hello all.
> >> >> >
> >> >> >
> >> >> > I am currently setting up a NiFi, 1.12.1, Cluster with LDAP
> authentication.
> >> >> > For now the accessPolicyProvider is the default one with the
> configuration template :
> >> >> > <accessPolicyProvider>
> >> >> > <identifier>file-access-policy-provider</identifier>
> >> >> >
> <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
> >> >> > <property name="User Group
> Provider">file-user-group-provider</property>
> >> >> > <property name="Authorizations
> File">./conf/authorizations.xml</property>
> >> >> > <property name="Initial Admin Identity"></property>
> >> >> > <property name="Legacy Authorized Users File"></property>
> >> >> > <property name="Node Identity 1"></property>
> >> >> > <property name="Node Group"></property>
> >> >> > </accessPolicyProvider>
> >> >> >
> >> >> > But I do not really understand the purpose of the Node Identity X
> property.
> >> >> > If I well understood, all nodes should have the same configuration
> file, and I should register all nodes identity.
> >> >> >
> >> >> > But what about if I want to add a new node in the cluster on the
> fly ?
> >> >> > Should I register a new node identity, and then I should change
> all nodes configurations ?
> >> >> > The comment, in the configuration file, mentions the configuration
> Node Group, The name of a group containing NiFi cluster nodes. The typical
> use for this is when nodes are dynamically added/removed from the cluster.
> >> >> > Should I just put a Node group name and this will do the trick ?
> >> >> >
> >> >> > What should I put ? At the following link,
> https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.0.3/bk_administration/content/cluster-node-identities.html,
> it is said something like : cn=nifi-1,ou=people,dc=example,dc=com
> >> >> > In that case, what should be the obejct class for the node
> cn=nifi-1 in the LDAP ?
> >> >> >
> >> >> > Any documentation links will be appreciated.
> >> >> >
> >> >> > Regards.
> >> >> >
> >> >> > Etienne Jouvin
>
Re: Ldap Cluster and Node Identity
Posted by Etienne Jouvin <la...@gmail.com>.
Ok fine.
I will try and response back here where everything works (or not, but I
will work)
I believe this part is not obvious and may be there is a lack in the
documentation ;)
Etienne
Le lun. 23 nov. 2020 à 18:39, Bryan Bende <bb...@gmail.com> a écrit :
> Yes it will be the DN of the server's certificate which comes from the
> keystore.
>
> NiFi will get an incoming request, see that there is an X509 cert,
> take the DN and go to the user group provider and ask for the user
> with this identity.
>
> On Mon, Nov 23, 2020 at 12:01 PM Etienne Jouvin <la...@gmail.com>
> wrote:
> >
> > Hum OK,
> >
> > I will give it a try.
> > But one more thing...
> >
> > If I only set the group node;
> > How NiFi will connect the node with the nodeId in the LDAP ?
> > Where does it take the nodeid value ?
> > Is it the value we set in the keystore / truststore, by default
> cn=localhost, dc=NIFI (something like this) ?
> >
> > Etienne
> >
> >
> > Le lun. 23 nov. 2020 à 17:54, Bryan Bende <bb...@gmail.com> a écrit :
> >>
> >> I don't really know the LDAP specifics too well, so I'm not actually
> sure.
> >>
> >> You just need the nodes to come back from the LDAP UserGroupProvider
> >> as if they were regular users and members of some group "foo", which
> >> you then put "foo" into the "Node Group".
> >>
> >> On Mon, Nov 23, 2020 at 11:50 AM Etienne Jouvin <
> lapinoujoujou@gmail.com> wrote:
> >> >
> >> > Thanks Bryan.
> >> >
> >> > With your answer.... I will go to the Node Group and assign node
> identities.
> >> > Better for deployment and setup on the fly, I guess.
> >> >
> >> > One more point, you said "creating ldap entries for your nodes and
> assigning them group membership in ldap". What type of objectClass would
> you assign to the node in LDAP ?
> >> > This is not inetOrgPerson. The node should not have password.
> >> > If I create groupOfMembers for each node, is it correct ?
> >> >
> >> >
> >> > Thanks
> >> >
> >> > Etienne
> >> >
> >> >
> >> >
> >> > Le lun. 23 nov. 2020 à 17:27, Bryan Bende <bb...@gmail.com> a écrit
> :
> >> >>
> >> >> Hello,
> >> >>
> >> >> "Node Identity" is similar to the "Initial Admin" concept, in that it
> >> >> sets up the policies for the initial nodes to have permissions to
> >> >> proxy.
> >> >>
> >> >> If you are creating ldap entries for your nodes and assigning them
> >> >> group membership in ldap, then yes you could put that group name as
> >> >> the "Node Group" and then you don't need to specify the "Node
> >> >> Identities".
> >> >>
> >> >> If you are creating the node users in NiFi's file-based user group
> >> >> provider then you need to use node identities, and when adding a new
> >> >> node to the cluster you'd have to add the user first through the
> >> >> UI/REST API and grant it proxy, then actually connect it to the
> >> >> cluster.
> >> >>
> >> >> Thanks,
> >> >>
> >> >> Bryan
> >> >>
> >> >>
> >> >> On Mon, Nov 23, 2020 at 7:58 AM Etienne Jouvin <
> lapinoujoujou@gmail.com> wrote:
> >> >> >
> >> >> > Hello all.
> >> >> >
> >> >> >
> >> >> > I am currently setting up a NiFi, 1.12.1, Cluster with LDAP
> authentication.
> >> >> > For now the accessPolicyProvider is the default one with the
> configuration template :
> >> >> > <accessPolicyProvider>
> >> >> > <identifier>file-access-policy-provider</identifier>
> >> >> >
> <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
> >> >> > <property name="User Group
> Provider">file-user-group-provider</property>
> >> >> > <property name="Authorizations
> File">./conf/authorizations.xml</property>
> >> >> > <property name="Initial Admin Identity"></property>
> >> >> > <property name="Legacy Authorized Users File"></property>
> >> >> > <property name="Node Identity 1"></property>
> >> >> > <property name="Node Group"></property>
> >> >> > </accessPolicyProvider>
> >> >> >
> >> >> > But I do not really understand the purpose of the Node Identity X
> property.
> >> >> > If I well understood, all nodes should have the same configuration
> file, and I should register all nodes identity.
> >> >> >
> >> >> > But what about if I want to add a new node in the cluster on the
> fly ?
> >> >> > Should I register a new node identity, and then I should change
> all nodes configurations ?
> >> >> > The comment, in the configuration file, mentions the configuration
> Node Group, The name of a group containing NiFi cluster nodes. The typical
> use for this is when nodes are dynamically added/removed from the cluster.
> >> >> > Should I just put a Node group name and this will do the trick ?
> >> >> >
> >> >> > What should I put ? At the following link,
> https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.0.3/bk_administration/content/cluster-node-identities.html,
> it is said something like : cn=nifi-1,ou=people,dc=example,dc=com
> >> >> > In that case, what should be the obejct class for the node
> cn=nifi-1 in the LDAP ?
> >> >> >
> >> >> > Any documentation links will be appreciated.
> >> >> >
> >> >> > Regards.
> >> >> >
> >> >> > Etienne Jouvin
>
Re: Ldap Cluster and Node Identity
Posted by Bryan Bende <bb...@gmail.com>.
Yes it will be the DN of the server's certificate which comes from the keystore.
NiFi will get an incoming request, see that there is an X509 cert,
take the DN and go to the user group provider and ask for the user
with this identity.
On Mon, Nov 23, 2020 at 12:01 PM Etienne Jouvin <la...@gmail.com> wrote:
>
> Hum OK,
>
> I will give it a try.
> But one more thing...
>
> If I only set the group node;
> How NiFi will connect the node with the nodeId in the LDAP ?
> Where does it take the nodeid value ?
> Is it the value we set in the keystore / truststore, by default cn=localhost, dc=NIFI (something like this) ?
>
> Etienne
>
>
> Le lun. 23 nov. 2020 à 17:54, Bryan Bende <bb...@gmail.com> a écrit :
>>
>> I don't really know the LDAP specifics too well, so I'm not actually sure.
>>
>> You just need the nodes to come back from the LDAP UserGroupProvider
>> as if they were regular users and members of some group "foo", which
>> you then put "foo" into the "Node Group".
>>
>> On Mon, Nov 23, 2020 at 11:50 AM Etienne Jouvin <la...@gmail.com> wrote:
>> >
>> > Thanks Bryan.
>> >
>> > With your answer.... I will go to the Node Group and assign node identities.
>> > Better for deployment and setup on the fly, I guess.
>> >
>> > One more point, you said "creating ldap entries for your nodes and assigning them group membership in ldap". What type of objectClass would you assign to the node in LDAP ?
>> > This is not inetOrgPerson. The node should not have password.
>> > If I create groupOfMembers for each node, is it correct ?
>> >
>> >
>> > Thanks
>> >
>> > Etienne
>> >
>> >
>> >
>> > Le lun. 23 nov. 2020 à 17:27, Bryan Bende <bb...@gmail.com> a écrit :
>> >>
>> >> Hello,
>> >>
>> >> "Node Identity" is similar to the "Initial Admin" concept, in that it
>> >> sets up the policies for the initial nodes to have permissions to
>> >> proxy.
>> >>
>> >> If you are creating ldap entries for your nodes and assigning them
>> >> group membership in ldap, then yes you could put that group name as
>> >> the "Node Group" and then you don't need to specify the "Node
>> >> Identities".
>> >>
>> >> If you are creating the node users in NiFi's file-based user group
>> >> provider then you need to use node identities, and when adding a new
>> >> node to the cluster you'd have to add the user first through the
>> >> UI/REST API and grant it proxy, then actually connect it to the
>> >> cluster.
>> >>
>> >> Thanks,
>> >>
>> >> Bryan
>> >>
>> >>
>> >> On Mon, Nov 23, 2020 at 7:58 AM Etienne Jouvin <la...@gmail.com> wrote:
>> >> >
>> >> > Hello all.
>> >> >
>> >> >
>> >> > I am currently setting up a NiFi, 1.12.1, Cluster with LDAP authentication.
>> >> > For now the accessPolicyProvider is the default one with the configuration template :
>> >> > <accessPolicyProvider>
>> >> > <identifier>file-access-policy-provider</identifier>
>> >> > <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
>> >> > <property name="User Group Provider">file-user-group-provider</property>
>> >> > <property name="Authorizations File">./conf/authorizations.xml</property>
>> >> > <property name="Initial Admin Identity"></property>
>> >> > <property name="Legacy Authorized Users File"></property>
>> >> > <property name="Node Identity 1"></property>
>> >> > <property name="Node Group"></property>
>> >> > </accessPolicyProvider>
>> >> >
>> >> > But I do not really understand the purpose of the Node Identity X property.
>> >> > If I well understood, all nodes should have the same configuration file, and I should register all nodes identity.
>> >> >
>> >> > But what about if I want to add a new node in the cluster on the fly ?
>> >> > Should I register a new node identity, and then I should change all nodes configurations ?
>> >> > The comment, in the configuration file, mentions the configuration Node Group, The name of a group containing NiFi cluster nodes. The typical use for this is when nodes are dynamically added/removed from the cluster.
>> >> > Should I just put a Node group name and this will do the trick ?
>> >> >
>> >> > What should I put ? At the following link, https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.0.3/bk_administration/content/cluster-node-identities.html, it is said something like : cn=nifi-1,ou=people,dc=example,dc=com
>> >> > In that case, what should be the obejct class for the node cn=nifi-1 in the LDAP ?
>> >> >
>> >> > Any documentation links will be appreciated.
>> >> >
>> >> > Regards.
>> >> >
>> >> > Etienne Jouvin
Re: Ldap Cluster and Node Identity
Posted by Etienne Jouvin <la...@gmail.com>.
Hum OK,
I will give it a try.
But one more thing...
If I only set the group node;
How NiFi will connect the node with the nodeId in the LDAP ?
Where does it take the nodeid value ?
Is it the value we set in the keystore / truststore, by default
cn=localhost, dc=NIFI (something like this) ?
Etienne
Le lun. 23 nov. 2020 à 17:54, Bryan Bende <bb...@gmail.com> a écrit :
> I don't really know the LDAP specifics too well, so I'm not actually sure.
>
> You just need the nodes to come back from the LDAP UserGroupProvider
> as if they were regular users and members of some group "foo", which
> you then put "foo" into the "Node Group".
>
> On Mon, Nov 23, 2020 at 11:50 AM Etienne Jouvin <la...@gmail.com>
> wrote:
> >
> > Thanks Bryan.
> >
> > With your answer.... I will go to the Node Group and assign node
> identities.
> > Better for deployment and setup on the fly, I guess.
> >
> > One more point, you said "creating ldap entries for your nodes and
> assigning them group membership in ldap". What type of objectClass would
> you assign to the node in LDAP ?
> > This is not inetOrgPerson. The node should not have password.
> > If I create groupOfMembers for each node, is it correct ?
> >
> >
> > Thanks
> >
> > Etienne
> >
> >
> >
> > Le lun. 23 nov. 2020 à 17:27, Bryan Bende <bb...@gmail.com> a écrit :
> >>
> >> Hello,
> >>
> >> "Node Identity" is similar to the "Initial Admin" concept, in that it
> >> sets up the policies for the initial nodes to have permissions to
> >> proxy.
> >>
> >> If you are creating ldap entries for your nodes and assigning them
> >> group membership in ldap, then yes you could put that group name as
> >> the "Node Group" and then you don't need to specify the "Node
> >> Identities".
> >>
> >> If you are creating the node users in NiFi's file-based user group
> >> provider then you need to use node identities, and when adding a new
> >> node to the cluster you'd have to add the user first through the
> >> UI/REST API and grant it proxy, then actually connect it to the
> >> cluster.
> >>
> >> Thanks,
> >>
> >> Bryan
> >>
> >>
> >> On Mon, Nov 23, 2020 at 7:58 AM Etienne Jouvin <la...@gmail.com>
> wrote:
> >> >
> >> > Hello all.
> >> >
> >> >
> >> > I am currently setting up a NiFi, 1.12.1, Cluster with LDAP
> authentication.
> >> > For now the accessPolicyProvider is the default one with the
> configuration template :
> >> > <accessPolicyProvider>
> >> > <identifier>file-access-policy-provider</identifier>
> >> >
> <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
> >> > <property name="User Group
> Provider">file-user-group-provider</property>
> >> > <property name="Authorizations
> File">./conf/authorizations.xml</property>
> >> > <property name="Initial Admin Identity"></property>
> >> > <property name="Legacy Authorized Users File"></property>
> >> > <property name="Node Identity 1"></property>
> >> > <property name="Node Group"></property>
> >> > </accessPolicyProvider>
> >> >
> >> > But I do not really understand the purpose of the Node Identity X
> property.
> >> > If I well understood, all nodes should have the same configuration
> file, and I should register all nodes identity.
> >> >
> >> > But what about if I want to add a new node in the cluster on the fly ?
> >> > Should I register a new node identity, and then I should change all
> nodes configurations ?
> >> > The comment, in the configuration file, mentions the configuration
> Node Group, The name of a group containing NiFi cluster nodes. The typical
> use for this is when nodes are dynamically added/removed from the cluster.
> >> > Should I just put a Node group name and this will do the trick ?
> >> >
> >> > What should I put ? At the following link,
> https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.0.3/bk_administration/content/cluster-node-identities.html,
> it is said something like : cn=nifi-1,ou=people,dc=example,dc=com
> >> > In that case, what should be the obejct class for the node cn=nifi-1
> in the LDAP ?
> >> >
> >> > Any documentation links will be appreciated.
> >> >
> >> > Regards.
> >> >
> >> > Etienne Jouvin
>
Re: Ldap Cluster and Node Identity
Posted by Bryan Bende <bb...@gmail.com>.
I don't really know the LDAP specifics too well, so I'm not actually sure.
You just need the nodes to come back from the LDAP UserGroupProvider
as if they were regular users and members of some group "foo", which
you then put "foo" into the "Node Group".
On Mon, Nov 23, 2020 at 11:50 AM Etienne Jouvin <la...@gmail.com> wrote:
>
> Thanks Bryan.
>
> With your answer.... I will go to the Node Group and assign node identities.
> Better for deployment and setup on the fly, I guess.
>
> One more point, you said "creating ldap entries for your nodes and assigning them group membership in ldap". What type of objectClass would you assign to the node in LDAP ?
> This is not inetOrgPerson. The node should not have password.
> If I create groupOfMembers for each node, is it correct ?
>
>
> Thanks
>
> Etienne
>
>
>
> Le lun. 23 nov. 2020 à 17:27, Bryan Bende <bb...@gmail.com> a écrit :
>>
>> Hello,
>>
>> "Node Identity" is similar to the "Initial Admin" concept, in that it
>> sets up the policies for the initial nodes to have permissions to
>> proxy.
>>
>> If you are creating ldap entries for your nodes and assigning them
>> group membership in ldap, then yes you could put that group name as
>> the "Node Group" and then you don't need to specify the "Node
>> Identities".
>>
>> If you are creating the node users in NiFi's file-based user group
>> provider then you need to use node identities, and when adding a new
>> node to the cluster you'd have to add the user first through the
>> UI/REST API and grant it proxy, then actually connect it to the
>> cluster.
>>
>> Thanks,
>>
>> Bryan
>>
>>
>> On Mon, Nov 23, 2020 at 7:58 AM Etienne Jouvin <la...@gmail.com> wrote:
>> >
>> > Hello all.
>> >
>> >
>> > I am currently setting up a NiFi, 1.12.1, Cluster with LDAP authentication.
>> > For now the accessPolicyProvider is the default one with the configuration template :
>> > <accessPolicyProvider>
>> > <identifier>file-access-policy-provider</identifier>
>> > <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
>> > <property name="User Group Provider">file-user-group-provider</property>
>> > <property name="Authorizations File">./conf/authorizations.xml</property>
>> > <property name="Initial Admin Identity"></property>
>> > <property name="Legacy Authorized Users File"></property>
>> > <property name="Node Identity 1"></property>
>> > <property name="Node Group"></property>
>> > </accessPolicyProvider>
>> >
>> > But I do not really understand the purpose of the Node Identity X property.
>> > If I well understood, all nodes should have the same configuration file, and I should register all nodes identity.
>> >
>> > But what about if I want to add a new node in the cluster on the fly ?
>> > Should I register a new node identity, and then I should change all nodes configurations ?
>> > The comment, in the configuration file, mentions the configuration Node Group, The name of a group containing NiFi cluster nodes. The typical use for this is when nodes are dynamically added/removed from the cluster.
>> > Should I just put a Node group name and this will do the trick ?
>> >
>> > What should I put ? At the following link, https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.0.3/bk_administration/content/cluster-node-identities.html, it is said something like : cn=nifi-1,ou=people,dc=example,dc=com
>> > In that case, what should be the obejct class for the node cn=nifi-1 in the LDAP ?
>> >
>> > Any documentation links will be appreciated.
>> >
>> > Regards.
>> >
>> > Etienne Jouvin
Re: Ldap Cluster and Node Identity
Posted by Etienne Jouvin <la...@gmail.com>.
Thanks Bryan.
With your answer.... I will go to the Node Group and assign node identities.
Better for deployment and setup on the fly, I guess.
One more point, you said "creating ldap entries for your nodes and
assigning them group membership in ldap". What type of objectClass would
you assign to the node in LDAP ?
This is not inetOrgPerson. The node should not have password.
If I create groupOfMembers for each node, is it correct ?
Thanks
Etienne
Le lun. 23 nov. 2020 à 17:27, Bryan Bende <bb...@gmail.com> a écrit :
> Hello,
>
> "Node Identity" is similar to the "Initial Admin" concept, in that it
> sets up the policies for the initial nodes to have permissions to
> proxy.
>
> If you are creating ldap entries for your nodes and assigning them
> group membership in ldap, then yes you could put that group name as
> the "Node Group" and then you don't need to specify the "Node
> Identities".
>
> If you are creating the node users in NiFi's file-based user group
> provider then you need to use node identities, and when adding a new
> node to the cluster you'd have to add the user first through the
> UI/REST API and grant it proxy, then actually connect it to the
> cluster.
>
> Thanks,
>
> Bryan
>
>
> On Mon, Nov 23, 2020 at 7:58 AM Etienne Jouvin <la...@gmail.com>
> wrote:
> >
> > Hello all.
> >
> >
> > I am currently setting up a NiFi, 1.12.1, Cluster with LDAP
> authentication.
> > For now the accessPolicyProvider is the default one with the
> configuration template :
> > <accessPolicyProvider>
> > <identifier>file-access-policy-provider</identifier>
> >
> <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
> > <property name="User Group
> Provider">file-user-group-provider</property>
> > <property name="Authorizations
> File">./conf/authorizations.xml</property>
> > <property name="Initial Admin Identity"></property>
> > <property name="Legacy Authorized Users File"></property>
> > <property name="Node Identity 1"></property>
> > <property name="Node Group"></property>
> > </accessPolicyProvider>
> >
> > But I do not really understand the purpose of the Node Identity X
> property.
> > If I well understood, all nodes should have the same configuration file,
> and I should register all nodes identity.
> >
> > But what about if I want to add a new node in the cluster on the fly ?
> > Should I register a new node identity, and then I should change all
> nodes configurations ?
> > The comment, in the configuration file, mentions the configuration Node
> Group, The name of a group containing NiFi cluster nodes. The typical use
> for this is when nodes are dynamically added/removed from the cluster.
> > Should I just put a Node group name and this will do the trick ?
> >
> > What should I put ? At the following link,
> https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.0.3/bk_administration/content/cluster-node-identities.html,
> it is said something like : cn=nifi-1,ou=people,dc=example,dc=com
> > In that case, what should be the obejct class for the node cn=nifi-1 in
> the LDAP ?
> >
> > Any documentation links will be appreciated.
> >
> > Regards.
> >
> > Etienne Jouvin
>
Re: Ldap Cluster and Node Identity
Posted by Bryan Bende <bb...@gmail.com>.
Hello,
"Node Identity" is similar to the "Initial Admin" concept, in that it
sets up the policies for the initial nodes to have permissions to
proxy.
If you are creating ldap entries for your nodes and assigning them
group membership in ldap, then yes you could put that group name as
the "Node Group" and then you don't need to specify the "Node
Identities".
If you are creating the node users in NiFi's file-based user group
provider then you need to use node identities, and when adding a new
node to the cluster you'd have to add the user first through the
UI/REST API and grant it proxy, then actually connect it to the
cluster.
Thanks,
Bryan
On Mon, Nov 23, 2020 at 7:58 AM Etienne Jouvin <la...@gmail.com> wrote:
>
> Hello all.
>
>
> I am currently setting up a NiFi, 1.12.1, Cluster with LDAP authentication.
> For now the accessPolicyProvider is the default one with the configuration template :
> <accessPolicyProvider>
> <identifier>file-access-policy-provider</identifier>
> <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
> <property name="User Group Provider">file-user-group-provider</property>
> <property name="Authorizations File">./conf/authorizations.xml</property>
> <property name="Initial Admin Identity"></property>
> <property name="Legacy Authorized Users File"></property>
> <property name="Node Identity 1"></property>
> <property name="Node Group"></property>
> </accessPolicyProvider>
>
> But I do not really understand the purpose of the Node Identity X property.
> If I well understood, all nodes should have the same configuration file, and I should register all nodes identity.
>
> But what about if I want to add a new node in the cluster on the fly ?
> Should I register a new node identity, and then I should change all nodes configurations ?
> The comment, in the configuration file, mentions the configuration Node Group, The name of a group containing NiFi cluster nodes. The typical use for this is when nodes are dynamically added/removed from the cluster.
> Should I just put a Node group name and this will do the trick ?
>
> What should I put ? At the following link, https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.0.3/bk_administration/content/cluster-node-identities.html, it is said something like : cn=nifi-1,ou=people,dc=example,dc=com
> In that case, what should be the obejct class for the node cn=nifi-1 in the LDAP ?
>
> Any documentation links will be appreciated.
>
> Regards.
>
> Etienne Jouvin