You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Scott Howell <sc...@mobilgov.com> on 2018/03/19 14:05:00 UTC
Issue with AWS ELB on secure nifi-registry
I was able to stand up nifi-registry behind an AWS ELB non-secure. Everything was working great and was able to access the UI anonymously. I set up the authorization just like on my nifi instances along with the authorizers and identity-provider. The service comes up without errors and everything looks good but the health check does not pass and I cannot access the UI to login. I was wondering if anyone else has ran into this issue using nifi-registry.
Re: Issue with AWS ELB on secure nifi-registry
Posted by Bryan Bende <bb...@gmail.com>.
I'm not sure you are even making it as far as authorization because I
think you would see unauthorized messages in the logs if that were the
case.
If you request the UI from your browser using the ELB URL, what page
is displayed? If you do the same thing using the direct URL to the
registry, is it any different?
Also, what values do you have set for nifi.registry.web.https.host=
and nifi.registry.security.needClientAuth= ?
On Mon, Mar 19, 2018 at 10:50 AM, Scott Howell <sc...@mobilgov.com> wrote:
> Thanks Kevin,
>
> I am just using the ELB to go from the public subnet to the private subnet. I will not have multiple instances running of registry.
>
> I will say on my authorizers.xml there is one difference between my nifi instance. On my nifi instance I am using file-provider for nifi.security.user.authorizer in my nifi.properties. I don’t think from reading the documents for nifi-registry that I can use that. If there is a way that might be my problem. I was running into some issues with my nifi instance when I was using managed-authorizers instead of file-provider.
>
>
>
>> On Mar 19, 2018, at 9:35 AM, Kevin Doran <kd...@apache.org> wrote:
>>
>> Hey Scott,
>>
>> Assuming you are using two-way TLS with client certificates for authentication, I recommend configuring your ELB for TCP passthrough so that the TLS handshake is between the end-client and the NiFi Registry Server (in other words, no decryption/termination of the TLS connection happens in the ELB). If you are using some other form of authentication (e.g., LDAP), you will need to configure your ELB to trust the self-signed key NiFi Registry is using. I'm not sure how to do that as I've never run an ELB with that configuration before.
>>
>> Also, just a note about using an ELB with NiFi Registry:
>>
>> NiFi Registry is currently only supports single-instance use as persisted data and in-memory state is not synced between multiple instances. Are you hoping to use the ELB for actual load balancing, or is it just to take advantage of other ELB features, such as forwarding and security group rules? If the plan is to load balance multiple Registry instances, just be aware that you will probably run into some unexpected behavior. (As you mentioned using authorization, that is one case where I know the in-memory cache of the persisted data will not refresh across instances, so even if you were using some sort of shared network file system attached to multiple Registry instances, such as EFS, it would not work the way you hope.)
>>
>> Hope this helps,
>> Kevin
>>
>> On 3/19/18, 10:20, "Scott Howell" <sc...@mobilgov.com> wrote:
>>
>> Thanks for the quick response.
>>
>> A couple of things I am seeing.
>>
>> 1. There is no error, I don’t see anything in the logs once the service comes up. This is because the health check is not even hitting the instance when secure.
>>
>> 2. Nothing interesting in the nifi-registry-app.logs. That was my concern because on my nifi instance I can see the health check hitting the instance from the ELB. This does not happen on the nifi-registry instance. I see the service startup and it tells me what domain and port I can access the UI but nothing else after that.
>>
>> 3. When I am on an instances in the same private subnet I am able to curl to the instance I get the TLS SSL which tells me the keystore is on the server. I am using a JKS keystore that is self-signed by the company I work for.
>>
>>> On Mar 19, 2018, at 9:10 AM, Bryan Bende <bb...@gmail.com> wrote:
>>>
>>> Hello,
>>>
>>> What error are you getting when you cannot access the UI?
>>>
>>> Is there anything interesting in nifi-registry-app.log regarding
>>> authentication/authorization when this happens?
>>>
>>> Can you access the UI securely without going through the ELB?
>>>
>>> Thanks,
>>>
>>> Bryan
>>>
>>>
>>> On Mon, Mar 19, 2018 at 10:05 AM, Scott Howell <sc...@mobilgov.com> wrote:
>>>> I was able to stand up nifi-registry behind an AWS ELB non-secure. Everything was working great and was able to access the UI anonymously. I set up the authorization just like on my nifi instances along with the authorizers and identity-provider. The service comes up without errors and everything looks good but the health check does not pass and I cannot access the UI to login. I was wondering if anyone else has ran into this issue using nifi-registry.
>>
>>
>>
>>
>
Re: Issue with AWS ELB on secure nifi-registry
Posted by Bryan Bende <bb...@gmail.com>.
Have you tried using the same 1 node config for the 2 node scenario?
I think since you have wildcard server certs, requests are going to
come from "CN=*.{redacted}.com, OU={redacted}, O={redacted}, L=Kansas
City, ST=Missouri, C=US" no matter which node the request comes from,
so there would be no way to know if the request was from
node1-nifi-dev or node2-nifi-dev.
So you really only have 1 server identity no matter how many nodes
your setup, and that identity would be just like you had in the 1 node
case.
I've never setup this scenario myself so I am mostly hypothesizing here.
On Wed, Mar 21, 2018 at 2:46 PM, Scott Howell <sc...@mobilgov.com> wrote:
> I do have a one node cluster working with the configuration below.
>
>
> This is the user.xml for my 2 node cluster
>
> <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
> <tenants>
> <groups/>
> <users>
> <user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53" identity="uid=scott,ou=users,dc=mobilgov,dc=com"/>
> <user identifier="727e6d3f-6e95-377b-b38f-d697163e2591" identity="CN=node1-nifi-dev.{redacted}.com, OU={redacted}, O={redacted}, L=Kansas City, ST=Missouri, C=US"/>
> <user identifier="1185a75e-586c-3a2c-b8df-93d0df3a4cb3" identity="CN=node2-nifi-dev.{redacted}.com, OU={redacted}, O={redacted}, L=Kansas City, ST=Missouri, C=US"/>
> </users>
> </tenants>
>
>
> Authorizations.xml
> ▽
> <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
> <authorizations>
> <policies>
> <policy identifier="f99bccd1-a30e-3e4a-98a2-dbc708edc67f" resource="/flow" action="R">
> <user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
> </policy>
> <policy identifier="b6df1162-ae29-3a55-ba31-36ce6ba674ea" resource="/data/process-groups/25d132eb-0162-1000-99a8-cc7e4dc0302d" action="R">
> <user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
> <user identifier="727e6d3f-6e95-377b-b38f-d697163e2591"/>
> <user identifier="1185a75e-586c-3a2c-b8df-93d0df3a4cb3"/>
> </policy>
> <policy identifier="46a35aa9-f909-3563-b73c-b5feac03cf6b" resource="/data/process-groups/25d132eb-0162-1000-99a8-cc7e4dc0302d" action="W">
> <user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
> <user identifier="727e6d3f-6e95-377b-b38f-d697163e2591"/>
> <user identifier="1185a75e-586c-3a2c-b8df-93d0df3a4cb3"/>
> </policy>
> <policy identifier="2037be0f-7a4d-3564-9230-cd338a255f03" resource="/process-groups/25d132eb-0162-1000-99a8-cc7e4dc0302d" action="R">
> <user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
> </policy>
> <policy identifier="f8327f78-a2fb-371b-abea-64079b60b938" resource="/process-groups/25d132eb-0162-1000-99a8-cc7e4dc0302d" action="W">
> <user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
> </policy>
> <policy identifier="b8775bd4-704a-34c6-987b-84f2daf7a515" resource="/restricted-components" action="W">
> <user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
> </policy>
> <policy identifier="627410be-1717-35b4-a06f-e9362b89e0b7" resource="/tenants" action="R">
> <user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
> </policy>
> <policy identifier="15e4e0bd-cb28-34fd-8587-f8d15162cba5" resource="/tenants" action="W">
> <user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
> </policy>
> <policy identifier="ff96062a-fa99-36dc-9942-0f6442ae7212" resource="/policies" action="R">
> <user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
> </policy>
> <policy identifier="ad99ea98-3af6-3561-ae27-5bf09e1d969d" resource="/policies" action="W">
> <user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
> </policy>
> <policy identifier="2e1015cb-0fed-3005-8e0d-722311f21a03" resource="/controller" action="R">
> <user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
> </policy>
> <policy identifier="c6322e6c-4cc1-3bcc-91b3-2ed2111674cf" resource="/controller" action="W">
> <user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
> </policy>
> <policy identifier="287edf48-da72-359b-8f61-da5d4c45a270" resource="/proxy" action="W">
> <user identifier="727e6d3f-6e95-377b-b38f-d697163e2591"/>
> <user identifier="1185a75e-586c-3a2c-b8df-93d0df3a4cb3"/>
> </policy>
> </policies>
> </authorizations>
>
> I get the untrusted proxy error when I have the nodes set like this.
>
>
>> On Mar 21, 2018, at 12:20 PM, Bryan Bende <bb...@gmail.com> wrote:
>>
>> Ok that looks correct for the 1-node case.
>>
>> So just to clarify what is working and not working...
>>
>> With the config in the last email, you have a 1 node cluster that is
>> working and you can get into the UI?
>>
>> For the two node case you would need each node to have a users.xml
>> with users for the two nodes, and an authorizations.xml with two
>> /proxy policies, one for each node.
>>
>> So if you do that then your cluster starts up, but when you access the
>> UI then you get the untrusted proxy?
>>
>>
>> On Wed, Mar 21, 2018 at 12:58 PM, Scott Howell <sc...@mobilgov.com> wrote:
>>> user.xml
>>>
>>> <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
>>> <tenants>
>>> <groups/>
>>> <users>
>>> <user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53" identity="uid=scott,ou=users,dc={redacted},dc=com"/>
>>> <user identifier="7f2b37ff-4d40-316a-843a-92c8523afc0b" identity="CN=*.{redacted}.com, OU={redacted}, O={redacted}, L=Kansas City, ST=Missouri, C=US"/>
>>> </users>
>>> </tenants>
>>>
>>> Authorizations.xml
>>> <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
>>> <authorizations>
>>> <policies>
>>> <policy identifier="f99bccd1-a30e-3e4a-98a2-dbc708edc67f" resource="/flow" action="R">
>>> <user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
>>> </policy>
>>> <policy identifier="ef96e849-629c-3f5e-97af-74efe29423bc" resource="/data/process-groups/4505628f-0162-1000-3b39-002cd06f74da" action="R">
>>> <user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
>>> <user identifier="7f2b37ff-4d40-316a-843a-92c8523afc0b"/>
>>> </policy>
>>> <policy identifier="fc29cd4c-ec37-3820-82b4-bbfd305b85ae" resource="/data/process-groups/4505628f-0162-1000-3b39-002cd06f74da" action="W">
>>> <user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
>>> <user identifier="7f2b37ff-4d40-316a-843a-92c8523afc0b"/>
>>> </policy>
>>> <policy identifier="75acccef-45ab-3b31-a49b-8cf88186c8bf" resource="/process-groups/4505628f-0162-1000-3b39-002cd06f74da" action="R">
>>> <user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
>>> </policy>
>>> <policy identifier="e9691c14-2540-3544-988a-654b79cf2370" resource="/process-groups/4505628f-0162-1000-3b39-002cd06f74da" action="W">
>>> <user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
>>> </policy>
>>> <policy identifier="b8775bd4-704a-34c6-987b-84f2daf7a515" resource="/restricted-components" action="W">
>>> <user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
>>> </policy>
>>> <policy identifier="627410be-1717-35b4-a06f-e9362b89e0b7" resource="/tenants" action="R">
>>> <user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
>>> </policy>
>>> <policy identifier="15e4e0bd-cb28-34fd-8587-f8d15162cba5" resource="/tenants" action="W">
>>> <user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
>>> </policy>
>>> <policy identifier="ff96062a-fa99-36dc-9942-0f6442ae7212" resource="/policies" action="R">
>>> <user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
>>> </policy>
>>> <policy identifier="ad99ea98-3af6-3561-ae27-5bf09e1d969d" resource="/policies" action="W">
>>> <user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
>>> </policy>
>>> <policy identifier="2e1015cb-0fed-3005-8e0d-722311f21a03" resource="/controller" action="R">
>>> <user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
>>> </policy>
>>> <policy identifier="c6322e6c-4cc1-3bcc-91b3-2ed2111674cf" resource="/controller" action="W">
>>> <user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
>>> </policy>
>>> <policy identifier="287edf48-da72-359b-8f61-da5d4c45a270" resource="/proxy" action="W">
>>> <user identifier="7f2b37ff-4d40-316a-843a-92c8523afc0b"/>
>>> </policy>
>>> </policies>
>>> </authorizations>
>>>
>>>> On Mar 21, 2018, at 11:49 AM, Bryan Bende <bb...@gmail.com> wrote:
>>>>
>>>> I've never used wildcard certs before so I'll have to defer to others
>>>> that might know if there is any issue with that.
>>>>
>>>> Could show the contents of these two files just so we can double check
>>>> the users/policies?
>>>>
>>>> <property name="Authorizations File">/opt/config/authorizations.xml</property>
>>>> <property name="Users File">/opt/config/users.xml</property>
>>>>
>>>> On Wed, Mar 21, 2018 at 12:37 PM, Scott Howell <sc...@mobilgov.com> wrote:
>>>>> Thanks I have checked that and the whitespace is correct in user.xml.
>>>>>
>>>>> I did make a change to my authorizer.xml
>>>>>
>>>>> <authorizers>
>>>>> <authorizer>
>>>>> <identifier>file-provider</identifier>
>>>>> <class>org.apache.nifi.authorization.FileAuthorizer</class>
>>>>> <property name="Authorizations File">/opt/config/authorizations.xml</property>
>>>>> <property name="Users File">/opt/config/users.xml</property>
>>>>> <property name="Initial Admin Identity">uid=scott,ou=users,dc=mobilgov,dc=com</property>
>>>>> <property name="Legacy Authorized Users File"></property>
>>>>>
>>>>> <property name="Node Identity 1">CN=*.{redacted}.com, OU={redacted}, O={redacted}, L=Kansas City, ST=Missouri, C=US</property>
>>>>> <property name="Node Identity 2">CN=*.{redacted}.com, OU={redacted}, O={redacted}, L=Kansas City, ST=Missouri, C=US</property>
>>>>> </authorizer>
>>>>> </authorizers>
>>>>>
>>>>> I made the CN=*.{redacted}.com just like the self-signed certificate showed. I now have a 1 node cluster up and running. It seems like NIfi isn’t taking into account the wildcard and treating it as a “*” instead.
>>>>>
>>>>>> On Mar 21, 2018, at 11:33 AM, Bryan Bende <bb...@gmail.com> wrote:
>>>>>>
>>>>>> There only needs to be W to /proxy so that part should be fine.
>>>>>>
>>>>>> After you edited the Node Identities, did you delete users.xml and
>>>>>> authorizations.xml?
>>>>>>
>>>>>> You would have to do that for those changes to take effect. You can
>>>>>> look in users.xml and see if you still have the user identities
>>>>>> without whitespace.
>>>>>>
>>>>>> On Wed, Mar 21, 2018 at 12:20 PM, Scott Howell <sc...@mobilgov.com> wrote:
>>>>>>> One other thing I am seeing and I don’t know if this is an issue or not in my authorizations.xml I do not have a policy for /proxy with action=“R” only action=“W”.
>>>>>>>
>>>>>>>> On Mar 21, 2018, at 11:03 AM, Scott Howell <sc...@mobilgov.com> wrote:
>>>>>>>>
>>>>>>>> Thanks for that. I am still getting this error in my nifi-user.log
>>>>>>>>
>>>>>>>> o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api: Untrusted proxy CN=*.{redacted}.com, OU={redacted}, O={redacted}, L=Kansas City, ST=Missouri, C=US
>>>>>>>>
>>>>>>>> Is there an issue with using a wildcard cert?
>>>>>>>>
>>>>>>>>
>>>>>>>>> On Mar 21, 2018, at 10:23 AM, Bryan Bende <bb...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>> All identity strings are case & whitespace sensitive.
>>>>>>>>>
>>>>>>>>> The node identities in your authorizers.xml have no whitespace, and
>>>>>>>>> the identity showing in the logs does.
>>>>>>>>>
>>>>>>>>> On Wed, Mar 21, 2018 at 11:05 AM, Scott Howell <sc...@mobilgov.com> wrote:
>>>>>>>>>> Thanks for all of the help with this. I have the cluster up and running. The
>>>>>>>>>> logs look great everything seems to be working but I cannot login into the
>>>>>>>>>> UI. I am using a wildcard self-signed certificate. The /proxy is in
>>>>>>>>>> authorizations.xml with the correct users for the nodes.
>>>>>>>>>>
>>>>>>>>>> The error I see with the UI :
>>>>>>>>>> is Untrusted proxy CN=*.{redacted}.com, OU={redacted}, O={redacted},
>>>>>>>>>> L=Kansas City, ST=Missouri, C=US
>>>>>>>>>>
>>>>>>>>>> I haven’t had much luck finding a lot of documentation or forum questions
>>>>>>>>>> with this kind of issue.
>>>>>>>>>>
>>>>>>>>>> My authorizers.xml looks like this
>>>>>>>>>> <authorizers>
>>>>>>>>>> <authorizer>
>>>>>>>>>> <identifier>file-provider</identifier>
>>>>>>>>>> <class>org.apache.nifi.authorization.FileAuthorizer</class>
>>>>>>>>>> <property name="Authorizations
>>>>>>>>>> File">/opt/config/authorizations.xml</property>
>>>>>>>>>> <property name="Users File">/opt/config/users.xml</property>
>>>>>>>>>> <property name="Initial Admin
>>>>>>>>>> Identity">uid=scott,ou=users,dc={redacted},dc=com</property>
>>>>>>>>>> <property name="Legacy Authorized Users File"></property>
>>>>>>>>>>
>>>>>>>>>> <property name="Node Identity
>>>>>>>>>> 1">CN=node-1-nifi-dev.{redacted}.com,OU={redacted},O={redacted},L=Kansas
>>>>>>>>>> City,ST=Missouri,C=US</property>
>>>>>>>>>> <property name="Node Identity
>>>>>>>>>> 2">CN=node-2-nifi-dev.{redacted}.com,OU={redacted},O={redacted},L=Kansas
>>>>>>>>>> City,ST=Missouri,C=US</property>
>>>>>>>>>> </authorizer>
>>>>>>>>>> </authorizers>
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>>
>>>>>>>>>> Scott
>>>>>>>>>>
>>>>>>>>>> On Mar 20, 2018, at 1:15 PM, Andy LoPresto <al...@apache.org> wrote:
>>>>>>>>>>
>>>>>>>>>> Scott,
>>>>>>>>>>
>>>>>>>>>> The original exception is "nested exception is
>>>>>>>>>> java.security.KeyStoreException: not found”. Can you verify that the
>>>>>>>>>> keystore you’ve provided is valid using the “keytool” command? In addition,
>>>>>>>>>> you will need a truststore as well. Try following Pierre's [1] or Bryan’s
>>>>>>>>>> [2] instructions for setting up a secure cluster.
>>>>>>>>>>
>>>>>>>>>> [1]
>>>>>>>>>> https://pierrevillard.com/2016/11/29/apache-nifi-1-1-0-secured-cluster-setup/
>>>>>>>>>> [2]
>>>>>>>>>> https://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-authorization-and-multi-tenancy
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Andy LoPresto
>>>>>>>>>> alopresto@apache.org
>>>>>>>>>> alopresto.apache@gmail.com
>>>>>>>>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>>>>>>>>>>
>>>>>>>>>> On Mar 20, 2018, at 11:05 AM, Scott Howell <sc...@mobilgov.com> wrote:
>>>>>>>>>>
>>>>>>>>>> Thanks for all of the help yesterday I was able to get a secure nifi and
>>>>>>>>>> nifi-registry up and communicating. I am now trying to figure out how to
>>>>>>>>>> create a secure cluster. I am currently getting this error when I start up
>>>>>>>>>> nifi.
>>>>>>>>>>
>>>>>>>>>> tion; nested exception is
>>>>>>>>>> org.springframework.beans.factory.BeanCreationException: Error creating bean
>>>>>>>>>> with name 'clusterCoordinationProtocolSenderListener' defined in class path
>>>>>>>>>> resource [nifi-cluster-protocol-context.xml]: Cannot resolve reference to
>>>>>>>>>> bean 'clusterCoordinationProtocolSender' while setting constructor argument;
>>>>>>>>>> nested exception is org.springframework.beans.factory.BeanCreationException:
>>>>>>>>>> Error creating bean with name 'clusterCoordinationProtocolSender' defined in
>>>>>>>>>> class path resource [nifi-cluster-protocol-context.xml]: Cannot resolve
>>>>>>>>>> reference to bean 'protocolSocketConfiguration' while setting constructor
>>>>>>>>>> argument; nested exception is
>>>>>>>>>> org.springframework.beans.factory.BeanCreationException: Error creating bean
>>>>>>>>>> with name 'protocolSocketConfiguration': FactoryBean threw exception on
>>>>>>>>>> object creation; nested exception is java.security.KeyStoreException: not
>>>>>>>>>> found
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175)
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1634)
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317)
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
>>>>>>>>>> ... 50 common frames omitted
>>>>>>>>>> Caused by: org.springframework.beans.factory.BeanCreationException: Error
>>>>>>>>>> creating bean with name 'clusterCoordinationProtocolSenderListener' defined
>>>>>>>>>> in class path resource [nifi-cluster-protocol-context.xml]: Cannot resolve
>>>>>>>>>> reference to bean 'clusterCoordinationProtocolSender' while setting
>>>>>>>>>> constructor argument; nested exception is
>>>>>>>>>> org.springframework.beans.factory.BeanCreationException: Error creating bean
>>>>>>>>>> with name 'clusterCoordinationProtocolSender' defined in class path resource
>>>>>>>>>> [nifi-cluster-protocol-context.xml]: Cannot resolve reference to bean
>>>>>>>>>> 'protocolSocketConfiguration' while setting constructor argument; nested
>>>>>>>>>> exception is org.springframework.beans.factory.BeanCreationException: Error
>>>>>>>>>> creating bean with name 'protocolSocketConfiguration': FactoryBean threw
>>>>>>>>>> exception on object creation; nested exception is
>>>>>>>>>> java.security.KeyStoreException: not found
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:359)
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:648)
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:145)
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1193)
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1095)
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
>>>>>>>>>> at
>>>>>>>>>> org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1084)
>>>>>>>>>> at
>>>>>>>>>> org.apache.nifi.cluster.spring.NodeClusterCoordinatorFactoryBean.getObject(NodeClusterCoordinatorFactoryBean.java:44)
>>>>>>>>>> at
>>>>>>>>>> org.apache.nifi.cluster.spring.NodeClusterCoordinatorFactoryBean.getObject(NodeClusterCoordinatorFactoryBean.java:34)
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
>>>>>>>>>> ... 55 common frames omitted
>>>>>>>>>> Caused by: org.springframework.beans.factory.BeanCreationException: Error
>>>>>>>>>> creating bean with name 'clusterCoordinationProtocolSender' defined in class
>>>>>>>>>> path resource [nifi-cluster-protocol-context.xml]: Cannot resolve reference
>>>>>>>>>> to bean 'protocolSocketConfiguration' while setting constructor argument;
>>>>>>>>>> nested exception is org.springframework.beans.factory.BeanCreationException:
>>>>>>>>>> Error creating bean with name 'protocolSocketConfiguration': FactoryBean
>>>>>>>>>> threw exception on object creation; nested exception is
>>>>>>>>>> java.security.KeyStoreException: not found
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:359)
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:648)
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:145)
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1193)
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1095)
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
>>>>>>>>>> ... 70 common frames omitted
>>>>>>>>>> Caused by: org.springframework.beans.factory.BeanCreationException: Error
>>>>>>>>>> creating bean with name 'protocolSocketConfiguration': FactoryBean threw
>>>>>>>>>> exception on object creation; nested exception is
>>>>>>>>>> java.security.KeyStoreException: not found
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175)
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1634)
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317)
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
>>>>>>>>>> ... 82 common frames omitted
>>>>>>>>>> Caused by: java.security.KeyStoreException: not found
>>>>>>>>>> at java.security.KeyStore.getInstance(KeyStore.java:851)
>>>>>>>>>> at
>>>>>>>>>> org.apache.nifi.security.util.KeyStoreUtils.getKeyStore(KeyStoreUtils.java:66)
>>>>>>>>>> at
>>>>>>>>>> org.apache.nifi.security.util.KeyStoreUtils.getTrustStore(KeyStoreUtils.java:80)
>>>>>>>>>> at
>>>>>>>>>> org.apache.nifi.io.socket.SSLContextFactory.<init>(SSLContextFactory.java:73)
>>>>>>>>>> at
>>>>>>>>>> org.apache.nifi.cluster.protocol.spring.SocketConfigurationFactoryBean.getObject(SocketConfigurationFactoryBean.java:45)
>>>>>>>>>> at
>>>>>>>>>> org.apache.nifi.cluster.protocol.spring.SocketConfigurationFactoryBean.getObject(SocketConfigurationFactoryBean.java:30)
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
>>>>>>>>>> ... 87 common frames omitted
>>>>>>>>>> Caused by: java.security.NoSuchAlgorithmException: KeyStore not available
>>>>>>>>>> at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
>>>>>>>>>> at java.security.Security.getImpl(Security.java:695)
>>>>>>>>>> at java.security.KeyStore.getInstance(KeyStore.java:848)
>>>>>>>>>> ... 93 common frames omitted
>>>>>>>>>>
>>>>>>>>>> My nifi.properties file is.
>>>>>>>>>>
>>>>>>>>>> # Licensed to the Apache Software Foundation (ASF) under one or more
>>>>>>>>>> # contributor license agreements. See the NOTICE file distributed with
>>>>>>>>>> # this work for additional information regarding copyright ownership.
>>>>>>>>>> # The ASF licenses this file to You under the Apache License, Version 2.0
>>>>>>>>>> # (the "License"); you may not use this file except in compliance with
>>>>>>>>>> # the License. You may obtain a copy of the License at
>>>>>>>>>> #
>>>>>>>>>> # http://www.apache.org/licenses/LICENSE-2.0
>>>>>>>>>> #
>>>>>>>>>> # Unless required by applicable law or agreed to in writing, software
>>>>>>>>>> # distributed under the License is distributed on an "AS IS" BASIS,
>>>>>>>>>> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>>>>>>>>>> # See the License for the specific language governing permissions and
>>>>>>>>>> # limitations under the License.
>>>>>>>>>>
>>>>>>>>>> # Core Properties #
>>>>>>>>>> nifi.version={{nifi_version}}
>>>>>>>>>> nifi.flow.configuration.file=/opt/config/flow.xml.gz
>>>>>>>>>> nifi.flow.configuration.archive.enabled=true
>>>>>>>>>> nifi.flow.configuration.archive.dir=/opt/config/archive/
>>>>>>>>>> nifi.flow.configuration.archive.max.time=30 days
>>>>>>>>>> nifi.flow.configuration.archive.max.storage=500 MB
>>>>>>>>>> nifi.flowcontroller.autoResumeState=true
>>>>>>>>>> nifi.flowcontroller.graceful.shutdown.period=10 sec
>>>>>>>>>> nifi.flowservice.writedelay.interval=500 ms
>>>>>>>>>> nifi.administrative.yield.duration=30 sec
>>>>>>>>>> # If a component has no work to do (is "bored"), how long should we wait
>>>>>>>>>> before checking again for work?
>>>>>>>>>> nifi.bored.yield.duration=10 millis
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> nifi.authorizer.configuration.file=/opt/config/authorizers.xml
>>>>>>>>>> nifi.login.identity.provider.configuration.file=/opt/config/login-identity-providers.xml
>>>>>>>>>> nifi.templates.directory=/opt/config/templates
>>>>>>>>>> nifi.ui.banner.text=
>>>>>>>>>> nifi.ui.autorefresh.interval=30 sec
>>>>>>>>>> nifi.nar.library.directory=/opt/nifi/lib
>>>>>>>>>> nifi.nar.library.directory.custom=/opt/config/processors
>>>>>>>>>> nifi.nar.working.directory=/opt/nifi/work/nar/
>>>>>>>>>> nifi.documentation.working.directory=./work/docs/components
>>>>>>>>>>
>>>>>>>>>> ####################
>>>>>>>>>> # State Management #
>>>>>>>>>> ####################
>>>>>>>>>> nifi.state.management.configuration.file=/opt/config/state-management.xml
>>>>>>>>>> # The ID of the local state provider
>>>>>>>>>> nifi.state.management.provider.local=local-provider
>>>>>>>>>> # The ID of the cluster-wide state provider. This will be ignored if NiFi is
>>>>>>>>>> not clustered but must be populated if running in a cluster.
>>>>>>>>>> nifi.state.management.provider.cluster=zk-provider
>>>>>>>>>> # Specifies whether or not this instance of NiFi should run an embedded
>>>>>>>>>> ZooKeeper server
>>>>>>>>>> nifi.state.management.embedded.zookeeper.start=false
>>>>>>>>>> # Properties file that provides the ZooKeeper properties to use if
>>>>>>>>>> <nifi.state.management.embedded.zookeeper.start> is set to true
>>>>>>>>>> nifi.state.management.embedded.zookeeper.properties=/opt/config/zookeeper.properties
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> # H2 Settings
>>>>>>>>>> nifi.database.directory=/opt/database_repository
>>>>>>>>>> nifi.h2.url.append=;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE
>>>>>>>>>>
>>>>>>>>>> # FlowFile Repository
>>>>>>>>>> nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository
>>>>>>>>>> nifi.flowfile.repository.directory=/opt/flowfile_repository
>>>>>>>>>> nifi.flowfile.repository.partitions=256
>>>>>>>>>> nifi.flowfile.repository.checkpoint.interval=2 mins
>>>>>>>>>> nifi.flowfile.repository.always.sync=false
>>>>>>>>>>
>>>>>>>>>> nifi.swap.manager.implementation=org.apache.nifi.controller.FileSystemSwapManager
>>>>>>>>>> nifi.queue.swap.threshold=20000
>>>>>>>>>> nifi.swap.in.period=5 sec
>>>>>>>>>> nifi.swap.in.threads=1
>>>>>>>>>> nifi.swap.out.period=5 sec
>>>>>>>>>> nifi.swap.out.threads=4
>>>>>>>>>>
>>>>>>>>>> # Content Repository
>>>>>>>>>> nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository
>>>>>>>>>> nifi.content.claim.max.appendable.size=10 MB
>>>>>>>>>> nifi.content.claim.max.flow.files=100
>>>>>>>>>> nifi.content.repository.directory.default=/opt/content_repository
>>>>>>>>>> nifi.content.repository.archive.max.retention.period=12 hours
>>>>>>>>>> nifi.content.repository.archive.max.usage.percentage=50%
>>>>>>>>>> nifi.content.repository.archive.enabled=true
>>>>>>>>>> nifi.content.repository.always.sync=false
>>>>>>>>>> nifi.content.viewer.url=/nifi-content-viewer/
>>>>>>>>>>
>>>>>>>>>> # Provenance Repository Properties
>>>>>>>>>> nifi.provenance.repository.implementation=org.apache.nifi.provenance.PersistentProvenanceRepository
>>>>>>>>>>
>>>>>>>>>> # Persistent Provenance Repository Properties
>>>>>>>>>> nifi.provenance.repository.directory.default=/opt/provenance_repository
>>>>>>>>>> nifi.provenance.repository.max.storage.time=24 hours
>>>>>>>>>> nifi.provenance.repository.max.storage.size=1 GB
>>>>>>>>>> nifi.provenance.repository.rollover.time=30 secs
>>>>>>>>>> nifi.provenance.repository.rollover.size=100 MB
>>>>>>>>>> nifi.provenance.repository.query.threads=2
>>>>>>>>>> nifi.provenance.repository.index.threads=1
>>>>>>>>>> nifi.provenance.repository.compress.on.rollover=true
>>>>>>>>>> nifi.provenance.repository.always.sync=false
>>>>>>>>>> nifi.provenance.repository.journal.count=16
>>>>>>>>>> # Comma-separated list of fields. Fields that are not indexed will not be
>>>>>>>>>> searchable. Valid fields are:
>>>>>>>>>> # EventType, FlowFileUUID, Filename, TransitURI, ProcessorID,
>>>>>>>>>> AlternateIdentifierURI, Relationship, Details
>>>>>>>>>> nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename,
>>>>>>>>>> ProcessorID, Relationship
>>>>>>>>>> # FlowFile Attributes that should be indexed and made searchable. Some
>>>>>>>>>> examples to consider are filename, uuid, mime.type
>>>>>>>>>> nifi.provenance.repository.indexed.attributes=
>>>>>>>>>> # Large values for the shard size will result in more Java heap usage when
>>>>>>>>>> searching the Provenance Repository
>>>>>>>>>> # but should provide better performance
>>>>>>>>>> nifi.provenance.repository.index.shard.size=500 MB
>>>>>>>>>> # Indicates the maximum length that a FlowFile attribute can be when
>>>>>>>>>> retrieving a Provenance Event from
>>>>>>>>>> # the repository. If the length of any attribute exceeds this value, it will
>>>>>>>>>> be truncated when the event is retrieved.
>>>>>>>>>> nifi.provenance.repository.max.attribute.length=65536
>>>>>>>>>>
>>>>>>>>>> # Volatile Provenance Respository Properties
>>>>>>>>>> nifi.provenance.repository.buffer.size=100000
>>>>>>>>>>
>>>>>>>>>> # Component Status Repository
>>>>>>>>>> nifi.components.status.repository.implementation=org.apache.nifi.controller.status.history.VolatileComponentStatusRepository
>>>>>>>>>> nifi.components.status.repository.buffer.size=1440
>>>>>>>>>> nifi.components.status.snapshot.frequency=1 min
>>>>>>>>>>
>>>>>>>>>> # Site to Site properties
>>>>>>>>>> nifi.remote.input.host=
>>>>>>>>>> nifi.remote.input.secure=false
>>>>>>>>>> nifi.remote.input.socket.port=9998
>>>>>>>>>> nifi.remote.input.http.enabled=false
>>>>>>>>>> nifi.remote.input.http.transaction.ttl=30 sec
>>>>>>>>>>
>>>>>>>>>> # web properties #
>>>>>>>>>> nifi.web.war.directory=/opt/nifi/lib
>>>>>>>>>> nifi.web.http.host=
>>>>>>>>>> nifi.web.http.port=
>>>>>>>>>> nifi.web.https.host={{redacted}}
>>>>>>>>>> nifi.web.https.port=8443
>>>>>>>>>> nifi.web.jetty.working.directory=/opt/nifi/work/jetty
>>>>>>>>>> nifi.web.jetty.threads=200
>>>>>>>>>>
>>>>>>>>>> # security properties #
>>>>>>>>>> nifi.sensitive.props.key=x0KDgO9L8lAhFGLdvu2VEjFVGc6Kg3V0R5I4bYwoqdgL47moo0wApKQtAVu1BvD
>>>>>>>>>> nifi.sensitive.props.key.protected=
>>>>>>>>>> nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
>>>>>>>>>> nifi.sensitive.props.provider=BC
>>>>>>>>>> nifi.sensitive.props.additional.keys=
>>>>>>>>>>
>>>>>>>>>> nifi.security.keystore=/opt/certs/payit_keystore
>>>>>>>>>> nifi.security.keystoreType=JKS
>>>>>>>>>> nifi.security.keystorePasswd={{keystore_password}}
>>>>>>>>>> nifi.security.keyPasswd=
>>>>>>>>>> nifi.security.truststore=
>>>>>>>>>> nifi.security.truststoreType=
>>>>>>>>>> nifi.security.truststorePasswd=
>>>>>>>>>> nifi.security.needClientAuth=false
>>>>>>>>>> nifi.security.user.authorizer=file-provider
>>>>>>>>>> nifi.security.user.login.identity.provider=ldap-provider
>>>>>>>>>> nifi.security.ocsp.responder.url=
>>>>>>>>>> nifi.security.ocsp.responder.certificate=
>>>>>>>>>>
>>>>>>>>>> # Identity Mapping Properties #
>>>>>>>>>> # These properties allow normalizing user identities such that identities
>>>>>>>>>> coming from different identity providers
>>>>>>>>>> # (certificates, LDAP, Kerberos) can be treated the same internally in NiFi.
>>>>>>>>>> The following example demonstrates normalizing
>>>>>>>>>> # DNs from certificates and principals from Kerberos into a common identity
>>>>>>>>>> string:
>>>>>>>>>> #
>>>>>>>>>> #nifi.security.identity.mapping.pattern.dn=^cn=(.*?),ou=(.*?),dc=(.*?),dc=(.*?),dc=(.*?)$
>>>>>>>>>> #nifi.security.identity.mapping.value.dn=$1
>>>>>>>>>> # nifi.security.identity.mapping.pattern.kerb=^(.*?)/instance@(.*?)$
>>>>>>>>>> # nifi.security.identity.mapping.value.kerb=$1@$2
>>>>>>>>>>
>>>>>>>>>> # cluster common properties (all nodes must have same values) #
>>>>>>>>>> nifi.cluster.protocol.heartbeat.interval=5 sec
>>>>>>>>>> nifi.cluster.protocol.is.secure=true
>>>>>>>>>>
>>>>>>>>>> # cluster node properties (only configure for cluster nodes) #
>>>>>>>>>> nifi.cluster.is.node=true
>>>>>>>>>> nifi.cluster.node.address=nifi-dev.mobilgov.com
>>>>>>>>>> nifi.cluster.node.protocol.port=9999
>>>>>>>>>> nifi.cluster.node.protocol.threads=10
>>>>>>>>>> nifi.cluster.node.event.history.size=25
>>>>>>>>>> nifi.cluster.node.connection.timeout=5 sec
>>>>>>>>>> nifi.cluster.node.read.timeout=5 sec
>>>>>>>>>> nifi.cluster.firewall.file=
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> # zookeeper properties, used for cluster management #
>>>>>>>>>> nifi.zookeeper.connect.string=internal-etcd-dev-EtcdLoadB-3RWA2WEFBBT8-2068560477.us-east-2.elb.amazonaws.com:2181,internal-etcd-dev-EtcdLoadB-3RWA2WEFBBT8-2068560477.us-east-2.elb.amazonaws.com2182,internal-etcd-dev-EtcdLoadB-3RWA2WEFBBT8-2068560477.us-east-2.elb.amazonaws.com:2183
>>>>>>>>>> nifi.zookeeper.connect.timeout=3 secs
>>>>>>>>>> nifi.zookeeper.session.timeout=3 secs
>>>>>>>>>> nifi.zookeeper.root.node=/nifi
>>>>>>>>>>
>>>>>>>>>> # kerberos #
>>>>>>>>>> nifi.kerberos.krb5.file=
>>>>>>>>>>
>>>>>>>>>> # kerberos service principle #
>>>>>>>>>> nifi.kerberos.service.principal=
>>>>>>>>>> nifi.kerberos.service.keytab.location=
>>>>>>>>>>
>>>>>>>>>> # kerberos spnego principle #
>>>>>>>>>> nifi.kerberos.spnego.principal=
>>>>>>>>>> nifi.kerberos.spnego.keytab.location=
>>>>>>>>>> nifi.kerberos.spnego.authentication.expiration=12 hours
>>>>>>>>>>
>>>>>>>>>> # external properties files for variable registry
>>>>>>>>>> # supports a comma delimited list of file locations
>>>>>>>>>> nifi.variable.registry.properties=
>>>>>>>>>>
>>>>>>>>>> I think I have everything set correctly but I have not been able to start an
>>>>>>>>>> instances up.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>>
>>>>>>>>>> Scott
>>>>>>>>>>
>>>>>>>>>> On Mar 19, 2018, at 4:35 PM, Bryan Bende <bb...@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>> The base file is here for comparison:
>>>>>>>>>>
>>>>>>>>>> https://github.com/apache/nifi-registry/blob/master/nifi-registry-resources/src/main/resources/conf/identity-providers.xml#L23
>>>>>>>>>>
>>>>>>>>>> On Mon, Mar 19, 2018 at 5:34 PM, Bryan Bende <bb...@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>> For your first file, is what you showed there actually wrapped in
>>>>>>>>>> <identityProviders> </identityProviders> or is it exactly what you
>>>>>>>>>> showed?
>>>>>>>>>>
>>>>>>>>>> It may just be that you only copied/pasted the one provider, but the
>>>>>>>>>> root element is not <provider>, so as it is shown there it would not
>>>>>>>>>> parse.
>>>>>>>>>>
>>>>>>>>>> On Mon, Mar 19, 2018 at 2:54 PM, Scott Howell <sc...@mobilgov.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Here is my file
>>>>>>>>>>
>>>>>>>>>> <provider>
>>>>>>>>>> <identifier>ldap-identity-provider</identifier>
>>>>>>>>>> <class>org.apache.nifi.registry.security.ldap.LdapProvider</class>
>>>>>>>>>> <property name="Authentication Strategy">SIMPLE</property>
>>>>>>>>>>
>>>>>>>>>> <property name="Manager DN">cn=Manager,dc=mobilgov,dc=com</property>
>>>>>>>>>> <property name="Manager Password”>redacted</property>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> <property name="Referral Strategy">FOLLOW</property>
>>>>>>>>>> <property name="Connect Timeout">10 secs</property>
>>>>>>>>>> <property name="Read Timeout">10 secs</property>
>>>>>>>>>>
>>>>>>>>>> <property name="Url”>redacted</property>
>>>>>>>>>> <property name="User Search
>>>>>>>>>> Base">ou=users,dc=mobilgov,dc=com</property>
>>>>>>>>>> <property name="User Search Filter">uid={0}</property>
>>>>>>>>>>
>>>>>>>>>> <property name="Identity Strategy">USE_DN</property>
>>>>>>>>>> <property name="Authentication Expiration">12 hours</property>
>>>>>>>>>> </provider>
>>>>>>>>>>
>>>>>>>>>> Here is my authorizers.xml
>>>>>>>>>>
>>>>>>>>>> <authorizers>
>>>>>>>>>>
>>>>>>>>>> <userGroupProvider>
>>>>>>>>>> <identifier>file-user-group-provider</identifier>
>>>>>>>>>>
>>>>>>>>>> <class>org.apache.nifi.registry.security.authorization.file.FileUserGroupProvider</class>
>>>>>>>>>> <property name="Users File">conf/users.xml</property>
>>>>>>>>>> <property name="Legacy Authorized Users File"></property>
>>>>>>>>>> <property name="Initial User Identity 1”>redacted</property>
>>>>>>>>>> </userGroupProvider>
>>>>>>>>>>
>>>>>>>>>> <accessPolicyProvider>
>>>>>>>>>> <identifier>file-access-policy-provider</identifier>
>>>>>>>>>>
>>>>>>>>>> <class>org.apache.nifi.registry.security.authorization.file.FileAccessPolicyProvider</class>
>>>>>>>>>> <property name="User Group
>>>>>>>>>> Provider">file-user-group-provider</property>
>>>>>>>>>> <property name="Authorizations
>>>>>>>>>> File">conf/authorizations.xml</property>
>>>>>>>>>> <property name="Initial Admin Identity”>redacted</property>
>>>>>>>>>> <property name="NiFi Identity 1"></property>
>>>>>>>>>> </accessPolicyProvider>
>>>>>>>>>>
>>>>>>>>>> <authorizer>
>>>>>>>>>> <identifier>managed-authorizer</identifier>
>>>>>>>>>>
>>>>>>>>>> <class>org.apache.nifi.registry.security.authorization.StandardManagedAuthorizer</class>
>>>>>>>>>> <property name="Access Policy
>>>>>>>>>> Provider">file-access-policy-provider</property>
>>>>>>>>>> </authorizer>
>>>>>>>>>> </authorizers>
>>>>>>>>>>
>>>>>>>>>> On Mar 19, 2018, at 12:59 PM, Bryan Bende <bb...@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>> It looks like that error would happen if your identity-providers.xml
>>>>>>>>>> contained invalid XML.
>>>>>>>>>>
>>>>>>>>>> Did you start by modifying the identity-providers.xml file that was
>>>>>>>>>> already there? Can you share the file, or the contents (removing
>>>>>>>>>> anything sensitive)?
>>>>>>>>>>
>>>>>>>>>> On Mon, Mar 19, 2018 at 1:09 PM, Scott Howell <sc...@mobilgov.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> So I was able to get the UI pulled up but now I am hitting a roadblock with
>>>>>>>>>> my identity-provider.xml.
>>>>>>>>>>
>>>>>>>>>> I am getting a number of errors like this:
>>>>>>>>>>
>>>>>>>>>> Caused by: org.springframework.beans.factory.BeanCreationException: Error
>>>>>>>>>> creating bean with name 'getIdentityProvider' defined in class path resource
>>>>>>>>>> [org/apache/nifi/registry/security/authentication/IdentityProviderFactory.class]:
>>>>>>>>>> Bean instantiation via factory method failed; nested exception is
>>>>>>>>>> org.springframework.beans.BeanInstantiationException: Failed to instantiate
>>>>>>>>>> [org.apache.nifi.registry.security.authentication.IdentityProvider]: Factory
>>>>>>>>>> method 'getIdentityProvider' threw exception; nested exception is
>>>>>>>>>> java.lang.Exception: Unable to load the login identity provider
>>>>>>>>>> configuration file at: /opt/nifi-registry-0.1.0/conf/identity-providers.xml
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:587)
>>>>>>>>>> ~[na:na]
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1250)
>>>>>>>>>> ~[na:na]
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1099)
>>>>>>>>>> ~[na:na]
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:545)
>>>>>>>>>> ~[na:na]
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:502)
>>>>>>>>>> ~[na:na]
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:312)
>>>>>>>>>> ~[na:na]
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228)
>>>>>>>>>> ~[na:na]
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:310)
>>>>>>>>>> ~[na:na]
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200)
>>>>>>>>>> ~[na:na]
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:251)
>>>>>>>>>> ~[na:na]
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1135)
>>>>>>>>>> ~[na:na]
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1062)
>>>>>>>>>> ~[na:na]
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:815)
>>>>>>>>>> ~[na:na]
>>>>>>>>>> at
>>>>>>>>>> org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:721)
>>>>>>>>>> ~[na:na]
>>>>>>>>>> ... 43 common frames omitted
>>>>>>>>>>
>>>>>>>>>> I know it has to do with the identity-provider.xml but I have my setup just
>>>>>>>>>> like the documentation ask for. I turned on debug but was not able to see
>>>>>>>>>> anything different or better explanation from it.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mar 19, 2018, at 10:06 AM, Kevin Doran <kd...@apache.org> wrote:
>>>>>>>>>>
>>>>>>>>>> Ok, that use case should be fine.
>>>>>>>>>>
>>>>>>>>>> If it were an authorization issue you would see something in the logs saying
>>>>>>>>>> that an authorization attempt failed and the server is responding with a
>>>>>>>>>> 403. Just to be sure, can you enable debug logging if you haven't already,
>>>>>>>>>> i.e., in your nifi-registry/conf/logback.xml file, change
>>>>>>>>>> 'org.apache.nifif.registry' to debug:
>>>>>>>>>>
>>>>>>>>>> <!-- valid logging levels: TRACE, DEBUG, INFO, WARN, ERROR -->
>>>>>>>>>> <logger name="org.apache.nifi.registry" level="DEBUG"/>
>>>>>>>>>>
>>>>>>>>>> If there is nothing being written to nifi-registry-app.log, it points
>>>>>>>>>> towards a connection issue, so I would double check your host, port, and TLS
>>>>>>>>>> settings. You'll have to get an HTTPS cert from a root CA or configure your
>>>>>>>>>> ELB to trust your company's self-signed cert (again, not sure if/how to do
>>>>>>>>>> this, but I assume there should be some way to configure it. It might
>>>>>>>>>> require settings not exposed in the AWS web console.)
>>>>>>>>>>
>>>>>>>>>> On 3/19/18, 10:51, "Scott Howell" <sc...@mobilgov.com> wrote:
>>>>>>>>>>
>>>>>>>>>> Thanks Kevin,
>>>>>>>>>>
>>>>>>>>>> I am just using the ELB to go from the public subnet to the private subnet.
>>>>>>>>>> I will not have multiple instances running of registry.
>>>>>>>>>>
>>>>>>>>>> I will say on my authorizers.xml there is one difference between my nifi
>>>>>>>>>> instance. On my nifi instance I am using file-provider for
>>>>>>>>>> nifi.security.user.authorizer in my nifi.properties. I don’t think from
>>>>>>>>>> reading the documents for nifi-registry that I can use that. If there is a
>>>>>>>>>> way that might be my problem. I was running into some issues with my nifi
>>>>>>>>>> instance when I was using managed-authorizers instead of file-provider.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mar 19, 2018, at 9:35 AM, Kevin Doran <kd...@apache.org> wrote:
>>>>>>>>>>
>>>>>>>>>> Hey Scott,
>>>>>>>>>>
>>>>>>>>>> Assuming you are using two-way TLS with client certificates for
>>>>>>>>>> authentication, I recommend configuring your ELB for TCP passthrough so that
>>>>>>>>>> the TLS handshake is between the end-client and the NiFi Registry Server (in
>>>>>>>>>> other words, no decryption/termination of the TLS connection happens in the
>>>>>>>>>> ELB). If you are using some other form of authentication (e.g., LDAP), you
>>>>>>>>>> will need to configure your ELB to trust the self-signed key NiFi Registry
>>>>>>>>>> is using. I'm not sure how to do that as I've never run an ELB with that
>>>>>>>>>> configuration before.
>>>>>>>>>>
>>>>>>>>>> Also, just a note about using an ELB with NiFi Registry:
>>>>>>>>>>
>>>>>>>>>> NiFi Registry is currently only supports single-instance use as persisted
>>>>>>>>>> data and in-memory state is not synced between multiple instances. Are you
>>>>>>>>>> hoping to use the ELB for actual load balancing, or is it just to take
>>>>>>>>>> advantage of other ELB features, such as forwarding and security group
>>>>>>>>>> rules? If the plan is to load balance multiple Registry instances, just be
>>>>>>>>>> aware that you will probably run into some unexpected behavior. (As you
>>>>>>>>>> mentioned using authorization, that is one case where I know the in-memory
>>>>>>>>>> cache of the persisted data will not refresh across instances, so even if
>>>>>>>>>> you were using some sort of shared network file system attached to multiple
>>>>>>>>>> Registry instances, such as EFS, it would not work the way you hope.)
>>>>>>>>>>
>>>>>>>>>> Hope this helps,
>>>>>>>>>> Kevin
>>>>>>>>>>
>>>>>>>>>> On 3/19/18, 10:20, "Scott Howell" <sc...@mobilgov.com> wrote:
>>>>>>>>>>
>>>>>>>>>> Thanks for the quick response.
>>>>>>>>>>
>>>>>>>>>> A couple of things I am seeing.
>>>>>>>>>>
>>>>>>>>>> 1. There is no error, I don’t see anything in the logs once the service
>>>>>>>>>> comes up. This is because the health check is not even hitting the instance
>>>>>>>>>> when secure.
>>>>>>>>>>
>>>>>>>>>> 2. Nothing interesting in the nifi-registry-app.logs. That was my concern
>>>>>>>>>> because on my nifi instance I can see the health check hitting the instance
>>>>>>>>>> from the ELB. This does not happen on the nifi-registry instance. I see the
>>>>>>>>>> service startup and it tells me what domain and port I can access the UI but
>>>>>>>>>> nothing else after that.
>>>>>>>>>>
>>>>>>>>>> 3. When I am on an instances in the same private subnet I am able to curl to
>>>>>>>>>> the instance I get the TLS SSL which tells me the keystore is on the server.
>>>>>>>>>> I am using a JKS keystore that is self-signed by the company I work for.
>>>>>>>>>>
>>>>>>>>>> On Mar 19, 2018, at 9:10 AM, Bryan Bende <bb...@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>> What error are you getting when you cannot access the UI?
>>>>>>>>>>
>>>>>>>>>> Is there anything interesting in nifi-registry-app.log regarding
>>>>>>>>>> authentication/authorization when this happens?
>>>>>>>>>>
>>>>>>>>>> Can you access the UI securely without going through the ELB?
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>>
>>>>>>>>>> Bryan
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Mar 19, 2018 at 10:05 AM, Scott Howell <sc...@mobilgov.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> I was able to stand up nifi-registry behind an AWS ELB non-secure.
>>>>>>>>>> Everything was working great and was able to access the UI anonymously. I
>>>>>>>>>> set up the authorization just like on my nifi instances along with the
>>>>>>>>>> authorizers and identity-provider. The service comes up without errors and
>>>>>>>>>> everything looks good but the health check does not pass and I cannot access
>>>>>>>>>> the UI to login. I was wondering if anyone else has ran into this issue
>>>>>>>>>> using nifi-registry.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>
>
Re: Issue with AWS ELB on secure nifi-registry
Posted by Scott Howell <sc...@mobilgov.com>.
I do have a one node cluster working with the configuration below.
This is the user.xml for my 2 node cluster
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<tenants>
<groups/>
<users>
<user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53" identity="uid=scott,ou=users,dc=mobilgov,dc=com"/>
<user identifier="727e6d3f-6e95-377b-b38f-d697163e2591" identity="CN=node1-nifi-dev.{redacted}.com, OU={redacted}, O={redacted}, L=Kansas City, ST=Missouri, C=US"/>
<user identifier="1185a75e-586c-3a2c-b8df-93d0df3a4cb3" identity="CN=node2-nifi-dev.{redacted}.com, OU={redacted}, O={redacted}, L=Kansas City, ST=Missouri, C=US"/>
</users>
</tenants>
Authorizations.xml
▽
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<authorizations>
<policies>
<policy identifier="f99bccd1-a30e-3e4a-98a2-dbc708edc67f" resource="/flow" action="R">
<user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
</policy>
<policy identifier="b6df1162-ae29-3a55-ba31-36ce6ba674ea" resource="/data/process-groups/25d132eb-0162-1000-99a8-cc7e4dc0302d" action="R">
<user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
<user identifier="727e6d3f-6e95-377b-b38f-d697163e2591"/>
<user identifier="1185a75e-586c-3a2c-b8df-93d0df3a4cb3"/>
</policy>
<policy identifier="46a35aa9-f909-3563-b73c-b5feac03cf6b" resource="/data/process-groups/25d132eb-0162-1000-99a8-cc7e4dc0302d" action="W">
<user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
<user identifier="727e6d3f-6e95-377b-b38f-d697163e2591"/>
<user identifier="1185a75e-586c-3a2c-b8df-93d0df3a4cb3"/>
</policy>
<policy identifier="2037be0f-7a4d-3564-9230-cd338a255f03" resource="/process-groups/25d132eb-0162-1000-99a8-cc7e4dc0302d" action="R">
<user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
</policy>
<policy identifier="f8327f78-a2fb-371b-abea-64079b60b938" resource="/process-groups/25d132eb-0162-1000-99a8-cc7e4dc0302d" action="W">
<user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
</policy>
<policy identifier="b8775bd4-704a-34c6-987b-84f2daf7a515" resource="/restricted-components" action="W">
<user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
</policy>
<policy identifier="627410be-1717-35b4-a06f-e9362b89e0b7" resource="/tenants" action="R">
<user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
</policy>
<policy identifier="15e4e0bd-cb28-34fd-8587-f8d15162cba5" resource="/tenants" action="W">
<user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
</policy>
<policy identifier="ff96062a-fa99-36dc-9942-0f6442ae7212" resource="/policies" action="R">
<user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
</policy>
<policy identifier="ad99ea98-3af6-3561-ae27-5bf09e1d969d" resource="/policies" action="W">
<user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
</policy>
<policy identifier="2e1015cb-0fed-3005-8e0d-722311f21a03" resource="/controller" action="R">
<user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
</policy>
<policy identifier="c6322e6c-4cc1-3bcc-91b3-2ed2111674cf" resource="/controller" action="W">
<user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
</policy>
<policy identifier="287edf48-da72-359b-8f61-da5d4c45a270" resource="/proxy" action="W">
<user identifier="727e6d3f-6e95-377b-b38f-d697163e2591"/>
<user identifier="1185a75e-586c-3a2c-b8df-93d0df3a4cb3"/>
</policy>
</policies>
</authorizations>
I get the untrusted proxy error when I have the nodes set like this.
> On Mar 21, 2018, at 12:20 PM, Bryan Bende <bb...@gmail.com> wrote:
>
> Ok that looks correct for the 1-node case.
>
> So just to clarify what is working and not working...
>
> With the config in the last email, you have a 1 node cluster that is
> working and you can get into the UI?
>
> For the two node case you would need each node to have a users.xml
> with users for the two nodes, and an authorizations.xml with two
> /proxy policies, one for each node.
>
> So if you do that then your cluster starts up, but when you access the
> UI then you get the untrusted proxy?
>
>
> On Wed, Mar 21, 2018 at 12:58 PM, Scott Howell <sc...@mobilgov.com> wrote:
>> user.xml
>>
>> <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
>> <tenants>
>> <groups/>
>> <users>
>> <user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53" identity="uid=scott,ou=users,dc={redacted},dc=com"/>
>> <user identifier="7f2b37ff-4d40-316a-843a-92c8523afc0b" identity="CN=*.{redacted}.com, OU={redacted}, O={redacted}, L=Kansas City, ST=Missouri, C=US"/>
>> </users>
>> </tenants>
>>
>> Authorizations.xml
>> <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
>> <authorizations>
>> <policies>
>> <policy identifier="f99bccd1-a30e-3e4a-98a2-dbc708edc67f" resource="/flow" action="R">
>> <user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
>> </policy>
>> <policy identifier="ef96e849-629c-3f5e-97af-74efe29423bc" resource="/data/process-groups/4505628f-0162-1000-3b39-002cd06f74da" action="R">
>> <user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
>> <user identifier="7f2b37ff-4d40-316a-843a-92c8523afc0b"/>
>> </policy>
>> <policy identifier="fc29cd4c-ec37-3820-82b4-bbfd305b85ae" resource="/data/process-groups/4505628f-0162-1000-3b39-002cd06f74da" action="W">
>> <user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
>> <user identifier="7f2b37ff-4d40-316a-843a-92c8523afc0b"/>
>> </policy>
>> <policy identifier="75acccef-45ab-3b31-a49b-8cf88186c8bf" resource="/process-groups/4505628f-0162-1000-3b39-002cd06f74da" action="R">
>> <user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
>> </policy>
>> <policy identifier="e9691c14-2540-3544-988a-654b79cf2370" resource="/process-groups/4505628f-0162-1000-3b39-002cd06f74da" action="W">
>> <user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
>> </policy>
>> <policy identifier="b8775bd4-704a-34c6-987b-84f2daf7a515" resource="/restricted-components" action="W">
>> <user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
>> </policy>
>> <policy identifier="627410be-1717-35b4-a06f-e9362b89e0b7" resource="/tenants" action="R">
>> <user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
>> </policy>
>> <policy identifier="15e4e0bd-cb28-34fd-8587-f8d15162cba5" resource="/tenants" action="W">
>> <user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
>> </policy>
>> <policy identifier="ff96062a-fa99-36dc-9942-0f6442ae7212" resource="/policies" action="R">
>> <user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
>> </policy>
>> <policy identifier="ad99ea98-3af6-3561-ae27-5bf09e1d969d" resource="/policies" action="W">
>> <user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
>> </policy>
>> <policy identifier="2e1015cb-0fed-3005-8e0d-722311f21a03" resource="/controller" action="R">
>> <user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
>> </policy>
>> <policy identifier="c6322e6c-4cc1-3bcc-91b3-2ed2111674cf" resource="/controller" action="W">
>> <user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
>> </policy>
>> <policy identifier="287edf48-da72-359b-8f61-da5d4c45a270" resource="/proxy" action="W">
>> <user identifier="7f2b37ff-4d40-316a-843a-92c8523afc0b"/>
>> </policy>
>> </policies>
>> </authorizations>
>>
>>> On Mar 21, 2018, at 11:49 AM, Bryan Bende <bb...@gmail.com> wrote:
>>>
>>> I've never used wildcard certs before so I'll have to defer to others
>>> that might know if there is any issue with that.
>>>
>>> Could show the contents of these two files just so we can double check
>>> the users/policies?
>>>
>>> <property name="Authorizations File">/opt/config/authorizations.xml</property>
>>> <property name="Users File">/opt/config/users.xml</property>
>>>
>>> On Wed, Mar 21, 2018 at 12:37 PM, Scott Howell <sc...@mobilgov.com> wrote:
>>>> Thanks I have checked that and the whitespace is correct in user.xml.
>>>>
>>>> I did make a change to my authorizer.xml
>>>>
>>>> <authorizers>
>>>> <authorizer>
>>>> <identifier>file-provider</identifier>
>>>> <class>org.apache.nifi.authorization.FileAuthorizer</class>
>>>> <property name="Authorizations File">/opt/config/authorizations.xml</property>
>>>> <property name="Users File">/opt/config/users.xml</property>
>>>> <property name="Initial Admin Identity">uid=scott,ou=users,dc=mobilgov,dc=com</property>
>>>> <property name="Legacy Authorized Users File"></property>
>>>>
>>>> <property name="Node Identity 1">CN=*.{redacted}.com, OU={redacted}, O={redacted}, L=Kansas City, ST=Missouri, C=US</property>
>>>> <property name="Node Identity 2">CN=*.{redacted}.com, OU={redacted}, O={redacted}, L=Kansas City, ST=Missouri, C=US</property>
>>>> </authorizer>
>>>> </authorizers>
>>>>
>>>> I made the CN=*.{redacted}.com just like the self-signed certificate showed. I now have a 1 node cluster up and running. It seems like NIfi isn’t taking into account the wildcard and treating it as a “*” instead.
>>>>
>>>>> On Mar 21, 2018, at 11:33 AM, Bryan Bende <bb...@gmail.com> wrote:
>>>>>
>>>>> There only needs to be W to /proxy so that part should be fine.
>>>>>
>>>>> After you edited the Node Identities, did you delete users.xml and
>>>>> authorizations.xml?
>>>>>
>>>>> You would have to do that for those changes to take effect. You can
>>>>> look in users.xml and see if you still have the user identities
>>>>> without whitespace.
>>>>>
>>>>> On Wed, Mar 21, 2018 at 12:20 PM, Scott Howell <sc...@mobilgov.com> wrote:
>>>>>> One other thing I am seeing and I don’t know if this is an issue or not in my authorizations.xml I do not have a policy for /proxy with action=“R” only action=“W”.
>>>>>>
>>>>>>> On Mar 21, 2018, at 11:03 AM, Scott Howell <sc...@mobilgov.com> wrote:
>>>>>>>
>>>>>>> Thanks for that. I am still getting this error in my nifi-user.log
>>>>>>>
>>>>>>> o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api: Untrusted proxy CN=*.{redacted}.com, OU={redacted}, O={redacted}, L=Kansas City, ST=Missouri, C=US
>>>>>>>
>>>>>>> Is there an issue with using a wildcard cert?
>>>>>>>
>>>>>>>
>>>>>>>> On Mar 21, 2018, at 10:23 AM, Bryan Bende <bb...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> All identity strings are case & whitespace sensitive.
>>>>>>>>
>>>>>>>> The node identities in your authorizers.xml have no whitespace, and
>>>>>>>> the identity showing in the logs does.
>>>>>>>>
>>>>>>>> On Wed, Mar 21, 2018 at 11:05 AM, Scott Howell <sc...@mobilgov.com> wrote:
>>>>>>>>> Thanks for all of the help with this. I have the cluster up and running. The
>>>>>>>>> logs look great everything seems to be working but I cannot login into the
>>>>>>>>> UI. I am using a wildcard self-signed certificate. The /proxy is in
>>>>>>>>> authorizations.xml with the correct users for the nodes.
>>>>>>>>>
>>>>>>>>> The error I see with the UI :
>>>>>>>>> is Untrusted proxy CN=*.{redacted}.com, OU={redacted}, O={redacted},
>>>>>>>>> L=Kansas City, ST=Missouri, C=US
>>>>>>>>>
>>>>>>>>> I haven’t had much luck finding a lot of documentation or forum questions
>>>>>>>>> with this kind of issue.
>>>>>>>>>
>>>>>>>>> My authorizers.xml looks like this
>>>>>>>>> <authorizers>
>>>>>>>>> <authorizer>
>>>>>>>>> <identifier>file-provider</identifier>
>>>>>>>>> <class>org.apache.nifi.authorization.FileAuthorizer</class>
>>>>>>>>> <property name="Authorizations
>>>>>>>>> File">/opt/config/authorizations.xml</property>
>>>>>>>>> <property name="Users File">/opt/config/users.xml</property>
>>>>>>>>> <property name="Initial Admin
>>>>>>>>> Identity">uid=scott,ou=users,dc={redacted},dc=com</property>
>>>>>>>>> <property name="Legacy Authorized Users File"></property>
>>>>>>>>>
>>>>>>>>> <property name="Node Identity
>>>>>>>>> 1">CN=node-1-nifi-dev.{redacted}.com,OU={redacted},O={redacted},L=Kansas
>>>>>>>>> City,ST=Missouri,C=US</property>
>>>>>>>>> <property name="Node Identity
>>>>>>>>> 2">CN=node-2-nifi-dev.{redacted}.com,OU={redacted},O={redacted},L=Kansas
>>>>>>>>> City,ST=Missouri,C=US</property>
>>>>>>>>> </authorizer>
>>>>>>>>> </authorizers>
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> Scott
>>>>>>>>>
>>>>>>>>> On Mar 20, 2018, at 1:15 PM, Andy LoPresto <al...@apache.org> wrote:
>>>>>>>>>
>>>>>>>>> Scott,
>>>>>>>>>
>>>>>>>>> The original exception is "nested exception is
>>>>>>>>> java.security.KeyStoreException: not found”. Can you verify that the
>>>>>>>>> keystore you’ve provided is valid using the “keytool” command? In addition,
>>>>>>>>> you will need a truststore as well. Try following Pierre's [1] or Bryan’s
>>>>>>>>> [2] instructions for setting up a secure cluster.
>>>>>>>>>
>>>>>>>>> [1]
>>>>>>>>> https://pierrevillard.com/2016/11/29/apache-nifi-1-1-0-secured-cluster-setup/
>>>>>>>>> [2]
>>>>>>>>> https://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-authorization-and-multi-tenancy
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Andy LoPresto
>>>>>>>>> alopresto@apache.org
>>>>>>>>> alopresto.apache@gmail.com
>>>>>>>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>>>>>>>>>
>>>>>>>>> On Mar 20, 2018, at 11:05 AM, Scott Howell <sc...@mobilgov.com> wrote:
>>>>>>>>>
>>>>>>>>> Thanks for all of the help yesterday I was able to get a secure nifi and
>>>>>>>>> nifi-registry up and communicating. I am now trying to figure out how to
>>>>>>>>> create a secure cluster. I am currently getting this error when I start up
>>>>>>>>> nifi.
>>>>>>>>>
>>>>>>>>> tion; nested exception is
>>>>>>>>> org.springframework.beans.factory.BeanCreationException: Error creating bean
>>>>>>>>> with name 'clusterCoordinationProtocolSenderListener' defined in class path
>>>>>>>>> resource [nifi-cluster-protocol-context.xml]: Cannot resolve reference to
>>>>>>>>> bean 'clusterCoordinationProtocolSender' while setting constructor argument;
>>>>>>>>> nested exception is org.springframework.beans.factory.BeanCreationException:
>>>>>>>>> Error creating bean with name 'clusterCoordinationProtocolSender' defined in
>>>>>>>>> class path resource [nifi-cluster-protocol-context.xml]: Cannot resolve
>>>>>>>>> reference to bean 'protocolSocketConfiguration' while setting constructor
>>>>>>>>> argument; nested exception is
>>>>>>>>> org.springframework.beans.factory.BeanCreationException: Error creating bean
>>>>>>>>> with name 'protocolSocketConfiguration': FactoryBean threw exception on
>>>>>>>>> object creation; nested exception is java.security.KeyStoreException: not
>>>>>>>>> found
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175)
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1634)
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317)
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
>>>>>>>>> ... 50 common frames omitted
>>>>>>>>> Caused by: org.springframework.beans.factory.BeanCreationException: Error
>>>>>>>>> creating bean with name 'clusterCoordinationProtocolSenderListener' defined
>>>>>>>>> in class path resource [nifi-cluster-protocol-context.xml]: Cannot resolve
>>>>>>>>> reference to bean 'clusterCoordinationProtocolSender' while setting
>>>>>>>>> constructor argument; nested exception is
>>>>>>>>> org.springframework.beans.factory.BeanCreationException: Error creating bean
>>>>>>>>> with name 'clusterCoordinationProtocolSender' defined in class path resource
>>>>>>>>> [nifi-cluster-protocol-context.xml]: Cannot resolve reference to bean
>>>>>>>>> 'protocolSocketConfiguration' while setting constructor argument; nested
>>>>>>>>> exception is org.springframework.beans.factory.BeanCreationException: Error
>>>>>>>>> creating bean with name 'protocolSocketConfiguration': FactoryBean threw
>>>>>>>>> exception on object creation; nested exception is
>>>>>>>>> java.security.KeyStoreException: not found
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:359)
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:648)
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:145)
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1193)
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1095)
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
>>>>>>>>> at
>>>>>>>>> org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1084)
>>>>>>>>> at
>>>>>>>>> org.apache.nifi.cluster.spring.NodeClusterCoordinatorFactoryBean.getObject(NodeClusterCoordinatorFactoryBean.java:44)
>>>>>>>>> at
>>>>>>>>> org.apache.nifi.cluster.spring.NodeClusterCoordinatorFactoryBean.getObject(NodeClusterCoordinatorFactoryBean.java:34)
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
>>>>>>>>> ... 55 common frames omitted
>>>>>>>>> Caused by: org.springframework.beans.factory.BeanCreationException: Error
>>>>>>>>> creating bean with name 'clusterCoordinationProtocolSender' defined in class
>>>>>>>>> path resource [nifi-cluster-protocol-context.xml]: Cannot resolve reference
>>>>>>>>> to bean 'protocolSocketConfiguration' while setting constructor argument;
>>>>>>>>> nested exception is org.springframework.beans.factory.BeanCreationException:
>>>>>>>>> Error creating bean with name 'protocolSocketConfiguration': FactoryBean
>>>>>>>>> threw exception on object creation; nested exception is
>>>>>>>>> java.security.KeyStoreException: not found
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:359)
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:648)
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:145)
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1193)
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1095)
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
>>>>>>>>> ... 70 common frames omitted
>>>>>>>>> Caused by: org.springframework.beans.factory.BeanCreationException: Error
>>>>>>>>> creating bean with name 'protocolSocketConfiguration': FactoryBean threw
>>>>>>>>> exception on object creation; nested exception is
>>>>>>>>> java.security.KeyStoreException: not found
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175)
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1634)
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317)
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
>>>>>>>>> ... 82 common frames omitted
>>>>>>>>> Caused by: java.security.KeyStoreException: not found
>>>>>>>>> at java.security.KeyStore.getInstance(KeyStore.java:851)
>>>>>>>>> at
>>>>>>>>> org.apache.nifi.security.util.KeyStoreUtils.getKeyStore(KeyStoreUtils.java:66)
>>>>>>>>> at
>>>>>>>>> org.apache.nifi.security.util.KeyStoreUtils.getTrustStore(KeyStoreUtils.java:80)
>>>>>>>>> at
>>>>>>>>> org.apache.nifi.io.socket.SSLContextFactory.<init>(SSLContextFactory.java:73)
>>>>>>>>> at
>>>>>>>>> org.apache.nifi.cluster.protocol.spring.SocketConfigurationFactoryBean.getObject(SocketConfigurationFactoryBean.java:45)
>>>>>>>>> at
>>>>>>>>> org.apache.nifi.cluster.protocol.spring.SocketConfigurationFactoryBean.getObject(SocketConfigurationFactoryBean.java:30)
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
>>>>>>>>> ... 87 common frames omitted
>>>>>>>>> Caused by: java.security.NoSuchAlgorithmException: KeyStore not available
>>>>>>>>> at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
>>>>>>>>> at java.security.Security.getImpl(Security.java:695)
>>>>>>>>> at java.security.KeyStore.getInstance(KeyStore.java:848)
>>>>>>>>> ... 93 common frames omitted
>>>>>>>>>
>>>>>>>>> My nifi.properties file is.
>>>>>>>>>
>>>>>>>>> # Licensed to the Apache Software Foundation (ASF) under one or more
>>>>>>>>> # contributor license agreements. See the NOTICE file distributed with
>>>>>>>>> # this work for additional information regarding copyright ownership.
>>>>>>>>> # The ASF licenses this file to You under the Apache License, Version 2.0
>>>>>>>>> # (the "License"); you may not use this file except in compliance with
>>>>>>>>> # the License. You may obtain a copy of the License at
>>>>>>>>> #
>>>>>>>>> # http://www.apache.org/licenses/LICENSE-2.0
>>>>>>>>> #
>>>>>>>>> # Unless required by applicable law or agreed to in writing, software
>>>>>>>>> # distributed under the License is distributed on an "AS IS" BASIS,
>>>>>>>>> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>>>>>>>>> # See the License for the specific language governing permissions and
>>>>>>>>> # limitations under the License.
>>>>>>>>>
>>>>>>>>> # Core Properties #
>>>>>>>>> nifi.version={{nifi_version}}
>>>>>>>>> nifi.flow.configuration.file=/opt/config/flow.xml.gz
>>>>>>>>> nifi.flow.configuration.archive.enabled=true
>>>>>>>>> nifi.flow.configuration.archive.dir=/opt/config/archive/
>>>>>>>>> nifi.flow.configuration.archive.max.time=30 days
>>>>>>>>> nifi.flow.configuration.archive.max.storage=500 MB
>>>>>>>>> nifi.flowcontroller.autoResumeState=true
>>>>>>>>> nifi.flowcontroller.graceful.shutdown.period=10 sec
>>>>>>>>> nifi.flowservice.writedelay.interval=500 ms
>>>>>>>>> nifi.administrative.yield.duration=30 sec
>>>>>>>>> # If a component has no work to do (is "bored"), how long should we wait
>>>>>>>>> before checking again for work?
>>>>>>>>> nifi.bored.yield.duration=10 millis
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> nifi.authorizer.configuration.file=/opt/config/authorizers.xml
>>>>>>>>> nifi.login.identity.provider.configuration.file=/opt/config/login-identity-providers.xml
>>>>>>>>> nifi.templates.directory=/opt/config/templates
>>>>>>>>> nifi.ui.banner.text=
>>>>>>>>> nifi.ui.autorefresh.interval=30 sec
>>>>>>>>> nifi.nar.library.directory=/opt/nifi/lib
>>>>>>>>> nifi.nar.library.directory.custom=/opt/config/processors
>>>>>>>>> nifi.nar.working.directory=/opt/nifi/work/nar/
>>>>>>>>> nifi.documentation.working.directory=./work/docs/components
>>>>>>>>>
>>>>>>>>> ####################
>>>>>>>>> # State Management #
>>>>>>>>> ####################
>>>>>>>>> nifi.state.management.configuration.file=/opt/config/state-management.xml
>>>>>>>>> # The ID of the local state provider
>>>>>>>>> nifi.state.management.provider.local=local-provider
>>>>>>>>> # The ID of the cluster-wide state provider. This will be ignored if NiFi is
>>>>>>>>> not clustered but must be populated if running in a cluster.
>>>>>>>>> nifi.state.management.provider.cluster=zk-provider
>>>>>>>>> # Specifies whether or not this instance of NiFi should run an embedded
>>>>>>>>> ZooKeeper server
>>>>>>>>> nifi.state.management.embedded.zookeeper.start=false
>>>>>>>>> # Properties file that provides the ZooKeeper properties to use if
>>>>>>>>> <nifi.state.management.embedded.zookeeper.start> is set to true
>>>>>>>>> nifi.state.management.embedded.zookeeper.properties=/opt/config/zookeeper.properties
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> # H2 Settings
>>>>>>>>> nifi.database.directory=/opt/database_repository
>>>>>>>>> nifi.h2.url.append=;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE
>>>>>>>>>
>>>>>>>>> # FlowFile Repository
>>>>>>>>> nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository
>>>>>>>>> nifi.flowfile.repository.directory=/opt/flowfile_repository
>>>>>>>>> nifi.flowfile.repository.partitions=256
>>>>>>>>> nifi.flowfile.repository.checkpoint.interval=2 mins
>>>>>>>>> nifi.flowfile.repository.always.sync=false
>>>>>>>>>
>>>>>>>>> nifi.swap.manager.implementation=org.apache.nifi.controller.FileSystemSwapManager
>>>>>>>>> nifi.queue.swap.threshold=20000
>>>>>>>>> nifi.swap.in.period=5 sec
>>>>>>>>> nifi.swap.in.threads=1
>>>>>>>>> nifi.swap.out.period=5 sec
>>>>>>>>> nifi.swap.out.threads=4
>>>>>>>>>
>>>>>>>>> # Content Repository
>>>>>>>>> nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository
>>>>>>>>> nifi.content.claim.max.appendable.size=10 MB
>>>>>>>>> nifi.content.claim.max.flow.files=100
>>>>>>>>> nifi.content.repository.directory.default=/opt/content_repository
>>>>>>>>> nifi.content.repository.archive.max.retention.period=12 hours
>>>>>>>>> nifi.content.repository.archive.max.usage.percentage=50%
>>>>>>>>> nifi.content.repository.archive.enabled=true
>>>>>>>>> nifi.content.repository.always.sync=false
>>>>>>>>> nifi.content.viewer.url=/nifi-content-viewer/
>>>>>>>>>
>>>>>>>>> # Provenance Repository Properties
>>>>>>>>> nifi.provenance.repository.implementation=org.apache.nifi.provenance.PersistentProvenanceRepository
>>>>>>>>>
>>>>>>>>> # Persistent Provenance Repository Properties
>>>>>>>>> nifi.provenance.repository.directory.default=/opt/provenance_repository
>>>>>>>>> nifi.provenance.repository.max.storage.time=24 hours
>>>>>>>>> nifi.provenance.repository.max.storage.size=1 GB
>>>>>>>>> nifi.provenance.repository.rollover.time=30 secs
>>>>>>>>> nifi.provenance.repository.rollover.size=100 MB
>>>>>>>>> nifi.provenance.repository.query.threads=2
>>>>>>>>> nifi.provenance.repository.index.threads=1
>>>>>>>>> nifi.provenance.repository.compress.on.rollover=true
>>>>>>>>> nifi.provenance.repository.always.sync=false
>>>>>>>>> nifi.provenance.repository.journal.count=16
>>>>>>>>> # Comma-separated list of fields. Fields that are not indexed will not be
>>>>>>>>> searchable. Valid fields are:
>>>>>>>>> # EventType, FlowFileUUID, Filename, TransitURI, ProcessorID,
>>>>>>>>> AlternateIdentifierURI, Relationship, Details
>>>>>>>>> nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename,
>>>>>>>>> ProcessorID, Relationship
>>>>>>>>> # FlowFile Attributes that should be indexed and made searchable. Some
>>>>>>>>> examples to consider are filename, uuid, mime.type
>>>>>>>>> nifi.provenance.repository.indexed.attributes=
>>>>>>>>> # Large values for the shard size will result in more Java heap usage when
>>>>>>>>> searching the Provenance Repository
>>>>>>>>> # but should provide better performance
>>>>>>>>> nifi.provenance.repository.index.shard.size=500 MB
>>>>>>>>> # Indicates the maximum length that a FlowFile attribute can be when
>>>>>>>>> retrieving a Provenance Event from
>>>>>>>>> # the repository. If the length of any attribute exceeds this value, it will
>>>>>>>>> be truncated when the event is retrieved.
>>>>>>>>> nifi.provenance.repository.max.attribute.length=65536
>>>>>>>>>
>>>>>>>>> # Volatile Provenance Respository Properties
>>>>>>>>> nifi.provenance.repository.buffer.size=100000
>>>>>>>>>
>>>>>>>>> # Component Status Repository
>>>>>>>>> nifi.components.status.repository.implementation=org.apache.nifi.controller.status.history.VolatileComponentStatusRepository
>>>>>>>>> nifi.components.status.repository.buffer.size=1440
>>>>>>>>> nifi.components.status.snapshot.frequency=1 min
>>>>>>>>>
>>>>>>>>> # Site to Site properties
>>>>>>>>> nifi.remote.input.host=
>>>>>>>>> nifi.remote.input.secure=false
>>>>>>>>> nifi.remote.input.socket.port=9998
>>>>>>>>> nifi.remote.input.http.enabled=false
>>>>>>>>> nifi.remote.input.http.transaction.ttl=30 sec
>>>>>>>>>
>>>>>>>>> # web properties #
>>>>>>>>> nifi.web.war.directory=/opt/nifi/lib
>>>>>>>>> nifi.web.http.host=
>>>>>>>>> nifi.web.http.port=
>>>>>>>>> nifi.web.https.host={{redacted}}
>>>>>>>>> nifi.web.https.port=8443
>>>>>>>>> nifi.web.jetty.working.directory=/opt/nifi/work/jetty
>>>>>>>>> nifi.web.jetty.threads=200
>>>>>>>>>
>>>>>>>>> # security properties #
>>>>>>>>> nifi.sensitive.props.key=x0KDgO9L8lAhFGLdvu2VEjFVGc6Kg3V0R5I4bYwoqdgL47moo0wApKQtAVu1BvD
>>>>>>>>> nifi.sensitive.props.key.protected=
>>>>>>>>> nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
>>>>>>>>> nifi.sensitive.props.provider=BC
>>>>>>>>> nifi.sensitive.props.additional.keys=
>>>>>>>>>
>>>>>>>>> nifi.security.keystore=/opt/certs/payit_keystore
>>>>>>>>> nifi.security.keystoreType=JKS
>>>>>>>>> nifi.security.keystorePasswd={{keystore_password}}
>>>>>>>>> nifi.security.keyPasswd=
>>>>>>>>> nifi.security.truststore=
>>>>>>>>> nifi.security.truststoreType=
>>>>>>>>> nifi.security.truststorePasswd=
>>>>>>>>> nifi.security.needClientAuth=false
>>>>>>>>> nifi.security.user.authorizer=file-provider
>>>>>>>>> nifi.security.user.login.identity.provider=ldap-provider
>>>>>>>>> nifi.security.ocsp.responder.url=
>>>>>>>>> nifi.security.ocsp.responder.certificate=
>>>>>>>>>
>>>>>>>>> # Identity Mapping Properties #
>>>>>>>>> # These properties allow normalizing user identities such that identities
>>>>>>>>> coming from different identity providers
>>>>>>>>> # (certificates, LDAP, Kerberos) can be treated the same internally in NiFi.
>>>>>>>>> The following example demonstrates normalizing
>>>>>>>>> # DNs from certificates and principals from Kerberos into a common identity
>>>>>>>>> string:
>>>>>>>>> #
>>>>>>>>> #nifi.security.identity.mapping.pattern.dn=^cn=(.*?),ou=(.*?),dc=(.*?),dc=(.*?),dc=(.*?)$
>>>>>>>>> #nifi.security.identity.mapping.value.dn=$1
>>>>>>>>> # nifi.security.identity.mapping.pattern.kerb=^(.*?)/instance@(.*?)$
>>>>>>>>> # nifi.security.identity.mapping.value.kerb=$1@$2
>>>>>>>>>
>>>>>>>>> # cluster common properties (all nodes must have same values) #
>>>>>>>>> nifi.cluster.protocol.heartbeat.interval=5 sec
>>>>>>>>> nifi.cluster.protocol.is.secure=true
>>>>>>>>>
>>>>>>>>> # cluster node properties (only configure for cluster nodes) #
>>>>>>>>> nifi.cluster.is.node=true
>>>>>>>>> nifi.cluster.node.address=nifi-dev.mobilgov.com
>>>>>>>>> nifi.cluster.node.protocol.port=9999
>>>>>>>>> nifi.cluster.node.protocol.threads=10
>>>>>>>>> nifi.cluster.node.event.history.size=25
>>>>>>>>> nifi.cluster.node.connection.timeout=5 sec
>>>>>>>>> nifi.cluster.node.read.timeout=5 sec
>>>>>>>>> nifi.cluster.firewall.file=
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> # zookeeper properties, used for cluster management #
>>>>>>>>> nifi.zookeeper.connect.string=internal-etcd-dev-EtcdLoadB-3RWA2WEFBBT8-2068560477.us-east-2.elb.amazonaws.com:2181,internal-etcd-dev-EtcdLoadB-3RWA2WEFBBT8-2068560477.us-east-2.elb.amazonaws.com2182,internal-etcd-dev-EtcdLoadB-3RWA2WEFBBT8-2068560477.us-east-2.elb.amazonaws.com:2183
>>>>>>>>> nifi.zookeeper.connect.timeout=3 secs
>>>>>>>>> nifi.zookeeper.session.timeout=3 secs
>>>>>>>>> nifi.zookeeper.root.node=/nifi
>>>>>>>>>
>>>>>>>>> # kerberos #
>>>>>>>>> nifi.kerberos.krb5.file=
>>>>>>>>>
>>>>>>>>> # kerberos service principle #
>>>>>>>>> nifi.kerberos.service.principal=
>>>>>>>>> nifi.kerberos.service.keytab.location=
>>>>>>>>>
>>>>>>>>> # kerberos spnego principle #
>>>>>>>>> nifi.kerberos.spnego.principal=
>>>>>>>>> nifi.kerberos.spnego.keytab.location=
>>>>>>>>> nifi.kerberos.spnego.authentication.expiration=12 hours
>>>>>>>>>
>>>>>>>>> # external properties files for variable registry
>>>>>>>>> # supports a comma delimited list of file locations
>>>>>>>>> nifi.variable.registry.properties=
>>>>>>>>>
>>>>>>>>> I think I have everything set correctly but I have not been able to start an
>>>>>>>>> instances up.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> Scott
>>>>>>>>>
>>>>>>>>> On Mar 19, 2018, at 4:35 PM, Bryan Bende <bb...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>> The base file is here for comparison:
>>>>>>>>>
>>>>>>>>> https://github.com/apache/nifi-registry/blob/master/nifi-registry-resources/src/main/resources/conf/identity-providers.xml#L23
>>>>>>>>>
>>>>>>>>> On Mon, Mar 19, 2018 at 5:34 PM, Bryan Bende <bb...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>> For your first file, is what you showed there actually wrapped in
>>>>>>>>> <identityProviders> </identityProviders> or is it exactly what you
>>>>>>>>> showed?
>>>>>>>>>
>>>>>>>>> It may just be that you only copied/pasted the one provider, but the
>>>>>>>>> root element is not <provider>, so as it is shown there it would not
>>>>>>>>> parse.
>>>>>>>>>
>>>>>>>>> On Mon, Mar 19, 2018 at 2:54 PM, Scott Howell <sc...@mobilgov.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Here is my file
>>>>>>>>>
>>>>>>>>> <provider>
>>>>>>>>> <identifier>ldap-identity-provider</identifier>
>>>>>>>>> <class>org.apache.nifi.registry.security.ldap.LdapProvider</class>
>>>>>>>>> <property name="Authentication Strategy">SIMPLE</property>
>>>>>>>>>
>>>>>>>>> <property name="Manager DN">cn=Manager,dc=mobilgov,dc=com</property>
>>>>>>>>> <property name="Manager Password”>redacted</property>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> <property name="Referral Strategy">FOLLOW</property>
>>>>>>>>> <property name="Connect Timeout">10 secs</property>
>>>>>>>>> <property name="Read Timeout">10 secs</property>
>>>>>>>>>
>>>>>>>>> <property name="Url”>redacted</property>
>>>>>>>>> <property name="User Search
>>>>>>>>> Base">ou=users,dc=mobilgov,dc=com</property>
>>>>>>>>> <property name="User Search Filter">uid={0}</property>
>>>>>>>>>
>>>>>>>>> <property name="Identity Strategy">USE_DN</property>
>>>>>>>>> <property name="Authentication Expiration">12 hours</property>
>>>>>>>>> </provider>
>>>>>>>>>
>>>>>>>>> Here is my authorizers.xml
>>>>>>>>>
>>>>>>>>> <authorizers>
>>>>>>>>>
>>>>>>>>> <userGroupProvider>
>>>>>>>>> <identifier>file-user-group-provider</identifier>
>>>>>>>>>
>>>>>>>>> <class>org.apache.nifi.registry.security.authorization.file.FileUserGroupProvider</class>
>>>>>>>>> <property name="Users File">conf/users.xml</property>
>>>>>>>>> <property name="Legacy Authorized Users File"></property>
>>>>>>>>> <property name="Initial User Identity 1”>redacted</property>
>>>>>>>>> </userGroupProvider>
>>>>>>>>>
>>>>>>>>> <accessPolicyProvider>
>>>>>>>>> <identifier>file-access-policy-provider</identifier>
>>>>>>>>>
>>>>>>>>> <class>org.apache.nifi.registry.security.authorization.file.FileAccessPolicyProvider</class>
>>>>>>>>> <property name="User Group
>>>>>>>>> Provider">file-user-group-provider</property>
>>>>>>>>> <property name="Authorizations
>>>>>>>>> File">conf/authorizations.xml</property>
>>>>>>>>> <property name="Initial Admin Identity”>redacted</property>
>>>>>>>>> <property name="NiFi Identity 1"></property>
>>>>>>>>> </accessPolicyProvider>
>>>>>>>>>
>>>>>>>>> <authorizer>
>>>>>>>>> <identifier>managed-authorizer</identifier>
>>>>>>>>>
>>>>>>>>> <class>org.apache.nifi.registry.security.authorization.StandardManagedAuthorizer</class>
>>>>>>>>> <property name="Access Policy
>>>>>>>>> Provider">file-access-policy-provider</property>
>>>>>>>>> </authorizer>
>>>>>>>>> </authorizers>
>>>>>>>>>
>>>>>>>>> On Mar 19, 2018, at 12:59 PM, Bryan Bende <bb...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>> It looks like that error would happen if your identity-providers.xml
>>>>>>>>> contained invalid XML.
>>>>>>>>>
>>>>>>>>> Did you start by modifying the identity-providers.xml file that was
>>>>>>>>> already there? Can you share the file, or the contents (removing
>>>>>>>>> anything sensitive)?
>>>>>>>>>
>>>>>>>>> On Mon, Mar 19, 2018 at 1:09 PM, Scott Howell <sc...@mobilgov.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> So I was able to get the UI pulled up but now I am hitting a roadblock with
>>>>>>>>> my identity-provider.xml.
>>>>>>>>>
>>>>>>>>> I am getting a number of errors like this:
>>>>>>>>>
>>>>>>>>> Caused by: org.springframework.beans.factory.BeanCreationException: Error
>>>>>>>>> creating bean with name 'getIdentityProvider' defined in class path resource
>>>>>>>>> [org/apache/nifi/registry/security/authentication/IdentityProviderFactory.class]:
>>>>>>>>> Bean instantiation via factory method failed; nested exception is
>>>>>>>>> org.springframework.beans.BeanInstantiationException: Failed to instantiate
>>>>>>>>> [org.apache.nifi.registry.security.authentication.IdentityProvider]: Factory
>>>>>>>>> method 'getIdentityProvider' threw exception; nested exception is
>>>>>>>>> java.lang.Exception: Unable to load the login identity provider
>>>>>>>>> configuration file at: /opt/nifi-registry-0.1.0/conf/identity-providers.xml
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:587)
>>>>>>>>> ~[na:na]
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1250)
>>>>>>>>> ~[na:na]
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1099)
>>>>>>>>> ~[na:na]
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:545)
>>>>>>>>> ~[na:na]
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:502)
>>>>>>>>> ~[na:na]
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:312)
>>>>>>>>> ~[na:na]
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228)
>>>>>>>>> ~[na:na]
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:310)
>>>>>>>>> ~[na:na]
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200)
>>>>>>>>> ~[na:na]
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:251)
>>>>>>>>> ~[na:na]
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1135)
>>>>>>>>> ~[na:na]
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1062)
>>>>>>>>> ~[na:na]
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:815)
>>>>>>>>> ~[na:na]
>>>>>>>>> at
>>>>>>>>> org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:721)
>>>>>>>>> ~[na:na]
>>>>>>>>> ... 43 common frames omitted
>>>>>>>>>
>>>>>>>>> I know it has to do with the identity-provider.xml but I have my setup just
>>>>>>>>> like the documentation ask for. I turned on debug but was not able to see
>>>>>>>>> anything different or better explanation from it.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mar 19, 2018, at 10:06 AM, Kevin Doran <kd...@apache.org> wrote:
>>>>>>>>>
>>>>>>>>> Ok, that use case should be fine.
>>>>>>>>>
>>>>>>>>> If it were an authorization issue you would see something in the logs saying
>>>>>>>>> that an authorization attempt failed and the server is responding with a
>>>>>>>>> 403. Just to be sure, can you enable debug logging if you haven't already,
>>>>>>>>> i.e., in your nifi-registry/conf/logback.xml file, change
>>>>>>>>> 'org.apache.nifif.registry' to debug:
>>>>>>>>>
>>>>>>>>> <!-- valid logging levels: TRACE, DEBUG, INFO, WARN, ERROR -->
>>>>>>>>> <logger name="org.apache.nifi.registry" level="DEBUG"/>
>>>>>>>>>
>>>>>>>>> If there is nothing being written to nifi-registry-app.log, it points
>>>>>>>>> towards a connection issue, so I would double check your host, port, and TLS
>>>>>>>>> settings. You'll have to get an HTTPS cert from a root CA or configure your
>>>>>>>>> ELB to trust your company's self-signed cert (again, not sure if/how to do
>>>>>>>>> this, but I assume there should be some way to configure it. It might
>>>>>>>>> require settings not exposed in the AWS web console.)
>>>>>>>>>
>>>>>>>>> On 3/19/18, 10:51, "Scott Howell" <sc...@mobilgov.com> wrote:
>>>>>>>>>
>>>>>>>>> Thanks Kevin,
>>>>>>>>>
>>>>>>>>> I am just using the ELB to go from the public subnet to the private subnet.
>>>>>>>>> I will not have multiple instances running of registry.
>>>>>>>>>
>>>>>>>>> I will say on my authorizers.xml there is one difference between my nifi
>>>>>>>>> instance. On my nifi instance I am using file-provider for
>>>>>>>>> nifi.security.user.authorizer in my nifi.properties. I don’t think from
>>>>>>>>> reading the documents for nifi-registry that I can use that. If there is a
>>>>>>>>> way that might be my problem. I was running into some issues with my nifi
>>>>>>>>> instance when I was using managed-authorizers instead of file-provider.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mar 19, 2018, at 9:35 AM, Kevin Doran <kd...@apache.org> wrote:
>>>>>>>>>
>>>>>>>>> Hey Scott,
>>>>>>>>>
>>>>>>>>> Assuming you are using two-way TLS with client certificates for
>>>>>>>>> authentication, I recommend configuring your ELB for TCP passthrough so that
>>>>>>>>> the TLS handshake is between the end-client and the NiFi Registry Server (in
>>>>>>>>> other words, no decryption/termination of the TLS connection happens in the
>>>>>>>>> ELB). If you are using some other form of authentication (e.g., LDAP), you
>>>>>>>>> will need to configure your ELB to trust the self-signed key NiFi Registry
>>>>>>>>> is using. I'm not sure how to do that as I've never run an ELB with that
>>>>>>>>> configuration before.
>>>>>>>>>
>>>>>>>>> Also, just a note about using an ELB with NiFi Registry:
>>>>>>>>>
>>>>>>>>> NiFi Registry is currently only supports single-instance use as persisted
>>>>>>>>> data and in-memory state is not synced between multiple instances. Are you
>>>>>>>>> hoping to use the ELB for actual load balancing, or is it just to take
>>>>>>>>> advantage of other ELB features, such as forwarding and security group
>>>>>>>>> rules? If the plan is to load balance multiple Registry instances, just be
>>>>>>>>> aware that you will probably run into some unexpected behavior. (As you
>>>>>>>>> mentioned using authorization, that is one case where I know the in-memory
>>>>>>>>> cache of the persisted data will not refresh across instances, so even if
>>>>>>>>> you were using some sort of shared network file system attached to multiple
>>>>>>>>> Registry instances, such as EFS, it would not work the way you hope.)
>>>>>>>>>
>>>>>>>>> Hope this helps,
>>>>>>>>> Kevin
>>>>>>>>>
>>>>>>>>> On 3/19/18, 10:20, "Scott Howell" <sc...@mobilgov.com> wrote:
>>>>>>>>>
>>>>>>>>> Thanks for the quick response.
>>>>>>>>>
>>>>>>>>> A couple of things I am seeing.
>>>>>>>>>
>>>>>>>>> 1. There is no error, I don’t see anything in the logs once the service
>>>>>>>>> comes up. This is because the health check is not even hitting the instance
>>>>>>>>> when secure.
>>>>>>>>>
>>>>>>>>> 2. Nothing interesting in the nifi-registry-app.logs. That was my concern
>>>>>>>>> because on my nifi instance I can see the health check hitting the instance
>>>>>>>>> from the ELB. This does not happen on the nifi-registry instance. I see the
>>>>>>>>> service startup and it tells me what domain and port I can access the UI but
>>>>>>>>> nothing else after that.
>>>>>>>>>
>>>>>>>>> 3. When I am on an instances in the same private subnet I am able to curl to
>>>>>>>>> the instance I get the TLS SSL which tells me the keystore is on the server.
>>>>>>>>> I am using a JKS keystore that is self-signed by the company I work for.
>>>>>>>>>
>>>>>>>>> On Mar 19, 2018, at 9:10 AM, Bryan Bende <bb...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> What error are you getting when you cannot access the UI?
>>>>>>>>>
>>>>>>>>> Is there anything interesting in nifi-registry-app.log regarding
>>>>>>>>> authentication/authorization when this happens?
>>>>>>>>>
>>>>>>>>> Can you access the UI securely without going through the ELB?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> Bryan
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Mar 19, 2018 at 10:05 AM, Scott Howell <sc...@mobilgov.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> I was able to stand up nifi-registry behind an AWS ELB non-secure.
>>>>>>>>> Everything was working great and was able to access the UI anonymously. I
>>>>>>>>> set up the authorization just like on my nifi instances along with the
>>>>>>>>> authorizers and identity-provider. The service comes up without errors and
>>>>>>>>> everything looks good but the health check does not pass and I cannot access
>>>>>>>>> the UI to login. I was wondering if anyone else has ran into this issue
>>>>>>>>> using nifi-registry.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>>
>
Re: Issue with AWS ELB on secure nifi-registry
Posted by Bryan Bende <bb...@gmail.com>.
Ok that looks correct for the 1-node case.
So just to clarify what is working and not working...
With the config in the last email, you have a 1 node cluster that is
working and you can get into the UI?
For the two node case you would need each node to have a users.xml
with users for the two nodes, and an authorizations.xml with two
/proxy policies, one for each node.
So if you do that then your cluster starts up, but when you access the
UI then you get the untrusted proxy?
On Wed, Mar 21, 2018 at 12:58 PM, Scott Howell <sc...@mobilgov.com> wrote:
> user.xml
>
> <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
> <tenants>
> <groups/>
> <users>
> <user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53" identity="uid=scott,ou=users,dc={redacted},dc=com"/>
> <user identifier="7f2b37ff-4d40-316a-843a-92c8523afc0b" identity="CN=*.{redacted}.com, OU={redacted}, O={redacted}, L=Kansas City, ST=Missouri, C=US"/>
> </users>
> </tenants>
>
> Authorizations.xml
> <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
> <authorizations>
> <policies>
> <policy identifier="f99bccd1-a30e-3e4a-98a2-dbc708edc67f" resource="/flow" action="R">
> <user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
> </policy>
> <policy identifier="ef96e849-629c-3f5e-97af-74efe29423bc" resource="/data/process-groups/4505628f-0162-1000-3b39-002cd06f74da" action="R">
> <user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
> <user identifier="7f2b37ff-4d40-316a-843a-92c8523afc0b"/>
> </policy>
> <policy identifier="fc29cd4c-ec37-3820-82b4-bbfd305b85ae" resource="/data/process-groups/4505628f-0162-1000-3b39-002cd06f74da" action="W">
> <user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
> <user identifier="7f2b37ff-4d40-316a-843a-92c8523afc0b"/>
> </policy>
> <policy identifier="75acccef-45ab-3b31-a49b-8cf88186c8bf" resource="/process-groups/4505628f-0162-1000-3b39-002cd06f74da" action="R">
> <user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
> </policy>
> <policy identifier="e9691c14-2540-3544-988a-654b79cf2370" resource="/process-groups/4505628f-0162-1000-3b39-002cd06f74da" action="W">
> <user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
> </policy>
> <policy identifier="b8775bd4-704a-34c6-987b-84f2daf7a515" resource="/restricted-components" action="W">
> <user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
> </policy>
> <policy identifier="627410be-1717-35b4-a06f-e9362b89e0b7" resource="/tenants" action="R">
> <user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
> </policy>
> <policy identifier="15e4e0bd-cb28-34fd-8587-f8d15162cba5" resource="/tenants" action="W">
> <user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
> </policy>
> <policy identifier="ff96062a-fa99-36dc-9942-0f6442ae7212" resource="/policies" action="R">
> <user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
> </policy>
> <policy identifier="ad99ea98-3af6-3561-ae27-5bf09e1d969d" resource="/policies" action="W">
> <user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
> </policy>
> <policy identifier="2e1015cb-0fed-3005-8e0d-722311f21a03" resource="/controller" action="R">
> <user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
> </policy>
> <policy identifier="c6322e6c-4cc1-3bcc-91b3-2ed2111674cf" resource="/controller" action="W">
> <user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
> </policy>
> <policy identifier="287edf48-da72-359b-8f61-da5d4c45a270" resource="/proxy" action="W">
> <user identifier="7f2b37ff-4d40-316a-843a-92c8523afc0b"/>
> </policy>
> </policies>
> </authorizations>
>
>> On Mar 21, 2018, at 11:49 AM, Bryan Bende <bb...@gmail.com> wrote:
>>
>> I've never used wildcard certs before so I'll have to defer to others
>> that might know if there is any issue with that.
>>
>> Could show the contents of these two files just so we can double check
>> the users/policies?
>>
>> <property name="Authorizations File">/opt/config/authorizations.xml</property>
>> <property name="Users File">/opt/config/users.xml</property>
>>
>> On Wed, Mar 21, 2018 at 12:37 PM, Scott Howell <sc...@mobilgov.com> wrote:
>>> Thanks I have checked that and the whitespace is correct in user.xml.
>>>
>>> I did make a change to my authorizer.xml
>>>
>>> <authorizers>
>>> <authorizer>
>>> <identifier>file-provider</identifier>
>>> <class>org.apache.nifi.authorization.FileAuthorizer</class>
>>> <property name="Authorizations File">/opt/config/authorizations.xml</property>
>>> <property name="Users File">/opt/config/users.xml</property>
>>> <property name="Initial Admin Identity">uid=scott,ou=users,dc=mobilgov,dc=com</property>
>>> <property name="Legacy Authorized Users File"></property>
>>>
>>> <property name="Node Identity 1">CN=*.{redacted}.com, OU={redacted}, O={redacted}, L=Kansas City, ST=Missouri, C=US</property>
>>> <property name="Node Identity 2">CN=*.{redacted}.com, OU={redacted}, O={redacted}, L=Kansas City, ST=Missouri, C=US</property>
>>> </authorizer>
>>> </authorizers>
>>>
>>> I made the CN=*.{redacted}.com just like the self-signed certificate showed. I now have a 1 node cluster up and running. It seems like NIfi isn’t taking into account the wildcard and treating it as a “*” instead.
>>>
>>>> On Mar 21, 2018, at 11:33 AM, Bryan Bende <bb...@gmail.com> wrote:
>>>>
>>>> There only needs to be W to /proxy so that part should be fine.
>>>>
>>>> After you edited the Node Identities, did you delete users.xml and
>>>> authorizations.xml?
>>>>
>>>> You would have to do that for those changes to take effect. You can
>>>> look in users.xml and see if you still have the user identities
>>>> without whitespace.
>>>>
>>>> On Wed, Mar 21, 2018 at 12:20 PM, Scott Howell <sc...@mobilgov.com> wrote:
>>>>> One other thing I am seeing and I don’t know if this is an issue or not in my authorizations.xml I do not have a policy for /proxy with action=“R” only action=“W”.
>>>>>
>>>>>> On Mar 21, 2018, at 11:03 AM, Scott Howell <sc...@mobilgov.com> wrote:
>>>>>>
>>>>>> Thanks for that. I am still getting this error in my nifi-user.log
>>>>>>
>>>>>> o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api: Untrusted proxy CN=*.{redacted}.com, OU={redacted}, O={redacted}, L=Kansas City, ST=Missouri, C=US
>>>>>>
>>>>>> Is there an issue with using a wildcard cert?
>>>>>>
>>>>>>
>>>>>>> On Mar 21, 2018, at 10:23 AM, Bryan Bende <bb...@gmail.com> wrote:
>>>>>>>
>>>>>>> All identity strings are case & whitespace sensitive.
>>>>>>>
>>>>>>> The node identities in your authorizers.xml have no whitespace, and
>>>>>>> the identity showing in the logs does.
>>>>>>>
>>>>>>> On Wed, Mar 21, 2018 at 11:05 AM, Scott Howell <sc...@mobilgov.com> wrote:
>>>>>>>> Thanks for all of the help with this. I have the cluster up and running. The
>>>>>>>> logs look great everything seems to be working but I cannot login into the
>>>>>>>> UI. I am using a wildcard self-signed certificate. The /proxy is in
>>>>>>>> authorizations.xml with the correct users for the nodes.
>>>>>>>>
>>>>>>>> The error I see with the UI :
>>>>>>>> is Untrusted proxy CN=*.{redacted}.com, OU={redacted}, O={redacted},
>>>>>>>> L=Kansas City, ST=Missouri, C=US
>>>>>>>>
>>>>>>>> I haven’t had much luck finding a lot of documentation or forum questions
>>>>>>>> with this kind of issue.
>>>>>>>>
>>>>>>>> My authorizers.xml looks like this
>>>>>>>> <authorizers>
>>>>>>>> <authorizer>
>>>>>>>> <identifier>file-provider</identifier>
>>>>>>>> <class>org.apache.nifi.authorization.FileAuthorizer</class>
>>>>>>>> <property name="Authorizations
>>>>>>>> File">/opt/config/authorizations.xml</property>
>>>>>>>> <property name="Users File">/opt/config/users.xml</property>
>>>>>>>> <property name="Initial Admin
>>>>>>>> Identity">uid=scott,ou=users,dc={redacted},dc=com</property>
>>>>>>>> <property name="Legacy Authorized Users File"></property>
>>>>>>>>
>>>>>>>> <property name="Node Identity
>>>>>>>> 1">CN=node-1-nifi-dev.{redacted}.com,OU={redacted},O={redacted},L=Kansas
>>>>>>>> City,ST=Missouri,C=US</property>
>>>>>>>> <property name="Node Identity
>>>>>>>> 2">CN=node-2-nifi-dev.{redacted}.com,OU={redacted},O={redacted},L=Kansas
>>>>>>>> City,ST=Missouri,C=US</property>
>>>>>>>> </authorizer>
>>>>>>>> </authorizers>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Scott
>>>>>>>>
>>>>>>>> On Mar 20, 2018, at 1:15 PM, Andy LoPresto <al...@apache.org> wrote:
>>>>>>>>
>>>>>>>> Scott,
>>>>>>>>
>>>>>>>> The original exception is "nested exception is
>>>>>>>> java.security.KeyStoreException: not found”. Can you verify that the
>>>>>>>> keystore you’ve provided is valid using the “keytool” command? In addition,
>>>>>>>> you will need a truststore as well. Try following Pierre's [1] or Bryan’s
>>>>>>>> [2] instructions for setting up a secure cluster.
>>>>>>>>
>>>>>>>> [1]
>>>>>>>> https://pierrevillard.com/2016/11/29/apache-nifi-1-1-0-secured-cluster-setup/
>>>>>>>> [2]
>>>>>>>> https://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-authorization-and-multi-tenancy
>>>>>>>>
>>>>>>>>
>>>>>>>> Andy LoPresto
>>>>>>>> alopresto@apache.org
>>>>>>>> alopresto.apache@gmail.com
>>>>>>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>>>>>>>>
>>>>>>>> On Mar 20, 2018, at 11:05 AM, Scott Howell <sc...@mobilgov.com> wrote:
>>>>>>>>
>>>>>>>> Thanks for all of the help yesterday I was able to get a secure nifi and
>>>>>>>> nifi-registry up and communicating. I am now trying to figure out how to
>>>>>>>> create a secure cluster. I am currently getting this error when I start up
>>>>>>>> nifi.
>>>>>>>>
>>>>>>>> tion; nested exception is
>>>>>>>> org.springframework.beans.factory.BeanCreationException: Error creating bean
>>>>>>>> with name 'clusterCoordinationProtocolSenderListener' defined in class path
>>>>>>>> resource [nifi-cluster-protocol-context.xml]: Cannot resolve reference to
>>>>>>>> bean 'clusterCoordinationProtocolSender' while setting constructor argument;
>>>>>>>> nested exception is org.springframework.beans.factory.BeanCreationException:
>>>>>>>> Error creating bean with name 'clusterCoordinationProtocolSender' defined in
>>>>>>>> class path resource [nifi-cluster-protocol-context.xml]: Cannot resolve
>>>>>>>> reference to bean 'protocolSocketConfiguration' while setting constructor
>>>>>>>> argument; nested exception is
>>>>>>>> org.springframework.beans.factory.BeanCreationException: Error creating bean
>>>>>>>> with name 'protocolSocketConfiguration': FactoryBean threw exception on
>>>>>>>> object creation; nested exception is java.security.KeyStoreException: not
>>>>>>>> found
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175)
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1634)
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317)
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
>>>>>>>> ... 50 common frames omitted
>>>>>>>> Caused by: org.springframework.beans.factory.BeanCreationException: Error
>>>>>>>> creating bean with name 'clusterCoordinationProtocolSenderListener' defined
>>>>>>>> in class path resource [nifi-cluster-protocol-context.xml]: Cannot resolve
>>>>>>>> reference to bean 'clusterCoordinationProtocolSender' while setting
>>>>>>>> constructor argument; nested exception is
>>>>>>>> org.springframework.beans.factory.BeanCreationException: Error creating bean
>>>>>>>> with name 'clusterCoordinationProtocolSender' defined in class path resource
>>>>>>>> [nifi-cluster-protocol-context.xml]: Cannot resolve reference to bean
>>>>>>>> 'protocolSocketConfiguration' while setting constructor argument; nested
>>>>>>>> exception is org.springframework.beans.factory.BeanCreationException: Error
>>>>>>>> creating bean with name 'protocolSocketConfiguration': FactoryBean threw
>>>>>>>> exception on object creation; nested exception is
>>>>>>>> java.security.KeyStoreException: not found
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:359)
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:648)
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:145)
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1193)
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1095)
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
>>>>>>>> at
>>>>>>>> org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1084)
>>>>>>>> at
>>>>>>>> org.apache.nifi.cluster.spring.NodeClusterCoordinatorFactoryBean.getObject(NodeClusterCoordinatorFactoryBean.java:44)
>>>>>>>> at
>>>>>>>> org.apache.nifi.cluster.spring.NodeClusterCoordinatorFactoryBean.getObject(NodeClusterCoordinatorFactoryBean.java:34)
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
>>>>>>>> ... 55 common frames omitted
>>>>>>>> Caused by: org.springframework.beans.factory.BeanCreationException: Error
>>>>>>>> creating bean with name 'clusterCoordinationProtocolSender' defined in class
>>>>>>>> path resource [nifi-cluster-protocol-context.xml]: Cannot resolve reference
>>>>>>>> to bean 'protocolSocketConfiguration' while setting constructor argument;
>>>>>>>> nested exception is org.springframework.beans.factory.BeanCreationException:
>>>>>>>> Error creating bean with name 'protocolSocketConfiguration': FactoryBean
>>>>>>>> threw exception on object creation; nested exception is
>>>>>>>> java.security.KeyStoreException: not found
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:359)
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:648)
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:145)
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1193)
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1095)
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
>>>>>>>> ... 70 common frames omitted
>>>>>>>> Caused by: org.springframework.beans.factory.BeanCreationException: Error
>>>>>>>> creating bean with name 'protocolSocketConfiguration': FactoryBean threw
>>>>>>>> exception on object creation; nested exception is
>>>>>>>> java.security.KeyStoreException: not found
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175)
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1634)
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317)
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
>>>>>>>> ... 82 common frames omitted
>>>>>>>> Caused by: java.security.KeyStoreException: not found
>>>>>>>> at java.security.KeyStore.getInstance(KeyStore.java:851)
>>>>>>>> at
>>>>>>>> org.apache.nifi.security.util.KeyStoreUtils.getKeyStore(KeyStoreUtils.java:66)
>>>>>>>> at
>>>>>>>> org.apache.nifi.security.util.KeyStoreUtils.getTrustStore(KeyStoreUtils.java:80)
>>>>>>>> at
>>>>>>>> org.apache.nifi.io.socket.SSLContextFactory.<init>(SSLContextFactory.java:73)
>>>>>>>> at
>>>>>>>> org.apache.nifi.cluster.protocol.spring.SocketConfigurationFactoryBean.getObject(SocketConfigurationFactoryBean.java:45)
>>>>>>>> at
>>>>>>>> org.apache.nifi.cluster.protocol.spring.SocketConfigurationFactoryBean.getObject(SocketConfigurationFactoryBean.java:30)
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
>>>>>>>> ... 87 common frames omitted
>>>>>>>> Caused by: java.security.NoSuchAlgorithmException: KeyStore not available
>>>>>>>> at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
>>>>>>>> at java.security.Security.getImpl(Security.java:695)
>>>>>>>> at java.security.KeyStore.getInstance(KeyStore.java:848)
>>>>>>>> ... 93 common frames omitted
>>>>>>>>
>>>>>>>> My nifi.properties file is.
>>>>>>>>
>>>>>>>> # Licensed to the Apache Software Foundation (ASF) under one or more
>>>>>>>> # contributor license agreements. See the NOTICE file distributed with
>>>>>>>> # this work for additional information regarding copyright ownership.
>>>>>>>> # The ASF licenses this file to You under the Apache License, Version 2.0
>>>>>>>> # (the "License"); you may not use this file except in compliance with
>>>>>>>> # the License. You may obtain a copy of the License at
>>>>>>>> #
>>>>>>>> # http://www.apache.org/licenses/LICENSE-2.0
>>>>>>>> #
>>>>>>>> # Unless required by applicable law or agreed to in writing, software
>>>>>>>> # distributed under the License is distributed on an "AS IS" BASIS,
>>>>>>>> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>>>>>>>> # See the License for the specific language governing permissions and
>>>>>>>> # limitations under the License.
>>>>>>>>
>>>>>>>> # Core Properties #
>>>>>>>> nifi.version={{nifi_version}}
>>>>>>>> nifi.flow.configuration.file=/opt/config/flow.xml.gz
>>>>>>>> nifi.flow.configuration.archive.enabled=true
>>>>>>>> nifi.flow.configuration.archive.dir=/opt/config/archive/
>>>>>>>> nifi.flow.configuration.archive.max.time=30 days
>>>>>>>> nifi.flow.configuration.archive.max.storage=500 MB
>>>>>>>> nifi.flowcontroller.autoResumeState=true
>>>>>>>> nifi.flowcontroller.graceful.shutdown.period=10 sec
>>>>>>>> nifi.flowservice.writedelay.interval=500 ms
>>>>>>>> nifi.administrative.yield.duration=30 sec
>>>>>>>> # If a component has no work to do (is "bored"), how long should we wait
>>>>>>>> before checking again for work?
>>>>>>>> nifi.bored.yield.duration=10 millis
>>>>>>>>
>>>>>>>>
>>>>>>>> nifi.authorizer.configuration.file=/opt/config/authorizers.xml
>>>>>>>> nifi.login.identity.provider.configuration.file=/opt/config/login-identity-providers.xml
>>>>>>>> nifi.templates.directory=/opt/config/templates
>>>>>>>> nifi.ui.banner.text=
>>>>>>>> nifi.ui.autorefresh.interval=30 sec
>>>>>>>> nifi.nar.library.directory=/opt/nifi/lib
>>>>>>>> nifi.nar.library.directory.custom=/opt/config/processors
>>>>>>>> nifi.nar.working.directory=/opt/nifi/work/nar/
>>>>>>>> nifi.documentation.working.directory=./work/docs/components
>>>>>>>>
>>>>>>>> ####################
>>>>>>>> # State Management #
>>>>>>>> ####################
>>>>>>>> nifi.state.management.configuration.file=/opt/config/state-management.xml
>>>>>>>> # The ID of the local state provider
>>>>>>>> nifi.state.management.provider.local=local-provider
>>>>>>>> # The ID of the cluster-wide state provider. This will be ignored if NiFi is
>>>>>>>> not clustered but must be populated if running in a cluster.
>>>>>>>> nifi.state.management.provider.cluster=zk-provider
>>>>>>>> # Specifies whether or not this instance of NiFi should run an embedded
>>>>>>>> ZooKeeper server
>>>>>>>> nifi.state.management.embedded.zookeeper.start=false
>>>>>>>> # Properties file that provides the ZooKeeper properties to use if
>>>>>>>> <nifi.state.management.embedded.zookeeper.start> is set to true
>>>>>>>> nifi.state.management.embedded.zookeeper.properties=/opt/config/zookeeper.properties
>>>>>>>>
>>>>>>>>
>>>>>>>> # H2 Settings
>>>>>>>> nifi.database.directory=/opt/database_repository
>>>>>>>> nifi.h2.url.append=;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE
>>>>>>>>
>>>>>>>> # FlowFile Repository
>>>>>>>> nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository
>>>>>>>> nifi.flowfile.repository.directory=/opt/flowfile_repository
>>>>>>>> nifi.flowfile.repository.partitions=256
>>>>>>>> nifi.flowfile.repository.checkpoint.interval=2 mins
>>>>>>>> nifi.flowfile.repository.always.sync=false
>>>>>>>>
>>>>>>>> nifi.swap.manager.implementation=org.apache.nifi.controller.FileSystemSwapManager
>>>>>>>> nifi.queue.swap.threshold=20000
>>>>>>>> nifi.swap.in.period=5 sec
>>>>>>>> nifi.swap.in.threads=1
>>>>>>>> nifi.swap.out.period=5 sec
>>>>>>>> nifi.swap.out.threads=4
>>>>>>>>
>>>>>>>> # Content Repository
>>>>>>>> nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository
>>>>>>>> nifi.content.claim.max.appendable.size=10 MB
>>>>>>>> nifi.content.claim.max.flow.files=100
>>>>>>>> nifi.content.repository.directory.default=/opt/content_repository
>>>>>>>> nifi.content.repository.archive.max.retention.period=12 hours
>>>>>>>> nifi.content.repository.archive.max.usage.percentage=50%
>>>>>>>> nifi.content.repository.archive.enabled=true
>>>>>>>> nifi.content.repository.always.sync=false
>>>>>>>> nifi.content.viewer.url=/nifi-content-viewer/
>>>>>>>>
>>>>>>>> # Provenance Repository Properties
>>>>>>>> nifi.provenance.repository.implementation=org.apache.nifi.provenance.PersistentProvenanceRepository
>>>>>>>>
>>>>>>>> # Persistent Provenance Repository Properties
>>>>>>>> nifi.provenance.repository.directory.default=/opt/provenance_repository
>>>>>>>> nifi.provenance.repository.max.storage.time=24 hours
>>>>>>>> nifi.provenance.repository.max.storage.size=1 GB
>>>>>>>> nifi.provenance.repository.rollover.time=30 secs
>>>>>>>> nifi.provenance.repository.rollover.size=100 MB
>>>>>>>> nifi.provenance.repository.query.threads=2
>>>>>>>> nifi.provenance.repository.index.threads=1
>>>>>>>> nifi.provenance.repository.compress.on.rollover=true
>>>>>>>> nifi.provenance.repository.always.sync=false
>>>>>>>> nifi.provenance.repository.journal.count=16
>>>>>>>> # Comma-separated list of fields. Fields that are not indexed will not be
>>>>>>>> searchable. Valid fields are:
>>>>>>>> # EventType, FlowFileUUID, Filename, TransitURI, ProcessorID,
>>>>>>>> AlternateIdentifierURI, Relationship, Details
>>>>>>>> nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename,
>>>>>>>> ProcessorID, Relationship
>>>>>>>> # FlowFile Attributes that should be indexed and made searchable. Some
>>>>>>>> examples to consider are filename, uuid, mime.type
>>>>>>>> nifi.provenance.repository.indexed.attributes=
>>>>>>>> # Large values for the shard size will result in more Java heap usage when
>>>>>>>> searching the Provenance Repository
>>>>>>>> # but should provide better performance
>>>>>>>> nifi.provenance.repository.index.shard.size=500 MB
>>>>>>>> # Indicates the maximum length that a FlowFile attribute can be when
>>>>>>>> retrieving a Provenance Event from
>>>>>>>> # the repository. If the length of any attribute exceeds this value, it will
>>>>>>>> be truncated when the event is retrieved.
>>>>>>>> nifi.provenance.repository.max.attribute.length=65536
>>>>>>>>
>>>>>>>> # Volatile Provenance Respository Properties
>>>>>>>> nifi.provenance.repository.buffer.size=100000
>>>>>>>>
>>>>>>>> # Component Status Repository
>>>>>>>> nifi.components.status.repository.implementation=org.apache.nifi.controller.status.history.VolatileComponentStatusRepository
>>>>>>>> nifi.components.status.repository.buffer.size=1440
>>>>>>>> nifi.components.status.snapshot.frequency=1 min
>>>>>>>>
>>>>>>>> # Site to Site properties
>>>>>>>> nifi.remote.input.host=
>>>>>>>> nifi.remote.input.secure=false
>>>>>>>> nifi.remote.input.socket.port=9998
>>>>>>>> nifi.remote.input.http.enabled=false
>>>>>>>> nifi.remote.input.http.transaction.ttl=30 sec
>>>>>>>>
>>>>>>>> # web properties #
>>>>>>>> nifi.web.war.directory=/opt/nifi/lib
>>>>>>>> nifi.web.http.host=
>>>>>>>> nifi.web.http.port=
>>>>>>>> nifi.web.https.host={{redacted}}
>>>>>>>> nifi.web.https.port=8443
>>>>>>>> nifi.web.jetty.working.directory=/opt/nifi/work/jetty
>>>>>>>> nifi.web.jetty.threads=200
>>>>>>>>
>>>>>>>> # security properties #
>>>>>>>> nifi.sensitive.props.key=x0KDgO9L8lAhFGLdvu2VEjFVGc6Kg3V0R5I4bYwoqdgL47moo0wApKQtAVu1BvD
>>>>>>>> nifi.sensitive.props.key.protected=
>>>>>>>> nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
>>>>>>>> nifi.sensitive.props.provider=BC
>>>>>>>> nifi.sensitive.props.additional.keys=
>>>>>>>>
>>>>>>>> nifi.security.keystore=/opt/certs/payit_keystore
>>>>>>>> nifi.security.keystoreType=JKS
>>>>>>>> nifi.security.keystorePasswd={{keystore_password}}
>>>>>>>> nifi.security.keyPasswd=
>>>>>>>> nifi.security.truststore=
>>>>>>>> nifi.security.truststoreType=
>>>>>>>> nifi.security.truststorePasswd=
>>>>>>>> nifi.security.needClientAuth=false
>>>>>>>> nifi.security.user.authorizer=file-provider
>>>>>>>> nifi.security.user.login.identity.provider=ldap-provider
>>>>>>>> nifi.security.ocsp.responder.url=
>>>>>>>> nifi.security.ocsp.responder.certificate=
>>>>>>>>
>>>>>>>> # Identity Mapping Properties #
>>>>>>>> # These properties allow normalizing user identities such that identities
>>>>>>>> coming from different identity providers
>>>>>>>> # (certificates, LDAP, Kerberos) can be treated the same internally in NiFi.
>>>>>>>> The following example demonstrates normalizing
>>>>>>>> # DNs from certificates and principals from Kerberos into a common identity
>>>>>>>> string:
>>>>>>>> #
>>>>>>>> #nifi.security.identity.mapping.pattern.dn=^cn=(.*?),ou=(.*?),dc=(.*?),dc=(.*?),dc=(.*?)$
>>>>>>>> #nifi.security.identity.mapping.value.dn=$1
>>>>>>>> # nifi.security.identity.mapping.pattern.kerb=^(.*?)/instance@(.*?)$
>>>>>>>> # nifi.security.identity.mapping.value.kerb=$1@$2
>>>>>>>>
>>>>>>>> # cluster common properties (all nodes must have same values) #
>>>>>>>> nifi.cluster.protocol.heartbeat.interval=5 sec
>>>>>>>> nifi.cluster.protocol.is.secure=true
>>>>>>>>
>>>>>>>> # cluster node properties (only configure for cluster nodes) #
>>>>>>>> nifi.cluster.is.node=true
>>>>>>>> nifi.cluster.node.address=nifi-dev.mobilgov.com
>>>>>>>> nifi.cluster.node.protocol.port=9999
>>>>>>>> nifi.cluster.node.protocol.threads=10
>>>>>>>> nifi.cluster.node.event.history.size=25
>>>>>>>> nifi.cluster.node.connection.timeout=5 sec
>>>>>>>> nifi.cluster.node.read.timeout=5 sec
>>>>>>>> nifi.cluster.firewall.file=
>>>>>>>>
>>>>>>>>
>>>>>>>> # zookeeper properties, used for cluster management #
>>>>>>>> nifi.zookeeper.connect.string=internal-etcd-dev-EtcdLoadB-3RWA2WEFBBT8-2068560477.us-east-2.elb.amazonaws.com:2181,internal-etcd-dev-EtcdLoadB-3RWA2WEFBBT8-2068560477.us-east-2.elb.amazonaws.com2182,internal-etcd-dev-EtcdLoadB-3RWA2WEFBBT8-2068560477.us-east-2.elb.amazonaws.com:2183
>>>>>>>> nifi.zookeeper.connect.timeout=3 secs
>>>>>>>> nifi.zookeeper.session.timeout=3 secs
>>>>>>>> nifi.zookeeper.root.node=/nifi
>>>>>>>>
>>>>>>>> # kerberos #
>>>>>>>> nifi.kerberos.krb5.file=
>>>>>>>>
>>>>>>>> # kerberos service principle #
>>>>>>>> nifi.kerberos.service.principal=
>>>>>>>> nifi.kerberos.service.keytab.location=
>>>>>>>>
>>>>>>>> # kerberos spnego principle #
>>>>>>>> nifi.kerberos.spnego.principal=
>>>>>>>> nifi.kerberos.spnego.keytab.location=
>>>>>>>> nifi.kerberos.spnego.authentication.expiration=12 hours
>>>>>>>>
>>>>>>>> # external properties files for variable registry
>>>>>>>> # supports a comma delimited list of file locations
>>>>>>>> nifi.variable.registry.properties=
>>>>>>>>
>>>>>>>> I think I have everything set correctly but I have not been able to start an
>>>>>>>> instances up.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Scott
>>>>>>>>
>>>>>>>> On Mar 19, 2018, at 4:35 PM, Bryan Bende <bb...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> The base file is here for comparison:
>>>>>>>>
>>>>>>>> https://github.com/apache/nifi-registry/blob/master/nifi-registry-resources/src/main/resources/conf/identity-providers.xml#L23
>>>>>>>>
>>>>>>>> On Mon, Mar 19, 2018 at 5:34 PM, Bryan Bende <bb...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> For your first file, is what you showed there actually wrapped in
>>>>>>>> <identityProviders> </identityProviders> or is it exactly what you
>>>>>>>> showed?
>>>>>>>>
>>>>>>>> It may just be that you only copied/pasted the one provider, but the
>>>>>>>> root element is not <provider>, so as it is shown there it would not
>>>>>>>> parse.
>>>>>>>>
>>>>>>>> On Mon, Mar 19, 2018 at 2:54 PM, Scott Howell <sc...@mobilgov.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Here is my file
>>>>>>>>
>>>>>>>> <provider>
>>>>>>>> <identifier>ldap-identity-provider</identifier>
>>>>>>>> <class>org.apache.nifi.registry.security.ldap.LdapProvider</class>
>>>>>>>> <property name="Authentication Strategy">SIMPLE</property>
>>>>>>>>
>>>>>>>> <property name="Manager DN">cn=Manager,dc=mobilgov,dc=com</property>
>>>>>>>> <property name="Manager Password”>redacted</property>
>>>>>>>>
>>>>>>>>
>>>>>>>> <property name="Referral Strategy">FOLLOW</property>
>>>>>>>> <property name="Connect Timeout">10 secs</property>
>>>>>>>> <property name="Read Timeout">10 secs</property>
>>>>>>>>
>>>>>>>> <property name="Url”>redacted</property>
>>>>>>>> <property name="User Search
>>>>>>>> Base">ou=users,dc=mobilgov,dc=com</property>
>>>>>>>> <property name="User Search Filter">uid={0}</property>
>>>>>>>>
>>>>>>>> <property name="Identity Strategy">USE_DN</property>
>>>>>>>> <property name="Authentication Expiration">12 hours</property>
>>>>>>>> </provider>
>>>>>>>>
>>>>>>>> Here is my authorizers.xml
>>>>>>>>
>>>>>>>> <authorizers>
>>>>>>>>
>>>>>>>> <userGroupProvider>
>>>>>>>> <identifier>file-user-group-provider</identifier>
>>>>>>>>
>>>>>>>> <class>org.apache.nifi.registry.security.authorization.file.FileUserGroupProvider</class>
>>>>>>>> <property name="Users File">conf/users.xml</property>
>>>>>>>> <property name="Legacy Authorized Users File"></property>
>>>>>>>> <property name="Initial User Identity 1”>redacted</property>
>>>>>>>> </userGroupProvider>
>>>>>>>>
>>>>>>>> <accessPolicyProvider>
>>>>>>>> <identifier>file-access-policy-provider</identifier>
>>>>>>>>
>>>>>>>> <class>org.apache.nifi.registry.security.authorization.file.FileAccessPolicyProvider</class>
>>>>>>>> <property name="User Group
>>>>>>>> Provider">file-user-group-provider</property>
>>>>>>>> <property name="Authorizations
>>>>>>>> File">conf/authorizations.xml</property>
>>>>>>>> <property name="Initial Admin Identity”>redacted</property>
>>>>>>>> <property name="NiFi Identity 1"></property>
>>>>>>>> </accessPolicyProvider>
>>>>>>>>
>>>>>>>> <authorizer>
>>>>>>>> <identifier>managed-authorizer</identifier>
>>>>>>>>
>>>>>>>> <class>org.apache.nifi.registry.security.authorization.StandardManagedAuthorizer</class>
>>>>>>>> <property name="Access Policy
>>>>>>>> Provider">file-access-policy-provider</property>
>>>>>>>> </authorizer>
>>>>>>>> </authorizers>
>>>>>>>>
>>>>>>>> On Mar 19, 2018, at 12:59 PM, Bryan Bende <bb...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> It looks like that error would happen if your identity-providers.xml
>>>>>>>> contained invalid XML.
>>>>>>>>
>>>>>>>> Did you start by modifying the identity-providers.xml file that was
>>>>>>>> already there? Can you share the file, or the contents (removing
>>>>>>>> anything sensitive)?
>>>>>>>>
>>>>>>>> On Mon, Mar 19, 2018 at 1:09 PM, Scott Howell <sc...@mobilgov.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> So I was able to get the UI pulled up but now I am hitting a roadblock with
>>>>>>>> my identity-provider.xml.
>>>>>>>>
>>>>>>>> I am getting a number of errors like this:
>>>>>>>>
>>>>>>>> Caused by: org.springframework.beans.factory.BeanCreationException: Error
>>>>>>>> creating bean with name 'getIdentityProvider' defined in class path resource
>>>>>>>> [org/apache/nifi/registry/security/authentication/IdentityProviderFactory.class]:
>>>>>>>> Bean instantiation via factory method failed; nested exception is
>>>>>>>> org.springframework.beans.BeanInstantiationException: Failed to instantiate
>>>>>>>> [org.apache.nifi.registry.security.authentication.IdentityProvider]: Factory
>>>>>>>> method 'getIdentityProvider' threw exception; nested exception is
>>>>>>>> java.lang.Exception: Unable to load the login identity provider
>>>>>>>> configuration file at: /opt/nifi-registry-0.1.0/conf/identity-providers.xml
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:587)
>>>>>>>> ~[na:na]
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1250)
>>>>>>>> ~[na:na]
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1099)
>>>>>>>> ~[na:na]
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:545)
>>>>>>>> ~[na:na]
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:502)
>>>>>>>> ~[na:na]
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:312)
>>>>>>>> ~[na:na]
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228)
>>>>>>>> ~[na:na]
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:310)
>>>>>>>> ~[na:na]
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200)
>>>>>>>> ~[na:na]
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:251)
>>>>>>>> ~[na:na]
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1135)
>>>>>>>> ~[na:na]
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1062)
>>>>>>>> ~[na:na]
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:815)
>>>>>>>> ~[na:na]
>>>>>>>> at
>>>>>>>> org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:721)
>>>>>>>> ~[na:na]
>>>>>>>> ... 43 common frames omitted
>>>>>>>>
>>>>>>>> I know it has to do with the identity-provider.xml but I have my setup just
>>>>>>>> like the documentation ask for. I turned on debug but was not able to see
>>>>>>>> anything different or better explanation from it.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mar 19, 2018, at 10:06 AM, Kevin Doran <kd...@apache.org> wrote:
>>>>>>>>
>>>>>>>> Ok, that use case should be fine.
>>>>>>>>
>>>>>>>> If it were an authorization issue you would see something in the logs saying
>>>>>>>> that an authorization attempt failed and the server is responding with a
>>>>>>>> 403. Just to be sure, can you enable debug logging if you haven't already,
>>>>>>>> i.e., in your nifi-registry/conf/logback.xml file, change
>>>>>>>> 'org.apache.nifif.registry' to debug:
>>>>>>>>
>>>>>>>> <!-- valid logging levels: TRACE, DEBUG, INFO, WARN, ERROR -->
>>>>>>>> <logger name="org.apache.nifi.registry" level="DEBUG"/>
>>>>>>>>
>>>>>>>> If there is nothing being written to nifi-registry-app.log, it points
>>>>>>>> towards a connection issue, so I would double check your host, port, and TLS
>>>>>>>> settings. You'll have to get an HTTPS cert from a root CA or configure your
>>>>>>>> ELB to trust your company's self-signed cert (again, not sure if/how to do
>>>>>>>> this, but I assume there should be some way to configure it. It might
>>>>>>>> require settings not exposed in the AWS web console.)
>>>>>>>>
>>>>>>>> On 3/19/18, 10:51, "Scott Howell" <sc...@mobilgov.com> wrote:
>>>>>>>>
>>>>>>>> Thanks Kevin,
>>>>>>>>
>>>>>>>> I am just using the ELB to go from the public subnet to the private subnet.
>>>>>>>> I will not have multiple instances running of registry.
>>>>>>>>
>>>>>>>> I will say on my authorizers.xml there is one difference between my nifi
>>>>>>>> instance. On my nifi instance I am using file-provider for
>>>>>>>> nifi.security.user.authorizer in my nifi.properties. I don’t think from
>>>>>>>> reading the documents for nifi-registry that I can use that. If there is a
>>>>>>>> way that might be my problem. I was running into some issues with my nifi
>>>>>>>> instance when I was using managed-authorizers instead of file-provider.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mar 19, 2018, at 9:35 AM, Kevin Doran <kd...@apache.org> wrote:
>>>>>>>>
>>>>>>>> Hey Scott,
>>>>>>>>
>>>>>>>> Assuming you are using two-way TLS with client certificates for
>>>>>>>> authentication, I recommend configuring your ELB for TCP passthrough so that
>>>>>>>> the TLS handshake is between the end-client and the NiFi Registry Server (in
>>>>>>>> other words, no decryption/termination of the TLS connection happens in the
>>>>>>>> ELB). If you are using some other form of authentication (e.g., LDAP), you
>>>>>>>> will need to configure your ELB to trust the self-signed key NiFi Registry
>>>>>>>> is using. I'm not sure how to do that as I've never run an ELB with that
>>>>>>>> configuration before.
>>>>>>>>
>>>>>>>> Also, just a note about using an ELB with NiFi Registry:
>>>>>>>>
>>>>>>>> NiFi Registry is currently only supports single-instance use as persisted
>>>>>>>> data and in-memory state is not synced between multiple instances. Are you
>>>>>>>> hoping to use the ELB for actual load balancing, or is it just to take
>>>>>>>> advantage of other ELB features, such as forwarding and security group
>>>>>>>> rules? If the plan is to load balance multiple Registry instances, just be
>>>>>>>> aware that you will probably run into some unexpected behavior. (As you
>>>>>>>> mentioned using authorization, that is one case where I know the in-memory
>>>>>>>> cache of the persisted data will not refresh across instances, so even if
>>>>>>>> you were using some sort of shared network file system attached to multiple
>>>>>>>> Registry instances, such as EFS, it would not work the way you hope.)
>>>>>>>>
>>>>>>>> Hope this helps,
>>>>>>>> Kevin
>>>>>>>>
>>>>>>>> On 3/19/18, 10:20, "Scott Howell" <sc...@mobilgov.com> wrote:
>>>>>>>>
>>>>>>>> Thanks for the quick response.
>>>>>>>>
>>>>>>>> A couple of things I am seeing.
>>>>>>>>
>>>>>>>> 1. There is no error, I don’t see anything in the logs once the service
>>>>>>>> comes up. This is because the health check is not even hitting the instance
>>>>>>>> when secure.
>>>>>>>>
>>>>>>>> 2. Nothing interesting in the nifi-registry-app.logs. That was my concern
>>>>>>>> because on my nifi instance I can see the health check hitting the instance
>>>>>>>> from the ELB. This does not happen on the nifi-registry instance. I see the
>>>>>>>> service startup and it tells me what domain and port I can access the UI but
>>>>>>>> nothing else after that.
>>>>>>>>
>>>>>>>> 3. When I am on an instances in the same private subnet I am able to curl to
>>>>>>>> the instance I get the TLS SSL which tells me the keystore is on the server.
>>>>>>>> I am using a JKS keystore that is self-signed by the company I work for.
>>>>>>>>
>>>>>>>> On Mar 19, 2018, at 9:10 AM, Bryan Bende <bb...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> What error are you getting when you cannot access the UI?
>>>>>>>>
>>>>>>>> Is there anything interesting in nifi-registry-app.log regarding
>>>>>>>> authentication/authorization when this happens?
>>>>>>>>
>>>>>>>> Can you access the UI securely without going through the ELB?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Bryan
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Mar 19, 2018 at 10:05 AM, Scott Howell <sc...@mobilgov.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> I was able to stand up nifi-registry behind an AWS ELB non-secure.
>>>>>>>> Everything was working great and was able to access the UI anonymously. I
>>>>>>>> set up the authorization just like on my nifi instances along with the
>>>>>>>> authorizers and identity-provider. The service comes up without errors and
>>>>>>>> everything looks good but the health check does not pass and I cannot access
>>>>>>>> the UI to login. I was wondering if anyone else has ran into this issue
>>>>>>>> using nifi-registry.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>
>
>
Re: Issue with AWS ELB on secure nifi-registry
Posted by Scott Howell <sc...@mobilgov.com>.
user.xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<tenants>
<groups/>
<users>
<user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53" identity="uid=scott,ou=users,dc={redacted},dc=com"/>
<user identifier="7f2b37ff-4d40-316a-843a-92c8523afc0b" identity="CN=*.{redacted}.com, OU={redacted}, O={redacted}, L=Kansas City, ST=Missouri, C=US"/>
</users>
</tenants>
Authorizations.xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<authorizations>
<policies>
<policy identifier="f99bccd1-a30e-3e4a-98a2-dbc708edc67f" resource="/flow" action="R">
<user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
</policy>
<policy identifier="ef96e849-629c-3f5e-97af-74efe29423bc" resource="/data/process-groups/4505628f-0162-1000-3b39-002cd06f74da" action="R">
<user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
<user identifier="7f2b37ff-4d40-316a-843a-92c8523afc0b"/>
</policy>
<policy identifier="fc29cd4c-ec37-3820-82b4-bbfd305b85ae" resource="/data/process-groups/4505628f-0162-1000-3b39-002cd06f74da" action="W">
<user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
<user identifier="7f2b37ff-4d40-316a-843a-92c8523afc0b"/>
</policy>
<policy identifier="75acccef-45ab-3b31-a49b-8cf88186c8bf" resource="/process-groups/4505628f-0162-1000-3b39-002cd06f74da" action="R">
<user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
</policy>
<policy identifier="e9691c14-2540-3544-988a-654b79cf2370" resource="/process-groups/4505628f-0162-1000-3b39-002cd06f74da" action="W">
<user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
</policy>
<policy identifier="b8775bd4-704a-34c6-987b-84f2daf7a515" resource="/restricted-components" action="W">
<user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
</policy>
<policy identifier="627410be-1717-35b4-a06f-e9362b89e0b7" resource="/tenants" action="R">
<user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
</policy>
<policy identifier="15e4e0bd-cb28-34fd-8587-f8d15162cba5" resource="/tenants" action="W">
<user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
</policy>
<policy identifier="ff96062a-fa99-36dc-9942-0f6442ae7212" resource="/policies" action="R">
<user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
</policy>
<policy identifier="ad99ea98-3af6-3561-ae27-5bf09e1d969d" resource="/policies" action="W">
<user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
</policy>
<policy identifier="2e1015cb-0fed-3005-8e0d-722311f21a03" resource="/controller" action="R">
<user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
</policy>
<policy identifier="c6322e6c-4cc1-3bcc-91b3-2ed2111674cf" resource="/controller" action="W">
<user identifier="4e9a2753-85a0-3c8e-96bf-6d5ef821fe53"/>
</policy>
<policy identifier="287edf48-da72-359b-8f61-da5d4c45a270" resource="/proxy" action="W">
<user identifier="7f2b37ff-4d40-316a-843a-92c8523afc0b"/>
</policy>
</policies>
</authorizations>
> On Mar 21, 2018, at 11:49 AM, Bryan Bende <bb...@gmail.com> wrote:
>
> I've never used wildcard certs before so I'll have to defer to others
> that might know if there is any issue with that.
>
> Could show the contents of these two files just so we can double check
> the users/policies?
>
> <property name="Authorizations File">/opt/config/authorizations.xml</property>
> <property name="Users File">/opt/config/users.xml</property>
>
> On Wed, Mar 21, 2018 at 12:37 PM, Scott Howell <sc...@mobilgov.com> wrote:
>> Thanks I have checked that and the whitespace is correct in user.xml.
>>
>> I did make a change to my authorizer.xml
>>
>> <authorizers>
>> <authorizer>
>> <identifier>file-provider</identifier>
>> <class>org.apache.nifi.authorization.FileAuthorizer</class>
>> <property name="Authorizations File">/opt/config/authorizations.xml</property>
>> <property name="Users File">/opt/config/users.xml</property>
>> <property name="Initial Admin Identity">uid=scott,ou=users,dc=mobilgov,dc=com</property>
>> <property name="Legacy Authorized Users File"></property>
>>
>> <property name="Node Identity 1">CN=*.{redacted}.com, OU={redacted}, O={redacted}, L=Kansas City, ST=Missouri, C=US</property>
>> <property name="Node Identity 2">CN=*.{redacted}.com, OU={redacted}, O={redacted}, L=Kansas City, ST=Missouri, C=US</property>
>> </authorizer>
>> </authorizers>
>>
>> I made the CN=*.{redacted}.com just like the self-signed certificate showed. I now have a 1 node cluster up and running. It seems like NIfi isn’t taking into account the wildcard and treating it as a “*” instead.
>>
>>> On Mar 21, 2018, at 11:33 AM, Bryan Bende <bb...@gmail.com> wrote:
>>>
>>> There only needs to be W to /proxy so that part should be fine.
>>>
>>> After you edited the Node Identities, did you delete users.xml and
>>> authorizations.xml?
>>>
>>> You would have to do that for those changes to take effect. You can
>>> look in users.xml and see if you still have the user identities
>>> without whitespace.
>>>
>>> On Wed, Mar 21, 2018 at 12:20 PM, Scott Howell <sc...@mobilgov.com> wrote:
>>>> One other thing I am seeing and I don’t know if this is an issue or not in my authorizations.xml I do not have a policy for /proxy with action=“R” only action=“W”.
>>>>
>>>>> On Mar 21, 2018, at 11:03 AM, Scott Howell <sc...@mobilgov.com> wrote:
>>>>>
>>>>> Thanks for that. I am still getting this error in my nifi-user.log
>>>>>
>>>>> o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api: Untrusted proxy CN=*.{redacted}.com, OU={redacted}, O={redacted}, L=Kansas City, ST=Missouri, C=US
>>>>>
>>>>> Is there an issue with using a wildcard cert?
>>>>>
>>>>>
>>>>>> On Mar 21, 2018, at 10:23 AM, Bryan Bende <bb...@gmail.com> wrote:
>>>>>>
>>>>>> All identity strings are case & whitespace sensitive.
>>>>>>
>>>>>> The node identities in your authorizers.xml have no whitespace, and
>>>>>> the identity showing in the logs does.
>>>>>>
>>>>>> On Wed, Mar 21, 2018 at 11:05 AM, Scott Howell <sc...@mobilgov.com> wrote:
>>>>>>> Thanks for all of the help with this. I have the cluster up and running. The
>>>>>>> logs look great everything seems to be working but I cannot login into the
>>>>>>> UI. I am using a wildcard self-signed certificate. The /proxy is in
>>>>>>> authorizations.xml with the correct users for the nodes.
>>>>>>>
>>>>>>> The error I see with the UI :
>>>>>>> is Untrusted proxy CN=*.{redacted}.com, OU={redacted}, O={redacted},
>>>>>>> L=Kansas City, ST=Missouri, C=US
>>>>>>>
>>>>>>> I haven’t had much luck finding a lot of documentation or forum questions
>>>>>>> with this kind of issue.
>>>>>>>
>>>>>>> My authorizers.xml looks like this
>>>>>>> <authorizers>
>>>>>>> <authorizer>
>>>>>>> <identifier>file-provider</identifier>
>>>>>>> <class>org.apache.nifi.authorization.FileAuthorizer</class>
>>>>>>> <property name="Authorizations
>>>>>>> File">/opt/config/authorizations.xml</property>
>>>>>>> <property name="Users File">/opt/config/users.xml</property>
>>>>>>> <property name="Initial Admin
>>>>>>> Identity">uid=scott,ou=users,dc={redacted},dc=com</property>
>>>>>>> <property name="Legacy Authorized Users File"></property>
>>>>>>>
>>>>>>> <property name="Node Identity
>>>>>>> 1">CN=node-1-nifi-dev.{redacted}.com,OU={redacted},O={redacted},L=Kansas
>>>>>>> City,ST=Missouri,C=US</property>
>>>>>>> <property name="Node Identity
>>>>>>> 2">CN=node-2-nifi-dev.{redacted}.com,OU={redacted},O={redacted},L=Kansas
>>>>>>> City,ST=Missouri,C=US</property>
>>>>>>> </authorizer>
>>>>>>> </authorizers>
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Scott
>>>>>>>
>>>>>>> On Mar 20, 2018, at 1:15 PM, Andy LoPresto <al...@apache.org> wrote:
>>>>>>>
>>>>>>> Scott,
>>>>>>>
>>>>>>> The original exception is "nested exception is
>>>>>>> java.security.KeyStoreException: not found”. Can you verify that the
>>>>>>> keystore you’ve provided is valid using the “keytool” command? In addition,
>>>>>>> you will need a truststore as well. Try following Pierre's [1] or Bryan’s
>>>>>>> [2] instructions for setting up a secure cluster.
>>>>>>>
>>>>>>> [1]
>>>>>>> https://pierrevillard.com/2016/11/29/apache-nifi-1-1-0-secured-cluster-setup/
>>>>>>> [2]
>>>>>>> https://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-authorization-and-multi-tenancy
>>>>>>>
>>>>>>>
>>>>>>> Andy LoPresto
>>>>>>> alopresto@apache.org
>>>>>>> alopresto.apache@gmail.com
>>>>>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>>>>>>>
>>>>>>> On Mar 20, 2018, at 11:05 AM, Scott Howell <sc...@mobilgov.com> wrote:
>>>>>>>
>>>>>>> Thanks for all of the help yesterday I was able to get a secure nifi and
>>>>>>> nifi-registry up and communicating. I am now trying to figure out how to
>>>>>>> create a secure cluster. I am currently getting this error when I start up
>>>>>>> nifi.
>>>>>>>
>>>>>>> tion; nested exception is
>>>>>>> org.springframework.beans.factory.BeanCreationException: Error creating bean
>>>>>>> with name 'clusterCoordinationProtocolSenderListener' defined in class path
>>>>>>> resource [nifi-cluster-protocol-context.xml]: Cannot resolve reference to
>>>>>>> bean 'clusterCoordinationProtocolSender' while setting constructor argument;
>>>>>>> nested exception is org.springframework.beans.factory.BeanCreationException:
>>>>>>> Error creating bean with name 'clusterCoordinationProtocolSender' defined in
>>>>>>> class path resource [nifi-cluster-protocol-context.xml]: Cannot resolve
>>>>>>> reference to bean 'protocolSocketConfiguration' while setting constructor
>>>>>>> argument; nested exception is
>>>>>>> org.springframework.beans.factory.BeanCreationException: Error creating bean
>>>>>>> with name 'protocolSocketConfiguration': FactoryBean threw exception on
>>>>>>> object creation; nested exception is java.security.KeyStoreException: not
>>>>>>> found
>>>>>>> at
>>>>>>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175)
>>>>>>> at
>>>>>>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
>>>>>>> at
>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1634)
>>>>>>> at
>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317)
>>>>>>> at
>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
>>>>>>> at
>>>>>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
>>>>>>> ... 50 common frames omitted
>>>>>>> Caused by: org.springframework.beans.factory.BeanCreationException: Error
>>>>>>> creating bean with name 'clusterCoordinationProtocolSenderListener' defined
>>>>>>> in class path resource [nifi-cluster-protocol-context.xml]: Cannot resolve
>>>>>>> reference to bean 'clusterCoordinationProtocolSender' while setting
>>>>>>> constructor argument; nested exception is
>>>>>>> org.springframework.beans.factory.BeanCreationException: Error creating bean
>>>>>>> with name 'clusterCoordinationProtocolSender' defined in class path resource
>>>>>>> [nifi-cluster-protocol-context.xml]: Cannot resolve reference to bean
>>>>>>> 'protocolSocketConfiguration' while setting constructor argument; nested
>>>>>>> exception is org.springframework.beans.factory.BeanCreationException: Error
>>>>>>> creating bean with name 'protocolSocketConfiguration': FactoryBean threw
>>>>>>> exception on object creation; nested exception is
>>>>>>> java.security.KeyStoreException: not found
>>>>>>> at
>>>>>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:359)
>>>>>>> at
>>>>>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)
>>>>>>> at
>>>>>>> org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:648)
>>>>>>> at
>>>>>>> org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:145)
>>>>>>> at
>>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1193)
>>>>>>> at
>>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1095)
>>>>>>> at
>>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
>>>>>>> at
>>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
>>>>>>> at
>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
>>>>>>> at
>>>>>>> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>>>>>>> at
>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
>>>>>>> at
>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
>>>>>>> at
>>>>>>> org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1084)
>>>>>>> at
>>>>>>> org.apache.nifi.cluster.spring.NodeClusterCoordinatorFactoryBean.getObject(NodeClusterCoordinatorFactoryBean.java:44)
>>>>>>> at
>>>>>>> org.apache.nifi.cluster.spring.NodeClusterCoordinatorFactoryBean.getObject(NodeClusterCoordinatorFactoryBean.java:34)
>>>>>>> at
>>>>>>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
>>>>>>> ... 55 common frames omitted
>>>>>>> Caused by: org.springframework.beans.factory.BeanCreationException: Error
>>>>>>> creating bean with name 'clusterCoordinationProtocolSender' defined in class
>>>>>>> path resource [nifi-cluster-protocol-context.xml]: Cannot resolve reference
>>>>>>> to bean 'protocolSocketConfiguration' while setting constructor argument;
>>>>>>> nested exception is org.springframework.beans.factory.BeanCreationException:
>>>>>>> Error creating bean with name 'protocolSocketConfiguration': FactoryBean
>>>>>>> threw exception on object creation; nested exception is
>>>>>>> java.security.KeyStoreException: not found
>>>>>>> at
>>>>>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:359)
>>>>>>> at
>>>>>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)
>>>>>>> at
>>>>>>> org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:648)
>>>>>>> at
>>>>>>> org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:145)
>>>>>>> at
>>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1193)
>>>>>>> at
>>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1095)
>>>>>>> at
>>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
>>>>>>> at
>>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
>>>>>>> at
>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
>>>>>>> at
>>>>>>> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>>>>>>> at
>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
>>>>>>> at
>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
>>>>>>> at
>>>>>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
>>>>>>> ... 70 common frames omitted
>>>>>>> Caused by: org.springframework.beans.factory.BeanCreationException: Error
>>>>>>> creating bean with name 'protocolSocketConfiguration': FactoryBean threw
>>>>>>> exception on object creation; nested exception is
>>>>>>> java.security.KeyStoreException: not found
>>>>>>> at
>>>>>>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175)
>>>>>>> at
>>>>>>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
>>>>>>> at
>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1634)
>>>>>>> at
>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317)
>>>>>>> at
>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
>>>>>>> at
>>>>>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
>>>>>>> ... 82 common frames omitted
>>>>>>> Caused by: java.security.KeyStoreException: not found
>>>>>>> at java.security.KeyStore.getInstance(KeyStore.java:851)
>>>>>>> at
>>>>>>> org.apache.nifi.security.util.KeyStoreUtils.getKeyStore(KeyStoreUtils.java:66)
>>>>>>> at
>>>>>>> org.apache.nifi.security.util.KeyStoreUtils.getTrustStore(KeyStoreUtils.java:80)
>>>>>>> at
>>>>>>> org.apache.nifi.io.socket.SSLContextFactory.<init>(SSLContextFactory.java:73)
>>>>>>> at
>>>>>>> org.apache.nifi.cluster.protocol.spring.SocketConfigurationFactoryBean.getObject(SocketConfigurationFactoryBean.java:45)
>>>>>>> at
>>>>>>> org.apache.nifi.cluster.protocol.spring.SocketConfigurationFactoryBean.getObject(SocketConfigurationFactoryBean.java:30)
>>>>>>> at
>>>>>>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
>>>>>>> ... 87 common frames omitted
>>>>>>> Caused by: java.security.NoSuchAlgorithmException: KeyStore not available
>>>>>>> at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
>>>>>>> at java.security.Security.getImpl(Security.java:695)
>>>>>>> at java.security.KeyStore.getInstance(KeyStore.java:848)
>>>>>>> ... 93 common frames omitted
>>>>>>>
>>>>>>> My nifi.properties file is.
>>>>>>>
>>>>>>> # Licensed to the Apache Software Foundation (ASF) under one or more
>>>>>>> # contributor license agreements. See the NOTICE file distributed with
>>>>>>> # this work for additional information regarding copyright ownership.
>>>>>>> # The ASF licenses this file to You under the Apache License, Version 2.0
>>>>>>> # (the "License"); you may not use this file except in compliance with
>>>>>>> # the License. You may obtain a copy of the License at
>>>>>>> #
>>>>>>> # http://www.apache.org/licenses/LICENSE-2.0
>>>>>>> #
>>>>>>> # Unless required by applicable law or agreed to in writing, software
>>>>>>> # distributed under the License is distributed on an "AS IS" BASIS,
>>>>>>> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>>>>>>> # See the License for the specific language governing permissions and
>>>>>>> # limitations under the License.
>>>>>>>
>>>>>>> # Core Properties #
>>>>>>> nifi.version={{nifi_version}}
>>>>>>> nifi.flow.configuration.file=/opt/config/flow.xml.gz
>>>>>>> nifi.flow.configuration.archive.enabled=true
>>>>>>> nifi.flow.configuration.archive.dir=/opt/config/archive/
>>>>>>> nifi.flow.configuration.archive.max.time=30 days
>>>>>>> nifi.flow.configuration.archive.max.storage=500 MB
>>>>>>> nifi.flowcontroller.autoResumeState=true
>>>>>>> nifi.flowcontroller.graceful.shutdown.period=10 sec
>>>>>>> nifi.flowservice.writedelay.interval=500 ms
>>>>>>> nifi.administrative.yield.duration=30 sec
>>>>>>> # If a component has no work to do (is "bored"), how long should we wait
>>>>>>> before checking again for work?
>>>>>>> nifi.bored.yield.duration=10 millis
>>>>>>>
>>>>>>>
>>>>>>> nifi.authorizer.configuration.file=/opt/config/authorizers.xml
>>>>>>> nifi.login.identity.provider.configuration.file=/opt/config/login-identity-providers.xml
>>>>>>> nifi.templates.directory=/opt/config/templates
>>>>>>> nifi.ui.banner.text=
>>>>>>> nifi.ui.autorefresh.interval=30 sec
>>>>>>> nifi.nar.library.directory=/opt/nifi/lib
>>>>>>> nifi.nar.library.directory.custom=/opt/config/processors
>>>>>>> nifi.nar.working.directory=/opt/nifi/work/nar/
>>>>>>> nifi.documentation.working.directory=./work/docs/components
>>>>>>>
>>>>>>> ####################
>>>>>>> # State Management #
>>>>>>> ####################
>>>>>>> nifi.state.management.configuration.file=/opt/config/state-management.xml
>>>>>>> # The ID of the local state provider
>>>>>>> nifi.state.management.provider.local=local-provider
>>>>>>> # The ID of the cluster-wide state provider. This will be ignored if NiFi is
>>>>>>> not clustered but must be populated if running in a cluster.
>>>>>>> nifi.state.management.provider.cluster=zk-provider
>>>>>>> # Specifies whether or not this instance of NiFi should run an embedded
>>>>>>> ZooKeeper server
>>>>>>> nifi.state.management.embedded.zookeeper.start=false
>>>>>>> # Properties file that provides the ZooKeeper properties to use if
>>>>>>> <nifi.state.management.embedded.zookeeper.start> is set to true
>>>>>>> nifi.state.management.embedded.zookeeper.properties=/opt/config/zookeeper.properties
>>>>>>>
>>>>>>>
>>>>>>> # H2 Settings
>>>>>>> nifi.database.directory=/opt/database_repository
>>>>>>> nifi.h2.url.append=;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE
>>>>>>>
>>>>>>> # FlowFile Repository
>>>>>>> nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository
>>>>>>> nifi.flowfile.repository.directory=/opt/flowfile_repository
>>>>>>> nifi.flowfile.repository.partitions=256
>>>>>>> nifi.flowfile.repository.checkpoint.interval=2 mins
>>>>>>> nifi.flowfile.repository.always.sync=false
>>>>>>>
>>>>>>> nifi.swap.manager.implementation=org.apache.nifi.controller.FileSystemSwapManager
>>>>>>> nifi.queue.swap.threshold=20000
>>>>>>> nifi.swap.in.period=5 sec
>>>>>>> nifi.swap.in.threads=1
>>>>>>> nifi.swap.out.period=5 sec
>>>>>>> nifi.swap.out.threads=4
>>>>>>>
>>>>>>> # Content Repository
>>>>>>> nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository
>>>>>>> nifi.content.claim.max.appendable.size=10 MB
>>>>>>> nifi.content.claim.max.flow.files=100
>>>>>>> nifi.content.repository.directory.default=/opt/content_repository
>>>>>>> nifi.content.repository.archive.max.retention.period=12 hours
>>>>>>> nifi.content.repository.archive.max.usage.percentage=50%
>>>>>>> nifi.content.repository.archive.enabled=true
>>>>>>> nifi.content.repository.always.sync=false
>>>>>>> nifi.content.viewer.url=/nifi-content-viewer/
>>>>>>>
>>>>>>> # Provenance Repository Properties
>>>>>>> nifi.provenance.repository.implementation=org.apache.nifi.provenance.PersistentProvenanceRepository
>>>>>>>
>>>>>>> # Persistent Provenance Repository Properties
>>>>>>> nifi.provenance.repository.directory.default=/opt/provenance_repository
>>>>>>> nifi.provenance.repository.max.storage.time=24 hours
>>>>>>> nifi.provenance.repository.max.storage.size=1 GB
>>>>>>> nifi.provenance.repository.rollover.time=30 secs
>>>>>>> nifi.provenance.repository.rollover.size=100 MB
>>>>>>> nifi.provenance.repository.query.threads=2
>>>>>>> nifi.provenance.repository.index.threads=1
>>>>>>> nifi.provenance.repository.compress.on.rollover=true
>>>>>>> nifi.provenance.repository.always.sync=false
>>>>>>> nifi.provenance.repository.journal.count=16
>>>>>>> # Comma-separated list of fields. Fields that are not indexed will not be
>>>>>>> searchable. Valid fields are:
>>>>>>> # EventType, FlowFileUUID, Filename, TransitURI, ProcessorID,
>>>>>>> AlternateIdentifierURI, Relationship, Details
>>>>>>> nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename,
>>>>>>> ProcessorID, Relationship
>>>>>>> # FlowFile Attributes that should be indexed and made searchable. Some
>>>>>>> examples to consider are filename, uuid, mime.type
>>>>>>> nifi.provenance.repository.indexed.attributes=
>>>>>>> # Large values for the shard size will result in more Java heap usage when
>>>>>>> searching the Provenance Repository
>>>>>>> # but should provide better performance
>>>>>>> nifi.provenance.repository.index.shard.size=500 MB
>>>>>>> # Indicates the maximum length that a FlowFile attribute can be when
>>>>>>> retrieving a Provenance Event from
>>>>>>> # the repository. If the length of any attribute exceeds this value, it will
>>>>>>> be truncated when the event is retrieved.
>>>>>>> nifi.provenance.repository.max.attribute.length=65536
>>>>>>>
>>>>>>> # Volatile Provenance Respository Properties
>>>>>>> nifi.provenance.repository.buffer.size=100000
>>>>>>>
>>>>>>> # Component Status Repository
>>>>>>> nifi.components.status.repository.implementation=org.apache.nifi.controller.status.history.VolatileComponentStatusRepository
>>>>>>> nifi.components.status.repository.buffer.size=1440
>>>>>>> nifi.components.status.snapshot.frequency=1 min
>>>>>>>
>>>>>>> # Site to Site properties
>>>>>>> nifi.remote.input.host=
>>>>>>> nifi.remote.input.secure=false
>>>>>>> nifi.remote.input.socket.port=9998
>>>>>>> nifi.remote.input.http.enabled=false
>>>>>>> nifi.remote.input.http.transaction.ttl=30 sec
>>>>>>>
>>>>>>> # web properties #
>>>>>>> nifi.web.war.directory=/opt/nifi/lib
>>>>>>> nifi.web.http.host=
>>>>>>> nifi.web.http.port=
>>>>>>> nifi.web.https.host={{redacted}}
>>>>>>> nifi.web.https.port=8443
>>>>>>> nifi.web.jetty.working.directory=/opt/nifi/work/jetty
>>>>>>> nifi.web.jetty.threads=200
>>>>>>>
>>>>>>> # security properties #
>>>>>>> nifi.sensitive.props.key=x0KDgO9L8lAhFGLdvu2VEjFVGc6Kg3V0R5I4bYwoqdgL47moo0wApKQtAVu1BvD
>>>>>>> nifi.sensitive.props.key.protected=
>>>>>>> nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
>>>>>>> nifi.sensitive.props.provider=BC
>>>>>>> nifi.sensitive.props.additional.keys=
>>>>>>>
>>>>>>> nifi.security.keystore=/opt/certs/payit_keystore
>>>>>>> nifi.security.keystoreType=JKS
>>>>>>> nifi.security.keystorePasswd={{keystore_password}}
>>>>>>> nifi.security.keyPasswd=
>>>>>>> nifi.security.truststore=
>>>>>>> nifi.security.truststoreType=
>>>>>>> nifi.security.truststorePasswd=
>>>>>>> nifi.security.needClientAuth=false
>>>>>>> nifi.security.user.authorizer=file-provider
>>>>>>> nifi.security.user.login.identity.provider=ldap-provider
>>>>>>> nifi.security.ocsp.responder.url=
>>>>>>> nifi.security.ocsp.responder.certificate=
>>>>>>>
>>>>>>> # Identity Mapping Properties #
>>>>>>> # These properties allow normalizing user identities such that identities
>>>>>>> coming from different identity providers
>>>>>>> # (certificates, LDAP, Kerberos) can be treated the same internally in NiFi.
>>>>>>> The following example demonstrates normalizing
>>>>>>> # DNs from certificates and principals from Kerberos into a common identity
>>>>>>> string:
>>>>>>> #
>>>>>>> #nifi.security.identity.mapping.pattern.dn=^cn=(.*?),ou=(.*?),dc=(.*?),dc=(.*?),dc=(.*?)$
>>>>>>> #nifi.security.identity.mapping.value.dn=$1
>>>>>>> # nifi.security.identity.mapping.pattern.kerb=^(.*?)/instance@(.*?)$
>>>>>>> # nifi.security.identity.mapping.value.kerb=$1@$2
>>>>>>>
>>>>>>> # cluster common properties (all nodes must have same values) #
>>>>>>> nifi.cluster.protocol.heartbeat.interval=5 sec
>>>>>>> nifi.cluster.protocol.is.secure=true
>>>>>>>
>>>>>>> # cluster node properties (only configure for cluster nodes) #
>>>>>>> nifi.cluster.is.node=true
>>>>>>> nifi.cluster.node.address=nifi-dev.mobilgov.com
>>>>>>> nifi.cluster.node.protocol.port=9999
>>>>>>> nifi.cluster.node.protocol.threads=10
>>>>>>> nifi.cluster.node.event.history.size=25
>>>>>>> nifi.cluster.node.connection.timeout=5 sec
>>>>>>> nifi.cluster.node.read.timeout=5 sec
>>>>>>> nifi.cluster.firewall.file=
>>>>>>>
>>>>>>>
>>>>>>> # zookeeper properties, used for cluster management #
>>>>>>> nifi.zookeeper.connect.string=internal-etcd-dev-EtcdLoadB-3RWA2WEFBBT8-2068560477.us-east-2.elb.amazonaws.com:2181,internal-etcd-dev-EtcdLoadB-3RWA2WEFBBT8-2068560477.us-east-2.elb.amazonaws.com2182,internal-etcd-dev-EtcdLoadB-3RWA2WEFBBT8-2068560477.us-east-2.elb.amazonaws.com:2183
>>>>>>> nifi.zookeeper.connect.timeout=3 secs
>>>>>>> nifi.zookeeper.session.timeout=3 secs
>>>>>>> nifi.zookeeper.root.node=/nifi
>>>>>>>
>>>>>>> # kerberos #
>>>>>>> nifi.kerberos.krb5.file=
>>>>>>>
>>>>>>> # kerberos service principle #
>>>>>>> nifi.kerberos.service.principal=
>>>>>>> nifi.kerberos.service.keytab.location=
>>>>>>>
>>>>>>> # kerberos spnego principle #
>>>>>>> nifi.kerberos.spnego.principal=
>>>>>>> nifi.kerberos.spnego.keytab.location=
>>>>>>> nifi.kerberos.spnego.authentication.expiration=12 hours
>>>>>>>
>>>>>>> # external properties files for variable registry
>>>>>>> # supports a comma delimited list of file locations
>>>>>>> nifi.variable.registry.properties=
>>>>>>>
>>>>>>> I think I have everything set correctly but I have not been able to start an
>>>>>>> instances up.
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Scott
>>>>>>>
>>>>>>> On Mar 19, 2018, at 4:35 PM, Bryan Bende <bb...@gmail.com> wrote:
>>>>>>>
>>>>>>> The base file is here for comparison:
>>>>>>>
>>>>>>> https://github.com/apache/nifi-registry/blob/master/nifi-registry-resources/src/main/resources/conf/identity-providers.xml#L23
>>>>>>>
>>>>>>> On Mon, Mar 19, 2018 at 5:34 PM, Bryan Bende <bb...@gmail.com> wrote:
>>>>>>>
>>>>>>> For your first file, is what you showed there actually wrapped in
>>>>>>> <identityProviders> </identityProviders> or is it exactly what you
>>>>>>> showed?
>>>>>>>
>>>>>>> It may just be that you only copied/pasted the one provider, but the
>>>>>>> root element is not <provider>, so as it is shown there it would not
>>>>>>> parse.
>>>>>>>
>>>>>>> On Mon, Mar 19, 2018 at 2:54 PM, Scott Howell <sc...@mobilgov.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Here is my file
>>>>>>>
>>>>>>> <provider>
>>>>>>> <identifier>ldap-identity-provider</identifier>
>>>>>>> <class>org.apache.nifi.registry.security.ldap.LdapProvider</class>
>>>>>>> <property name="Authentication Strategy">SIMPLE</property>
>>>>>>>
>>>>>>> <property name="Manager DN">cn=Manager,dc=mobilgov,dc=com</property>
>>>>>>> <property name="Manager Password”>redacted</property>
>>>>>>>
>>>>>>>
>>>>>>> <property name="Referral Strategy">FOLLOW</property>
>>>>>>> <property name="Connect Timeout">10 secs</property>
>>>>>>> <property name="Read Timeout">10 secs</property>
>>>>>>>
>>>>>>> <property name="Url”>redacted</property>
>>>>>>> <property name="User Search
>>>>>>> Base">ou=users,dc=mobilgov,dc=com</property>
>>>>>>> <property name="User Search Filter">uid={0}</property>
>>>>>>>
>>>>>>> <property name="Identity Strategy">USE_DN</property>
>>>>>>> <property name="Authentication Expiration">12 hours</property>
>>>>>>> </provider>
>>>>>>>
>>>>>>> Here is my authorizers.xml
>>>>>>>
>>>>>>> <authorizers>
>>>>>>>
>>>>>>> <userGroupProvider>
>>>>>>> <identifier>file-user-group-provider</identifier>
>>>>>>>
>>>>>>> <class>org.apache.nifi.registry.security.authorization.file.FileUserGroupProvider</class>
>>>>>>> <property name="Users File">conf/users.xml</property>
>>>>>>> <property name="Legacy Authorized Users File"></property>
>>>>>>> <property name="Initial User Identity 1”>redacted</property>
>>>>>>> </userGroupProvider>
>>>>>>>
>>>>>>> <accessPolicyProvider>
>>>>>>> <identifier>file-access-policy-provider</identifier>
>>>>>>>
>>>>>>> <class>org.apache.nifi.registry.security.authorization.file.FileAccessPolicyProvider</class>
>>>>>>> <property name="User Group
>>>>>>> Provider">file-user-group-provider</property>
>>>>>>> <property name="Authorizations
>>>>>>> File">conf/authorizations.xml</property>
>>>>>>> <property name="Initial Admin Identity”>redacted</property>
>>>>>>> <property name="NiFi Identity 1"></property>
>>>>>>> </accessPolicyProvider>
>>>>>>>
>>>>>>> <authorizer>
>>>>>>> <identifier>managed-authorizer</identifier>
>>>>>>>
>>>>>>> <class>org.apache.nifi.registry.security.authorization.StandardManagedAuthorizer</class>
>>>>>>> <property name="Access Policy
>>>>>>> Provider">file-access-policy-provider</property>
>>>>>>> </authorizer>
>>>>>>> </authorizers>
>>>>>>>
>>>>>>> On Mar 19, 2018, at 12:59 PM, Bryan Bende <bb...@gmail.com> wrote:
>>>>>>>
>>>>>>> It looks like that error would happen if your identity-providers.xml
>>>>>>> contained invalid XML.
>>>>>>>
>>>>>>> Did you start by modifying the identity-providers.xml file that was
>>>>>>> already there? Can you share the file, or the contents (removing
>>>>>>> anything sensitive)?
>>>>>>>
>>>>>>> On Mon, Mar 19, 2018 at 1:09 PM, Scott Howell <sc...@mobilgov.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> So I was able to get the UI pulled up but now I am hitting a roadblock with
>>>>>>> my identity-provider.xml.
>>>>>>>
>>>>>>> I am getting a number of errors like this:
>>>>>>>
>>>>>>> Caused by: org.springframework.beans.factory.BeanCreationException: Error
>>>>>>> creating bean with name 'getIdentityProvider' defined in class path resource
>>>>>>> [org/apache/nifi/registry/security/authentication/IdentityProviderFactory.class]:
>>>>>>> Bean instantiation via factory method failed; nested exception is
>>>>>>> org.springframework.beans.BeanInstantiationException: Failed to instantiate
>>>>>>> [org.apache.nifi.registry.security.authentication.IdentityProvider]: Factory
>>>>>>> method 'getIdentityProvider' threw exception; nested exception is
>>>>>>> java.lang.Exception: Unable to load the login identity provider
>>>>>>> configuration file at: /opt/nifi-registry-0.1.0/conf/identity-providers.xml
>>>>>>> at
>>>>>>> org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:587)
>>>>>>> ~[na:na]
>>>>>>> at
>>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1250)
>>>>>>> ~[na:na]
>>>>>>> at
>>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1099)
>>>>>>> ~[na:na]
>>>>>>> at
>>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:545)
>>>>>>> ~[na:na]
>>>>>>> at
>>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:502)
>>>>>>> ~[na:na]
>>>>>>> at
>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:312)
>>>>>>> ~[na:na]
>>>>>>> at
>>>>>>> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228)
>>>>>>> ~[na:na]
>>>>>>> at
>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:310)
>>>>>>> ~[na:na]
>>>>>>> at
>>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200)
>>>>>>> ~[na:na]
>>>>>>> at
>>>>>>> org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:251)
>>>>>>> ~[na:na]
>>>>>>> at
>>>>>>> org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1135)
>>>>>>> ~[na:na]
>>>>>>> at
>>>>>>> org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1062)
>>>>>>> ~[na:na]
>>>>>>> at
>>>>>>> org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:815)
>>>>>>> ~[na:na]
>>>>>>> at
>>>>>>> org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:721)
>>>>>>> ~[na:na]
>>>>>>> ... 43 common frames omitted
>>>>>>>
>>>>>>> I know it has to do with the identity-provider.xml but I have my setup just
>>>>>>> like the documentation ask for. I turned on debug but was not able to see
>>>>>>> anything different or better explanation from it.
>>>>>>>
>>>>>>>
>>>>>>> On Mar 19, 2018, at 10:06 AM, Kevin Doran <kd...@apache.org> wrote:
>>>>>>>
>>>>>>> Ok, that use case should be fine.
>>>>>>>
>>>>>>> If it were an authorization issue you would see something in the logs saying
>>>>>>> that an authorization attempt failed and the server is responding with a
>>>>>>> 403. Just to be sure, can you enable debug logging if you haven't already,
>>>>>>> i.e., in your nifi-registry/conf/logback.xml file, change
>>>>>>> 'org.apache.nifif.registry' to debug:
>>>>>>>
>>>>>>> <!-- valid logging levels: TRACE, DEBUG, INFO, WARN, ERROR -->
>>>>>>> <logger name="org.apache.nifi.registry" level="DEBUG"/>
>>>>>>>
>>>>>>> If there is nothing being written to nifi-registry-app.log, it points
>>>>>>> towards a connection issue, so I would double check your host, port, and TLS
>>>>>>> settings. You'll have to get an HTTPS cert from a root CA or configure your
>>>>>>> ELB to trust your company's self-signed cert (again, not sure if/how to do
>>>>>>> this, but I assume there should be some way to configure it. It might
>>>>>>> require settings not exposed in the AWS web console.)
>>>>>>>
>>>>>>> On 3/19/18, 10:51, "Scott Howell" <sc...@mobilgov.com> wrote:
>>>>>>>
>>>>>>> Thanks Kevin,
>>>>>>>
>>>>>>> I am just using the ELB to go from the public subnet to the private subnet.
>>>>>>> I will not have multiple instances running of registry.
>>>>>>>
>>>>>>> I will say on my authorizers.xml there is one difference between my nifi
>>>>>>> instance. On my nifi instance I am using file-provider for
>>>>>>> nifi.security.user.authorizer in my nifi.properties. I don’t think from
>>>>>>> reading the documents for nifi-registry that I can use that. If there is a
>>>>>>> way that might be my problem. I was running into some issues with my nifi
>>>>>>> instance when I was using managed-authorizers instead of file-provider.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Mar 19, 2018, at 9:35 AM, Kevin Doran <kd...@apache.org> wrote:
>>>>>>>
>>>>>>> Hey Scott,
>>>>>>>
>>>>>>> Assuming you are using two-way TLS with client certificates for
>>>>>>> authentication, I recommend configuring your ELB for TCP passthrough so that
>>>>>>> the TLS handshake is between the end-client and the NiFi Registry Server (in
>>>>>>> other words, no decryption/termination of the TLS connection happens in the
>>>>>>> ELB). If you are using some other form of authentication (e.g., LDAP), you
>>>>>>> will need to configure your ELB to trust the self-signed key NiFi Registry
>>>>>>> is using. I'm not sure how to do that as I've never run an ELB with that
>>>>>>> configuration before.
>>>>>>>
>>>>>>> Also, just a note about using an ELB with NiFi Registry:
>>>>>>>
>>>>>>> NiFi Registry is currently only supports single-instance use as persisted
>>>>>>> data and in-memory state is not synced between multiple instances. Are you
>>>>>>> hoping to use the ELB for actual load balancing, or is it just to take
>>>>>>> advantage of other ELB features, such as forwarding and security group
>>>>>>> rules? If the plan is to load balance multiple Registry instances, just be
>>>>>>> aware that you will probably run into some unexpected behavior. (As you
>>>>>>> mentioned using authorization, that is one case where I know the in-memory
>>>>>>> cache of the persisted data will not refresh across instances, so even if
>>>>>>> you were using some sort of shared network file system attached to multiple
>>>>>>> Registry instances, such as EFS, it would not work the way you hope.)
>>>>>>>
>>>>>>> Hope this helps,
>>>>>>> Kevin
>>>>>>>
>>>>>>> On 3/19/18, 10:20, "Scott Howell" <sc...@mobilgov.com> wrote:
>>>>>>>
>>>>>>> Thanks for the quick response.
>>>>>>>
>>>>>>> A couple of things I am seeing.
>>>>>>>
>>>>>>> 1. There is no error, I don’t see anything in the logs once the service
>>>>>>> comes up. This is because the health check is not even hitting the instance
>>>>>>> when secure.
>>>>>>>
>>>>>>> 2. Nothing interesting in the nifi-registry-app.logs. That was my concern
>>>>>>> because on my nifi instance I can see the health check hitting the instance
>>>>>>> from the ELB. This does not happen on the nifi-registry instance. I see the
>>>>>>> service startup and it tells me what domain and port I can access the UI but
>>>>>>> nothing else after that.
>>>>>>>
>>>>>>> 3. When I am on an instances in the same private subnet I am able to curl to
>>>>>>> the instance I get the TLS SSL which tells me the keystore is on the server.
>>>>>>> I am using a JKS keystore that is self-signed by the company I work for.
>>>>>>>
>>>>>>> On Mar 19, 2018, at 9:10 AM, Bryan Bende <bb...@gmail.com> wrote:
>>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> What error are you getting when you cannot access the UI?
>>>>>>>
>>>>>>> Is there anything interesting in nifi-registry-app.log regarding
>>>>>>> authentication/authorization when this happens?
>>>>>>>
>>>>>>> Can you access the UI securely without going through the ELB?
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Bryan
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Mar 19, 2018 at 10:05 AM, Scott Howell <sc...@mobilgov.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> I was able to stand up nifi-registry behind an AWS ELB non-secure.
>>>>>>> Everything was working great and was able to access the UI anonymously. I
>>>>>>> set up the authorization just like on my nifi instances along with the
>>>>>>> authorizers and identity-provider. The service comes up without errors and
>>>>>>> everything looks good but the health check does not pass and I cannot access
>>>>>>> the UI to login. I was wondering if anyone else has ran into this issue
>>>>>>> using nifi-registry.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>>
>
Re: Issue with AWS ELB on secure nifi-registry
Posted by Bryan Bende <bb...@gmail.com>.
I've never used wildcard certs before so I'll have to defer to others
that might know if there is any issue with that.
Could show the contents of these two files just so we can double check
the users/policies?
<property name="Authorizations File">/opt/config/authorizations.xml</property>
<property name="Users File">/opt/config/users.xml</property>
On Wed, Mar 21, 2018 at 12:37 PM, Scott Howell <sc...@mobilgov.com> wrote:
> Thanks I have checked that and the whitespace is correct in user.xml.
>
> I did make a change to my authorizer.xml
>
> <authorizers>
> <authorizer>
> <identifier>file-provider</identifier>
> <class>org.apache.nifi.authorization.FileAuthorizer</class>
> <property name="Authorizations File">/opt/config/authorizations.xml</property>
> <property name="Users File">/opt/config/users.xml</property>
> <property name="Initial Admin Identity">uid=scott,ou=users,dc=mobilgov,dc=com</property>
> <property name="Legacy Authorized Users File"></property>
>
> <property name="Node Identity 1">CN=*.{redacted}.com, OU={redacted}, O={redacted}, L=Kansas City, ST=Missouri, C=US</property>
> <property name="Node Identity 2">CN=*.{redacted}.com, OU={redacted}, O={redacted}, L=Kansas City, ST=Missouri, C=US</property>
> </authorizer>
> </authorizers>
>
> I made the CN=*.{redacted}.com just like the self-signed certificate showed. I now have a 1 node cluster up and running. It seems like NIfi isn’t taking into account the wildcard and treating it as a “*” instead.
>
>> On Mar 21, 2018, at 11:33 AM, Bryan Bende <bb...@gmail.com> wrote:
>>
>> There only needs to be W to /proxy so that part should be fine.
>>
>> After you edited the Node Identities, did you delete users.xml and
>> authorizations.xml?
>>
>> You would have to do that for those changes to take effect. You can
>> look in users.xml and see if you still have the user identities
>> without whitespace.
>>
>> On Wed, Mar 21, 2018 at 12:20 PM, Scott Howell <sc...@mobilgov.com> wrote:
>>> One other thing I am seeing and I don’t know if this is an issue or not in my authorizations.xml I do not have a policy for /proxy with action=“R” only action=“W”.
>>>
>>>> On Mar 21, 2018, at 11:03 AM, Scott Howell <sc...@mobilgov.com> wrote:
>>>>
>>>> Thanks for that. I am still getting this error in my nifi-user.log
>>>>
>>>> o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api: Untrusted proxy CN=*.{redacted}.com, OU={redacted}, O={redacted}, L=Kansas City, ST=Missouri, C=US
>>>>
>>>> Is there an issue with using a wildcard cert?
>>>>
>>>>
>>>>> On Mar 21, 2018, at 10:23 AM, Bryan Bende <bb...@gmail.com> wrote:
>>>>>
>>>>> All identity strings are case & whitespace sensitive.
>>>>>
>>>>> The node identities in your authorizers.xml have no whitespace, and
>>>>> the identity showing in the logs does.
>>>>>
>>>>> On Wed, Mar 21, 2018 at 11:05 AM, Scott Howell <sc...@mobilgov.com> wrote:
>>>>>> Thanks for all of the help with this. I have the cluster up and running. The
>>>>>> logs look great everything seems to be working but I cannot login into the
>>>>>> UI. I am using a wildcard self-signed certificate. The /proxy is in
>>>>>> authorizations.xml with the correct users for the nodes.
>>>>>>
>>>>>> The error I see with the UI :
>>>>>> is Untrusted proxy CN=*.{redacted}.com, OU={redacted}, O={redacted},
>>>>>> L=Kansas City, ST=Missouri, C=US
>>>>>>
>>>>>> I haven’t had much luck finding a lot of documentation or forum questions
>>>>>> with this kind of issue.
>>>>>>
>>>>>> My authorizers.xml looks like this
>>>>>> <authorizers>
>>>>>> <authorizer>
>>>>>> <identifier>file-provider</identifier>
>>>>>> <class>org.apache.nifi.authorization.FileAuthorizer</class>
>>>>>> <property name="Authorizations
>>>>>> File">/opt/config/authorizations.xml</property>
>>>>>> <property name="Users File">/opt/config/users.xml</property>
>>>>>> <property name="Initial Admin
>>>>>> Identity">uid=scott,ou=users,dc={redacted},dc=com</property>
>>>>>> <property name="Legacy Authorized Users File"></property>
>>>>>>
>>>>>> <property name="Node Identity
>>>>>> 1">CN=node-1-nifi-dev.{redacted}.com,OU={redacted},O={redacted},L=Kansas
>>>>>> City,ST=Missouri,C=US</property>
>>>>>> <property name="Node Identity
>>>>>> 2">CN=node-2-nifi-dev.{redacted}.com,OU={redacted},O={redacted},L=Kansas
>>>>>> City,ST=Missouri,C=US</property>
>>>>>> </authorizer>
>>>>>> </authorizers>
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Scott
>>>>>>
>>>>>> On Mar 20, 2018, at 1:15 PM, Andy LoPresto <al...@apache.org> wrote:
>>>>>>
>>>>>> Scott,
>>>>>>
>>>>>> The original exception is "nested exception is
>>>>>> java.security.KeyStoreException: not found”. Can you verify that the
>>>>>> keystore you’ve provided is valid using the “keytool” command? In addition,
>>>>>> you will need a truststore as well. Try following Pierre's [1] or Bryan’s
>>>>>> [2] instructions for setting up a secure cluster.
>>>>>>
>>>>>> [1]
>>>>>> https://pierrevillard.com/2016/11/29/apache-nifi-1-1-0-secured-cluster-setup/
>>>>>> [2]
>>>>>> https://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-authorization-and-multi-tenancy
>>>>>>
>>>>>>
>>>>>> Andy LoPresto
>>>>>> alopresto@apache.org
>>>>>> alopresto.apache@gmail.com
>>>>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>>>>>>
>>>>>> On Mar 20, 2018, at 11:05 AM, Scott Howell <sc...@mobilgov.com> wrote:
>>>>>>
>>>>>> Thanks for all of the help yesterday I was able to get a secure nifi and
>>>>>> nifi-registry up and communicating. I am now trying to figure out how to
>>>>>> create a secure cluster. I am currently getting this error when I start up
>>>>>> nifi.
>>>>>>
>>>>>> tion; nested exception is
>>>>>> org.springframework.beans.factory.BeanCreationException: Error creating bean
>>>>>> with name 'clusterCoordinationProtocolSenderListener' defined in class path
>>>>>> resource [nifi-cluster-protocol-context.xml]: Cannot resolve reference to
>>>>>> bean 'clusterCoordinationProtocolSender' while setting constructor argument;
>>>>>> nested exception is org.springframework.beans.factory.BeanCreationException:
>>>>>> Error creating bean with name 'clusterCoordinationProtocolSender' defined in
>>>>>> class path resource [nifi-cluster-protocol-context.xml]: Cannot resolve
>>>>>> reference to bean 'protocolSocketConfiguration' while setting constructor
>>>>>> argument; nested exception is
>>>>>> org.springframework.beans.factory.BeanCreationException: Error creating bean
>>>>>> with name 'protocolSocketConfiguration': FactoryBean threw exception on
>>>>>> object creation; nested exception is java.security.KeyStoreException: not
>>>>>> found
>>>>>> at
>>>>>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175)
>>>>>> at
>>>>>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
>>>>>> at
>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1634)
>>>>>> at
>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317)
>>>>>> at
>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
>>>>>> at
>>>>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
>>>>>> ... 50 common frames omitted
>>>>>> Caused by: org.springframework.beans.factory.BeanCreationException: Error
>>>>>> creating bean with name 'clusterCoordinationProtocolSenderListener' defined
>>>>>> in class path resource [nifi-cluster-protocol-context.xml]: Cannot resolve
>>>>>> reference to bean 'clusterCoordinationProtocolSender' while setting
>>>>>> constructor argument; nested exception is
>>>>>> org.springframework.beans.factory.BeanCreationException: Error creating bean
>>>>>> with name 'clusterCoordinationProtocolSender' defined in class path resource
>>>>>> [nifi-cluster-protocol-context.xml]: Cannot resolve reference to bean
>>>>>> 'protocolSocketConfiguration' while setting constructor argument; nested
>>>>>> exception is org.springframework.beans.factory.BeanCreationException: Error
>>>>>> creating bean with name 'protocolSocketConfiguration': FactoryBean threw
>>>>>> exception on object creation; nested exception is
>>>>>> java.security.KeyStoreException: not found
>>>>>> at
>>>>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:359)
>>>>>> at
>>>>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)
>>>>>> at
>>>>>> org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:648)
>>>>>> at
>>>>>> org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:145)
>>>>>> at
>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1193)
>>>>>> at
>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1095)
>>>>>> at
>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
>>>>>> at
>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
>>>>>> at
>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
>>>>>> at
>>>>>> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>>>>>> at
>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
>>>>>> at
>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
>>>>>> at
>>>>>> org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1084)
>>>>>> at
>>>>>> org.apache.nifi.cluster.spring.NodeClusterCoordinatorFactoryBean.getObject(NodeClusterCoordinatorFactoryBean.java:44)
>>>>>> at
>>>>>> org.apache.nifi.cluster.spring.NodeClusterCoordinatorFactoryBean.getObject(NodeClusterCoordinatorFactoryBean.java:34)
>>>>>> at
>>>>>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
>>>>>> ... 55 common frames omitted
>>>>>> Caused by: org.springframework.beans.factory.BeanCreationException: Error
>>>>>> creating bean with name 'clusterCoordinationProtocolSender' defined in class
>>>>>> path resource [nifi-cluster-protocol-context.xml]: Cannot resolve reference
>>>>>> to bean 'protocolSocketConfiguration' while setting constructor argument;
>>>>>> nested exception is org.springframework.beans.factory.BeanCreationException:
>>>>>> Error creating bean with name 'protocolSocketConfiguration': FactoryBean
>>>>>> threw exception on object creation; nested exception is
>>>>>> java.security.KeyStoreException: not found
>>>>>> at
>>>>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:359)
>>>>>> at
>>>>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)
>>>>>> at
>>>>>> org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:648)
>>>>>> at
>>>>>> org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:145)
>>>>>> at
>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1193)
>>>>>> at
>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1095)
>>>>>> at
>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
>>>>>> at
>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
>>>>>> at
>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
>>>>>> at
>>>>>> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>>>>>> at
>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
>>>>>> at
>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
>>>>>> at
>>>>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
>>>>>> ... 70 common frames omitted
>>>>>> Caused by: org.springframework.beans.factory.BeanCreationException: Error
>>>>>> creating bean with name 'protocolSocketConfiguration': FactoryBean threw
>>>>>> exception on object creation; nested exception is
>>>>>> java.security.KeyStoreException: not found
>>>>>> at
>>>>>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175)
>>>>>> at
>>>>>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
>>>>>> at
>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1634)
>>>>>> at
>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317)
>>>>>> at
>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
>>>>>> at
>>>>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
>>>>>> ... 82 common frames omitted
>>>>>> Caused by: java.security.KeyStoreException: not found
>>>>>> at java.security.KeyStore.getInstance(KeyStore.java:851)
>>>>>> at
>>>>>> org.apache.nifi.security.util.KeyStoreUtils.getKeyStore(KeyStoreUtils.java:66)
>>>>>> at
>>>>>> org.apache.nifi.security.util.KeyStoreUtils.getTrustStore(KeyStoreUtils.java:80)
>>>>>> at
>>>>>> org.apache.nifi.io.socket.SSLContextFactory.<init>(SSLContextFactory.java:73)
>>>>>> at
>>>>>> org.apache.nifi.cluster.protocol.spring.SocketConfigurationFactoryBean.getObject(SocketConfigurationFactoryBean.java:45)
>>>>>> at
>>>>>> org.apache.nifi.cluster.protocol.spring.SocketConfigurationFactoryBean.getObject(SocketConfigurationFactoryBean.java:30)
>>>>>> at
>>>>>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
>>>>>> ... 87 common frames omitted
>>>>>> Caused by: java.security.NoSuchAlgorithmException: KeyStore not available
>>>>>> at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
>>>>>> at java.security.Security.getImpl(Security.java:695)
>>>>>> at java.security.KeyStore.getInstance(KeyStore.java:848)
>>>>>> ... 93 common frames omitted
>>>>>>
>>>>>> My nifi.properties file is.
>>>>>>
>>>>>> # Licensed to the Apache Software Foundation (ASF) under one or more
>>>>>> # contributor license agreements. See the NOTICE file distributed with
>>>>>> # this work for additional information regarding copyright ownership.
>>>>>> # The ASF licenses this file to You under the Apache License, Version 2.0
>>>>>> # (the "License"); you may not use this file except in compliance with
>>>>>> # the License. You may obtain a copy of the License at
>>>>>> #
>>>>>> # http://www.apache.org/licenses/LICENSE-2.0
>>>>>> #
>>>>>> # Unless required by applicable law or agreed to in writing, software
>>>>>> # distributed under the License is distributed on an "AS IS" BASIS,
>>>>>> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>>>>>> # See the License for the specific language governing permissions and
>>>>>> # limitations under the License.
>>>>>>
>>>>>> # Core Properties #
>>>>>> nifi.version={{nifi_version}}
>>>>>> nifi.flow.configuration.file=/opt/config/flow.xml.gz
>>>>>> nifi.flow.configuration.archive.enabled=true
>>>>>> nifi.flow.configuration.archive.dir=/opt/config/archive/
>>>>>> nifi.flow.configuration.archive.max.time=30 days
>>>>>> nifi.flow.configuration.archive.max.storage=500 MB
>>>>>> nifi.flowcontroller.autoResumeState=true
>>>>>> nifi.flowcontroller.graceful.shutdown.period=10 sec
>>>>>> nifi.flowservice.writedelay.interval=500 ms
>>>>>> nifi.administrative.yield.duration=30 sec
>>>>>> # If a component has no work to do (is "bored"), how long should we wait
>>>>>> before checking again for work?
>>>>>> nifi.bored.yield.duration=10 millis
>>>>>>
>>>>>>
>>>>>> nifi.authorizer.configuration.file=/opt/config/authorizers.xml
>>>>>> nifi.login.identity.provider.configuration.file=/opt/config/login-identity-providers.xml
>>>>>> nifi.templates.directory=/opt/config/templates
>>>>>> nifi.ui.banner.text=
>>>>>> nifi.ui.autorefresh.interval=30 sec
>>>>>> nifi.nar.library.directory=/opt/nifi/lib
>>>>>> nifi.nar.library.directory.custom=/opt/config/processors
>>>>>> nifi.nar.working.directory=/opt/nifi/work/nar/
>>>>>> nifi.documentation.working.directory=./work/docs/components
>>>>>>
>>>>>> ####################
>>>>>> # State Management #
>>>>>> ####################
>>>>>> nifi.state.management.configuration.file=/opt/config/state-management.xml
>>>>>> # The ID of the local state provider
>>>>>> nifi.state.management.provider.local=local-provider
>>>>>> # The ID of the cluster-wide state provider. This will be ignored if NiFi is
>>>>>> not clustered but must be populated if running in a cluster.
>>>>>> nifi.state.management.provider.cluster=zk-provider
>>>>>> # Specifies whether or not this instance of NiFi should run an embedded
>>>>>> ZooKeeper server
>>>>>> nifi.state.management.embedded.zookeeper.start=false
>>>>>> # Properties file that provides the ZooKeeper properties to use if
>>>>>> <nifi.state.management.embedded.zookeeper.start> is set to true
>>>>>> nifi.state.management.embedded.zookeeper.properties=/opt/config/zookeeper.properties
>>>>>>
>>>>>>
>>>>>> # H2 Settings
>>>>>> nifi.database.directory=/opt/database_repository
>>>>>> nifi.h2.url.append=;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE
>>>>>>
>>>>>> # FlowFile Repository
>>>>>> nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository
>>>>>> nifi.flowfile.repository.directory=/opt/flowfile_repository
>>>>>> nifi.flowfile.repository.partitions=256
>>>>>> nifi.flowfile.repository.checkpoint.interval=2 mins
>>>>>> nifi.flowfile.repository.always.sync=false
>>>>>>
>>>>>> nifi.swap.manager.implementation=org.apache.nifi.controller.FileSystemSwapManager
>>>>>> nifi.queue.swap.threshold=20000
>>>>>> nifi.swap.in.period=5 sec
>>>>>> nifi.swap.in.threads=1
>>>>>> nifi.swap.out.period=5 sec
>>>>>> nifi.swap.out.threads=4
>>>>>>
>>>>>> # Content Repository
>>>>>> nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository
>>>>>> nifi.content.claim.max.appendable.size=10 MB
>>>>>> nifi.content.claim.max.flow.files=100
>>>>>> nifi.content.repository.directory.default=/opt/content_repository
>>>>>> nifi.content.repository.archive.max.retention.period=12 hours
>>>>>> nifi.content.repository.archive.max.usage.percentage=50%
>>>>>> nifi.content.repository.archive.enabled=true
>>>>>> nifi.content.repository.always.sync=false
>>>>>> nifi.content.viewer.url=/nifi-content-viewer/
>>>>>>
>>>>>> # Provenance Repository Properties
>>>>>> nifi.provenance.repository.implementation=org.apache.nifi.provenance.PersistentProvenanceRepository
>>>>>>
>>>>>> # Persistent Provenance Repository Properties
>>>>>> nifi.provenance.repository.directory.default=/opt/provenance_repository
>>>>>> nifi.provenance.repository.max.storage.time=24 hours
>>>>>> nifi.provenance.repository.max.storage.size=1 GB
>>>>>> nifi.provenance.repository.rollover.time=30 secs
>>>>>> nifi.provenance.repository.rollover.size=100 MB
>>>>>> nifi.provenance.repository.query.threads=2
>>>>>> nifi.provenance.repository.index.threads=1
>>>>>> nifi.provenance.repository.compress.on.rollover=true
>>>>>> nifi.provenance.repository.always.sync=false
>>>>>> nifi.provenance.repository.journal.count=16
>>>>>> # Comma-separated list of fields. Fields that are not indexed will not be
>>>>>> searchable. Valid fields are:
>>>>>> # EventType, FlowFileUUID, Filename, TransitURI, ProcessorID,
>>>>>> AlternateIdentifierURI, Relationship, Details
>>>>>> nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename,
>>>>>> ProcessorID, Relationship
>>>>>> # FlowFile Attributes that should be indexed and made searchable. Some
>>>>>> examples to consider are filename, uuid, mime.type
>>>>>> nifi.provenance.repository.indexed.attributes=
>>>>>> # Large values for the shard size will result in more Java heap usage when
>>>>>> searching the Provenance Repository
>>>>>> # but should provide better performance
>>>>>> nifi.provenance.repository.index.shard.size=500 MB
>>>>>> # Indicates the maximum length that a FlowFile attribute can be when
>>>>>> retrieving a Provenance Event from
>>>>>> # the repository. If the length of any attribute exceeds this value, it will
>>>>>> be truncated when the event is retrieved.
>>>>>> nifi.provenance.repository.max.attribute.length=65536
>>>>>>
>>>>>> # Volatile Provenance Respository Properties
>>>>>> nifi.provenance.repository.buffer.size=100000
>>>>>>
>>>>>> # Component Status Repository
>>>>>> nifi.components.status.repository.implementation=org.apache.nifi.controller.status.history.VolatileComponentStatusRepository
>>>>>> nifi.components.status.repository.buffer.size=1440
>>>>>> nifi.components.status.snapshot.frequency=1 min
>>>>>>
>>>>>> # Site to Site properties
>>>>>> nifi.remote.input.host=
>>>>>> nifi.remote.input.secure=false
>>>>>> nifi.remote.input.socket.port=9998
>>>>>> nifi.remote.input.http.enabled=false
>>>>>> nifi.remote.input.http.transaction.ttl=30 sec
>>>>>>
>>>>>> # web properties #
>>>>>> nifi.web.war.directory=/opt/nifi/lib
>>>>>> nifi.web.http.host=
>>>>>> nifi.web.http.port=
>>>>>> nifi.web.https.host={{redacted}}
>>>>>> nifi.web.https.port=8443
>>>>>> nifi.web.jetty.working.directory=/opt/nifi/work/jetty
>>>>>> nifi.web.jetty.threads=200
>>>>>>
>>>>>> # security properties #
>>>>>> nifi.sensitive.props.key=x0KDgO9L8lAhFGLdvu2VEjFVGc6Kg3V0R5I4bYwoqdgL47moo0wApKQtAVu1BvD
>>>>>> nifi.sensitive.props.key.protected=
>>>>>> nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
>>>>>> nifi.sensitive.props.provider=BC
>>>>>> nifi.sensitive.props.additional.keys=
>>>>>>
>>>>>> nifi.security.keystore=/opt/certs/payit_keystore
>>>>>> nifi.security.keystoreType=JKS
>>>>>> nifi.security.keystorePasswd={{keystore_password}}
>>>>>> nifi.security.keyPasswd=
>>>>>> nifi.security.truststore=
>>>>>> nifi.security.truststoreType=
>>>>>> nifi.security.truststorePasswd=
>>>>>> nifi.security.needClientAuth=false
>>>>>> nifi.security.user.authorizer=file-provider
>>>>>> nifi.security.user.login.identity.provider=ldap-provider
>>>>>> nifi.security.ocsp.responder.url=
>>>>>> nifi.security.ocsp.responder.certificate=
>>>>>>
>>>>>> # Identity Mapping Properties #
>>>>>> # These properties allow normalizing user identities such that identities
>>>>>> coming from different identity providers
>>>>>> # (certificates, LDAP, Kerberos) can be treated the same internally in NiFi.
>>>>>> The following example demonstrates normalizing
>>>>>> # DNs from certificates and principals from Kerberos into a common identity
>>>>>> string:
>>>>>> #
>>>>>> #nifi.security.identity.mapping.pattern.dn=^cn=(.*?),ou=(.*?),dc=(.*?),dc=(.*?),dc=(.*?)$
>>>>>> #nifi.security.identity.mapping.value.dn=$1
>>>>>> # nifi.security.identity.mapping.pattern.kerb=^(.*?)/instance@(.*?)$
>>>>>> # nifi.security.identity.mapping.value.kerb=$1@$2
>>>>>>
>>>>>> # cluster common properties (all nodes must have same values) #
>>>>>> nifi.cluster.protocol.heartbeat.interval=5 sec
>>>>>> nifi.cluster.protocol.is.secure=true
>>>>>>
>>>>>> # cluster node properties (only configure for cluster nodes) #
>>>>>> nifi.cluster.is.node=true
>>>>>> nifi.cluster.node.address=nifi-dev.mobilgov.com
>>>>>> nifi.cluster.node.protocol.port=9999
>>>>>> nifi.cluster.node.protocol.threads=10
>>>>>> nifi.cluster.node.event.history.size=25
>>>>>> nifi.cluster.node.connection.timeout=5 sec
>>>>>> nifi.cluster.node.read.timeout=5 sec
>>>>>> nifi.cluster.firewall.file=
>>>>>>
>>>>>>
>>>>>> # zookeeper properties, used for cluster management #
>>>>>> nifi.zookeeper.connect.string=internal-etcd-dev-EtcdLoadB-3RWA2WEFBBT8-2068560477.us-east-2.elb.amazonaws.com:2181,internal-etcd-dev-EtcdLoadB-3RWA2WEFBBT8-2068560477.us-east-2.elb.amazonaws.com2182,internal-etcd-dev-EtcdLoadB-3RWA2WEFBBT8-2068560477.us-east-2.elb.amazonaws.com:2183
>>>>>> nifi.zookeeper.connect.timeout=3 secs
>>>>>> nifi.zookeeper.session.timeout=3 secs
>>>>>> nifi.zookeeper.root.node=/nifi
>>>>>>
>>>>>> # kerberos #
>>>>>> nifi.kerberos.krb5.file=
>>>>>>
>>>>>> # kerberos service principle #
>>>>>> nifi.kerberos.service.principal=
>>>>>> nifi.kerberos.service.keytab.location=
>>>>>>
>>>>>> # kerberos spnego principle #
>>>>>> nifi.kerberos.spnego.principal=
>>>>>> nifi.kerberos.spnego.keytab.location=
>>>>>> nifi.kerberos.spnego.authentication.expiration=12 hours
>>>>>>
>>>>>> # external properties files for variable registry
>>>>>> # supports a comma delimited list of file locations
>>>>>> nifi.variable.registry.properties=
>>>>>>
>>>>>> I think I have everything set correctly but I have not been able to start an
>>>>>> instances up.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Scott
>>>>>>
>>>>>> On Mar 19, 2018, at 4:35 PM, Bryan Bende <bb...@gmail.com> wrote:
>>>>>>
>>>>>> The base file is here for comparison:
>>>>>>
>>>>>> https://github.com/apache/nifi-registry/blob/master/nifi-registry-resources/src/main/resources/conf/identity-providers.xml#L23
>>>>>>
>>>>>> On Mon, Mar 19, 2018 at 5:34 PM, Bryan Bende <bb...@gmail.com> wrote:
>>>>>>
>>>>>> For your first file, is what you showed there actually wrapped in
>>>>>> <identityProviders> </identityProviders> or is it exactly what you
>>>>>> showed?
>>>>>>
>>>>>> It may just be that you only copied/pasted the one provider, but the
>>>>>> root element is not <provider>, so as it is shown there it would not
>>>>>> parse.
>>>>>>
>>>>>> On Mon, Mar 19, 2018 at 2:54 PM, Scott Howell <sc...@mobilgov.com>
>>>>>> wrote:
>>>>>>
>>>>>> Here is my file
>>>>>>
>>>>>> <provider>
>>>>>> <identifier>ldap-identity-provider</identifier>
>>>>>> <class>org.apache.nifi.registry.security.ldap.LdapProvider</class>
>>>>>> <property name="Authentication Strategy">SIMPLE</property>
>>>>>>
>>>>>> <property name="Manager DN">cn=Manager,dc=mobilgov,dc=com</property>
>>>>>> <property name="Manager Password”>redacted</property>
>>>>>>
>>>>>>
>>>>>> <property name="Referral Strategy">FOLLOW</property>
>>>>>> <property name="Connect Timeout">10 secs</property>
>>>>>> <property name="Read Timeout">10 secs</property>
>>>>>>
>>>>>> <property name="Url”>redacted</property>
>>>>>> <property name="User Search
>>>>>> Base">ou=users,dc=mobilgov,dc=com</property>
>>>>>> <property name="User Search Filter">uid={0}</property>
>>>>>>
>>>>>> <property name="Identity Strategy">USE_DN</property>
>>>>>> <property name="Authentication Expiration">12 hours</property>
>>>>>> </provider>
>>>>>>
>>>>>> Here is my authorizers.xml
>>>>>>
>>>>>> <authorizers>
>>>>>>
>>>>>> <userGroupProvider>
>>>>>> <identifier>file-user-group-provider</identifier>
>>>>>>
>>>>>> <class>org.apache.nifi.registry.security.authorization.file.FileUserGroupProvider</class>
>>>>>> <property name="Users File">conf/users.xml</property>
>>>>>> <property name="Legacy Authorized Users File"></property>
>>>>>> <property name="Initial User Identity 1”>redacted</property>
>>>>>> </userGroupProvider>
>>>>>>
>>>>>> <accessPolicyProvider>
>>>>>> <identifier>file-access-policy-provider</identifier>
>>>>>>
>>>>>> <class>org.apache.nifi.registry.security.authorization.file.FileAccessPolicyProvider</class>
>>>>>> <property name="User Group
>>>>>> Provider">file-user-group-provider</property>
>>>>>> <property name="Authorizations
>>>>>> File">conf/authorizations.xml</property>
>>>>>> <property name="Initial Admin Identity”>redacted</property>
>>>>>> <property name="NiFi Identity 1"></property>
>>>>>> </accessPolicyProvider>
>>>>>>
>>>>>> <authorizer>
>>>>>> <identifier>managed-authorizer</identifier>
>>>>>>
>>>>>> <class>org.apache.nifi.registry.security.authorization.StandardManagedAuthorizer</class>
>>>>>> <property name="Access Policy
>>>>>> Provider">file-access-policy-provider</property>
>>>>>> </authorizer>
>>>>>> </authorizers>
>>>>>>
>>>>>> On Mar 19, 2018, at 12:59 PM, Bryan Bende <bb...@gmail.com> wrote:
>>>>>>
>>>>>> It looks like that error would happen if your identity-providers.xml
>>>>>> contained invalid XML.
>>>>>>
>>>>>> Did you start by modifying the identity-providers.xml file that was
>>>>>> already there? Can you share the file, or the contents (removing
>>>>>> anything sensitive)?
>>>>>>
>>>>>> On Mon, Mar 19, 2018 at 1:09 PM, Scott Howell <sc...@mobilgov.com>
>>>>>> wrote:
>>>>>>
>>>>>> So I was able to get the UI pulled up but now I am hitting a roadblock with
>>>>>> my identity-provider.xml.
>>>>>>
>>>>>> I am getting a number of errors like this:
>>>>>>
>>>>>> Caused by: org.springframework.beans.factory.BeanCreationException: Error
>>>>>> creating bean with name 'getIdentityProvider' defined in class path resource
>>>>>> [org/apache/nifi/registry/security/authentication/IdentityProviderFactory.class]:
>>>>>> Bean instantiation via factory method failed; nested exception is
>>>>>> org.springframework.beans.BeanInstantiationException: Failed to instantiate
>>>>>> [org.apache.nifi.registry.security.authentication.IdentityProvider]: Factory
>>>>>> method 'getIdentityProvider' threw exception; nested exception is
>>>>>> java.lang.Exception: Unable to load the login identity provider
>>>>>> configuration file at: /opt/nifi-registry-0.1.0/conf/identity-providers.xml
>>>>>> at
>>>>>> org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:587)
>>>>>> ~[na:na]
>>>>>> at
>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1250)
>>>>>> ~[na:na]
>>>>>> at
>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1099)
>>>>>> ~[na:na]
>>>>>> at
>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:545)
>>>>>> ~[na:na]
>>>>>> at
>>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:502)
>>>>>> ~[na:na]
>>>>>> at
>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:312)
>>>>>> ~[na:na]
>>>>>> at
>>>>>> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228)
>>>>>> ~[na:na]
>>>>>> at
>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:310)
>>>>>> ~[na:na]
>>>>>> at
>>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200)
>>>>>> ~[na:na]
>>>>>> at
>>>>>> org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:251)
>>>>>> ~[na:na]
>>>>>> at
>>>>>> org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1135)
>>>>>> ~[na:na]
>>>>>> at
>>>>>> org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1062)
>>>>>> ~[na:na]
>>>>>> at
>>>>>> org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:815)
>>>>>> ~[na:na]
>>>>>> at
>>>>>> org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:721)
>>>>>> ~[na:na]
>>>>>> ... 43 common frames omitted
>>>>>>
>>>>>> I know it has to do with the identity-provider.xml but I have my setup just
>>>>>> like the documentation ask for. I turned on debug but was not able to see
>>>>>> anything different or better explanation from it.
>>>>>>
>>>>>>
>>>>>> On Mar 19, 2018, at 10:06 AM, Kevin Doran <kd...@apache.org> wrote:
>>>>>>
>>>>>> Ok, that use case should be fine.
>>>>>>
>>>>>> If it were an authorization issue you would see something in the logs saying
>>>>>> that an authorization attempt failed and the server is responding with a
>>>>>> 403. Just to be sure, can you enable debug logging if you haven't already,
>>>>>> i.e., in your nifi-registry/conf/logback.xml file, change
>>>>>> 'org.apache.nifif.registry' to debug:
>>>>>>
>>>>>> <!-- valid logging levels: TRACE, DEBUG, INFO, WARN, ERROR -->
>>>>>> <logger name="org.apache.nifi.registry" level="DEBUG"/>
>>>>>>
>>>>>> If there is nothing being written to nifi-registry-app.log, it points
>>>>>> towards a connection issue, so I would double check your host, port, and TLS
>>>>>> settings. You'll have to get an HTTPS cert from a root CA or configure your
>>>>>> ELB to trust your company's self-signed cert (again, not sure if/how to do
>>>>>> this, but I assume there should be some way to configure it. It might
>>>>>> require settings not exposed in the AWS web console.)
>>>>>>
>>>>>> On 3/19/18, 10:51, "Scott Howell" <sc...@mobilgov.com> wrote:
>>>>>>
>>>>>> Thanks Kevin,
>>>>>>
>>>>>> I am just using the ELB to go from the public subnet to the private subnet.
>>>>>> I will not have multiple instances running of registry.
>>>>>>
>>>>>> I will say on my authorizers.xml there is one difference between my nifi
>>>>>> instance. On my nifi instance I am using file-provider for
>>>>>> nifi.security.user.authorizer in my nifi.properties. I don’t think from
>>>>>> reading the documents for nifi-registry that I can use that. If there is a
>>>>>> way that might be my problem. I was running into some issues with my nifi
>>>>>> instance when I was using managed-authorizers instead of file-provider.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mar 19, 2018, at 9:35 AM, Kevin Doran <kd...@apache.org> wrote:
>>>>>>
>>>>>> Hey Scott,
>>>>>>
>>>>>> Assuming you are using two-way TLS with client certificates for
>>>>>> authentication, I recommend configuring your ELB for TCP passthrough so that
>>>>>> the TLS handshake is between the end-client and the NiFi Registry Server (in
>>>>>> other words, no decryption/termination of the TLS connection happens in the
>>>>>> ELB). If you are using some other form of authentication (e.g., LDAP), you
>>>>>> will need to configure your ELB to trust the self-signed key NiFi Registry
>>>>>> is using. I'm not sure how to do that as I've never run an ELB with that
>>>>>> configuration before.
>>>>>>
>>>>>> Also, just a note about using an ELB with NiFi Registry:
>>>>>>
>>>>>> NiFi Registry is currently only supports single-instance use as persisted
>>>>>> data and in-memory state is not synced between multiple instances. Are you
>>>>>> hoping to use the ELB for actual load balancing, or is it just to take
>>>>>> advantage of other ELB features, such as forwarding and security group
>>>>>> rules? If the plan is to load balance multiple Registry instances, just be
>>>>>> aware that you will probably run into some unexpected behavior. (As you
>>>>>> mentioned using authorization, that is one case where I know the in-memory
>>>>>> cache of the persisted data will not refresh across instances, so even if
>>>>>> you were using some sort of shared network file system attached to multiple
>>>>>> Registry instances, such as EFS, it would not work the way you hope.)
>>>>>>
>>>>>> Hope this helps,
>>>>>> Kevin
>>>>>>
>>>>>> On 3/19/18, 10:20, "Scott Howell" <sc...@mobilgov.com> wrote:
>>>>>>
>>>>>> Thanks for the quick response.
>>>>>>
>>>>>> A couple of things I am seeing.
>>>>>>
>>>>>> 1. There is no error, I don’t see anything in the logs once the service
>>>>>> comes up. This is because the health check is not even hitting the instance
>>>>>> when secure.
>>>>>>
>>>>>> 2. Nothing interesting in the nifi-registry-app.logs. That was my concern
>>>>>> because on my nifi instance I can see the health check hitting the instance
>>>>>> from the ELB. This does not happen on the nifi-registry instance. I see the
>>>>>> service startup and it tells me what domain and port I can access the UI but
>>>>>> nothing else after that.
>>>>>>
>>>>>> 3. When I am on an instances in the same private subnet I am able to curl to
>>>>>> the instance I get the TLS SSL which tells me the keystore is on the server.
>>>>>> I am using a JKS keystore that is self-signed by the company I work for.
>>>>>>
>>>>>> On Mar 19, 2018, at 9:10 AM, Bryan Bende <bb...@gmail.com> wrote:
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> What error are you getting when you cannot access the UI?
>>>>>>
>>>>>> Is there anything interesting in nifi-registry-app.log regarding
>>>>>> authentication/authorization when this happens?
>>>>>>
>>>>>> Can you access the UI securely without going through the ELB?
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Bryan
>>>>>>
>>>>>>
>>>>>> On Mon, Mar 19, 2018 at 10:05 AM, Scott Howell <sc...@mobilgov.com>
>>>>>> wrote:
>>>>>>
>>>>>> I was able to stand up nifi-registry behind an AWS ELB non-secure.
>>>>>> Everything was working great and was able to access the UI anonymously. I
>>>>>> set up the authorization just like on my nifi instances along with the
>>>>>> authorizers and identity-provider. The service comes up without errors and
>>>>>> everything looks good but the health check does not pass and I cannot access
>>>>>> the UI to login. I was wondering if anyone else has ran into this issue
>>>>>> using nifi-registry.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>
>>>
>>
>
>
Re: Issue with AWS ELB on secure nifi-registry
Posted by Scott Howell <sc...@mobilgov.com>.
Thanks I have checked that and the whitespace is correct in user.xml.
I did make a change to my authorizer.xml
<authorizers>
<authorizer>
<identifier>file-provider</identifier>
<class>org.apache.nifi.authorization.FileAuthorizer</class>
<property name="Authorizations File">/opt/config/authorizations.xml</property>
<property name="Users File">/opt/config/users.xml</property>
<property name="Initial Admin Identity">uid=scott,ou=users,dc=mobilgov,dc=com</property>
<property name="Legacy Authorized Users File"></property>
<property name="Node Identity 1">CN=*.{redacted}.com, OU={redacted}, O={redacted}, L=Kansas City, ST=Missouri, C=US</property>
<property name="Node Identity 2">CN=*.{redacted}.com, OU={redacted}, O={redacted}, L=Kansas City, ST=Missouri, C=US</property>
</authorizer>
</authorizers>
I made the CN=*.{redacted}.com just like the self-signed certificate showed. I now have a 1 node cluster up and running. It seems like NIfi isn’t taking into account the wildcard and treating it as a “*” instead.
> On Mar 21, 2018, at 11:33 AM, Bryan Bende <bb...@gmail.com> wrote:
>
> There only needs to be W to /proxy so that part should be fine.
>
> After you edited the Node Identities, did you delete users.xml and
> authorizations.xml?
>
> You would have to do that for those changes to take effect. You can
> look in users.xml and see if you still have the user identities
> without whitespace.
>
> On Wed, Mar 21, 2018 at 12:20 PM, Scott Howell <sc...@mobilgov.com> wrote:
>> One other thing I am seeing and I don’t know if this is an issue or not in my authorizations.xml I do not have a policy for /proxy with action=“R” only action=“W”.
>>
>>> On Mar 21, 2018, at 11:03 AM, Scott Howell <sc...@mobilgov.com> wrote:
>>>
>>> Thanks for that. I am still getting this error in my nifi-user.log
>>>
>>> o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api: Untrusted proxy CN=*.{redacted}.com, OU={redacted}, O={redacted}, L=Kansas City, ST=Missouri, C=US
>>>
>>> Is there an issue with using a wildcard cert?
>>>
>>>
>>>> On Mar 21, 2018, at 10:23 AM, Bryan Bende <bb...@gmail.com> wrote:
>>>>
>>>> All identity strings are case & whitespace sensitive.
>>>>
>>>> The node identities in your authorizers.xml have no whitespace, and
>>>> the identity showing in the logs does.
>>>>
>>>> On Wed, Mar 21, 2018 at 11:05 AM, Scott Howell <sc...@mobilgov.com> wrote:
>>>>> Thanks for all of the help with this. I have the cluster up and running. The
>>>>> logs look great everything seems to be working but I cannot login into the
>>>>> UI. I am using a wildcard self-signed certificate. The /proxy is in
>>>>> authorizations.xml with the correct users for the nodes.
>>>>>
>>>>> The error I see with the UI :
>>>>> is Untrusted proxy CN=*.{redacted}.com, OU={redacted}, O={redacted},
>>>>> L=Kansas City, ST=Missouri, C=US
>>>>>
>>>>> I haven’t had much luck finding a lot of documentation or forum questions
>>>>> with this kind of issue.
>>>>>
>>>>> My authorizers.xml looks like this
>>>>> <authorizers>
>>>>> <authorizer>
>>>>> <identifier>file-provider</identifier>
>>>>> <class>org.apache.nifi.authorization.FileAuthorizer</class>
>>>>> <property name="Authorizations
>>>>> File">/opt/config/authorizations.xml</property>
>>>>> <property name="Users File">/opt/config/users.xml</property>
>>>>> <property name="Initial Admin
>>>>> Identity">uid=scott,ou=users,dc={redacted},dc=com</property>
>>>>> <property name="Legacy Authorized Users File"></property>
>>>>>
>>>>> <property name="Node Identity
>>>>> 1">CN=node-1-nifi-dev.{redacted}.com,OU={redacted},O={redacted},L=Kansas
>>>>> City,ST=Missouri,C=US</property>
>>>>> <property name="Node Identity
>>>>> 2">CN=node-2-nifi-dev.{redacted}.com,OU={redacted},O={redacted},L=Kansas
>>>>> City,ST=Missouri,C=US</property>
>>>>> </authorizer>
>>>>> </authorizers>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Scott
>>>>>
>>>>> On Mar 20, 2018, at 1:15 PM, Andy LoPresto <al...@apache.org> wrote:
>>>>>
>>>>> Scott,
>>>>>
>>>>> The original exception is "nested exception is
>>>>> java.security.KeyStoreException: not found”. Can you verify that the
>>>>> keystore you’ve provided is valid using the “keytool” command? In addition,
>>>>> you will need a truststore as well. Try following Pierre's [1] or Bryan’s
>>>>> [2] instructions for setting up a secure cluster.
>>>>>
>>>>> [1]
>>>>> https://pierrevillard.com/2016/11/29/apache-nifi-1-1-0-secured-cluster-setup/
>>>>> [2]
>>>>> https://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-authorization-and-multi-tenancy
>>>>>
>>>>>
>>>>> Andy LoPresto
>>>>> alopresto@apache.org
>>>>> alopresto.apache@gmail.com
>>>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>>>>>
>>>>> On Mar 20, 2018, at 11:05 AM, Scott Howell <sc...@mobilgov.com> wrote:
>>>>>
>>>>> Thanks for all of the help yesterday I was able to get a secure nifi and
>>>>> nifi-registry up and communicating. I am now trying to figure out how to
>>>>> create a secure cluster. I am currently getting this error when I start up
>>>>> nifi.
>>>>>
>>>>> tion; nested exception is
>>>>> org.springframework.beans.factory.BeanCreationException: Error creating bean
>>>>> with name 'clusterCoordinationProtocolSenderListener' defined in class path
>>>>> resource [nifi-cluster-protocol-context.xml]: Cannot resolve reference to
>>>>> bean 'clusterCoordinationProtocolSender' while setting constructor argument;
>>>>> nested exception is org.springframework.beans.factory.BeanCreationException:
>>>>> Error creating bean with name 'clusterCoordinationProtocolSender' defined in
>>>>> class path resource [nifi-cluster-protocol-context.xml]: Cannot resolve
>>>>> reference to bean 'protocolSocketConfiguration' while setting constructor
>>>>> argument; nested exception is
>>>>> org.springframework.beans.factory.BeanCreationException: Error creating bean
>>>>> with name 'protocolSocketConfiguration': FactoryBean threw exception on
>>>>> object creation; nested exception is java.security.KeyStoreException: not
>>>>> found
>>>>> at
>>>>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175)
>>>>> at
>>>>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1634)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
>>>>> at
>>>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
>>>>> ... 50 common frames omitted
>>>>> Caused by: org.springframework.beans.factory.BeanCreationException: Error
>>>>> creating bean with name 'clusterCoordinationProtocolSenderListener' defined
>>>>> in class path resource [nifi-cluster-protocol-context.xml]: Cannot resolve
>>>>> reference to bean 'clusterCoordinationProtocolSender' while setting
>>>>> constructor argument; nested exception is
>>>>> org.springframework.beans.factory.BeanCreationException: Error creating bean
>>>>> with name 'clusterCoordinationProtocolSender' defined in class path resource
>>>>> [nifi-cluster-protocol-context.xml]: Cannot resolve reference to bean
>>>>> 'protocolSocketConfiguration' while setting constructor argument; nested
>>>>> exception is org.springframework.beans.factory.BeanCreationException: Error
>>>>> creating bean with name 'protocolSocketConfiguration': FactoryBean threw
>>>>> exception on object creation; nested exception is
>>>>> java.security.KeyStoreException: not found
>>>>> at
>>>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:359)
>>>>> at
>>>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)
>>>>> at
>>>>> org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:648)
>>>>> at
>>>>> org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:145)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1193)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1095)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
>>>>> at
>>>>> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
>>>>> at
>>>>> org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1084)
>>>>> at
>>>>> org.apache.nifi.cluster.spring.NodeClusterCoordinatorFactoryBean.getObject(NodeClusterCoordinatorFactoryBean.java:44)
>>>>> at
>>>>> org.apache.nifi.cluster.spring.NodeClusterCoordinatorFactoryBean.getObject(NodeClusterCoordinatorFactoryBean.java:34)
>>>>> at
>>>>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
>>>>> ... 55 common frames omitted
>>>>> Caused by: org.springframework.beans.factory.BeanCreationException: Error
>>>>> creating bean with name 'clusterCoordinationProtocolSender' defined in class
>>>>> path resource [nifi-cluster-protocol-context.xml]: Cannot resolve reference
>>>>> to bean 'protocolSocketConfiguration' while setting constructor argument;
>>>>> nested exception is org.springframework.beans.factory.BeanCreationException:
>>>>> Error creating bean with name 'protocolSocketConfiguration': FactoryBean
>>>>> threw exception on object creation; nested exception is
>>>>> java.security.KeyStoreException: not found
>>>>> at
>>>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:359)
>>>>> at
>>>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)
>>>>> at
>>>>> org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:648)
>>>>> at
>>>>> org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:145)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1193)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1095)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
>>>>> at
>>>>> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
>>>>> at
>>>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
>>>>> ... 70 common frames omitted
>>>>> Caused by: org.springframework.beans.factory.BeanCreationException: Error
>>>>> creating bean with name 'protocolSocketConfiguration': FactoryBean threw
>>>>> exception on object creation; nested exception is
>>>>> java.security.KeyStoreException: not found
>>>>> at
>>>>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175)
>>>>> at
>>>>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1634)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
>>>>> at
>>>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
>>>>> ... 82 common frames omitted
>>>>> Caused by: java.security.KeyStoreException: not found
>>>>> at java.security.KeyStore.getInstance(KeyStore.java:851)
>>>>> at
>>>>> org.apache.nifi.security.util.KeyStoreUtils.getKeyStore(KeyStoreUtils.java:66)
>>>>> at
>>>>> org.apache.nifi.security.util.KeyStoreUtils.getTrustStore(KeyStoreUtils.java:80)
>>>>> at
>>>>> org.apache.nifi.io.socket.SSLContextFactory.<init>(SSLContextFactory.java:73)
>>>>> at
>>>>> org.apache.nifi.cluster.protocol.spring.SocketConfigurationFactoryBean.getObject(SocketConfigurationFactoryBean.java:45)
>>>>> at
>>>>> org.apache.nifi.cluster.protocol.spring.SocketConfigurationFactoryBean.getObject(SocketConfigurationFactoryBean.java:30)
>>>>> at
>>>>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
>>>>> ... 87 common frames omitted
>>>>> Caused by: java.security.NoSuchAlgorithmException: KeyStore not available
>>>>> at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
>>>>> at java.security.Security.getImpl(Security.java:695)
>>>>> at java.security.KeyStore.getInstance(KeyStore.java:848)
>>>>> ... 93 common frames omitted
>>>>>
>>>>> My nifi.properties file is.
>>>>>
>>>>> # Licensed to the Apache Software Foundation (ASF) under one or more
>>>>> # contributor license agreements. See the NOTICE file distributed with
>>>>> # this work for additional information regarding copyright ownership.
>>>>> # The ASF licenses this file to You under the Apache License, Version 2.0
>>>>> # (the "License"); you may not use this file except in compliance with
>>>>> # the License. You may obtain a copy of the License at
>>>>> #
>>>>> # http://www.apache.org/licenses/LICENSE-2.0
>>>>> #
>>>>> # Unless required by applicable law or agreed to in writing, software
>>>>> # distributed under the License is distributed on an "AS IS" BASIS,
>>>>> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>>>>> # See the License for the specific language governing permissions and
>>>>> # limitations under the License.
>>>>>
>>>>> # Core Properties #
>>>>> nifi.version={{nifi_version}}
>>>>> nifi.flow.configuration.file=/opt/config/flow.xml.gz
>>>>> nifi.flow.configuration.archive.enabled=true
>>>>> nifi.flow.configuration.archive.dir=/opt/config/archive/
>>>>> nifi.flow.configuration.archive.max.time=30 days
>>>>> nifi.flow.configuration.archive.max.storage=500 MB
>>>>> nifi.flowcontroller.autoResumeState=true
>>>>> nifi.flowcontroller.graceful.shutdown.period=10 sec
>>>>> nifi.flowservice.writedelay.interval=500 ms
>>>>> nifi.administrative.yield.duration=30 sec
>>>>> # If a component has no work to do (is "bored"), how long should we wait
>>>>> before checking again for work?
>>>>> nifi.bored.yield.duration=10 millis
>>>>>
>>>>>
>>>>> nifi.authorizer.configuration.file=/opt/config/authorizers.xml
>>>>> nifi.login.identity.provider.configuration.file=/opt/config/login-identity-providers.xml
>>>>> nifi.templates.directory=/opt/config/templates
>>>>> nifi.ui.banner.text=
>>>>> nifi.ui.autorefresh.interval=30 sec
>>>>> nifi.nar.library.directory=/opt/nifi/lib
>>>>> nifi.nar.library.directory.custom=/opt/config/processors
>>>>> nifi.nar.working.directory=/opt/nifi/work/nar/
>>>>> nifi.documentation.working.directory=./work/docs/components
>>>>>
>>>>> ####################
>>>>> # State Management #
>>>>> ####################
>>>>> nifi.state.management.configuration.file=/opt/config/state-management.xml
>>>>> # The ID of the local state provider
>>>>> nifi.state.management.provider.local=local-provider
>>>>> # The ID of the cluster-wide state provider. This will be ignored if NiFi is
>>>>> not clustered but must be populated if running in a cluster.
>>>>> nifi.state.management.provider.cluster=zk-provider
>>>>> # Specifies whether or not this instance of NiFi should run an embedded
>>>>> ZooKeeper server
>>>>> nifi.state.management.embedded.zookeeper.start=false
>>>>> # Properties file that provides the ZooKeeper properties to use if
>>>>> <nifi.state.management.embedded.zookeeper.start> is set to true
>>>>> nifi.state.management.embedded.zookeeper.properties=/opt/config/zookeeper.properties
>>>>>
>>>>>
>>>>> # H2 Settings
>>>>> nifi.database.directory=/opt/database_repository
>>>>> nifi.h2.url.append=;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE
>>>>>
>>>>> # FlowFile Repository
>>>>> nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository
>>>>> nifi.flowfile.repository.directory=/opt/flowfile_repository
>>>>> nifi.flowfile.repository.partitions=256
>>>>> nifi.flowfile.repository.checkpoint.interval=2 mins
>>>>> nifi.flowfile.repository.always.sync=false
>>>>>
>>>>> nifi.swap.manager.implementation=org.apache.nifi.controller.FileSystemSwapManager
>>>>> nifi.queue.swap.threshold=20000
>>>>> nifi.swap.in.period=5 sec
>>>>> nifi.swap.in.threads=1
>>>>> nifi.swap.out.period=5 sec
>>>>> nifi.swap.out.threads=4
>>>>>
>>>>> # Content Repository
>>>>> nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository
>>>>> nifi.content.claim.max.appendable.size=10 MB
>>>>> nifi.content.claim.max.flow.files=100
>>>>> nifi.content.repository.directory.default=/opt/content_repository
>>>>> nifi.content.repository.archive.max.retention.period=12 hours
>>>>> nifi.content.repository.archive.max.usage.percentage=50%
>>>>> nifi.content.repository.archive.enabled=true
>>>>> nifi.content.repository.always.sync=false
>>>>> nifi.content.viewer.url=/nifi-content-viewer/
>>>>>
>>>>> # Provenance Repository Properties
>>>>> nifi.provenance.repository.implementation=org.apache.nifi.provenance.PersistentProvenanceRepository
>>>>>
>>>>> # Persistent Provenance Repository Properties
>>>>> nifi.provenance.repository.directory.default=/opt/provenance_repository
>>>>> nifi.provenance.repository.max.storage.time=24 hours
>>>>> nifi.provenance.repository.max.storage.size=1 GB
>>>>> nifi.provenance.repository.rollover.time=30 secs
>>>>> nifi.provenance.repository.rollover.size=100 MB
>>>>> nifi.provenance.repository.query.threads=2
>>>>> nifi.provenance.repository.index.threads=1
>>>>> nifi.provenance.repository.compress.on.rollover=true
>>>>> nifi.provenance.repository.always.sync=false
>>>>> nifi.provenance.repository.journal.count=16
>>>>> # Comma-separated list of fields. Fields that are not indexed will not be
>>>>> searchable. Valid fields are:
>>>>> # EventType, FlowFileUUID, Filename, TransitURI, ProcessorID,
>>>>> AlternateIdentifierURI, Relationship, Details
>>>>> nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename,
>>>>> ProcessorID, Relationship
>>>>> # FlowFile Attributes that should be indexed and made searchable. Some
>>>>> examples to consider are filename, uuid, mime.type
>>>>> nifi.provenance.repository.indexed.attributes=
>>>>> # Large values for the shard size will result in more Java heap usage when
>>>>> searching the Provenance Repository
>>>>> # but should provide better performance
>>>>> nifi.provenance.repository.index.shard.size=500 MB
>>>>> # Indicates the maximum length that a FlowFile attribute can be when
>>>>> retrieving a Provenance Event from
>>>>> # the repository. If the length of any attribute exceeds this value, it will
>>>>> be truncated when the event is retrieved.
>>>>> nifi.provenance.repository.max.attribute.length=65536
>>>>>
>>>>> # Volatile Provenance Respository Properties
>>>>> nifi.provenance.repository.buffer.size=100000
>>>>>
>>>>> # Component Status Repository
>>>>> nifi.components.status.repository.implementation=org.apache.nifi.controller.status.history.VolatileComponentStatusRepository
>>>>> nifi.components.status.repository.buffer.size=1440
>>>>> nifi.components.status.snapshot.frequency=1 min
>>>>>
>>>>> # Site to Site properties
>>>>> nifi.remote.input.host=
>>>>> nifi.remote.input.secure=false
>>>>> nifi.remote.input.socket.port=9998
>>>>> nifi.remote.input.http.enabled=false
>>>>> nifi.remote.input.http.transaction.ttl=30 sec
>>>>>
>>>>> # web properties #
>>>>> nifi.web.war.directory=/opt/nifi/lib
>>>>> nifi.web.http.host=
>>>>> nifi.web.http.port=
>>>>> nifi.web.https.host={{redacted}}
>>>>> nifi.web.https.port=8443
>>>>> nifi.web.jetty.working.directory=/opt/nifi/work/jetty
>>>>> nifi.web.jetty.threads=200
>>>>>
>>>>> # security properties #
>>>>> nifi.sensitive.props.key=x0KDgO9L8lAhFGLdvu2VEjFVGc6Kg3V0R5I4bYwoqdgL47moo0wApKQtAVu1BvD
>>>>> nifi.sensitive.props.key.protected=
>>>>> nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
>>>>> nifi.sensitive.props.provider=BC
>>>>> nifi.sensitive.props.additional.keys=
>>>>>
>>>>> nifi.security.keystore=/opt/certs/payit_keystore
>>>>> nifi.security.keystoreType=JKS
>>>>> nifi.security.keystorePasswd={{keystore_password}}
>>>>> nifi.security.keyPasswd=
>>>>> nifi.security.truststore=
>>>>> nifi.security.truststoreType=
>>>>> nifi.security.truststorePasswd=
>>>>> nifi.security.needClientAuth=false
>>>>> nifi.security.user.authorizer=file-provider
>>>>> nifi.security.user.login.identity.provider=ldap-provider
>>>>> nifi.security.ocsp.responder.url=
>>>>> nifi.security.ocsp.responder.certificate=
>>>>>
>>>>> # Identity Mapping Properties #
>>>>> # These properties allow normalizing user identities such that identities
>>>>> coming from different identity providers
>>>>> # (certificates, LDAP, Kerberos) can be treated the same internally in NiFi.
>>>>> The following example demonstrates normalizing
>>>>> # DNs from certificates and principals from Kerberos into a common identity
>>>>> string:
>>>>> #
>>>>> #nifi.security.identity.mapping.pattern.dn=^cn=(.*?),ou=(.*?),dc=(.*?),dc=(.*?),dc=(.*?)$
>>>>> #nifi.security.identity.mapping.value.dn=$1
>>>>> # nifi.security.identity.mapping.pattern.kerb=^(.*?)/instance@(.*?)$
>>>>> # nifi.security.identity.mapping.value.kerb=$1@$2
>>>>>
>>>>> # cluster common properties (all nodes must have same values) #
>>>>> nifi.cluster.protocol.heartbeat.interval=5 sec
>>>>> nifi.cluster.protocol.is.secure=true
>>>>>
>>>>> # cluster node properties (only configure for cluster nodes) #
>>>>> nifi.cluster.is.node=true
>>>>> nifi.cluster.node.address=nifi-dev.mobilgov.com
>>>>> nifi.cluster.node.protocol.port=9999
>>>>> nifi.cluster.node.protocol.threads=10
>>>>> nifi.cluster.node.event.history.size=25
>>>>> nifi.cluster.node.connection.timeout=5 sec
>>>>> nifi.cluster.node.read.timeout=5 sec
>>>>> nifi.cluster.firewall.file=
>>>>>
>>>>>
>>>>> # zookeeper properties, used for cluster management #
>>>>> nifi.zookeeper.connect.string=internal-etcd-dev-EtcdLoadB-3RWA2WEFBBT8-2068560477.us-east-2.elb.amazonaws.com:2181,internal-etcd-dev-EtcdLoadB-3RWA2WEFBBT8-2068560477.us-east-2.elb.amazonaws.com2182,internal-etcd-dev-EtcdLoadB-3RWA2WEFBBT8-2068560477.us-east-2.elb.amazonaws.com:2183
>>>>> nifi.zookeeper.connect.timeout=3 secs
>>>>> nifi.zookeeper.session.timeout=3 secs
>>>>> nifi.zookeeper.root.node=/nifi
>>>>>
>>>>> # kerberos #
>>>>> nifi.kerberos.krb5.file=
>>>>>
>>>>> # kerberos service principle #
>>>>> nifi.kerberos.service.principal=
>>>>> nifi.kerberos.service.keytab.location=
>>>>>
>>>>> # kerberos spnego principle #
>>>>> nifi.kerberos.spnego.principal=
>>>>> nifi.kerberos.spnego.keytab.location=
>>>>> nifi.kerberos.spnego.authentication.expiration=12 hours
>>>>>
>>>>> # external properties files for variable registry
>>>>> # supports a comma delimited list of file locations
>>>>> nifi.variable.registry.properties=
>>>>>
>>>>> I think I have everything set correctly but I have not been able to start an
>>>>> instances up.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Scott
>>>>>
>>>>> On Mar 19, 2018, at 4:35 PM, Bryan Bende <bb...@gmail.com> wrote:
>>>>>
>>>>> The base file is here for comparison:
>>>>>
>>>>> https://github.com/apache/nifi-registry/blob/master/nifi-registry-resources/src/main/resources/conf/identity-providers.xml#L23
>>>>>
>>>>> On Mon, Mar 19, 2018 at 5:34 PM, Bryan Bende <bb...@gmail.com> wrote:
>>>>>
>>>>> For your first file, is what you showed there actually wrapped in
>>>>> <identityProviders> </identityProviders> or is it exactly what you
>>>>> showed?
>>>>>
>>>>> It may just be that you only copied/pasted the one provider, but the
>>>>> root element is not <provider>, so as it is shown there it would not
>>>>> parse.
>>>>>
>>>>> On Mon, Mar 19, 2018 at 2:54 PM, Scott Howell <sc...@mobilgov.com>
>>>>> wrote:
>>>>>
>>>>> Here is my file
>>>>>
>>>>> <provider>
>>>>> <identifier>ldap-identity-provider</identifier>
>>>>> <class>org.apache.nifi.registry.security.ldap.LdapProvider</class>
>>>>> <property name="Authentication Strategy">SIMPLE</property>
>>>>>
>>>>> <property name="Manager DN">cn=Manager,dc=mobilgov,dc=com</property>
>>>>> <property name="Manager Password”>redacted</property>
>>>>>
>>>>>
>>>>> <property name="Referral Strategy">FOLLOW</property>
>>>>> <property name="Connect Timeout">10 secs</property>
>>>>> <property name="Read Timeout">10 secs</property>
>>>>>
>>>>> <property name="Url”>redacted</property>
>>>>> <property name="User Search
>>>>> Base">ou=users,dc=mobilgov,dc=com</property>
>>>>> <property name="User Search Filter">uid={0}</property>
>>>>>
>>>>> <property name="Identity Strategy">USE_DN</property>
>>>>> <property name="Authentication Expiration">12 hours</property>
>>>>> </provider>
>>>>>
>>>>> Here is my authorizers.xml
>>>>>
>>>>> <authorizers>
>>>>>
>>>>> <userGroupProvider>
>>>>> <identifier>file-user-group-provider</identifier>
>>>>>
>>>>> <class>org.apache.nifi.registry.security.authorization.file.FileUserGroupProvider</class>
>>>>> <property name="Users File">conf/users.xml</property>
>>>>> <property name="Legacy Authorized Users File"></property>
>>>>> <property name="Initial User Identity 1”>redacted</property>
>>>>> </userGroupProvider>
>>>>>
>>>>> <accessPolicyProvider>
>>>>> <identifier>file-access-policy-provider</identifier>
>>>>>
>>>>> <class>org.apache.nifi.registry.security.authorization.file.FileAccessPolicyProvider</class>
>>>>> <property name="User Group
>>>>> Provider">file-user-group-provider</property>
>>>>> <property name="Authorizations
>>>>> File">conf/authorizations.xml</property>
>>>>> <property name="Initial Admin Identity”>redacted</property>
>>>>> <property name="NiFi Identity 1"></property>
>>>>> </accessPolicyProvider>
>>>>>
>>>>> <authorizer>
>>>>> <identifier>managed-authorizer</identifier>
>>>>>
>>>>> <class>org.apache.nifi.registry.security.authorization.StandardManagedAuthorizer</class>
>>>>> <property name="Access Policy
>>>>> Provider">file-access-policy-provider</property>
>>>>> </authorizer>
>>>>> </authorizers>
>>>>>
>>>>> On Mar 19, 2018, at 12:59 PM, Bryan Bende <bb...@gmail.com> wrote:
>>>>>
>>>>> It looks like that error would happen if your identity-providers.xml
>>>>> contained invalid XML.
>>>>>
>>>>> Did you start by modifying the identity-providers.xml file that was
>>>>> already there? Can you share the file, or the contents (removing
>>>>> anything sensitive)?
>>>>>
>>>>> On Mon, Mar 19, 2018 at 1:09 PM, Scott Howell <sc...@mobilgov.com>
>>>>> wrote:
>>>>>
>>>>> So I was able to get the UI pulled up but now I am hitting a roadblock with
>>>>> my identity-provider.xml.
>>>>>
>>>>> I am getting a number of errors like this:
>>>>>
>>>>> Caused by: org.springframework.beans.factory.BeanCreationException: Error
>>>>> creating bean with name 'getIdentityProvider' defined in class path resource
>>>>> [org/apache/nifi/registry/security/authentication/IdentityProviderFactory.class]:
>>>>> Bean instantiation via factory method failed; nested exception is
>>>>> org.springframework.beans.BeanInstantiationException: Failed to instantiate
>>>>> [org.apache.nifi.registry.security.authentication.IdentityProvider]: Factory
>>>>> method 'getIdentityProvider' threw exception; nested exception is
>>>>> java.lang.Exception: Unable to load the login identity provider
>>>>> configuration file at: /opt/nifi-registry-0.1.0/conf/identity-providers.xml
>>>>> at
>>>>> org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:587)
>>>>> ~[na:na]
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1250)
>>>>> ~[na:na]
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1099)
>>>>> ~[na:na]
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:545)
>>>>> ~[na:na]
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:502)
>>>>> ~[na:na]
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:312)
>>>>> ~[na:na]
>>>>> at
>>>>> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228)
>>>>> ~[na:na]
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:310)
>>>>> ~[na:na]
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200)
>>>>> ~[na:na]
>>>>> at
>>>>> org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:251)
>>>>> ~[na:na]
>>>>> at
>>>>> org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1135)
>>>>> ~[na:na]
>>>>> at
>>>>> org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1062)
>>>>> ~[na:na]
>>>>> at
>>>>> org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:815)
>>>>> ~[na:na]
>>>>> at
>>>>> org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:721)
>>>>> ~[na:na]
>>>>> ... 43 common frames omitted
>>>>>
>>>>> I know it has to do with the identity-provider.xml but I have my setup just
>>>>> like the documentation ask for. I turned on debug but was not able to see
>>>>> anything different or better explanation from it.
>>>>>
>>>>>
>>>>> On Mar 19, 2018, at 10:06 AM, Kevin Doran <kd...@apache.org> wrote:
>>>>>
>>>>> Ok, that use case should be fine.
>>>>>
>>>>> If it were an authorization issue you would see something in the logs saying
>>>>> that an authorization attempt failed and the server is responding with a
>>>>> 403. Just to be sure, can you enable debug logging if you haven't already,
>>>>> i.e., in your nifi-registry/conf/logback.xml file, change
>>>>> 'org.apache.nifif.registry' to debug:
>>>>>
>>>>> <!-- valid logging levels: TRACE, DEBUG, INFO, WARN, ERROR -->
>>>>> <logger name="org.apache.nifi.registry" level="DEBUG"/>
>>>>>
>>>>> If there is nothing being written to nifi-registry-app.log, it points
>>>>> towards a connection issue, so I would double check your host, port, and TLS
>>>>> settings. You'll have to get an HTTPS cert from a root CA or configure your
>>>>> ELB to trust your company's self-signed cert (again, not sure if/how to do
>>>>> this, but I assume there should be some way to configure it. It might
>>>>> require settings not exposed in the AWS web console.)
>>>>>
>>>>> On 3/19/18, 10:51, "Scott Howell" <sc...@mobilgov.com> wrote:
>>>>>
>>>>> Thanks Kevin,
>>>>>
>>>>> I am just using the ELB to go from the public subnet to the private subnet.
>>>>> I will not have multiple instances running of registry.
>>>>>
>>>>> I will say on my authorizers.xml there is one difference between my nifi
>>>>> instance. On my nifi instance I am using file-provider for
>>>>> nifi.security.user.authorizer in my nifi.properties. I don’t think from
>>>>> reading the documents for nifi-registry that I can use that. If there is a
>>>>> way that might be my problem. I was running into some issues with my nifi
>>>>> instance when I was using managed-authorizers instead of file-provider.
>>>>>
>>>>>
>>>>>
>>>>> On Mar 19, 2018, at 9:35 AM, Kevin Doran <kd...@apache.org> wrote:
>>>>>
>>>>> Hey Scott,
>>>>>
>>>>> Assuming you are using two-way TLS with client certificates for
>>>>> authentication, I recommend configuring your ELB for TCP passthrough so that
>>>>> the TLS handshake is between the end-client and the NiFi Registry Server (in
>>>>> other words, no decryption/termination of the TLS connection happens in the
>>>>> ELB). If you are using some other form of authentication (e.g., LDAP), you
>>>>> will need to configure your ELB to trust the self-signed key NiFi Registry
>>>>> is using. I'm not sure how to do that as I've never run an ELB with that
>>>>> configuration before.
>>>>>
>>>>> Also, just a note about using an ELB with NiFi Registry:
>>>>>
>>>>> NiFi Registry is currently only supports single-instance use as persisted
>>>>> data and in-memory state is not synced between multiple instances. Are you
>>>>> hoping to use the ELB for actual load balancing, or is it just to take
>>>>> advantage of other ELB features, such as forwarding and security group
>>>>> rules? If the plan is to load balance multiple Registry instances, just be
>>>>> aware that you will probably run into some unexpected behavior. (As you
>>>>> mentioned using authorization, that is one case where I know the in-memory
>>>>> cache of the persisted data will not refresh across instances, so even if
>>>>> you were using some sort of shared network file system attached to multiple
>>>>> Registry instances, such as EFS, it would not work the way you hope.)
>>>>>
>>>>> Hope this helps,
>>>>> Kevin
>>>>>
>>>>> On 3/19/18, 10:20, "Scott Howell" <sc...@mobilgov.com> wrote:
>>>>>
>>>>> Thanks for the quick response.
>>>>>
>>>>> A couple of things I am seeing.
>>>>>
>>>>> 1. There is no error, I don’t see anything in the logs once the service
>>>>> comes up. This is because the health check is not even hitting the instance
>>>>> when secure.
>>>>>
>>>>> 2. Nothing interesting in the nifi-registry-app.logs. That was my concern
>>>>> because on my nifi instance I can see the health check hitting the instance
>>>>> from the ELB. This does not happen on the nifi-registry instance. I see the
>>>>> service startup and it tells me what domain and port I can access the UI but
>>>>> nothing else after that.
>>>>>
>>>>> 3. When I am on an instances in the same private subnet I am able to curl to
>>>>> the instance I get the TLS SSL which tells me the keystore is on the server.
>>>>> I am using a JKS keystore that is self-signed by the company I work for.
>>>>>
>>>>> On Mar 19, 2018, at 9:10 AM, Bryan Bende <bb...@gmail.com> wrote:
>>>>>
>>>>> Hello,
>>>>>
>>>>> What error are you getting when you cannot access the UI?
>>>>>
>>>>> Is there anything interesting in nifi-registry-app.log regarding
>>>>> authentication/authorization when this happens?
>>>>>
>>>>> Can you access the UI securely without going through the ELB?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Bryan
>>>>>
>>>>>
>>>>> On Mon, Mar 19, 2018 at 10:05 AM, Scott Howell <sc...@mobilgov.com>
>>>>> wrote:
>>>>>
>>>>> I was able to stand up nifi-registry behind an AWS ELB non-secure.
>>>>> Everything was working great and was able to access the UI anonymously. I
>>>>> set up the authorization just like on my nifi instances along with the
>>>>> authorizers and identity-provider. The service comes up without errors and
>>>>> everything looks good but the health check does not pass and I cannot access
>>>>> the UI to login. I was wondering if anyone else has ran into this issue
>>>>> using nifi-registry.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>
>>
>>
>
Re: Issue with AWS ELB on secure nifi-registry
Posted by Bryan Bende <bb...@gmail.com>.
There only needs to be W to /proxy so that part should be fine.
After you edited the Node Identities, did you delete users.xml and
authorizations.xml?
You would have to do that for those changes to take effect. You can
look in users.xml and see if you still have the user identities
without whitespace.
On Wed, Mar 21, 2018 at 12:20 PM, Scott Howell <sc...@mobilgov.com> wrote:
> One other thing I am seeing and I don’t know if this is an issue or not in my authorizations.xml I do not have a policy for /proxy with action=“R” only action=“W”.
>
>> On Mar 21, 2018, at 11:03 AM, Scott Howell <sc...@mobilgov.com> wrote:
>>
>> Thanks for that. I am still getting this error in my nifi-user.log
>>
>> o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api: Untrusted proxy CN=*.{redacted}.com, OU={redacted}, O={redacted}, L=Kansas City, ST=Missouri, C=US
>>
>> Is there an issue with using a wildcard cert?
>>
>>
>>> On Mar 21, 2018, at 10:23 AM, Bryan Bende <bb...@gmail.com> wrote:
>>>
>>> All identity strings are case & whitespace sensitive.
>>>
>>> The node identities in your authorizers.xml have no whitespace, and
>>> the identity showing in the logs does.
>>>
>>> On Wed, Mar 21, 2018 at 11:05 AM, Scott Howell <sc...@mobilgov.com> wrote:
>>>> Thanks for all of the help with this. I have the cluster up and running. The
>>>> logs look great everything seems to be working but I cannot login into the
>>>> UI. I am using a wildcard self-signed certificate. The /proxy is in
>>>> authorizations.xml with the correct users for the nodes.
>>>>
>>>> The error I see with the UI :
>>>> is Untrusted proxy CN=*.{redacted}.com, OU={redacted}, O={redacted},
>>>> L=Kansas City, ST=Missouri, C=US
>>>>
>>>> I haven’t had much luck finding a lot of documentation or forum questions
>>>> with this kind of issue.
>>>>
>>>> My authorizers.xml looks like this
>>>> <authorizers>
>>>> <authorizer>
>>>> <identifier>file-provider</identifier>
>>>> <class>org.apache.nifi.authorization.FileAuthorizer</class>
>>>> <property name="Authorizations
>>>> File">/opt/config/authorizations.xml</property>
>>>> <property name="Users File">/opt/config/users.xml</property>
>>>> <property name="Initial Admin
>>>> Identity">uid=scott,ou=users,dc={redacted},dc=com</property>
>>>> <property name="Legacy Authorized Users File"></property>
>>>>
>>>> <property name="Node Identity
>>>> 1">CN=node-1-nifi-dev.{redacted}.com,OU={redacted},O={redacted},L=Kansas
>>>> City,ST=Missouri,C=US</property>
>>>> <property name="Node Identity
>>>> 2">CN=node-2-nifi-dev.{redacted}.com,OU={redacted},O={redacted},L=Kansas
>>>> City,ST=Missouri,C=US</property>
>>>> </authorizer>
>>>> </authorizers>
>>>>
>>>> Thanks,
>>>>
>>>> Scott
>>>>
>>>> On Mar 20, 2018, at 1:15 PM, Andy LoPresto <al...@apache.org> wrote:
>>>>
>>>> Scott,
>>>>
>>>> The original exception is "nested exception is
>>>> java.security.KeyStoreException: not found”. Can you verify that the
>>>> keystore you’ve provided is valid using the “keytool” command? In addition,
>>>> you will need a truststore as well. Try following Pierre's [1] or Bryan’s
>>>> [2] instructions for setting up a secure cluster.
>>>>
>>>> [1]
>>>> https://pierrevillard.com/2016/11/29/apache-nifi-1-1-0-secured-cluster-setup/
>>>> [2]
>>>> https://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-authorization-and-multi-tenancy
>>>>
>>>>
>>>> Andy LoPresto
>>>> alopresto@apache.org
>>>> alopresto.apache@gmail.com
>>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>>>>
>>>> On Mar 20, 2018, at 11:05 AM, Scott Howell <sc...@mobilgov.com> wrote:
>>>>
>>>> Thanks for all of the help yesterday I was able to get a secure nifi and
>>>> nifi-registry up and communicating. I am now trying to figure out how to
>>>> create a secure cluster. I am currently getting this error when I start up
>>>> nifi.
>>>>
>>>> tion; nested exception is
>>>> org.springframework.beans.factory.BeanCreationException: Error creating bean
>>>> with name 'clusterCoordinationProtocolSenderListener' defined in class path
>>>> resource [nifi-cluster-protocol-context.xml]: Cannot resolve reference to
>>>> bean 'clusterCoordinationProtocolSender' while setting constructor argument;
>>>> nested exception is org.springframework.beans.factory.BeanCreationException:
>>>> Error creating bean with name 'clusterCoordinationProtocolSender' defined in
>>>> class path resource [nifi-cluster-protocol-context.xml]: Cannot resolve
>>>> reference to bean 'protocolSocketConfiguration' while setting constructor
>>>> argument; nested exception is
>>>> org.springframework.beans.factory.BeanCreationException: Error creating bean
>>>> with name 'protocolSocketConfiguration': FactoryBean threw exception on
>>>> object creation; nested exception is java.security.KeyStoreException: not
>>>> found
>>>> at
>>>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175)
>>>> at
>>>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
>>>> at
>>>> org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1634)
>>>> at
>>>> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317)
>>>> at
>>>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
>>>> at
>>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
>>>> ... 50 common frames omitted
>>>> Caused by: org.springframework.beans.factory.BeanCreationException: Error
>>>> creating bean with name 'clusterCoordinationProtocolSenderListener' defined
>>>> in class path resource [nifi-cluster-protocol-context.xml]: Cannot resolve
>>>> reference to bean 'clusterCoordinationProtocolSender' while setting
>>>> constructor argument; nested exception is
>>>> org.springframework.beans.factory.BeanCreationException: Error creating bean
>>>> with name 'clusterCoordinationProtocolSender' defined in class path resource
>>>> [nifi-cluster-protocol-context.xml]: Cannot resolve reference to bean
>>>> 'protocolSocketConfiguration' while setting constructor argument; nested
>>>> exception is org.springframework.beans.factory.BeanCreationException: Error
>>>> creating bean with name 'protocolSocketConfiguration': FactoryBean threw
>>>> exception on object creation; nested exception is
>>>> java.security.KeyStoreException: not found
>>>> at
>>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:359)
>>>> at
>>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)
>>>> at
>>>> org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:648)
>>>> at
>>>> org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:145)
>>>> at
>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1193)
>>>> at
>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1095)
>>>> at
>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
>>>> at
>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
>>>> at
>>>> org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
>>>> at
>>>> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>>>> at
>>>> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
>>>> at
>>>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
>>>> at
>>>> org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1084)
>>>> at
>>>> org.apache.nifi.cluster.spring.NodeClusterCoordinatorFactoryBean.getObject(NodeClusterCoordinatorFactoryBean.java:44)
>>>> at
>>>> org.apache.nifi.cluster.spring.NodeClusterCoordinatorFactoryBean.getObject(NodeClusterCoordinatorFactoryBean.java:34)
>>>> at
>>>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
>>>> ... 55 common frames omitted
>>>> Caused by: org.springframework.beans.factory.BeanCreationException: Error
>>>> creating bean with name 'clusterCoordinationProtocolSender' defined in class
>>>> path resource [nifi-cluster-protocol-context.xml]: Cannot resolve reference
>>>> to bean 'protocolSocketConfiguration' while setting constructor argument;
>>>> nested exception is org.springframework.beans.factory.BeanCreationException:
>>>> Error creating bean with name 'protocolSocketConfiguration': FactoryBean
>>>> threw exception on object creation; nested exception is
>>>> java.security.KeyStoreException: not found
>>>> at
>>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:359)
>>>> at
>>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)
>>>> at
>>>> org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:648)
>>>> at
>>>> org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:145)
>>>> at
>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1193)
>>>> at
>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1095)
>>>> at
>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
>>>> at
>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
>>>> at
>>>> org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
>>>> at
>>>> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>>>> at
>>>> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
>>>> at
>>>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
>>>> at
>>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
>>>> ... 70 common frames omitted
>>>> Caused by: org.springframework.beans.factory.BeanCreationException: Error
>>>> creating bean with name 'protocolSocketConfiguration': FactoryBean threw
>>>> exception on object creation; nested exception is
>>>> java.security.KeyStoreException: not found
>>>> at
>>>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175)
>>>> at
>>>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
>>>> at
>>>> org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1634)
>>>> at
>>>> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317)
>>>> at
>>>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
>>>> at
>>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
>>>> ... 82 common frames omitted
>>>> Caused by: java.security.KeyStoreException: not found
>>>> at java.security.KeyStore.getInstance(KeyStore.java:851)
>>>> at
>>>> org.apache.nifi.security.util.KeyStoreUtils.getKeyStore(KeyStoreUtils.java:66)
>>>> at
>>>> org.apache.nifi.security.util.KeyStoreUtils.getTrustStore(KeyStoreUtils.java:80)
>>>> at
>>>> org.apache.nifi.io.socket.SSLContextFactory.<init>(SSLContextFactory.java:73)
>>>> at
>>>> org.apache.nifi.cluster.protocol.spring.SocketConfigurationFactoryBean.getObject(SocketConfigurationFactoryBean.java:45)
>>>> at
>>>> org.apache.nifi.cluster.protocol.spring.SocketConfigurationFactoryBean.getObject(SocketConfigurationFactoryBean.java:30)
>>>> at
>>>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
>>>> ... 87 common frames omitted
>>>> Caused by: java.security.NoSuchAlgorithmException: KeyStore not available
>>>> at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
>>>> at java.security.Security.getImpl(Security.java:695)
>>>> at java.security.KeyStore.getInstance(KeyStore.java:848)
>>>> ... 93 common frames omitted
>>>>
>>>> My nifi.properties file is.
>>>>
>>>> # Licensed to the Apache Software Foundation (ASF) under one or more
>>>> # contributor license agreements. See the NOTICE file distributed with
>>>> # this work for additional information regarding copyright ownership.
>>>> # The ASF licenses this file to You under the Apache License, Version 2.0
>>>> # (the "License"); you may not use this file except in compliance with
>>>> # the License. You may obtain a copy of the License at
>>>> #
>>>> # http://www.apache.org/licenses/LICENSE-2.0
>>>> #
>>>> # Unless required by applicable law or agreed to in writing, software
>>>> # distributed under the License is distributed on an "AS IS" BASIS,
>>>> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>>>> # See the License for the specific language governing permissions and
>>>> # limitations under the License.
>>>>
>>>> # Core Properties #
>>>> nifi.version={{nifi_version}}
>>>> nifi.flow.configuration.file=/opt/config/flow.xml.gz
>>>> nifi.flow.configuration.archive.enabled=true
>>>> nifi.flow.configuration.archive.dir=/opt/config/archive/
>>>> nifi.flow.configuration.archive.max.time=30 days
>>>> nifi.flow.configuration.archive.max.storage=500 MB
>>>> nifi.flowcontroller.autoResumeState=true
>>>> nifi.flowcontroller.graceful.shutdown.period=10 sec
>>>> nifi.flowservice.writedelay.interval=500 ms
>>>> nifi.administrative.yield.duration=30 sec
>>>> # If a component has no work to do (is "bored"), how long should we wait
>>>> before checking again for work?
>>>> nifi.bored.yield.duration=10 millis
>>>>
>>>>
>>>> nifi.authorizer.configuration.file=/opt/config/authorizers.xml
>>>> nifi.login.identity.provider.configuration.file=/opt/config/login-identity-providers.xml
>>>> nifi.templates.directory=/opt/config/templates
>>>> nifi.ui.banner.text=
>>>> nifi.ui.autorefresh.interval=30 sec
>>>> nifi.nar.library.directory=/opt/nifi/lib
>>>> nifi.nar.library.directory.custom=/opt/config/processors
>>>> nifi.nar.working.directory=/opt/nifi/work/nar/
>>>> nifi.documentation.working.directory=./work/docs/components
>>>>
>>>> ####################
>>>> # State Management #
>>>> ####################
>>>> nifi.state.management.configuration.file=/opt/config/state-management.xml
>>>> # The ID of the local state provider
>>>> nifi.state.management.provider.local=local-provider
>>>> # The ID of the cluster-wide state provider. This will be ignored if NiFi is
>>>> not clustered but must be populated if running in a cluster.
>>>> nifi.state.management.provider.cluster=zk-provider
>>>> # Specifies whether or not this instance of NiFi should run an embedded
>>>> ZooKeeper server
>>>> nifi.state.management.embedded.zookeeper.start=false
>>>> # Properties file that provides the ZooKeeper properties to use if
>>>> <nifi.state.management.embedded.zookeeper.start> is set to true
>>>> nifi.state.management.embedded.zookeeper.properties=/opt/config/zookeeper.properties
>>>>
>>>>
>>>> # H2 Settings
>>>> nifi.database.directory=/opt/database_repository
>>>> nifi.h2.url.append=;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE
>>>>
>>>> # FlowFile Repository
>>>> nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository
>>>> nifi.flowfile.repository.directory=/opt/flowfile_repository
>>>> nifi.flowfile.repository.partitions=256
>>>> nifi.flowfile.repository.checkpoint.interval=2 mins
>>>> nifi.flowfile.repository.always.sync=false
>>>>
>>>> nifi.swap.manager.implementation=org.apache.nifi.controller.FileSystemSwapManager
>>>> nifi.queue.swap.threshold=20000
>>>> nifi.swap.in.period=5 sec
>>>> nifi.swap.in.threads=1
>>>> nifi.swap.out.period=5 sec
>>>> nifi.swap.out.threads=4
>>>>
>>>> # Content Repository
>>>> nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository
>>>> nifi.content.claim.max.appendable.size=10 MB
>>>> nifi.content.claim.max.flow.files=100
>>>> nifi.content.repository.directory.default=/opt/content_repository
>>>> nifi.content.repository.archive.max.retention.period=12 hours
>>>> nifi.content.repository.archive.max.usage.percentage=50%
>>>> nifi.content.repository.archive.enabled=true
>>>> nifi.content.repository.always.sync=false
>>>> nifi.content.viewer.url=/nifi-content-viewer/
>>>>
>>>> # Provenance Repository Properties
>>>> nifi.provenance.repository.implementation=org.apache.nifi.provenance.PersistentProvenanceRepository
>>>>
>>>> # Persistent Provenance Repository Properties
>>>> nifi.provenance.repository.directory.default=/opt/provenance_repository
>>>> nifi.provenance.repository.max.storage.time=24 hours
>>>> nifi.provenance.repository.max.storage.size=1 GB
>>>> nifi.provenance.repository.rollover.time=30 secs
>>>> nifi.provenance.repository.rollover.size=100 MB
>>>> nifi.provenance.repository.query.threads=2
>>>> nifi.provenance.repository.index.threads=1
>>>> nifi.provenance.repository.compress.on.rollover=true
>>>> nifi.provenance.repository.always.sync=false
>>>> nifi.provenance.repository.journal.count=16
>>>> # Comma-separated list of fields. Fields that are not indexed will not be
>>>> searchable. Valid fields are:
>>>> # EventType, FlowFileUUID, Filename, TransitURI, ProcessorID,
>>>> AlternateIdentifierURI, Relationship, Details
>>>> nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename,
>>>> ProcessorID, Relationship
>>>> # FlowFile Attributes that should be indexed and made searchable. Some
>>>> examples to consider are filename, uuid, mime.type
>>>> nifi.provenance.repository.indexed.attributes=
>>>> # Large values for the shard size will result in more Java heap usage when
>>>> searching the Provenance Repository
>>>> # but should provide better performance
>>>> nifi.provenance.repository.index.shard.size=500 MB
>>>> # Indicates the maximum length that a FlowFile attribute can be when
>>>> retrieving a Provenance Event from
>>>> # the repository. If the length of any attribute exceeds this value, it will
>>>> be truncated when the event is retrieved.
>>>> nifi.provenance.repository.max.attribute.length=65536
>>>>
>>>> # Volatile Provenance Respository Properties
>>>> nifi.provenance.repository.buffer.size=100000
>>>>
>>>> # Component Status Repository
>>>> nifi.components.status.repository.implementation=org.apache.nifi.controller.status.history.VolatileComponentStatusRepository
>>>> nifi.components.status.repository.buffer.size=1440
>>>> nifi.components.status.snapshot.frequency=1 min
>>>>
>>>> # Site to Site properties
>>>> nifi.remote.input.host=
>>>> nifi.remote.input.secure=false
>>>> nifi.remote.input.socket.port=9998
>>>> nifi.remote.input.http.enabled=false
>>>> nifi.remote.input.http.transaction.ttl=30 sec
>>>>
>>>> # web properties #
>>>> nifi.web.war.directory=/opt/nifi/lib
>>>> nifi.web.http.host=
>>>> nifi.web.http.port=
>>>> nifi.web.https.host={{redacted}}
>>>> nifi.web.https.port=8443
>>>> nifi.web.jetty.working.directory=/opt/nifi/work/jetty
>>>> nifi.web.jetty.threads=200
>>>>
>>>> # security properties #
>>>> nifi.sensitive.props.key=x0KDgO9L8lAhFGLdvu2VEjFVGc6Kg3V0R5I4bYwoqdgL47moo0wApKQtAVu1BvD
>>>> nifi.sensitive.props.key.protected=
>>>> nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
>>>> nifi.sensitive.props.provider=BC
>>>> nifi.sensitive.props.additional.keys=
>>>>
>>>> nifi.security.keystore=/opt/certs/payit_keystore
>>>> nifi.security.keystoreType=JKS
>>>> nifi.security.keystorePasswd={{keystore_password}}
>>>> nifi.security.keyPasswd=
>>>> nifi.security.truststore=
>>>> nifi.security.truststoreType=
>>>> nifi.security.truststorePasswd=
>>>> nifi.security.needClientAuth=false
>>>> nifi.security.user.authorizer=file-provider
>>>> nifi.security.user.login.identity.provider=ldap-provider
>>>> nifi.security.ocsp.responder.url=
>>>> nifi.security.ocsp.responder.certificate=
>>>>
>>>> # Identity Mapping Properties #
>>>> # These properties allow normalizing user identities such that identities
>>>> coming from different identity providers
>>>> # (certificates, LDAP, Kerberos) can be treated the same internally in NiFi.
>>>> The following example demonstrates normalizing
>>>> # DNs from certificates and principals from Kerberos into a common identity
>>>> string:
>>>> #
>>>> #nifi.security.identity.mapping.pattern.dn=^cn=(.*?),ou=(.*?),dc=(.*?),dc=(.*?),dc=(.*?)$
>>>> #nifi.security.identity.mapping.value.dn=$1
>>>> # nifi.security.identity.mapping.pattern.kerb=^(.*?)/instance@(.*?)$
>>>> # nifi.security.identity.mapping.value.kerb=$1@$2
>>>>
>>>> # cluster common properties (all nodes must have same values) #
>>>> nifi.cluster.protocol.heartbeat.interval=5 sec
>>>> nifi.cluster.protocol.is.secure=true
>>>>
>>>> # cluster node properties (only configure for cluster nodes) #
>>>> nifi.cluster.is.node=true
>>>> nifi.cluster.node.address=nifi-dev.mobilgov.com
>>>> nifi.cluster.node.protocol.port=9999
>>>> nifi.cluster.node.protocol.threads=10
>>>> nifi.cluster.node.event.history.size=25
>>>> nifi.cluster.node.connection.timeout=5 sec
>>>> nifi.cluster.node.read.timeout=5 sec
>>>> nifi.cluster.firewall.file=
>>>>
>>>>
>>>> # zookeeper properties, used for cluster management #
>>>> nifi.zookeeper.connect.string=internal-etcd-dev-EtcdLoadB-3RWA2WEFBBT8-2068560477.us-east-2.elb.amazonaws.com:2181,internal-etcd-dev-EtcdLoadB-3RWA2WEFBBT8-2068560477.us-east-2.elb.amazonaws.com2182,internal-etcd-dev-EtcdLoadB-3RWA2WEFBBT8-2068560477.us-east-2.elb.amazonaws.com:2183
>>>> nifi.zookeeper.connect.timeout=3 secs
>>>> nifi.zookeeper.session.timeout=3 secs
>>>> nifi.zookeeper.root.node=/nifi
>>>>
>>>> # kerberos #
>>>> nifi.kerberos.krb5.file=
>>>>
>>>> # kerberos service principle #
>>>> nifi.kerberos.service.principal=
>>>> nifi.kerberos.service.keytab.location=
>>>>
>>>> # kerberos spnego principle #
>>>> nifi.kerberos.spnego.principal=
>>>> nifi.kerberos.spnego.keytab.location=
>>>> nifi.kerberos.spnego.authentication.expiration=12 hours
>>>>
>>>> # external properties files for variable registry
>>>> # supports a comma delimited list of file locations
>>>> nifi.variable.registry.properties=
>>>>
>>>> I think I have everything set correctly but I have not been able to start an
>>>> instances up.
>>>>
>>>> Thanks,
>>>>
>>>> Scott
>>>>
>>>> On Mar 19, 2018, at 4:35 PM, Bryan Bende <bb...@gmail.com> wrote:
>>>>
>>>> The base file is here for comparison:
>>>>
>>>> https://github.com/apache/nifi-registry/blob/master/nifi-registry-resources/src/main/resources/conf/identity-providers.xml#L23
>>>>
>>>> On Mon, Mar 19, 2018 at 5:34 PM, Bryan Bende <bb...@gmail.com> wrote:
>>>>
>>>> For your first file, is what you showed there actually wrapped in
>>>> <identityProviders> </identityProviders> or is it exactly what you
>>>> showed?
>>>>
>>>> It may just be that you only copied/pasted the one provider, but the
>>>> root element is not <provider>, so as it is shown there it would not
>>>> parse.
>>>>
>>>> On Mon, Mar 19, 2018 at 2:54 PM, Scott Howell <sc...@mobilgov.com>
>>>> wrote:
>>>>
>>>> Here is my file
>>>>
>>>> <provider>
>>>> <identifier>ldap-identity-provider</identifier>
>>>> <class>org.apache.nifi.registry.security.ldap.LdapProvider</class>
>>>> <property name="Authentication Strategy">SIMPLE</property>
>>>>
>>>> <property name="Manager DN">cn=Manager,dc=mobilgov,dc=com</property>
>>>> <property name="Manager Password”>redacted</property>
>>>>
>>>>
>>>> <property name="Referral Strategy">FOLLOW</property>
>>>> <property name="Connect Timeout">10 secs</property>
>>>> <property name="Read Timeout">10 secs</property>
>>>>
>>>> <property name="Url”>redacted</property>
>>>> <property name="User Search
>>>> Base">ou=users,dc=mobilgov,dc=com</property>
>>>> <property name="User Search Filter">uid={0}</property>
>>>>
>>>> <property name="Identity Strategy">USE_DN</property>
>>>> <property name="Authentication Expiration">12 hours</property>
>>>> </provider>
>>>>
>>>> Here is my authorizers.xml
>>>>
>>>> <authorizers>
>>>>
>>>> <userGroupProvider>
>>>> <identifier>file-user-group-provider</identifier>
>>>>
>>>> <class>org.apache.nifi.registry.security.authorization.file.FileUserGroupProvider</class>
>>>> <property name="Users File">conf/users.xml</property>
>>>> <property name="Legacy Authorized Users File"></property>
>>>> <property name="Initial User Identity 1”>redacted</property>
>>>> </userGroupProvider>
>>>>
>>>> <accessPolicyProvider>
>>>> <identifier>file-access-policy-provider</identifier>
>>>>
>>>> <class>org.apache.nifi.registry.security.authorization.file.FileAccessPolicyProvider</class>
>>>> <property name="User Group
>>>> Provider">file-user-group-provider</property>
>>>> <property name="Authorizations
>>>> File">conf/authorizations.xml</property>
>>>> <property name="Initial Admin Identity”>redacted</property>
>>>> <property name="NiFi Identity 1"></property>
>>>> </accessPolicyProvider>
>>>>
>>>> <authorizer>
>>>> <identifier>managed-authorizer</identifier>
>>>>
>>>> <class>org.apache.nifi.registry.security.authorization.StandardManagedAuthorizer</class>
>>>> <property name="Access Policy
>>>> Provider">file-access-policy-provider</property>
>>>> </authorizer>
>>>> </authorizers>
>>>>
>>>> On Mar 19, 2018, at 12:59 PM, Bryan Bende <bb...@gmail.com> wrote:
>>>>
>>>> It looks like that error would happen if your identity-providers.xml
>>>> contained invalid XML.
>>>>
>>>> Did you start by modifying the identity-providers.xml file that was
>>>> already there? Can you share the file, or the contents (removing
>>>> anything sensitive)?
>>>>
>>>> On Mon, Mar 19, 2018 at 1:09 PM, Scott Howell <sc...@mobilgov.com>
>>>> wrote:
>>>>
>>>> So I was able to get the UI pulled up but now I am hitting a roadblock with
>>>> my identity-provider.xml.
>>>>
>>>> I am getting a number of errors like this:
>>>>
>>>> Caused by: org.springframework.beans.factory.BeanCreationException: Error
>>>> creating bean with name 'getIdentityProvider' defined in class path resource
>>>> [org/apache/nifi/registry/security/authentication/IdentityProviderFactory.class]:
>>>> Bean instantiation via factory method failed; nested exception is
>>>> org.springframework.beans.BeanInstantiationException: Failed to instantiate
>>>> [org.apache.nifi.registry.security.authentication.IdentityProvider]: Factory
>>>> method 'getIdentityProvider' threw exception; nested exception is
>>>> java.lang.Exception: Unable to load the login identity provider
>>>> configuration file at: /opt/nifi-registry-0.1.0/conf/identity-providers.xml
>>>> at
>>>> org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:587)
>>>> ~[na:na]
>>>> at
>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1250)
>>>> ~[na:na]
>>>> at
>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1099)
>>>> ~[na:na]
>>>> at
>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:545)
>>>> ~[na:na]
>>>> at
>>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:502)
>>>> ~[na:na]
>>>> at
>>>> org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:312)
>>>> ~[na:na]
>>>> at
>>>> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228)
>>>> ~[na:na]
>>>> at
>>>> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:310)
>>>> ~[na:na]
>>>> at
>>>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200)
>>>> ~[na:na]
>>>> at
>>>> org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:251)
>>>> ~[na:na]
>>>> at
>>>> org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1135)
>>>> ~[na:na]
>>>> at
>>>> org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1062)
>>>> ~[na:na]
>>>> at
>>>> org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:815)
>>>> ~[na:na]
>>>> at
>>>> org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:721)
>>>> ~[na:na]
>>>> ... 43 common frames omitted
>>>>
>>>> I know it has to do with the identity-provider.xml but I have my setup just
>>>> like the documentation ask for. I turned on debug but was not able to see
>>>> anything different or better explanation from it.
>>>>
>>>>
>>>> On Mar 19, 2018, at 10:06 AM, Kevin Doran <kd...@apache.org> wrote:
>>>>
>>>> Ok, that use case should be fine.
>>>>
>>>> If it were an authorization issue you would see something in the logs saying
>>>> that an authorization attempt failed and the server is responding with a
>>>> 403. Just to be sure, can you enable debug logging if you haven't already,
>>>> i.e., in your nifi-registry/conf/logback.xml file, change
>>>> 'org.apache.nifif.registry' to debug:
>>>>
>>>> <!-- valid logging levels: TRACE, DEBUG, INFO, WARN, ERROR -->
>>>> <logger name="org.apache.nifi.registry" level="DEBUG"/>
>>>>
>>>> If there is nothing being written to nifi-registry-app.log, it points
>>>> towards a connection issue, so I would double check your host, port, and TLS
>>>> settings. You'll have to get an HTTPS cert from a root CA or configure your
>>>> ELB to trust your company's self-signed cert (again, not sure if/how to do
>>>> this, but I assume there should be some way to configure it. It might
>>>> require settings not exposed in the AWS web console.)
>>>>
>>>> On 3/19/18, 10:51, "Scott Howell" <sc...@mobilgov.com> wrote:
>>>>
>>>> Thanks Kevin,
>>>>
>>>> I am just using the ELB to go from the public subnet to the private subnet.
>>>> I will not have multiple instances running of registry.
>>>>
>>>> I will say on my authorizers.xml there is one difference between my nifi
>>>> instance. On my nifi instance I am using file-provider for
>>>> nifi.security.user.authorizer in my nifi.properties. I don’t think from
>>>> reading the documents for nifi-registry that I can use that. If there is a
>>>> way that might be my problem. I was running into some issues with my nifi
>>>> instance when I was using managed-authorizers instead of file-provider.
>>>>
>>>>
>>>>
>>>> On Mar 19, 2018, at 9:35 AM, Kevin Doran <kd...@apache.org> wrote:
>>>>
>>>> Hey Scott,
>>>>
>>>> Assuming you are using two-way TLS with client certificates for
>>>> authentication, I recommend configuring your ELB for TCP passthrough so that
>>>> the TLS handshake is between the end-client and the NiFi Registry Server (in
>>>> other words, no decryption/termination of the TLS connection happens in the
>>>> ELB). If you are using some other form of authentication (e.g., LDAP), you
>>>> will need to configure your ELB to trust the self-signed key NiFi Registry
>>>> is using. I'm not sure how to do that as I've never run an ELB with that
>>>> configuration before.
>>>>
>>>> Also, just a note about using an ELB with NiFi Registry:
>>>>
>>>> NiFi Registry is currently only supports single-instance use as persisted
>>>> data and in-memory state is not synced between multiple instances. Are you
>>>> hoping to use the ELB for actual load balancing, or is it just to take
>>>> advantage of other ELB features, such as forwarding and security group
>>>> rules? If the plan is to load balance multiple Registry instances, just be
>>>> aware that you will probably run into some unexpected behavior. (As you
>>>> mentioned using authorization, that is one case where I know the in-memory
>>>> cache of the persisted data will not refresh across instances, so even if
>>>> you were using some sort of shared network file system attached to multiple
>>>> Registry instances, such as EFS, it would not work the way you hope.)
>>>>
>>>> Hope this helps,
>>>> Kevin
>>>>
>>>> On 3/19/18, 10:20, "Scott Howell" <sc...@mobilgov.com> wrote:
>>>>
>>>> Thanks for the quick response.
>>>>
>>>> A couple of things I am seeing.
>>>>
>>>> 1. There is no error, I don’t see anything in the logs once the service
>>>> comes up. This is because the health check is not even hitting the instance
>>>> when secure.
>>>>
>>>> 2. Nothing interesting in the nifi-registry-app.logs. That was my concern
>>>> because on my nifi instance I can see the health check hitting the instance
>>>> from the ELB. This does not happen on the nifi-registry instance. I see the
>>>> service startup and it tells me what domain and port I can access the UI but
>>>> nothing else after that.
>>>>
>>>> 3. When I am on an instances in the same private subnet I am able to curl to
>>>> the instance I get the TLS SSL which tells me the keystore is on the server.
>>>> I am using a JKS keystore that is self-signed by the company I work for.
>>>>
>>>> On Mar 19, 2018, at 9:10 AM, Bryan Bende <bb...@gmail.com> wrote:
>>>>
>>>> Hello,
>>>>
>>>> What error are you getting when you cannot access the UI?
>>>>
>>>> Is there anything interesting in nifi-registry-app.log regarding
>>>> authentication/authorization when this happens?
>>>>
>>>> Can you access the UI securely without going through the ELB?
>>>>
>>>> Thanks,
>>>>
>>>> Bryan
>>>>
>>>>
>>>> On Mon, Mar 19, 2018 at 10:05 AM, Scott Howell <sc...@mobilgov.com>
>>>> wrote:
>>>>
>>>> I was able to stand up nifi-registry behind an AWS ELB non-secure.
>>>> Everything was working great and was able to access the UI anonymously. I
>>>> set up the authorization just like on my nifi instances along with the
>>>> authorizers and identity-provider. The service comes up without errors and
>>>> everything looks good but the health check does not pass and I cannot access
>>>> the UI to login. I was wondering if anyone else has ran into this issue
>>>> using nifi-registry.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>
>
>
Re: Issue with AWS ELB on secure nifi-registry
Posted by Scott Howell <sc...@mobilgov.com>.
One other thing I am seeing and I don’t know if this is an issue or not in my authorizations.xml I do not have a policy for /proxy with action=“R” only action=“W”.
> On Mar 21, 2018, at 11:03 AM, Scott Howell <sc...@mobilgov.com> wrote:
>
> Thanks for that. I am still getting this error in my nifi-user.log
>
> o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api: Untrusted proxy CN=*.{redacted}.com, OU={redacted}, O={redacted}, L=Kansas City, ST=Missouri, C=US
>
> Is there an issue with using a wildcard cert?
>
>
>> On Mar 21, 2018, at 10:23 AM, Bryan Bende <bb...@gmail.com> wrote:
>>
>> All identity strings are case & whitespace sensitive.
>>
>> The node identities in your authorizers.xml have no whitespace, and
>> the identity showing in the logs does.
>>
>> On Wed, Mar 21, 2018 at 11:05 AM, Scott Howell <sc...@mobilgov.com> wrote:
>>> Thanks for all of the help with this. I have the cluster up and running. The
>>> logs look great everything seems to be working but I cannot login into the
>>> UI. I am using a wildcard self-signed certificate. The /proxy is in
>>> authorizations.xml with the correct users for the nodes.
>>>
>>> The error I see with the UI :
>>> is Untrusted proxy CN=*.{redacted}.com, OU={redacted}, O={redacted},
>>> L=Kansas City, ST=Missouri, C=US
>>>
>>> I haven’t had much luck finding a lot of documentation or forum questions
>>> with this kind of issue.
>>>
>>> My authorizers.xml looks like this
>>> <authorizers>
>>> <authorizer>
>>> <identifier>file-provider</identifier>
>>> <class>org.apache.nifi.authorization.FileAuthorizer</class>
>>> <property name="Authorizations
>>> File">/opt/config/authorizations.xml</property>
>>> <property name="Users File">/opt/config/users.xml</property>
>>> <property name="Initial Admin
>>> Identity">uid=scott,ou=users,dc={redacted},dc=com</property>
>>> <property name="Legacy Authorized Users File"></property>
>>>
>>> <property name="Node Identity
>>> 1">CN=node-1-nifi-dev.{redacted}.com,OU={redacted},O={redacted},L=Kansas
>>> City,ST=Missouri,C=US</property>
>>> <property name="Node Identity
>>> 2">CN=node-2-nifi-dev.{redacted}.com,OU={redacted},O={redacted},L=Kansas
>>> City,ST=Missouri,C=US</property>
>>> </authorizer>
>>> </authorizers>
>>>
>>> Thanks,
>>>
>>> Scott
>>>
>>> On Mar 20, 2018, at 1:15 PM, Andy LoPresto <al...@apache.org> wrote:
>>>
>>> Scott,
>>>
>>> The original exception is "nested exception is
>>> java.security.KeyStoreException: not found”. Can you verify that the
>>> keystore you’ve provided is valid using the “keytool” command? In addition,
>>> you will need a truststore as well. Try following Pierre's [1] or Bryan’s
>>> [2] instructions for setting up a secure cluster.
>>>
>>> [1]
>>> https://pierrevillard.com/2016/11/29/apache-nifi-1-1-0-secured-cluster-setup/
>>> [2]
>>> https://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-authorization-and-multi-tenancy
>>>
>>>
>>> Andy LoPresto
>>> alopresto@apache.org
>>> alopresto.apache@gmail.com
>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>>>
>>> On Mar 20, 2018, at 11:05 AM, Scott Howell <sc...@mobilgov.com> wrote:
>>>
>>> Thanks for all of the help yesterday I was able to get a secure nifi and
>>> nifi-registry up and communicating. I am now trying to figure out how to
>>> create a secure cluster. I am currently getting this error when I start up
>>> nifi.
>>>
>>> tion; nested exception is
>>> org.springframework.beans.factory.BeanCreationException: Error creating bean
>>> with name 'clusterCoordinationProtocolSenderListener' defined in class path
>>> resource [nifi-cluster-protocol-context.xml]: Cannot resolve reference to
>>> bean 'clusterCoordinationProtocolSender' while setting constructor argument;
>>> nested exception is org.springframework.beans.factory.BeanCreationException:
>>> Error creating bean with name 'clusterCoordinationProtocolSender' defined in
>>> class path resource [nifi-cluster-protocol-context.xml]: Cannot resolve
>>> reference to bean 'protocolSocketConfiguration' while setting constructor
>>> argument; nested exception is
>>> org.springframework.beans.factory.BeanCreationException: Error creating bean
>>> with name 'protocolSocketConfiguration': FactoryBean threw exception on
>>> object creation; nested exception is java.security.KeyStoreException: not
>>> found
>>> at
>>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175)
>>> at
>>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
>>> at
>>> org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1634)
>>> at
>>> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317)
>>> at
>>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
>>> at
>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
>>> ... 50 common frames omitted
>>> Caused by: org.springframework.beans.factory.BeanCreationException: Error
>>> creating bean with name 'clusterCoordinationProtocolSenderListener' defined
>>> in class path resource [nifi-cluster-protocol-context.xml]: Cannot resolve
>>> reference to bean 'clusterCoordinationProtocolSender' while setting
>>> constructor argument; nested exception is
>>> org.springframework.beans.factory.BeanCreationException: Error creating bean
>>> with name 'clusterCoordinationProtocolSender' defined in class path resource
>>> [nifi-cluster-protocol-context.xml]: Cannot resolve reference to bean
>>> 'protocolSocketConfiguration' while setting constructor argument; nested
>>> exception is org.springframework.beans.factory.BeanCreationException: Error
>>> creating bean with name 'protocolSocketConfiguration': FactoryBean threw
>>> exception on object creation; nested exception is
>>> java.security.KeyStoreException: not found
>>> at
>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:359)
>>> at
>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)
>>> at
>>> org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:648)
>>> at
>>> org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:145)
>>> at
>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1193)
>>> at
>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1095)
>>> at
>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
>>> at
>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
>>> at
>>> org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
>>> at
>>> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>>> at
>>> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
>>> at
>>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
>>> at
>>> org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1084)
>>> at
>>> org.apache.nifi.cluster.spring.NodeClusterCoordinatorFactoryBean.getObject(NodeClusterCoordinatorFactoryBean.java:44)
>>> at
>>> org.apache.nifi.cluster.spring.NodeClusterCoordinatorFactoryBean.getObject(NodeClusterCoordinatorFactoryBean.java:34)
>>> at
>>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
>>> ... 55 common frames omitted
>>> Caused by: org.springframework.beans.factory.BeanCreationException: Error
>>> creating bean with name 'clusterCoordinationProtocolSender' defined in class
>>> path resource [nifi-cluster-protocol-context.xml]: Cannot resolve reference
>>> to bean 'protocolSocketConfiguration' while setting constructor argument;
>>> nested exception is org.springframework.beans.factory.BeanCreationException:
>>> Error creating bean with name 'protocolSocketConfiguration': FactoryBean
>>> threw exception on object creation; nested exception is
>>> java.security.KeyStoreException: not found
>>> at
>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:359)
>>> at
>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)
>>> at
>>> org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:648)
>>> at
>>> org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:145)
>>> at
>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1193)
>>> at
>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1095)
>>> at
>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
>>> at
>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
>>> at
>>> org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
>>> at
>>> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>>> at
>>> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
>>> at
>>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
>>> at
>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
>>> ... 70 common frames omitted
>>> Caused by: org.springframework.beans.factory.BeanCreationException: Error
>>> creating bean with name 'protocolSocketConfiguration': FactoryBean threw
>>> exception on object creation; nested exception is
>>> java.security.KeyStoreException: not found
>>> at
>>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175)
>>> at
>>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
>>> at
>>> org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1634)
>>> at
>>> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317)
>>> at
>>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
>>> at
>>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
>>> ... 82 common frames omitted
>>> Caused by: java.security.KeyStoreException: not found
>>> at java.security.KeyStore.getInstance(KeyStore.java:851)
>>> at
>>> org.apache.nifi.security.util.KeyStoreUtils.getKeyStore(KeyStoreUtils.java:66)
>>> at
>>> org.apache.nifi.security.util.KeyStoreUtils.getTrustStore(KeyStoreUtils.java:80)
>>> at
>>> org.apache.nifi.io.socket.SSLContextFactory.<init>(SSLContextFactory.java:73)
>>> at
>>> org.apache.nifi.cluster.protocol.spring.SocketConfigurationFactoryBean.getObject(SocketConfigurationFactoryBean.java:45)
>>> at
>>> org.apache.nifi.cluster.protocol.spring.SocketConfigurationFactoryBean.getObject(SocketConfigurationFactoryBean.java:30)
>>> at
>>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
>>> ... 87 common frames omitted
>>> Caused by: java.security.NoSuchAlgorithmException: KeyStore not available
>>> at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
>>> at java.security.Security.getImpl(Security.java:695)
>>> at java.security.KeyStore.getInstance(KeyStore.java:848)
>>> ... 93 common frames omitted
>>>
>>> My nifi.properties file is.
>>>
>>> # Licensed to the Apache Software Foundation (ASF) under one or more
>>> # contributor license agreements. See the NOTICE file distributed with
>>> # this work for additional information regarding copyright ownership.
>>> # The ASF licenses this file to You under the Apache License, Version 2.0
>>> # (the "License"); you may not use this file except in compliance with
>>> # the License. You may obtain a copy of the License at
>>> #
>>> # http://www.apache.org/licenses/LICENSE-2.0
>>> #
>>> # Unless required by applicable law or agreed to in writing, software
>>> # distributed under the License is distributed on an "AS IS" BASIS,
>>> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>>> # See the License for the specific language governing permissions and
>>> # limitations under the License.
>>>
>>> # Core Properties #
>>> nifi.version={{nifi_version}}
>>> nifi.flow.configuration.file=/opt/config/flow.xml.gz
>>> nifi.flow.configuration.archive.enabled=true
>>> nifi.flow.configuration.archive.dir=/opt/config/archive/
>>> nifi.flow.configuration.archive.max.time=30 days
>>> nifi.flow.configuration.archive.max.storage=500 MB
>>> nifi.flowcontroller.autoResumeState=true
>>> nifi.flowcontroller.graceful.shutdown.period=10 sec
>>> nifi.flowservice.writedelay.interval=500 ms
>>> nifi.administrative.yield.duration=30 sec
>>> # If a component has no work to do (is "bored"), how long should we wait
>>> before checking again for work?
>>> nifi.bored.yield.duration=10 millis
>>>
>>>
>>> nifi.authorizer.configuration.file=/opt/config/authorizers.xml
>>> nifi.login.identity.provider.configuration.file=/opt/config/login-identity-providers.xml
>>> nifi.templates.directory=/opt/config/templates
>>> nifi.ui.banner.text=
>>> nifi.ui.autorefresh.interval=30 sec
>>> nifi.nar.library.directory=/opt/nifi/lib
>>> nifi.nar.library.directory.custom=/opt/config/processors
>>> nifi.nar.working.directory=/opt/nifi/work/nar/
>>> nifi.documentation.working.directory=./work/docs/components
>>>
>>> ####################
>>> # State Management #
>>> ####################
>>> nifi.state.management.configuration.file=/opt/config/state-management.xml
>>> # The ID of the local state provider
>>> nifi.state.management.provider.local=local-provider
>>> # The ID of the cluster-wide state provider. This will be ignored if NiFi is
>>> not clustered but must be populated if running in a cluster.
>>> nifi.state.management.provider.cluster=zk-provider
>>> # Specifies whether or not this instance of NiFi should run an embedded
>>> ZooKeeper server
>>> nifi.state.management.embedded.zookeeper.start=false
>>> # Properties file that provides the ZooKeeper properties to use if
>>> <nifi.state.management.embedded.zookeeper.start> is set to true
>>> nifi.state.management.embedded.zookeeper.properties=/opt/config/zookeeper.properties
>>>
>>>
>>> # H2 Settings
>>> nifi.database.directory=/opt/database_repository
>>> nifi.h2.url.append=;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE
>>>
>>> # FlowFile Repository
>>> nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository
>>> nifi.flowfile.repository.directory=/opt/flowfile_repository
>>> nifi.flowfile.repository.partitions=256
>>> nifi.flowfile.repository.checkpoint.interval=2 mins
>>> nifi.flowfile.repository.always.sync=false
>>>
>>> nifi.swap.manager.implementation=org.apache.nifi.controller.FileSystemSwapManager
>>> nifi.queue.swap.threshold=20000
>>> nifi.swap.in.period=5 sec
>>> nifi.swap.in.threads=1
>>> nifi.swap.out.period=5 sec
>>> nifi.swap.out.threads=4
>>>
>>> # Content Repository
>>> nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository
>>> nifi.content.claim.max.appendable.size=10 MB
>>> nifi.content.claim.max.flow.files=100
>>> nifi.content.repository.directory.default=/opt/content_repository
>>> nifi.content.repository.archive.max.retention.period=12 hours
>>> nifi.content.repository.archive.max.usage.percentage=50%
>>> nifi.content.repository.archive.enabled=true
>>> nifi.content.repository.always.sync=false
>>> nifi.content.viewer.url=/nifi-content-viewer/
>>>
>>> # Provenance Repository Properties
>>> nifi.provenance.repository.implementation=org.apache.nifi.provenance.PersistentProvenanceRepository
>>>
>>> # Persistent Provenance Repository Properties
>>> nifi.provenance.repository.directory.default=/opt/provenance_repository
>>> nifi.provenance.repository.max.storage.time=24 hours
>>> nifi.provenance.repository.max.storage.size=1 GB
>>> nifi.provenance.repository.rollover.time=30 secs
>>> nifi.provenance.repository.rollover.size=100 MB
>>> nifi.provenance.repository.query.threads=2
>>> nifi.provenance.repository.index.threads=1
>>> nifi.provenance.repository.compress.on.rollover=true
>>> nifi.provenance.repository.always.sync=false
>>> nifi.provenance.repository.journal.count=16
>>> # Comma-separated list of fields. Fields that are not indexed will not be
>>> searchable. Valid fields are:
>>> # EventType, FlowFileUUID, Filename, TransitURI, ProcessorID,
>>> AlternateIdentifierURI, Relationship, Details
>>> nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename,
>>> ProcessorID, Relationship
>>> # FlowFile Attributes that should be indexed and made searchable. Some
>>> examples to consider are filename, uuid, mime.type
>>> nifi.provenance.repository.indexed.attributes=
>>> # Large values for the shard size will result in more Java heap usage when
>>> searching the Provenance Repository
>>> # but should provide better performance
>>> nifi.provenance.repository.index.shard.size=500 MB
>>> # Indicates the maximum length that a FlowFile attribute can be when
>>> retrieving a Provenance Event from
>>> # the repository. If the length of any attribute exceeds this value, it will
>>> be truncated when the event is retrieved.
>>> nifi.provenance.repository.max.attribute.length=65536
>>>
>>> # Volatile Provenance Respository Properties
>>> nifi.provenance.repository.buffer.size=100000
>>>
>>> # Component Status Repository
>>> nifi.components.status.repository.implementation=org.apache.nifi.controller.status.history.VolatileComponentStatusRepository
>>> nifi.components.status.repository.buffer.size=1440
>>> nifi.components.status.snapshot.frequency=1 min
>>>
>>> # Site to Site properties
>>> nifi.remote.input.host=
>>> nifi.remote.input.secure=false
>>> nifi.remote.input.socket.port=9998
>>> nifi.remote.input.http.enabled=false
>>> nifi.remote.input.http.transaction.ttl=30 sec
>>>
>>> # web properties #
>>> nifi.web.war.directory=/opt/nifi/lib
>>> nifi.web.http.host=
>>> nifi.web.http.port=
>>> nifi.web.https.host={{redacted}}
>>> nifi.web.https.port=8443
>>> nifi.web.jetty.working.directory=/opt/nifi/work/jetty
>>> nifi.web.jetty.threads=200
>>>
>>> # security properties #
>>> nifi.sensitive.props.key=x0KDgO9L8lAhFGLdvu2VEjFVGc6Kg3V0R5I4bYwoqdgL47moo0wApKQtAVu1BvD
>>> nifi.sensitive.props.key.protected=
>>> nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
>>> nifi.sensitive.props.provider=BC
>>> nifi.sensitive.props.additional.keys=
>>>
>>> nifi.security.keystore=/opt/certs/payit_keystore
>>> nifi.security.keystoreType=JKS
>>> nifi.security.keystorePasswd={{keystore_password}}
>>> nifi.security.keyPasswd=
>>> nifi.security.truststore=
>>> nifi.security.truststoreType=
>>> nifi.security.truststorePasswd=
>>> nifi.security.needClientAuth=false
>>> nifi.security.user.authorizer=file-provider
>>> nifi.security.user.login.identity.provider=ldap-provider
>>> nifi.security.ocsp.responder.url=
>>> nifi.security.ocsp.responder.certificate=
>>>
>>> # Identity Mapping Properties #
>>> # These properties allow normalizing user identities such that identities
>>> coming from different identity providers
>>> # (certificates, LDAP, Kerberos) can be treated the same internally in NiFi.
>>> The following example demonstrates normalizing
>>> # DNs from certificates and principals from Kerberos into a common identity
>>> string:
>>> #
>>> #nifi.security.identity.mapping.pattern.dn=^cn=(.*?),ou=(.*?),dc=(.*?),dc=(.*?),dc=(.*?)$
>>> #nifi.security.identity.mapping.value.dn=$1
>>> # nifi.security.identity.mapping.pattern.kerb=^(.*?)/instance@(.*?)$
>>> # nifi.security.identity.mapping.value.kerb=$1@$2
>>>
>>> # cluster common properties (all nodes must have same values) #
>>> nifi.cluster.protocol.heartbeat.interval=5 sec
>>> nifi.cluster.protocol.is.secure=true
>>>
>>> # cluster node properties (only configure for cluster nodes) #
>>> nifi.cluster.is.node=true
>>> nifi.cluster.node.address=nifi-dev.mobilgov.com
>>> nifi.cluster.node.protocol.port=9999
>>> nifi.cluster.node.protocol.threads=10
>>> nifi.cluster.node.event.history.size=25
>>> nifi.cluster.node.connection.timeout=5 sec
>>> nifi.cluster.node.read.timeout=5 sec
>>> nifi.cluster.firewall.file=
>>>
>>>
>>> # zookeeper properties, used for cluster management #
>>> nifi.zookeeper.connect.string=internal-etcd-dev-EtcdLoadB-3RWA2WEFBBT8-2068560477.us-east-2.elb.amazonaws.com:2181,internal-etcd-dev-EtcdLoadB-3RWA2WEFBBT8-2068560477.us-east-2.elb.amazonaws.com2182,internal-etcd-dev-EtcdLoadB-3RWA2WEFBBT8-2068560477.us-east-2.elb.amazonaws.com:2183
>>> nifi.zookeeper.connect.timeout=3 secs
>>> nifi.zookeeper.session.timeout=3 secs
>>> nifi.zookeeper.root.node=/nifi
>>>
>>> # kerberos #
>>> nifi.kerberos.krb5.file=
>>>
>>> # kerberos service principle #
>>> nifi.kerberos.service.principal=
>>> nifi.kerberos.service.keytab.location=
>>>
>>> # kerberos spnego principle #
>>> nifi.kerberos.spnego.principal=
>>> nifi.kerberos.spnego.keytab.location=
>>> nifi.kerberos.spnego.authentication.expiration=12 hours
>>>
>>> # external properties files for variable registry
>>> # supports a comma delimited list of file locations
>>> nifi.variable.registry.properties=
>>>
>>> I think I have everything set correctly but I have not been able to start an
>>> instances up.
>>>
>>> Thanks,
>>>
>>> Scott
>>>
>>> On Mar 19, 2018, at 4:35 PM, Bryan Bende <bb...@gmail.com> wrote:
>>>
>>> The base file is here for comparison:
>>>
>>> https://github.com/apache/nifi-registry/blob/master/nifi-registry-resources/src/main/resources/conf/identity-providers.xml#L23
>>>
>>> On Mon, Mar 19, 2018 at 5:34 PM, Bryan Bende <bb...@gmail.com> wrote:
>>>
>>> For your first file, is what you showed there actually wrapped in
>>> <identityProviders> </identityProviders> or is it exactly what you
>>> showed?
>>>
>>> It may just be that you only copied/pasted the one provider, but the
>>> root element is not <provider>, so as it is shown there it would not
>>> parse.
>>>
>>> On Mon, Mar 19, 2018 at 2:54 PM, Scott Howell <sc...@mobilgov.com>
>>> wrote:
>>>
>>> Here is my file
>>>
>>> <provider>
>>> <identifier>ldap-identity-provider</identifier>
>>> <class>org.apache.nifi.registry.security.ldap.LdapProvider</class>
>>> <property name="Authentication Strategy">SIMPLE</property>
>>>
>>> <property name="Manager DN">cn=Manager,dc=mobilgov,dc=com</property>
>>> <property name="Manager Password”>redacted</property>
>>>
>>>
>>> <property name="Referral Strategy">FOLLOW</property>
>>> <property name="Connect Timeout">10 secs</property>
>>> <property name="Read Timeout">10 secs</property>
>>>
>>> <property name="Url”>redacted</property>
>>> <property name="User Search
>>> Base">ou=users,dc=mobilgov,dc=com</property>
>>> <property name="User Search Filter">uid={0}</property>
>>>
>>> <property name="Identity Strategy">USE_DN</property>
>>> <property name="Authentication Expiration">12 hours</property>
>>> </provider>
>>>
>>> Here is my authorizers.xml
>>>
>>> <authorizers>
>>>
>>> <userGroupProvider>
>>> <identifier>file-user-group-provider</identifier>
>>>
>>> <class>org.apache.nifi.registry.security.authorization.file.FileUserGroupProvider</class>
>>> <property name="Users File">conf/users.xml</property>
>>> <property name="Legacy Authorized Users File"></property>
>>> <property name="Initial User Identity 1”>redacted</property>
>>> </userGroupProvider>
>>>
>>> <accessPolicyProvider>
>>> <identifier>file-access-policy-provider</identifier>
>>>
>>> <class>org.apache.nifi.registry.security.authorization.file.FileAccessPolicyProvider</class>
>>> <property name="User Group
>>> Provider">file-user-group-provider</property>
>>> <property name="Authorizations
>>> File">conf/authorizations.xml</property>
>>> <property name="Initial Admin Identity”>redacted</property>
>>> <property name="NiFi Identity 1"></property>
>>> </accessPolicyProvider>
>>>
>>> <authorizer>
>>> <identifier>managed-authorizer</identifier>
>>>
>>> <class>org.apache.nifi.registry.security.authorization.StandardManagedAuthorizer</class>
>>> <property name="Access Policy
>>> Provider">file-access-policy-provider</property>
>>> </authorizer>
>>> </authorizers>
>>>
>>> On Mar 19, 2018, at 12:59 PM, Bryan Bende <bb...@gmail.com> wrote:
>>>
>>> It looks like that error would happen if your identity-providers.xml
>>> contained invalid XML.
>>>
>>> Did you start by modifying the identity-providers.xml file that was
>>> already there? Can you share the file, or the contents (removing
>>> anything sensitive)?
>>>
>>> On Mon, Mar 19, 2018 at 1:09 PM, Scott Howell <sc...@mobilgov.com>
>>> wrote:
>>>
>>> So I was able to get the UI pulled up but now I am hitting a roadblock with
>>> my identity-provider.xml.
>>>
>>> I am getting a number of errors like this:
>>>
>>> Caused by: org.springframework.beans.factory.BeanCreationException: Error
>>> creating bean with name 'getIdentityProvider' defined in class path resource
>>> [org/apache/nifi/registry/security/authentication/IdentityProviderFactory.class]:
>>> Bean instantiation via factory method failed; nested exception is
>>> org.springframework.beans.BeanInstantiationException: Failed to instantiate
>>> [org.apache.nifi.registry.security.authentication.IdentityProvider]: Factory
>>> method 'getIdentityProvider' threw exception; nested exception is
>>> java.lang.Exception: Unable to load the login identity provider
>>> configuration file at: /opt/nifi-registry-0.1.0/conf/identity-providers.xml
>>> at
>>> org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:587)
>>> ~[na:na]
>>> at
>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1250)
>>> ~[na:na]
>>> at
>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1099)
>>> ~[na:na]
>>> at
>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:545)
>>> ~[na:na]
>>> at
>>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:502)
>>> ~[na:na]
>>> at
>>> org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:312)
>>> ~[na:na]
>>> at
>>> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228)
>>> ~[na:na]
>>> at
>>> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:310)
>>> ~[na:na]
>>> at
>>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200)
>>> ~[na:na]
>>> at
>>> org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:251)
>>> ~[na:na]
>>> at
>>> org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1135)
>>> ~[na:na]
>>> at
>>> org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1062)
>>> ~[na:na]
>>> at
>>> org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:815)
>>> ~[na:na]
>>> at
>>> org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:721)
>>> ~[na:na]
>>> ... 43 common frames omitted
>>>
>>> I know it has to do with the identity-provider.xml but I have my setup just
>>> like the documentation ask for. I turned on debug but was not able to see
>>> anything different or better explanation from it.
>>>
>>>
>>> On Mar 19, 2018, at 10:06 AM, Kevin Doran <kd...@apache.org> wrote:
>>>
>>> Ok, that use case should be fine.
>>>
>>> If it were an authorization issue you would see something in the logs saying
>>> that an authorization attempt failed and the server is responding with a
>>> 403. Just to be sure, can you enable debug logging if you haven't already,
>>> i.e., in your nifi-registry/conf/logback.xml file, change
>>> 'org.apache.nifif.registry' to debug:
>>>
>>> <!-- valid logging levels: TRACE, DEBUG, INFO, WARN, ERROR -->
>>> <logger name="org.apache.nifi.registry" level="DEBUG"/>
>>>
>>> If there is nothing being written to nifi-registry-app.log, it points
>>> towards a connection issue, so I would double check your host, port, and TLS
>>> settings. You'll have to get an HTTPS cert from a root CA or configure your
>>> ELB to trust your company's self-signed cert (again, not sure if/how to do
>>> this, but I assume there should be some way to configure it. It might
>>> require settings not exposed in the AWS web console.)
>>>
>>> On 3/19/18, 10:51, "Scott Howell" <sc...@mobilgov.com> wrote:
>>>
>>> Thanks Kevin,
>>>
>>> I am just using the ELB to go from the public subnet to the private subnet.
>>> I will not have multiple instances running of registry.
>>>
>>> I will say on my authorizers.xml there is one difference between my nifi
>>> instance. On my nifi instance I am using file-provider for
>>> nifi.security.user.authorizer in my nifi.properties. I don’t think from
>>> reading the documents for nifi-registry that I can use that. If there is a
>>> way that might be my problem. I was running into some issues with my nifi
>>> instance when I was using managed-authorizers instead of file-provider.
>>>
>>>
>>>
>>> On Mar 19, 2018, at 9:35 AM, Kevin Doran <kd...@apache.org> wrote:
>>>
>>> Hey Scott,
>>>
>>> Assuming you are using two-way TLS with client certificates for
>>> authentication, I recommend configuring your ELB for TCP passthrough so that
>>> the TLS handshake is between the end-client and the NiFi Registry Server (in
>>> other words, no decryption/termination of the TLS connection happens in the
>>> ELB). If you are using some other form of authentication (e.g., LDAP), you
>>> will need to configure your ELB to trust the self-signed key NiFi Registry
>>> is using. I'm not sure how to do that as I've never run an ELB with that
>>> configuration before.
>>>
>>> Also, just a note about using an ELB with NiFi Registry:
>>>
>>> NiFi Registry is currently only supports single-instance use as persisted
>>> data and in-memory state is not synced between multiple instances. Are you
>>> hoping to use the ELB for actual load balancing, or is it just to take
>>> advantage of other ELB features, such as forwarding and security group
>>> rules? If the plan is to load balance multiple Registry instances, just be
>>> aware that you will probably run into some unexpected behavior. (As you
>>> mentioned using authorization, that is one case where I know the in-memory
>>> cache of the persisted data will not refresh across instances, so even if
>>> you were using some sort of shared network file system attached to multiple
>>> Registry instances, such as EFS, it would not work the way you hope.)
>>>
>>> Hope this helps,
>>> Kevin
>>>
>>> On 3/19/18, 10:20, "Scott Howell" <sc...@mobilgov.com> wrote:
>>>
>>> Thanks for the quick response.
>>>
>>> A couple of things I am seeing.
>>>
>>> 1. There is no error, I don’t see anything in the logs once the service
>>> comes up. This is because the health check is not even hitting the instance
>>> when secure.
>>>
>>> 2. Nothing interesting in the nifi-registry-app.logs. That was my concern
>>> because on my nifi instance I can see the health check hitting the instance
>>> from the ELB. This does not happen on the nifi-registry instance. I see the
>>> service startup and it tells me what domain and port I can access the UI but
>>> nothing else after that.
>>>
>>> 3. When I am on an instances in the same private subnet I am able to curl to
>>> the instance I get the TLS SSL which tells me the keystore is on the server.
>>> I am using a JKS keystore that is self-signed by the company I work for.
>>>
>>> On Mar 19, 2018, at 9:10 AM, Bryan Bende <bb...@gmail.com> wrote:
>>>
>>> Hello,
>>>
>>> What error are you getting when you cannot access the UI?
>>>
>>> Is there anything interesting in nifi-registry-app.log regarding
>>> authentication/authorization when this happens?
>>>
>>> Can you access the UI securely without going through the ELB?
>>>
>>> Thanks,
>>>
>>> Bryan
>>>
>>>
>>> On Mon, Mar 19, 2018 at 10:05 AM, Scott Howell <sc...@mobilgov.com>
>>> wrote:
>>>
>>> I was able to stand up nifi-registry behind an AWS ELB non-secure.
>>> Everything was working great and was able to access the UI anonymously. I
>>> set up the authorization just like on my nifi instances along with the
>>> authorizers and identity-provider. The service comes up without errors and
>>> everything looks good but the health check does not pass and I cannot access
>>> the UI to login. I was wondering if anyone else has ran into this issue
>>> using nifi-registry.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>
Re: Issue with AWS ELB on secure nifi-registry
Posted by Scott Howell <sc...@mobilgov.com>.
Thanks for that. I am still getting this error in my nifi-user.log
o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api: Untrusted proxy CN=*.{redacted}.com, OU={redacted}, O={redacted}, L=Kansas City, ST=Missouri, C=US
Is there an issue with using a wildcard cert?
> On Mar 21, 2018, at 10:23 AM, Bryan Bende <bb...@gmail.com> wrote:
>
> All identity strings are case & whitespace sensitive.
>
> The node identities in your authorizers.xml have no whitespace, and
> the identity showing in the logs does.
>
> On Wed, Mar 21, 2018 at 11:05 AM, Scott Howell <sc...@mobilgov.com> wrote:
>> Thanks for all of the help with this. I have the cluster up and running. The
>> logs look great everything seems to be working but I cannot login into the
>> UI. I am using a wildcard self-signed certificate. The /proxy is in
>> authorizations.xml with the correct users for the nodes.
>>
>> The error I see with the UI :
>> is Untrusted proxy CN=*.{redacted}.com, OU={redacted}, O={redacted},
>> L=Kansas City, ST=Missouri, C=US
>>
>> I haven’t had much luck finding a lot of documentation or forum questions
>> with this kind of issue.
>>
>> My authorizers.xml looks like this
>> <authorizers>
>> <authorizer>
>> <identifier>file-provider</identifier>
>> <class>org.apache.nifi.authorization.FileAuthorizer</class>
>> <property name="Authorizations
>> File">/opt/config/authorizations.xml</property>
>> <property name="Users File">/opt/config/users.xml</property>
>> <property name="Initial Admin
>> Identity">uid=scott,ou=users,dc={redacted},dc=com</property>
>> <property name="Legacy Authorized Users File"></property>
>>
>> <property name="Node Identity
>> 1">CN=node-1-nifi-dev.{redacted}.com,OU={redacted},O={redacted},L=Kansas
>> City,ST=Missouri,C=US</property>
>> <property name="Node Identity
>> 2">CN=node-2-nifi-dev.{redacted}.com,OU={redacted},O={redacted},L=Kansas
>> City,ST=Missouri,C=US</property>
>> </authorizer>
>> </authorizers>
>>
>> Thanks,
>>
>> Scott
>>
>> On Mar 20, 2018, at 1:15 PM, Andy LoPresto <al...@apache.org> wrote:
>>
>> Scott,
>>
>> The original exception is "nested exception is
>> java.security.KeyStoreException: not found”. Can you verify that the
>> keystore you’ve provided is valid using the “keytool” command? In addition,
>> you will need a truststore as well. Try following Pierre's [1] or Bryan’s
>> [2] instructions for setting up a secure cluster.
>>
>> [1]
>> https://pierrevillard.com/2016/11/29/apache-nifi-1-1-0-secured-cluster-setup/
>> [2]
>> https://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-authorization-and-multi-tenancy
>>
>>
>> Andy LoPresto
>> alopresto@apache.org
>> alopresto.apache@gmail.com
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>>
>> On Mar 20, 2018, at 11:05 AM, Scott Howell <sc...@mobilgov.com> wrote:
>>
>> Thanks for all of the help yesterday I was able to get a secure nifi and
>> nifi-registry up and communicating. I am now trying to figure out how to
>> create a secure cluster. I am currently getting this error when I start up
>> nifi.
>>
>> tion; nested exception is
>> org.springframework.beans.factory.BeanCreationException: Error creating bean
>> with name 'clusterCoordinationProtocolSenderListener' defined in class path
>> resource [nifi-cluster-protocol-context.xml]: Cannot resolve reference to
>> bean 'clusterCoordinationProtocolSender' while setting constructor argument;
>> nested exception is org.springframework.beans.factory.BeanCreationException:
>> Error creating bean with name 'clusterCoordinationProtocolSender' defined in
>> class path resource [nifi-cluster-protocol-context.xml]: Cannot resolve
>> reference to bean 'protocolSocketConfiguration' while setting constructor
>> argument; nested exception is
>> org.springframework.beans.factory.BeanCreationException: Error creating bean
>> with name 'protocolSocketConfiguration': FactoryBean threw exception on
>> object creation; nested exception is java.security.KeyStoreException: not
>> found
>> at
>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175)
>> at
>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
>> at
>> org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1634)
>> at
>> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317)
>> at
>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
>> at
>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
>> ... 50 common frames omitted
>> Caused by: org.springframework.beans.factory.BeanCreationException: Error
>> creating bean with name 'clusterCoordinationProtocolSenderListener' defined
>> in class path resource [nifi-cluster-protocol-context.xml]: Cannot resolve
>> reference to bean 'clusterCoordinationProtocolSender' while setting
>> constructor argument; nested exception is
>> org.springframework.beans.factory.BeanCreationException: Error creating bean
>> with name 'clusterCoordinationProtocolSender' defined in class path resource
>> [nifi-cluster-protocol-context.xml]: Cannot resolve reference to bean
>> 'protocolSocketConfiguration' while setting constructor argument; nested
>> exception is org.springframework.beans.factory.BeanCreationException: Error
>> creating bean with name 'protocolSocketConfiguration': FactoryBean threw
>> exception on object creation; nested exception is
>> java.security.KeyStoreException: not found
>> at
>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:359)
>> at
>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)
>> at
>> org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:648)
>> at
>> org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:145)
>> at
>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1193)
>> at
>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1095)
>> at
>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
>> at
>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
>> at
>> org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
>> at
>> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>> at
>> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
>> at
>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
>> at
>> org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1084)
>> at
>> org.apache.nifi.cluster.spring.NodeClusterCoordinatorFactoryBean.getObject(NodeClusterCoordinatorFactoryBean.java:44)
>> at
>> org.apache.nifi.cluster.spring.NodeClusterCoordinatorFactoryBean.getObject(NodeClusterCoordinatorFactoryBean.java:34)
>> at
>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
>> ... 55 common frames omitted
>> Caused by: org.springframework.beans.factory.BeanCreationException: Error
>> creating bean with name 'clusterCoordinationProtocolSender' defined in class
>> path resource [nifi-cluster-protocol-context.xml]: Cannot resolve reference
>> to bean 'protocolSocketConfiguration' while setting constructor argument;
>> nested exception is org.springframework.beans.factory.BeanCreationException:
>> Error creating bean with name 'protocolSocketConfiguration': FactoryBean
>> threw exception on object creation; nested exception is
>> java.security.KeyStoreException: not found
>> at
>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:359)
>> at
>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)
>> at
>> org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:648)
>> at
>> org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:145)
>> at
>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1193)
>> at
>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1095)
>> at
>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
>> at
>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
>> at
>> org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
>> at
>> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>> at
>> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
>> at
>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
>> at
>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
>> ... 70 common frames omitted
>> Caused by: org.springframework.beans.factory.BeanCreationException: Error
>> creating bean with name 'protocolSocketConfiguration': FactoryBean threw
>> exception on object creation; nested exception is
>> java.security.KeyStoreException: not found
>> at
>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175)
>> at
>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
>> at
>> org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1634)
>> at
>> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317)
>> at
>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
>> at
>> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
>> ... 82 common frames omitted
>> Caused by: java.security.KeyStoreException: not found
>> at java.security.KeyStore.getInstance(KeyStore.java:851)
>> at
>> org.apache.nifi.security.util.KeyStoreUtils.getKeyStore(KeyStoreUtils.java:66)
>> at
>> org.apache.nifi.security.util.KeyStoreUtils.getTrustStore(KeyStoreUtils.java:80)
>> at
>> org.apache.nifi.io.socket.SSLContextFactory.<init>(SSLContextFactory.java:73)
>> at
>> org.apache.nifi.cluster.protocol.spring.SocketConfigurationFactoryBean.getObject(SocketConfigurationFactoryBean.java:45)
>> at
>> org.apache.nifi.cluster.protocol.spring.SocketConfigurationFactoryBean.getObject(SocketConfigurationFactoryBean.java:30)
>> at
>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
>> ... 87 common frames omitted
>> Caused by: java.security.NoSuchAlgorithmException: KeyStore not available
>> at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
>> at java.security.Security.getImpl(Security.java:695)
>> at java.security.KeyStore.getInstance(KeyStore.java:848)
>> ... 93 common frames omitted
>>
>> My nifi.properties file is.
>>
>> # Licensed to the Apache Software Foundation (ASF) under one or more
>> # contributor license agreements. See the NOTICE file distributed with
>> # this work for additional information regarding copyright ownership.
>> # The ASF licenses this file to You under the Apache License, Version 2.0
>> # (the "License"); you may not use this file except in compliance with
>> # the License. You may obtain a copy of the License at
>> #
>> # http://www.apache.org/licenses/LICENSE-2.0
>> #
>> # Unless required by applicable law or agreed to in writing, software
>> # distributed under the License is distributed on an "AS IS" BASIS,
>> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>> # See the License for the specific language governing permissions and
>> # limitations under the License.
>>
>> # Core Properties #
>> nifi.version={{nifi_version}}
>> nifi.flow.configuration.file=/opt/config/flow.xml.gz
>> nifi.flow.configuration.archive.enabled=true
>> nifi.flow.configuration.archive.dir=/opt/config/archive/
>> nifi.flow.configuration.archive.max.time=30 days
>> nifi.flow.configuration.archive.max.storage=500 MB
>> nifi.flowcontroller.autoResumeState=true
>> nifi.flowcontroller.graceful.shutdown.period=10 sec
>> nifi.flowservice.writedelay.interval=500 ms
>> nifi.administrative.yield.duration=30 sec
>> # If a component has no work to do (is "bored"), how long should we wait
>> before checking again for work?
>> nifi.bored.yield.duration=10 millis
>>
>>
>> nifi.authorizer.configuration.file=/opt/config/authorizers.xml
>> nifi.login.identity.provider.configuration.file=/opt/config/login-identity-providers.xml
>> nifi.templates.directory=/opt/config/templates
>> nifi.ui.banner.text=
>> nifi.ui.autorefresh.interval=30 sec
>> nifi.nar.library.directory=/opt/nifi/lib
>> nifi.nar.library.directory.custom=/opt/config/processors
>> nifi.nar.working.directory=/opt/nifi/work/nar/
>> nifi.documentation.working.directory=./work/docs/components
>>
>> ####################
>> # State Management #
>> ####################
>> nifi.state.management.configuration.file=/opt/config/state-management.xml
>> # The ID of the local state provider
>> nifi.state.management.provider.local=local-provider
>> # The ID of the cluster-wide state provider. This will be ignored if NiFi is
>> not clustered but must be populated if running in a cluster.
>> nifi.state.management.provider.cluster=zk-provider
>> # Specifies whether or not this instance of NiFi should run an embedded
>> ZooKeeper server
>> nifi.state.management.embedded.zookeeper.start=false
>> # Properties file that provides the ZooKeeper properties to use if
>> <nifi.state.management.embedded.zookeeper.start> is set to true
>> nifi.state.management.embedded.zookeeper.properties=/opt/config/zookeeper.properties
>>
>>
>> # H2 Settings
>> nifi.database.directory=/opt/database_repository
>> nifi.h2.url.append=;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE
>>
>> # FlowFile Repository
>> nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository
>> nifi.flowfile.repository.directory=/opt/flowfile_repository
>> nifi.flowfile.repository.partitions=256
>> nifi.flowfile.repository.checkpoint.interval=2 mins
>> nifi.flowfile.repository.always.sync=false
>>
>> nifi.swap.manager.implementation=org.apache.nifi.controller.FileSystemSwapManager
>> nifi.queue.swap.threshold=20000
>> nifi.swap.in.period=5 sec
>> nifi.swap.in.threads=1
>> nifi.swap.out.period=5 sec
>> nifi.swap.out.threads=4
>>
>> # Content Repository
>> nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository
>> nifi.content.claim.max.appendable.size=10 MB
>> nifi.content.claim.max.flow.files=100
>> nifi.content.repository.directory.default=/opt/content_repository
>> nifi.content.repository.archive.max.retention.period=12 hours
>> nifi.content.repository.archive.max.usage.percentage=50%
>> nifi.content.repository.archive.enabled=true
>> nifi.content.repository.always.sync=false
>> nifi.content.viewer.url=/nifi-content-viewer/
>>
>> # Provenance Repository Properties
>> nifi.provenance.repository.implementation=org.apache.nifi.provenance.PersistentProvenanceRepository
>>
>> # Persistent Provenance Repository Properties
>> nifi.provenance.repository.directory.default=/opt/provenance_repository
>> nifi.provenance.repository.max.storage.time=24 hours
>> nifi.provenance.repository.max.storage.size=1 GB
>> nifi.provenance.repository.rollover.time=30 secs
>> nifi.provenance.repository.rollover.size=100 MB
>> nifi.provenance.repository.query.threads=2
>> nifi.provenance.repository.index.threads=1
>> nifi.provenance.repository.compress.on.rollover=true
>> nifi.provenance.repository.always.sync=false
>> nifi.provenance.repository.journal.count=16
>> # Comma-separated list of fields. Fields that are not indexed will not be
>> searchable. Valid fields are:
>> # EventType, FlowFileUUID, Filename, TransitURI, ProcessorID,
>> AlternateIdentifierURI, Relationship, Details
>> nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename,
>> ProcessorID, Relationship
>> # FlowFile Attributes that should be indexed and made searchable. Some
>> examples to consider are filename, uuid, mime.type
>> nifi.provenance.repository.indexed.attributes=
>> # Large values for the shard size will result in more Java heap usage when
>> searching the Provenance Repository
>> # but should provide better performance
>> nifi.provenance.repository.index.shard.size=500 MB
>> # Indicates the maximum length that a FlowFile attribute can be when
>> retrieving a Provenance Event from
>> # the repository. If the length of any attribute exceeds this value, it will
>> be truncated when the event is retrieved.
>> nifi.provenance.repository.max.attribute.length=65536
>>
>> # Volatile Provenance Respository Properties
>> nifi.provenance.repository.buffer.size=100000
>>
>> # Component Status Repository
>> nifi.components.status.repository.implementation=org.apache.nifi.controller.status.history.VolatileComponentStatusRepository
>> nifi.components.status.repository.buffer.size=1440
>> nifi.components.status.snapshot.frequency=1 min
>>
>> # Site to Site properties
>> nifi.remote.input.host=
>> nifi.remote.input.secure=false
>> nifi.remote.input.socket.port=9998
>> nifi.remote.input.http.enabled=false
>> nifi.remote.input.http.transaction.ttl=30 sec
>>
>> # web properties #
>> nifi.web.war.directory=/opt/nifi/lib
>> nifi.web.http.host=
>> nifi.web.http.port=
>> nifi.web.https.host={{redacted}}
>> nifi.web.https.port=8443
>> nifi.web.jetty.working.directory=/opt/nifi/work/jetty
>> nifi.web.jetty.threads=200
>>
>> # security properties #
>> nifi.sensitive.props.key=x0KDgO9L8lAhFGLdvu2VEjFVGc6Kg3V0R5I4bYwoqdgL47moo0wApKQtAVu1BvD
>> nifi.sensitive.props.key.protected=
>> nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
>> nifi.sensitive.props.provider=BC
>> nifi.sensitive.props.additional.keys=
>>
>> nifi.security.keystore=/opt/certs/payit_keystore
>> nifi.security.keystoreType=JKS
>> nifi.security.keystorePasswd={{keystore_password}}
>> nifi.security.keyPasswd=
>> nifi.security.truststore=
>> nifi.security.truststoreType=
>> nifi.security.truststorePasswd=
>> nifi.security.needClientAuth=false
>> nifi.security.user.authorizer=file-provider
>> nifi.security.user.login.identity.provider=ldap-provider
>> nifi.security.ocsp.responder.url=
>> nifi.security.ocsp.responder.certificate=
>>
>> # Identity Mapping Properties #
>> # These properties allow normalizing user identities such that identities
>> coming from different identity providers
>> # (certificates, LDAP, Kerberos) can be treated the same internally in NiFi.
>> The following example demonstrates normalizing
>> # DNs from certificates and principals from Kerberos into a common identity
>> string:
>> #
>> #nifi.security.identity.mapping.pattern.dn=^cn=(.*?),ou=(.*?),dc=(.*?),dc=(.*?),dc=(.*?)$
>> #nifi.security.identity.mapping.value.dn=$1
>> # nifi.security.identity.mapping.pattern.kerb=^(.*?)/instance@(.*?)$
>> # nifi.security.identity.mapping.value.kerb=$1@$2
>>
>> # cluster common properties (all nodes must have same values) #
>> nifi.cluster.protocol.heartbeat.interval=5 sec
>> nifi.cluster.protocol.is.secure=true
>>
>> # cluster node properties (only configure for cluster nodes) #
>> nifi.cluster.is.node=true
>> nifi.cluster.node.address=nifi-dev.mobilgov.com
>> nifi.cluster.node.protocol.port=9999
>> nifi.cluster.node.protocol.threads=10
>> nifi.cluster.node.event.history.size=25
>> nifi.cluster.node.connection.timeout=5 sec
>> nifi.cluster.node.read.timeout=5 sec
>> nifi.cluster.firewall.file=
>>
>>
>> # zookeeper properties, used for cluster management #
>> nifi.zookeeper.connect.string=internal-etcd-dev-EtcdLoadB-3RWA2WEFBBT8-2068560477.us-east-2.elb.amazonaws.com:2181,internal-etcd-dev-EtcdLoadB-3RWA2WEFBBT8-2068560477.us-east-2.elb.amazonaws.com2182,internal-etcd-dev-EtcdLoadB-3RWA2WEFBBT8-2068560477.us-east-2.elb.amazonaws.com:2183
>> nifi.zookeeper.connect.timeout=3 secs
>> nifi.zookeeper.session.timeout=3 secs
>> nifi.zookeeper.root.node=/nifi
>>
>> # kerberos #
>> nifi.kerberos.krb5.file=
>>
>> # kerberos service principle #
>> nifi.kerberos.service.principal=
>> nifi.kerberos.service.keytab.location=
>>
>> # kerberos spnego principle #
>> nifi.kerberos.spnego.principal=
>> nifi.kerberos.spnego.keytab.location=
>> nifi.kerberos.spnego.authentication.expiration=12 hours
>>
>> # external properties files for variable registry
>> # supports a comma delimited list of file locations
>> nifi.variable.registry.properties=
>>
>> I think I have everything set correctly but I have not been able to start an
>> instances up.
>>
>> Thanks,
>>
>> Scott
>>
>> On Mar 19, 2018, at 4:35 PM, Bryan Bende <bb...@gmail.com> wrote:
>>
>> The base file is here for comparison:
>>
>> https://github.com/apache/nifi-registry/blob/master/nifi-registry-resources/src/main/resources/conf/identity-providers.xml#L23
>>
>> On Mon, Mar 19, 2018 at 5:34 PM, Bryan Bende <bb...@gmail.com> wrote:
>>
>> For your first file, is what you showed there actually wrapped in
>> <identityProviders> </identityProviders> or is it exactly what you
>> showed?
>>
>> It may just be that you only copied/pasted the one provider, but the
>> root element is not <provider>, so as it is shown there it would not
>> parse.
>>
>> On Mon, Mar 19, 2018 at 2:54 PM, Scott Howell <sc...@mobilgov.com>
>> wrote:
>>
>> Here is my file
>>
>> <provider>
>> <identifier>ldap-identity-provider</identifier>
>> <class>org.apache.nifi.registry.security.ldap.LdapProvider</class>
>> <property name="Authentication Strategy">SIMPLE</property>
>>
>> <property name="Manager DN">cn=Manager,dc=mobilgov,dc=com</property>
>> <property name="Manager Password”>redacted</property>
>>
>>
>> <property name="Referral Strategy">FOLLOW</property>
>> <property name="Connect Timeout">10 secs</property>
>> <property name="Read Timeout">10 secs</property>
>>
>> <property name="Url”>redacted</property>
>> <property name="User Search
>> Base">ou=users,dc=mobilgov,dc=com</property>
>> <property name="User Search Filter">uid={0}</property>
>>
>> <property name="Identity Strategy">USE_DN</property>
>> <property name="Authentication Expiration">12 hours</property>
>> </provider>
>>
>> Here is my authorizers.xml
>>
>> <authorizers>
>>
>> <userGroupProvider>
>> <identifier>file-user-group-provider</identifier>
>>
>> <class>org.apache.nifi.registry.security.authorization.file.FileUserGroupProvider</class>
>> <property name="Users File">conf/users.xml</property>
>> <property name="Legacy Authorized Users File"></property>
>> <property name="Initial User Identity 1”>redacted</property>
>> </userGroupProvider>
>>
>> <accessPolicyProvider>
>> <identifier>file-access-policy-provider</identifier>
>>
>> <class>org.apache.nifi.registry.security.authorization.file.FileAccessPolicyProvider</class>
>> <property name="User Group
>> Provider">file-user-group-provider</property>
>> <property name="Authorizations
>> File">conf/authorizations.xml</property>
>> <property name="Initial Admin Identity”>redacted</property>
>> <property name="NiFi Identity 1"></property>
>> </accessPolicyProvider>
>>
>> <authorizer>
>> <identifier>managed-authorizer</identifier>
>>
>> <class>org.apache.nifi.registry.security.authorization.StandardManagedAuthorizer</class>
>> <property name="Access Policy
>> Provider">file-access-policy-provider</property>
>> </authorizer>
>> </authorizers>
>>
>> On Mar 19, 2018, at 12:59 PM, Bryan Bende <bb...@gmail.com> wrote:
>>
>> It looks like that error would happen if your identity-providers.xml
>> contained invalid XML.
>>
>> Did you start by modifying the identity-providers.xml file that was
>> already there? Can you share the file, or the contents (removing
>> anything sensitive)?
>>
>> On Mon, Mar 19, 2018 at 1:09 PM, Scott Howell <sc...@mobilgov.com>
>> wrote:
>>
>> So I was able to get the UI pulled up but now I am hitting a roadblock with
>> my identity-provider.xml.
>>
>> I am getting a number of errors like this:
>>
>> Caused by: org.springframework.beans.factory.BeanCreationException: Error
>> creating bean with name 'getIdentityProvider' defined in class path resource
>> [org/apache/nifi/registry/security/authentication/IdentityProviderFactory.class]:
>> Bean instantiation via factory method failed; nested exception is
>> org.springframework.beans.BeanInstantiationException: Failed to instantiate
>> [org.apache.nifi.registry.security.authentication.IdentityProvider]: Factory
>> method 'getIdentityProvider' threw exception; nested exception is
>> java.lang.Exception: Unable to load the login identity provider
>> configuration file at: /opt/nifi-registry-0.1.0/conf/identity-providers.xml
>> at
>> org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:587)
>> ~[na:na]
>> at
>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1250)
>> ~[na:na]
>> at
>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1099)
>> ~[na:na]
>> at
>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:545)
>> ~[na:na]
>> at
>> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:502)
>> ~[na:na]
>> at
>> org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:312)
>> ~[na:na]
>> at
>> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228)
>> ~[na:na]
>> at
>> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:310)
>> ~[na:na]
>> at
>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200)
>> ~[na:na]
>> at
>> org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:251)
>> ~[na:na]
>> at
>> org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1135)
>> ~[na:na]
>> at
>> org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1062)
>> ~[na:na]
>> at
>> org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:815)
>> ~[na:na]
>> at
>> org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:721)
>> ~[na:na]
>> ... 43 common frames omitted
>>
>> I know it has to do with the identity-provider.xml but I have my setup just
>> like the documentation ask for. I turned on debug but was not able to see
>> anything different or better explanation from it.
>>
>>
>> On Mar 19, 2018, at 10:06 AM, Kevin Doran <kd...@apache.org> wrote:
>>
>> Ok, that use case should be fine.
>>
>> If it were an authorization issue you would see something in the logs saying
>> that an authorization attempt failed and the server is responding with a
>> 403. Just to be sure, can you enable debug logging if you haven't already,
>> i.e., in your nifi-registry/conf/logback.xml file, change
>> 'org.apache.nifif.registry' to debug:
>>
>> <!-- valid logging levels: TRACE, DEBUG, INFO, WARN, ERROR -->
>> <logger name="org.apache.nifi.registry" level="DEBUG"/>
>>
>> If there is nothing being written to nifi-registry-app.log, it points
>> towards a connection issue, so I would double check your host, port, and TLS
>> settings. You'll have to get an HTTPS cert from a root CA or configure your
>> ELB to trust your company's self-signed cert (again, not sure if/how to do
>> this, but I assume there should be some way to configure it. It might
>> require settings not exposed in the AWS web console.)
>>
>> On 3/19/18, 10:51, "Scott Howell" <sc...@mobilgov.com> wrote:
>>
>> Thanks Kevin,
>>
>> I am just using the ELB to go from the public subnet to the private subnet.
>> I will not have multiple instances running of registry.
>>
>> I will say on my authorizers.xml there is one difference between my nifi
>> instance. On my nifi instance I am using file-provider for
>> nifi.security.user.authorizer in my nifi.properties. I don’t think from
>> reading the documents for nifi-registry that I can use that. If there is a
>> way that might be my problem. I was running into some issues with my nifi
>> instance when I was using managed-authorizers instead of file-provider.
>>
>>
>>
>> On Mar 19, 2018, at 9:35 AM, Kevin Doran <kd...@apache.org> wrote:
>>
>> Hey Scott,
>>
>> Assuming you are using two-way TLS with client certificates for
>> authentication, I recommend configuring your ELB for TCP passthrough so that
>> the TLS handshake is between the end-client and the NiFi Registry Server (in
>> other words, no decryption/termination of the TLS connection happens in the
>> ELB). If you are using some other form of authentication (e.g., LDAP), you
>> will need to configure your ELB to trust the self-signed key NiFi Registry
>> is using. I'm not sure how to do that as I've never run an ELB with that
>> configuration before.
>>
>> Also, just a note about using an ELB with NiFi Registry:
>>
>> NiFi Registry is currently only supports single-instance use as persisted
>> data and in-memory state is not synced between multiple instances. Are you
>> hoping to use the ELB for actual load balancing, or is it just to take
>> advantage of other ELB features, such as forwarding and security group
>> rules? If the plan is to load balance multiple Registry instances, just be
>> aware that you will probably run into some unexpected behavior. (As you
>> mentioned using authorization, that is one case where I know the in-memory
>> cache of the persisted data will not refresh across instances, so even if
>> you were using some sort of shared network file system attached to multiple
>> Registry instances, such as EFS, it would not work the way you hope.)
>>
>> Hope this helps,
>> Kevin
>>
>> On 3/19/18, 10:20, "Scott Howell" <sc...@mobilgov.com> wrote:
>>
>> Thanks for the quick response.
>>
>> A couple of things I am seeing.
>>
>> 1. There is no error, I don’t see anything in the logs once the service
>> comes up. This is because the health check is not even hitting the instance
>> when secure.
>>
>> 2. Nothing interesting in the nifi-registry-app.logs. That was my concern
>> because on my nifi instance I can see the health check hitting the instance
>> from the ELB. This does not happen on the nifi-registry instance. I see the
>> service startup and it tells me what domain and port I can access the UI but
>> nothing else after that.
>>
>> 3. When I am on an instances in the same private subnet I am able to curl to
>> the instance I get the TLS SSL which tells me the keystore is on the server.
>> I am using a JKS keystore that is self-signed by the company I work for.
>>
>> On Mar 19, 2018, at 9:10 AM, Bryan Bende <bb...@gmail.com> wrote:
>>
>> Hello,
>>
>> What error are you getting when you cannot access the UI?
>>
>> Is there anything interesting in nifi-registry-app.log regarding
>> authentication/authorization when this happens?
>>
>> Can you access the UI securely without going through the ELB?
>>
>> Thanks,
>>
>> Bryan
>>
>>
>> On Mon, Mar 19, 2018 at 10:05 AM, Scott Howell <sc...@mobilgov.com>
>> wrote:
>>
>> I was able to stand up nifi-registry behind an AWS ELB non-secure.
>> Everything was working great and was able to access the UI anonymously. I
>> set up the authorization just like on my nifi instances along with the
>> authorizers and identity-provider. The service comes up without errors and
>> everything looks good but the health check does not pass and I cannot access
>> the UI to login. I was wondering if anyone else has ran into this issue
>> using nifi-registry.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
Re: Issue with AWS ELB on secure nifi-registry
Posted by Bryan Bende <bb...@gmail.com>.
All identity strings are case & whitespace sensitive.
The node identities in your authorizers.xml have no whitespace, and
the identity showing in the logs does.
On Wed, Mar 21, 2018 at 11:05 AM, Scott Howell <sc...@mobilgov.com> wrote:
> Thanks for all of the help with this. I have the cluster up and running. The
> logs look great everything seems to be working but I cannot login into the
> UI. I am using a wildcard self-signed certificate. The /proxy is in
> authorizations.xml with the correct users for the nodes.
>
> The error I see with the UI :
> is Untrusted proxy CN=*.{redacted}.com, OU={redacted}, O={redacted},
> L=Kansas City, ST=Missouri, C=US
>
> I haven’t had much luck finding a lot of documentation or forum questions
> with this kind of issue.
>
> My authorizers.xml looks like this
> <authorizers>
> <authorizer>
> <identifier>file-provider</identifier>
> <class>org.apache.nifi.authorization.FileAuthorizer</class>
> <property name="Authorizations
> File">/opt/config/authorizations.xml</property>
> <property name="Users File">/opt/config/users.xml</property>
> <property name="Initial Admin
> Identity">uid=scott,ou=users,dc={redacted},dc=com</property>
> <property name="Legacy Authorized Users File"></property>
>
> <property name="Node Identity
> 1">CN=node-1-nifi-dev.{redacted}.com,OU={redacted},O={redacted},L=Kansas
> City,ST=Missouri,C=US</property>
> <property name="Node Identity
> 2">CN=node-2-nifi-dev.{redacted}.com,OU={redacted},O={redacted},L=Kansas
> City,ST=Missouri,C=US</property>
> </authorizer>
> </authorizers>
>
> Thanks,
>
> Scott
>
> On Mar 20, 2018, at 1:15 PM, Andy LoPresto <al...@apache.org> wrote:
>
> Scott,
>
> The original exception is "nested exception is
> java.security.KeyStoreException: not found”. Can you verify that the
> keystore you’ve provided is valid using the “keytool” command? In addition,
> you will need a truststore as well. Try following Pierre's [1] or Bryan’s
> [2] instructions for setting up a secure cluster.
>
> [1]
> https://pierrevillard.com/2016/11/29/apache-nifi-1-1-0-secured-cluster-setup/
> [2]
> https://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-authorization-and-multi-tenancy
>
>
> Andy LoPresto
> alopresto@apache.org
> alopresto.apache@gmail.com
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
> On Mar 20, 2018, at 11:05 AM, Scott Howell <sc...@mobilgov.com> wrote:
>
> Thanks for all of the help yesterday I was able to get a secure nifi and
> nifi-registry up and communicating. I am now trying to figure out how to
> create a secure cluster. I am currently getting this error when I start up
> nifi.
>
> tion; nested exception is
> org.springframework.beans.factory.BeanCreationException: Error creating bean
> with name 'clusterCoordinationProtocolSenderListener' defined in class path
> resource [nifi-cluster-protocol-context.xml]: Cannot resolve reference to
> bean 'clusterCoordinationProtocolSender' while setting constructor argument;
> nested exception is org.springframework.beans.factory.BeanCreationException:
> Error creating bean with name 'clusterCoordinationProtocolSender' defined in
> class path resource [nifi-cluster-protocol-context.xml]: Cannot resolve
> reference to bean 'protocolSocketConfiguration' while setting constructor
> argument; nested exception is
> org.springframework.beans.factory.BeanCreationException: Error creating bean
> with name 'protocolSocketConfiguration': FactoryBean threw exception on
> object creation; nested exception is java.security.KeyStoreException: not
> found
> at
> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175)
> at
> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1634)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
> at
> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
> ... 50 common frames omitted
> Caused by: org.springframework.beans.factory.BeanCreationException: Error
> creating bean with name 'clusterCoordinationProtocolSenderListener' defined
> in class path resource [nifi-cluster-protocol-context.xml]: Cannot resolve
> reference to bean 'clusterCoordinationProtocolSender' while setting
> constructor argument; nested exception is
> org.springframework.beans.factory.BeanCreationException: Error creating bean
> with name 'clusterCoordinationProtocolSender' defined in class path resource
> [nifi-cluster-protocol-context.xml]: Cannot resolve reference to bean
> 'protocolSocketConfiguration' while setting constructor argument; nested
> exception is org.springframework.beans.factory.BeanCreationException: Error
> creating bean with name 'protocolSocketConfiguration': FactoryBean threw
> exception on object creation; nested exception is
> java.security.KeyStoreException: not found
> at
> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:359)
> at
> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)
> at
> org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:648)
> at
> org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:145)
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1193)
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1095)
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
> at
> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
> at
> org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1084)
> at
> org.apache.nifi.cluster.spring.NodeClusterCoordinatorFactoryBean.getObject(NodeClusterCoordinatorFactoryBean.java:44)
> at
> org.apache.nifi.cluster.spring.NodeClusterCoordinatorFactoryBean.getObject(NodeClusterCoordinatorFactoryBean.java:34)
> at
> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
> ... 55 common frames omitted
> Caused by: org.springframework.beans.factory.BeanCreationException: Error
> creating bean with name 'clusterCoordinationProtocolSender' defined in class
> path resource [nifi-cluster-protocol-context.xml]: Cannot resolve reference
> to bean 'protocolSocketConfiguration' while setting constructor argument;
> nested exception is org.springframework.beans.factory.BeanCreationException:
> Error creating bean with name 'protocolSocketConfiguration': FactoryBean
> threw exception on object creation; nested exception is
> java.security.KeyStoreException: not found
> at
> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:359)
> at
> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)
> at
> org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:648)
> at
> org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:145)
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1193)
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1095)
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
> at
> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
> at
> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
> ... 70 common frames omitted
> Caused by: org.springframework.beans.factory.BeanCreationException: Error
> creating bean with name 'protocolSocketConfiguration': FactoryBean threw
> exception on object creation; nested exception is
> java.security.KeyStoreException: not found
> at
> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175)
> at
> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1634)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
> at
> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
> ... 82 common frames omitted
> Caused by: java.security.KeyStoreException: not found
> at java.security.KeyStore.getInstance(KeyStore.java:851)
> at
> org.apache.nifi.security.util.KeyStoreUtils.getKeyStore(KeyStoreUtils.java:66)
> at
> org.apache.nifi.security.util.KeyStoreUtils.getTrustStore(KeyStoreUtils.java:80)
> at
> org.apache.nifi.io.socket.SSLContextFactory.<init>(SSLContextFactory.java:73)
> at
> org.apache.nifi.cluster.protocol.spring.SocketConfigurationFactoryBean.getObject(SocketConfigurationFactoryBean.java:45)
> at
> org.apache.nifi.cluster.protocol.spring.SocketConfigurationFactoryBean.getObject(SocketConfigurationFactoryBean.java:30)
> at
> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
> ... 87 common frames omitted
> Caused by: java.security.NoSuchAlgorithmException: KeyStore not available
> at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
> at java.security.Security.getImpl(Security.java:695)
> at java.security.KeyStore.getInstance(KeyStore.java:848)
> ... 93 common frames omitted
>
> My nifi.properties file is.
>
> # Licensed to the Apache Software Foundation (ASF) under one or more
> # contributor license agreements. See the NOTICE file distributed with
> # this work for additional information regarding copyright ownership.
> # The ASF licenses this file to You under the Apache License, Version 2.0
> # (the "License"); you may not use this file except in compliance with
> # the License. You may obtain a copy of the License at
> #
> # http://www.apache.org/licenses/LICENSE-2.0
> #
> # Unless required by applicable law or agreed to in writing, software
> # distributed under the License is distributed on an "AS IS" BASIS,
> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> # See the License for the specific language governing permissions and
> # limitations under the License.
>
> # Core Properties #
> nifi.version={{nifi_version}}
> nifi.flow.configuration.file=/opt/config/flow.xml.gz
> nifi.flow.configuration.archive.enabled=true
> nifi.flow.configuration.archive.dir=/opt/config/archive/
> nifi.flow.configuration.archive.max.time=30 days
> nifi.flow.configuration.archive.max.storage=500 MB
> nifi.flowcontroller.autoResumeState=true
> nifi.flowcontroller.graceful.shutdown.period=10 sec
> nifi.flowservice.writedelay.interval=500 ms
> nifi.administrative.yield.duration=30 sec
> # If a component has no work to do (is "bored"), how long should we wait
> before checking again for work?
> nifi.bored.yield.duration=10 millis
>
>
> nifi.authorizer.configuration.file=/opt/config/authorizers.xml
> nifi.login.identity.provider.configuration.file=/opt/config/login-identity-providers.xml
> nifi.templates.directory=/opt/config/templates
> nifi.ui.banner.text=
> nifi.ui.autorefresh.interval=30 sec
> nifi.nar.library.directory=/opt/nifi/lib
> nifi.nar.library.directory.custom=/opt/config/processors
> nifi.nar.working.directory=/opt/nifi/work/nar/
> nifi.documentation.working.directory=./work/docs/components
>
> ####################
> # State Management #
> ####################
> nifi.state.management.configuration.file=/opt/config/state-management.xml
> # The ID of the local state provider
> nifi.state.management.provider.local=local-provider
> # The ID of the cluster-wide state provider. This will be ignored if NiFi is
> not clustered but must be populated if running in a cluster.
> nifi.state.management.provider.cluster=zk-provider
> # Specifies whether or not this instance of NiFi should run an embedded
> ZooKeeper server
> nifi.state.management.embedded.zookeeper.start=false
> # Properties file that provides the ZooKeeper properties to use if
> <nifi.state.management.embedded.zookeeper.start> is set to true
> nifi.state.management.embedded.zookeeper.properties=/opt/config/zookeeper.properties
>
>
> # H2 Settings
> nifi.database.directory=/opt/database_repository
> nifi.h2.url.append=;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE
>
> # FlowFile Repository
> nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository
> nifi.flowfile.repository.directory=/opt/flowfile_repository
> nifi.flowfile.repository.partitions=256
> nifi.flowfile.repository.checkpoint.interval=2 mins
> nifi.flowfile.repository.always.sync=false
>
> nifi.swap.manager.implementation=org.apache.nifi.controller.FileSystemSwapManager
> nifi.queue.swap.threshold=20000
> nifi.swap.in.period=5 sec
> nifi.swap.in.threads=1
> nifi.swap.out.period=5 sec
> nifi.swap.out.threads=4
>
> # Content Repository
> nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository
> nifi.content.claim.max.appendable.size=10 MB
> nifi.content.claim.max.flow.files=100
> nifi.content.repository.directory.default=/opt/content_repository
> nifi.content.repository.archive.max.retention.period=12 hours
> nifi.content.repository.archive.max.usage.percentage=50%
> nifi.content.repository.archive.enabled=true
> nifi.content.repository.always.sync=false
> nifi.content.viewer.url=/nifi-content-viewer/
>
> # Provenance Repository Properties
> nifi.provenance.repository.implementation=org.apache.nifi.provenance.PersistentProvenanceRepository
>
> # Persistent Provenance Repository Properties
> nifi.provenance.repository.directory.default=/opt/provenance_repository
> nifi.provenance.repository.max.storage.time=24 hours
> nifi.provenance.repository.max.storage.size=1 GB
> nifi.provenance.repository.rollover.time=30 secs
> nifi.provenance.repository.rollover.size=100 MB
> nifi.provenance.repository.query.threads=2
> nifi.provenance.repository.index.threads=1
> nifi.provenance.repository.compress.on.rollover=true
> nifi.provenance.repository.always.sync=false
> nifi.provenance.repository.journal.count=16
> # Comma-separated list of fields. Fields that are not indexed will not be
> searchable. Valid fields are:
> # EventType, FlowFileUUID, Filename, TransitURI, ProcessorID,
> AlternateIdentifierURI, Relationship, Details
> nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename,
> ProcessorID, Relationship
> # FlowFile Attributes that should be indexed and made searchable. Some
> examples to consider are filename, uuid, mime.type
> nifi.provenance.repository.indexed.attributes=
> # Large values for the shard size will result in more Java heap usage when
> searching the Provenance Repository
> # but should provide better performance
> nifi.provenance.repository.index.shard.size=500 MB
> # Indicates the maximum length that a FlowFile attribute can be when
> retrieving a Provenance Event from
> # the repository. If the length of any attribute exceeds this value, it will
> be truncated when the event is retrieved.
> nifi.provenance.repository.max.attribute.length=65536
>
> # Volatile Provenance Respository Properties
> nifi.provenance.repository.buffer.size=100000
>
> # Component Status Repository
> nifi.components.status.repository.implementation=org.apache.nifi.controller.status.history.VolatileComponentStatusRepository
> nifi.components.status.repository.buffer.size=1440
> nifi.components.status.snapshot.frequency=1 min
>
> # Site to Site properties
> nifi.remote.input.host=
> nifi.remote.input.secure=false
> nifi.remote.input.socket.port=9998
> nifi.remote.input.http.enabled=false
> nifi.remote.input.http.transaction.ttl=30 sec
>
> # web properties #
> nifi.web.war.directory=/opt/nifi/lib
> nifi.web.http.host=
> nifi.web.http.port=
> nifi.web.https.host={{redacted}}
> nifi.web.https.port=8443
> nifi.web.jetty.working.directory=/opt/nifi/work/jetty
> nifi.web.jetty.threads=200
>
> # security properties #
> nifi.sensitive.props.key=x0KDgO9L8lAhFGLdvu2VEjFVGc6Kg3V0R5I4bYwoqdgL47moo0wApKQtAVu1BvD
> nifi.sensitive.props.key.protected=
> nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
> nifi.sensitive.props.provider=BC
> nifi.sensitive.props.additional.keys=
>
> nifi.security.keystore=/opt/certs/payit_keystore
> nifi.security.keystoreType=JKS
> nifi.security.keystorePasswd={{keystore_password}}
> nifi.security.keyPasswd=
> nifi.security.truststore=
> nifi.security.truststoreType=
> nifi.security.truststorePasswd=
> nifi.security.needClientAuth=false
> nifi.security.user.authorizer=file-provider
> nifi.security.user.login.identity.provider=ldap-provider
> nifi.security.ocsp.responder.url=
> nifi.security.ocsp.responder.certificate=
>
> # Identity Mapping Properties #
> # These properties allow normalizing user identities such that identities
> coming from different identity providers
> # (certificates, LDAP, Kerberos) can be treated the same internally in NiFi.
> The following example demonstrates normalizing
> # DNs from certificates and principals from Kerberos into a common identity
> string:
> #
> #nifi.security.identity.mapping.pattern.dn=^cn=(.*?),ou=(.*?),dc=(.*?),dc=(.*?),dc=(.*?)$
> #nifi.security.identity.mapping.value.dn=$1
> # nifi.security.identity.mapping.pattern.kerb=^(.*?)/instance@(.*?)$
> # nifi.security.identity.mapping.value.kerb=$1@$2
>
> # cluster common properties (all nodes must have same values) #
> nifi.cluster.protocol.heartbeat.interval=5 sec
> nifi.cluster.protocol.is.secure=true
>
> # cluster node properties (only configure for cluster nodes) #
> nifi.cluster.is.node=true
> nifi.cluster.node.address=nifi-dev.mobilgov.com
> nifi.cluster.node.protocol.port=9999
> nifi.cluster.node.protocol.threads=10
> nifi.cluster.node.event.history.size=25
> nifi.cluster.node.connection.timeout=5 sec
> nifi.cluster.node.read.timeout=5 sec
> nifi.cluster.firewall.file=
>
>
> # zookeeper properties, used for cluster management #
> nifi.zookeeper.connect.string=internal-etcd-dev-EtcdLoadB-3RWA2WEFBBT8-2068560477.us-east-2.elb.amazonaws.com:2181,internal-etcd-dev-EtcdLoadB-3RWA2WEFBBT8-2068560477.us-east-2.elb.amazonaws.com2182,internal-etcd-dev-EtcdLoadB-3RWA2WEFBBT8-2068560477.us-east-2.elb.amazonaws.com:2183
> nifi.zookeeper.connect.timeout=3 secs
> nifi.zookeeper.session.timeout=3 secs
> nifi.zookeeper.root.node=/nifi
>
> # kerberos #
> nifi.kerberos.krb5.file=
>
> # kerberos service principle #
> nifi.kerberos.service.principal=
> nifi.kerberos.service.keytab.location=
>
> # kerberos spnego principle #
> nifi.kerberos.spnego.principal=
> nifi.kerberos.spnego.keytab.location=
> nifi.kerberos.spnego.authentication.expiration=12 hours
>
> # external properties files for variable registry
> # supports a comma delimited list of file locations
> nifi.variable.registry.properties=
>
> I think I have everything set correctly but I have not been able to start an
> instances up.
>
> Thanks,
>
> Scott
>
> On Mar 19, 2018, at 4:35 PM, Bryan Bende <bb...@gmail.com> wrote:
>
> The base file is here for comparison:
>
> https://github.com/apache/nifi-registry/blob/master/nifi-registry-resources/src/main/resources/conf/identity-providers.xml#L23
>
> On Mon, Mar 19, 2018 at 5:34 PM, Bryan Bende <bb...@gmail.com> wrote:
>
> For your first file, is what you showed there actually wrapped in
> <identityProviders> </identityProviders> or is it exactly what you
> showed?
>
> It may just be that you only copied/pasted the one provider, but the
> root element is not <provider>, so as it is shown there it would not
> parse.
>
> On Mon, Mar 19, 2018 at 2:54 PM, Scott Howell <sc...@mobilgov.com>
> wrote:
>
> Here is my file
>
> <provider>
> <identifier>ldap-identity-provider</identifier>
> <class>org.apache.nifi.registry.security.ldap.LdapProvider</class>
> <property name="Authentication Strategy">SIMPLE</property>
>
> <property name="Manager DN">cn=Manager,dc=mobilgov,dc=com</property>
> <property name="Manager Password”>redacted</property>
>
>
> <property name="Referral Strategy">FOLLOW</property>
> <property name="Connect Timeout">10 secs</property>
> <property name="Read Timeout">10 secs</property>
>
> <property name="Url”>redacted</property>
> <property name="User Search
> Base">ou=users,dc=mobilgov,dc=com</property>
> <property name="User Search Filter">uid={0}</property>
>
> <property name="Identity Strategy">USE_DN</property>
> <property name="Authentication Expiration">12 hours</property>
> </provider>
>
> Here is my authorizers.xml
>
> <authorizers>
>
> <userGroupProvider>
> <identifier>file-user-group-provider</identifier>
>
> <class>org.apache.nifi.registry.security.authorization.file.FileUserGroupProvider</class>
> <property name="Users File">conf/users.xml</property>
> <property name="Legacy Authorized Users File"></property>
> <property name="Initial User Identity 1”>redacted</property>
> </userGroupProvider>
>
> <accessPolicyProvider>
> <identifier>file-access-policy-provider</identifier>
>
> <class>org.apache.nifi.registry.security.authorization.file.FileAccessPolicyProvider</class>
> <property name="User Group
> Provider">file-user-group-provider</property>
> <property name="Authorizations
> File">conf/authorizations.xml</property>
> <property name="Initial Admin Identity”>redacted</property>
> <property name="NiFi Identity 1"></property>
> </accessPolicyProvider>
>
> <authorizer>
> <identifier>managed-authorizer</identifier>
>
> <class>org.apache.nifi.registry.security.authorization.StandardManagedAuthorizer</class>
> <property name="Access Policy
> Provider">file-access-policy-provider</property>
> </authorizer>
> </authorizers>
>
> On Mar 19, 2018, at 12:59 PM, Bryan Bende <bb...@gmail.com> wrote:
>
> It looks like that error would happen if your identity-providers.xml
> contained invalid XML.
>
> Did you start by modifying the identity-providers.xml file that was
> already there? Can you share the file, or the contents (removing
> anything sensitive)?
>
> On Mon, Mar 19, 2018 at 1:09 PM, Scott Howell <sc...@mobilgov.com>
> wrote:
>
> So I was able to get the UI pulled up but now I am hitting a roadblock with
> my identity-provider.xml.
>
> I am getting a number of errors like this:
>
> Caused by: org.springframework.beans.factory.BeanCreationException: Error
> creating bean with name 'getIdentityProvider' defined in class path resource
> [org/apache/nifi/registry/security/authentication/IdentityProviderFactory.class]:
> Bean instantiation via factory method failed; nested exception is
> org.springframework.beans.BeanInstantiationException: Failed to instantiate
> [org.apache.nifi.registry.security.authentication.IdentityProvider]: Factory
> method 'getIdentityProvider' threw exception; nested exception is
> java.lang.Exception: Unable to load the login identity provider
> configuration file at: /opt/nifi-registry-0.1.0/conf/identity-providers.xml
> at
> org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:587)
> ~[na:na]
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1250)
> ~[na:na]
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1099)
> ~[na:na]
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:545)
> ~[na:na]
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:502)
> ~[na:na]
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:312)
> ~[na:na]
> at
> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228)
> ~[na:na]
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:310)
> ~[na:na]
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200)
> ~[na:na]
> at
> org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:251)
> ~[na:na]
> at
> org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1135)
> ~[na:na]
> at
> org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1062)
> ~[na:na]
> at
> org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:815)
> ~[na:na]
> at
> org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:721)
> ~[na:na]
> ... 43 common frames omitted
>
> I know it has to do with the identity-provider.xml but I have my setup just
> like the documentation ask for. I turned on debug but was not able to see
> anything different or better explanation from it.
>
>
> On Mar 19, 2018, at 10:06 AM, Kevin Doran <kd...@apache.org> wrote:
>
> Ok, that use case should be fine.
>
> If it were an authorization issue you would see something in the logs saying
> that an authorization attempt failed and the server is responding with a
> 403. Just to be sure, can you enable debug logging if you haven't already,
> i.e., in your nifi-registry/conf/logback.xml file, change
> 'org.apache.nifif.registry' to debug:
>
> <!-- valid logging levels: TRACE, DEBUG, INFO, WARN, ERROR -->
> <logger name="org.apache.nifi.registry" level="DEBUG"/>
>
> If there is nothing being written to nifi-registry-app.log, it points
> towards a connection issue, so I would double check your host, port, and TLS
> settings. You'll have to get an HTTPS cert from a root CA or configure your
> ELB to trust your company's self-signed cert (again, not sure if/how to do
> this, but I assume there should be some way to configure it. It might
> require settings not exposed in the AWS web console.)
>
> On 3/19/18, 10:51, "Scott Howell" <sc...@mobilgov.com> wrote:
>
> Thanks Kevin,
>
> I am just using the ELB to go from the public subnet to the private subnet.
> I will not have multiple instances running of registry.
>
> I will say on my authorizers.xml there is one difference between my nifi
> instance. On my nifi instance I am using file-provider for
> nifi.security.user.authorizer in my nifi.properties. I don’t think from
> reading the documents for nifi-registry that I can use that. If there is a
> way that might be my problem. I was running into some issues with my nifi
> instance when I was using managed-authorizers instead of file-provider.
>
>
>
> On Mar 19, 2018, at 9:35 AM, Kevin Doran <kd...@apache.org> wrote:
>
> Hey Scott,
>
> Assuming you are using two-way TLS with client certificates for
> authentication, I recommend configuring your ELB for TCP passthrough so that
> the TLS handshake is between the end-client and the NiFi Registry Server (in
> other words, no decryption/termination of the TLS connection happens in the
> ELB). If you are using some other form of authentication (e.g., LDAP), you
> will need to configure your ELB to trust the self-signed key NiFi Registry
> is using. I'm not sure how to do that as I've never run an ELB with that
> configuration before.
>
> Also, just a note about using an ELB with NiFi Registry:
>
> NiFi Registry is currently only supports single-instance use as persisted
> data and in-memory state is not synced between multiple instances. Are you
> hoping to use the ELB for actual load balancing, or is it just to take
> advantage of other ELB features, such as forwarding and security group
> rules? If the plan is to load balance multiple Registry instances, just be
> aware that you will probably run into some unexpected behavior. (As you
> mentioned using authorization, that is one case where I know the in-memory
> cache of the persisted data will not refresh across instances, so even if
> you were using some sort of shared network file system attached to multiple
> Registry instances, such as EFS, it would not work the way you hope.)
>
> Hope this helps,
> Kevin
>
> On 3/19/18, 10:20, "Scott Howell" <sc...@mobilgov.com> wrote:
>
> Thanks for the quick response.
>
> A couple of things I am seeing.
>
> 1. There is no error, I don’t see anything in the logs once the service
> comes up. This is because the health check is not even hitting the instance
> when secure.
>
> 2. Nothing interesting in the nifi-registry-app.logs. That was my concern
> because on my nifi instance I can see the health check hitting the instance
> from the ELB. This does not happen on the nifi-registry instance. I see the
> service startup and it tells me what domain and port I can access the UI but
> nothing else after that.
>
> 3. When I am on an instances in the same private subnet I am able to curl to
> the instance I get the TLS SSL which tells me the keystore is on the server.
> I am using a JKS keystore that is self-signed by the company I work for.
>
> On Mar 19, 2018, at 9:10 AM, Bryan Bende <bb...@gmail.com> wrote:
>
> Hello,
>
> What error are you getting when you cannot access the UI?
>
> Is there anything interesting in nifi-registry-app.log regarding
> authentication/authorization when this happens?
>
> Can you access the UI securely without going through the ELB?
>
> Thanks,
>
> Bryan
>
>
> On Mon, Mar 19, 2018 at 10:05 AM, Scott Howell <sc...@mobilgov.com>
> wrote:
>
> I was able to stand up nifi-registry behind an AWS ELB non-secure.
> Everything was working great and was able to access the UI anonymously. I
> set up the authorization just like on my nifi instances along with the
> authorizers and identity-provider. The service comes up without errors and
> everything looks good but the health check does not pass and I cannot access
> the UI to login. I was wondering if anyone else has ran into this issue
> using nifi-registry.
>
>
>
>
>
>
>
>
>
>
>
>
>
>
Re: Issue with AWS ELB on secure nifi-registry
Posted by Scott Howell <sc...@mobilgov.com>.
Thanks for all of the help with this. I have the cluster up and running. The logs look great everything seems to be working but I cannot login into the UI. I am using a wildcard self-signed certificate. The /proxy is in authorizations.xml with the correct users for the nodes.
The error I see with the UI :
is Untrusted proxy CN=*.{redacted}.com, OU={redacted}, O={redacted}, L=Kansas City, ST=Missouri, C=US
I haven’t had much luck finding a lot of documentation or forum questions with this kind of issue.
My authorizers.xml looks like this
<authorizers>
<authorizer>
<identifier>file-provider</identifier>
<class>org.apache.nifi.authorization.FileAuthorizer</class>
<property name="Authorizations File">/opt/config/authorizations.xml</property>
<property name="Users File">/opt/config/users.xml</property>
<property name="Initial Admin Identity">uid=scott,ou=users,dc={redacted},dc=com</property>
<property name="Legacy Authorized Users File"></property>
<property name="Node Identity 1">CN=node-1-nifi-dev.{redacted}.com,OU={redacted},O={redacted},L=Kansas City,ST=Missouri,C=US</property>
<property name="Node Identity 2">CN=node-2-nifi-dev.{redacted}.com,OU={redacted},O={redacted},L=Kansas City,ST=Missouri,C=US</property>
</authorizer>
</authorizers>
Thanks,
Scott
> On Mar 20, 2018, at 1:15 PM, Andy LoPresto <al...@apache.org> wrote:
>
> Scott,
>
> The original exception is "nested exception is java.security.KeyStoreException: not found”. Can you verify that the keystore you’ve provided is valid using the “keytool” command? In addition, you will need a truststore as well. Try following Pierre's [1] or Bryan’s [2] instructions for setting up a secure cluster.
>
> [1] https://pierrevillard.com/2016/11/29/apache-nifi-1-1-0-secured-cluster-setup/ <https://pierrevillard.com/2016/11/29/apache-nifi-1-1-0-secured-cluster-setup/>
> [2] https://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-authorization-and-multi-tenancy <https://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-authorization-and-multi-tenancy>
>
>
> Andy LoPresto
> alopresto@apache.org <ma...@apache.org>
> alopresto.apache@gmail.com <ma...@gmail.com>
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
>> On Mar 20, 2018, at 11:05 AM, Scott Howell <scotthowell@mobilgov.com <ma...@mobilgov.com>> wrote:
>>
>> Thanks for all of the help yesterday I was able to get a secure nifi and nifi-registry up and communicating. I am now trying to figure out how to create a secure cluster. I am currently getting this error when I start up nifi.
>>
>> tion; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'clusterCoordinationProtocolSenderListener' defined in class path resource [nifi-cluster-protocol-context.xml]: Cannot resolve reference to bean 'clusterCoordinationProtocolSender' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'clusterCoordinationProtocolSender' defined in class path resource [nifi-cluster-protocol-context.xml]: Cannot resolve reference to bean 'protocolSocketConfiguration' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'protocolSocketConfiguration': FactoryBean threw exception on object creation; nested exception is java.security.KeyStoreException: not found
>> at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175)
>> at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
>> at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1634)
>> at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317)
>> at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
>> at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
>> ... 50 common frames omitted
>> Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'clusterCoordinationProtocolSenderListener' defined in class path resource [nifi-cluster-protocol-context.xml]: Cannot resolve reference to bean 'clusterCoordinationProtocolSender' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'clusterCoordinationProtocolSender' defined in class path resource [nifi-cluster-protocol-context.xml]: Cannot resolve reference to bean 'protocolSocketConfiguration' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'protocolSocketConfiguration': FactoryBean threw exception on object creation; nested exception is java.security.KeyStoreException: not found
>> at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:359)
>> at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)
>> at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:648)
>> at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:145)
>> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1193)
>> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1095)
>> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
>> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
>> at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
>> at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>> at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
>> at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
>> at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1084)
>> at org.apache.nifi.cluster.spring.NodeClusterCoordinatorFactoryBean.getObject(NodeClusterCoordinatorFactoryBean.java:44)
>> at org.apache.nifi.cluster.spring.NodeClusterCoordinatorFactoryBean.getObject(NodeClusterCoordinatorFactoryBean.java:34)
>> at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
>> ... 55 common frames omitted
>> Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'clusterCoordinationProtocolSender' defined in class path resource [nifi-cluster-protocol-context.xml]: Cannot resolve reference to bean 'protocolSocketConfiguration' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'protocolSocketConfiguration': FactoryBean threw exception on object creation; nested exception is java.security.KeyStoreException: not found
>> at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:359)
>> at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)
>> at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:648)
>> at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:145)
>> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1193)
>> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1095)
>> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
>> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
>> at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
>> at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>> at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
>> at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
>> at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
>> ... 70 common frames omitted
>> Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'protocolSocketConfiguration': FactoryBean threw exception on object creation; nested exception is java.security.KeyStoreException: not found
>> at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175)
>> at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
>> at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1634)
>> at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317)
>> at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
>> at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
>> ... 82 common frames omitted
>> Caused by: java.security.KeyStoreException: not found
>> at java.security.KeyStore.getInstance(KeyStore.java:851)
>> at org.apache.nifi.security.util.KeyStoreUtils.getKeyStore(KeyStoreUtils.java:66)
>> at org.apache.nifi.security.util.KeyStoreUtils.getTrustStore(KeyStoreUtils.java:80)
>> at org.apache.nifi.io.socket.SSLContextFactory.<init>(SSLContextFactory.java:73)
>> at org.apache.nifi.cluster.protocol.spring.SocketConfigurationFactoryBean.getObject(SocketConfigurationFactoryBean.java:45)
>> at org.apache.nifi.cluster.protocol.spring.SocketConfigurationFactoryBean.getObject(SocketConfigurationFactoryBean.java:30)
>> at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
>> ... 87 common frames omitted
>> Caused by: java.security.NoSuchAlgorithmException: KeyStore not available
>> at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
>> at java.security.Security.getImpl(Security.java:695)
>> at java.security.KeyStore.getInstance(KeyStore.java:848)
>> ... 93 common frames omitted
>>
>> My nifi.properties file is.
>>
>> # Licensed to the Apache Software Foundation (ASF) under one or more
>> # contributor license agreements. See the NOTICE file distributed with
>> # this work for additional information regarding copyright ownership.
>> # The ASF licenses this file to You under the Apache License, Version 2.0
>> # (the "License"); you may not use this file except in compliance with
>> # the License. You may obtain a copy of the License at
>> #
>> # http://www.apache.org/licenses/LICENSE-2.0 <http://www.apache.org/licenses/LICENSE-2.0>
>> #
>> # Unless required by applicable law or agreed to in writing, software
>> # distributed under the License is distributed on an "AS IS" BASIS,
>> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>> # See the License for the specific language governing permissions and
>> # limitations under the License.
>>
>> # Core Properties #
>> nifi.version={{nifi_version}}
>> nifi.flow.configuration.file=/opt/config/flow.xml.gz
>> nifi.flow.configuration.archive.enabled=true
>> nifi.flow.configuration.archive.dir=/opt/config/archive/
>> nifi.flow.configuration.archive.max.time=30 days
>> nifi.flow.configuration.archive.max.storage=500 MB
>> nifi.flowcontroller.autoResumeState=true
>> nifi.flowcontroller.graceful.shutdown.period=10 sec
>> nifi.flowservice.writedelay.interval=500 ms
>> nifi.administrative.yield.duration=30 sec
>> # If a component has no work to do (is "bored"), how long should we wait before checking again for work?
>> nifi.bored.yield.duration=10 millis
>>
>>
>> nifi.authorizer.configuration.file=/opt/config/authorizers.xml
>> nifi.login.identity.provider.configuration.file=/opt/config/login-identity-providers.xml
>> nifi.templates.directory=/opt/config/templates
>> nifi.ui.banner.text=
>> nifi.ui.autorefresh.interval=30 sec
>> nifi.nar.library.directory=/opt/nifi/lib
>> nifi.nar.library.directory.custom=/opt/config/processors
>> nifi.nar.working.directory=/opt/nifi/work/nar/
>> nifi.documentation.working.directory=./work/docs/components
>>
>> ####################
>> # State Management #
>> ####################
>> nifi.state.management.configuration.file=/opt/config/state-management.xml
>> # The ID of the local state provider
>> nifi.state.management.provider.local=local-provider
>> # The ID of the cluster-wide state provider. This will be ignored if NiFi is not clustered but must be populated if running in a cluster.
>> nifi.state.management.provider.cluster=zk-provider
>> # Specifies whether or not this instance of NiFi should run an embedded ZooKeeper server
>> nifi.state.management.embedded.zookeeper.start=false
>> # Properties file that provides the ZooKeeper properties to use if <nifi.state.management.embedded.zookeeper.start> is set to true
>> nifi.state.management.embedded.zookeeper.properties=/opt/config/zookeeper.properties
>>
>>
>> # H2 Settings
>> nifi.database.directory=/opt/database_repository
>> nifi.h2.url.append=;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE
>>
>> # FlowFile Repository
>> nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository
>> nifi.flowfile.repository.directory=/opt/flowfile_repository
>> nifi.flowfile.repository.partitions=256
>> nifi.flowfile.repository.checkpoint.interval=2 mins
>> nifi.flowfile.repository.always.sync=false
>>
>> nifi.swap.manager.implementation=org.apache.nifi.controller.FileSystemSwapManager
>> nifi.queue.swap.threshold=20000
>> nifi.swap.in.period=5 sec
>> nifi.swap.in.threads=1
>> nifi.swap.out.period=5 sec
>> nifi.swap.out.threads=4
>>
>> # Content Repository
>> nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository
>> nifi.content.claim.max.appendable.size=10 MB
>> nifi.content.claim.max.flow.files=100
>> nifi.content.repository.directory.default=/opt/content_repository
>> nifi.content.repository.archive.max.retention.period=12 hours
>> nifi.content.repository.archive.max.usage.percentage=50%
>> nifi.content.repository.archive.enabled=true
>> nifi.content.repository.always.sync=false
>> nifi.content.viewer.url=/nifi-content-viewer/
>>
>> # Provenance Repository Properties
>> nifi.provenance.repository.implementation=org.apache.nifi.provenance.PersistentProvenanceRepository
>>
>> # Persistent Provenance Repository Properties
>> nifi.provenance.repository.directory.default=/opt/provenance_repository
>> nifi.provenance.repository.max.storage.time=24 hours
>> nifi.provenance.repository.max.storage.size=1 GB
>> nifi.provenance.repository.rollover.time=30 secs
>> nifi.provenance.repository.rollover.size=100 MB
>> nifi.provenance.repository.query.threads=2
>> nifi.provenance.repository.index.threads=1
>> nifi.provenance.repository.compress.on.rollover=true
>> nifi.provenance.repository.always.sync=false
>> nifi.provenance.repository.journal.count=16
>> # Comma-separated list of fields. Fields that are not indexed will not be searchable. Valid fields are:
>> # EventType, FlowFileUUID, Filename, TransitURI, ProcessorID, AlternateIdentifierURI, Relationship, Details
>> nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename, ProcessorID, Relationship
>> # FlowFile Attributes that should be indexed and made searchable. Some examples to consider are filename, uuid, mime.type
>> nifi.provenance.repository.indexed.attributes=
>> # Large values for the shard size will result in more Java heap usage when searching the Provenance Repository
>> # but should provide better performance
>> nifi.provenance.repository.index.shard.size=500 MB
>> # Indicates the maximum length that a FlowFile attribute can be when retrieving a Provenance Event from
>> # the repository. If the length of any attribute exceeds this value, it will be truncated when the event is retrieved.
>> nifi.provenance.repository.max.attribute.length=65536
>>
>> # Volatile Provenance Respository Properties
>> nifi.provenance.repository.buffer.size=100000
>>
>> # Component Status Repository
>> nifi.components.status.repository.implementation=org.apache.nifi.controller.status.history.VolatileComponentStatusRepository
>> nifi.components.status.repository.buffer.size=1440
>> nifi.components.status.snapshot.frequency=1 min
>>
>> # Site to Site properties
>> nifi.remote.input.host=
>> nifi.remote.input.secure=false
>> nifi.remote.input.socket.port=9998
>> nifi.remote.input.http.enabled=false
>> nifi.remote.input.http.transaction.ttl=30 sec
>>
>> # web properties #
>> nifi.web.war.directory=/opt/nifi/lib
>> nifi.web.http.host=
>> nifi.web.http.port=
>> nifi.web.https.host={{redacted}}
>> nifi.web.https.port=8443
>> nifi.web.jetty.working.directory=/opt/nifi/work/jetty
>> nifi.web.jetty.threads=200
>>
>> # security properties #
>> nifi.sensitive.props.key=x0KDgO9L8lAhFGLdvu2VEjFVGc6Kg3V0R5I4bYwoqdgL47moo0wApKQtAVu1BvD
>> nifi.sensitive.props.key.protected=
>> nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
>> nifi.sensitive.props.provider=BC
>> nifi.sensitive.props.additional.keys=
>>
>> nifi.security.keystore=/opt/certs/payit_keystore
>> nifi.security.keystoreType=JKS
>> nifi.security.keystorePasswd={{keystore_password}}
>> nifi.security.keyPasswd=
>> nifi.security.truststore=
>> nifi.security.truststoreType=
>> nifi.security.truststorePasswd=
>> nifi.security.needClientAuth=false
>> nifi.security.user.authorizer=file-provider
>> nifi.security.user.login.identity.provider=ldap-provider
>> nifi.security.ocsp.responder.url=
>> nifi.security.ocsp.responder.certificate=
>>
>> # Identity Mapping Properties #
>> # These properties allow normalizing user identities such that identities coming from different identity providers
>> # (certificates, LDAP, Kerberos) can be treated the same internally in NiFi. The following example demonstrates normalizing
>> # DNs from certificates and principals from Kerberos into a common identity string:
>> #
>> #nifi.security.identity.mapping.pattern.dn=^cn=(.*?),ou=(.*?),dc=(.*?),dc=(.*?),dc=(.*?)$
>> #nifi.security.identity.mapping.value.dn=$1
>> # nifi.security.identity.mapping.pattern.kerb=^(.*?)/instance@(.*?)$
>> # nifi.security.identity.mapping.value.kerb=$1@$2
>>
>> # cluster common properties (all nodes must have same values) #
>> nifi.cluster.protocol.heartbeat.interval=5 sec
>> nifi.cluster.protocol.is.secure=true
>>
>> # cluster node properties (only configure for cluster nodes) #
>> nifi.cluster.is.node=true
>> nifi.cluster.node.address=nifi-dev.mobilgov.com <http://nifi-dev.mobilgov.com/>
>> nifi.cluster.node.protocol.port=9999
>> nifi.cluster.node.protocol.threads=10
>> nifi.cluster.node.event.history.size=25
>> nifi.cluster.node.connection.timeout=5 sec
>> nifi.cluster.node.read.timeout=5 sec
>> nifi.cluster.firewall.file=
>>
>>
>> # zookeeper properties, used for cluster management #
>> nifi.zookeeper.connect.string=internal-etcd-dev-EtcdLoadB-3RWA2WEFBBT8-2068560477.us-east-2.elb.amazonaws.com <http://internal-etcd-dev-etcdloadb-3rwa2wefbbt8-2068560477.us-east-2.elb.amazonaws.com/>:2181,internal-etcd-dev-EtcdLoadB-3RWA2WEFBBT8-2068560477.us <http://internal-etcd-dev-etcdloadb-3rwa2wefbbt8-2068560477.us/>-east-2.elb.amazonaws.com2182,internal-etcd-dev-EtcdLoadB-3RWA2WEFBBT8-2068560477.us-east-2.elb.amazonaws.com <http://internal-etcd-dev-etcdloadb-3rwa2wefbbt8-2068560477.us-east-2.elb.amazonaws.com/>:2183
>> nifi.zookeeper.connect.timeout=3 secs
>> nifi.zookeeper.session.timeout=3 secs
>> nifi.zookeeper.root.node=/nifi
>>
>> # kerberos #
>> nifi.kerberos.krb5.file=
>>
>> # kerberos service principle #
>> nifi.kerberos.service.principal=
>> nifi.kerberos.service.keytab.location=
>>
>> # kerberos spnego principle #
>> nifi.kerberos.spnego.principal=
>> nifi.kerberos.spnego.keytab.location=
>> nifi.kerberos.spnego.authentication.expiration=12 hours
>>
>> # external properties files for variable registry
>> # supports a comma delimited list of file locations
>> nifi.variable.registry.properties=
>>
>> I think I have everything set correctly but I have not been able to start an instances up.
>>
>> Thanks,
>>
>> Scott
>>
>>> On Mar 19, 2018, at 4:35 PM, Bryan Bende <bbende@gmail.com <ma...@gmail.com>> wrote:
>>>
>>> The base file is here for comparison:
>>>
>>> https://github.com/apache/nifi-registry/blob/master/nifi-registry-resources/src/main/resources/conf/identity-providers.xml#L23 <https://github.com/apache/nifi-registry/blob/master/nifi-registry-resources/src/main/resources/conf/identity-providers.xml#L23>
>>>
>>> On Mon, Mar 19, 2018 at 5:34 PM, Bryan Bende <bb...@gmail.com> wrote:
>>>> For your first file, is what you showed there actually wrapped in
>>>> <identityProviders> </identityProviders> or is it exactly what you
>>>> showed?
>>>>
>>>> It may just be that you only copied/pasted the one provider, but the
>>>> root element is not <provider>, so as it is shown there it would not
>>>> parse.
>>>>
>>>> On Mon, Mar 19, 2018 at 2:54 PM, Scott Howell <sc...@mobilgov.com> wrote:
>>>>> Here is my file
>>>>>
>>>>> <provider>
>>>>> <identifier>ldap-identity-provider</identifier>
>>>>> <class>org.apache.nifi.registry.security.ldap.LdapProvider</class>
>>>>> <property name="Authentication Strategy">SIMPLE</property>
>>>>>
>>>>> <property name="Manager DN">cn=Manager,dc=mobilgov,dc=com</property>
>>>>> <property name="Manager Password”>redacted</property>
>>>>>
>>>>>
>>>>> <property name="Referral Strategy">FOLLOW</property>
>>>>> <property name="Connect Timeout">10 secs</property>
>>>>> <property name="Read Timeout">10 secs</property>
>>>>>
>>>>> <property name="Url”>redacted</property>
>>>>> <property name="User Search Base">ou=users,dc=mobilgov,dc=com</property>
>>>>> <property name="User Search Filter">uid={0}</property>
>>>>>
>>>>> <property name="Identity Strategy">USE_DN</property>
>>>>> <property name="Authentication Expiration">12 hours</property>
>>>>> </provider>
>>>>>
>>>>> Here is my authorizers.xml
>>>>>
>>>>> <authorizers>
>>>>>
>>>>> <userGroupProvider>
>>>>> <identifier>file-user-group-provider</identifier>
>>>>> <class>org.apache.nifi.registry.security.authorization.file.FileUserGroupProvider</class>
>>>>> <property name="Users File">conf/users.xml</property>
>>>>> <property name="Legacy Authorized Users File"></property>
>>>>> <property name="Initial User Identity 1”>redacted</property>
>>>>> </userGroupProvider>
>>>>>
>>>>> <accessPolicyProvider>
>>>>> <identifier>file-access-policy-provider</identifier>
>>>>> <class>org.apache.nifi.registry.security.authorization.file.FileAccessPolicyProvider</class>
>>>>> <property name="User Group Provider">file-user-group-provider</property>
>>>>> <property name="Authorizations File">conf/authorizations.xml</property>
>>>>> <property name="Initial Admin Identity”>redacted</property>
>>>>> <property name="NiFi Identity 1"></property>
>>>>> </accessPolicyProvider>
>>>>>
>>>>> <authorizer>
>>>>> <identifier>managed-authorizer</identifier>
>>>>> <class>org.apache.nifi.registry.security.authorization.StandardManagedAuthorizer</class>
>>>>> <property name="Access Policy Provider">file-access-policy-provider</property>
>>>>> </authorizer>
>>>>> </authorizers>
>>>>>
>>>>>> On Mar 19, 2018, at 12:59 PM, Bryan Bende <bb...@gmail.com> wrote:
>>>>>>
>>>>>> It looks like that error would happen if your identity-providers.xml
>>>>>> contained invalid XML.
>>>>>>
>>>>>> Did you start by modifying the identity-providers.xml file that was
>>>>>> already there? Can you share the file, or the contents (removing
>>>>>> anything sensitive)?
>>>>>>
>>>>>> On Mon, Mar 19, 2018 at 1:09 PM, Scott Howell <sc...@mobilgov.com> wrote:
>>>>>>> So I was able to get the UI pulled up but now I am hitting a roadblock with my identity-provider.xml.
>>>>>>>
>>>>>>> I am getting a number of errors like this:
>>>>>>>
>>>>>>> Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'getIdentityProvider' defined in class path resource [org/apache/nifi/registry/security/authentication/IdentityProviderFactory.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.nifi.registry.security.authentication.IdentityProvider]: Factory method 'getIdentityProvider' threw exception; nested exception is java.lang.Exception: Unable to load the login identity provider configuration file at: /opt/nifi-registry-0.1.0/conf/identity-providers.xml
>>>>>>> at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:587) ~[na:na]
>>>>>>> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1250) ~[na:na]
>>>>>>> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1099) ~[na:na]
>>>>>>> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:545) ~[na:na]
>>>>>>> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:502) ~[na:na]
>>>>>>> at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:312) ~[na:na]
>>>>>>> at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228) ~[na:na]
>>>>>>> at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:310) ~[na:na]
>>>>>>> at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200) ~[na:na]
>>>>>>> at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:251) ~[na:na]
>>>>>>> at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1135) ~[na:na]
>>>>>>> at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1062) ~[na:na]
>>>>>>> at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:815) ~[na:na]
>>>>>>> at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:721) ~[na:na]
>>>>>>> ... 43 common frames omitted
>>>>>>>
>>>>>>> I know it has to do with the identity-provider.xml but I have my setup just like the documentation ask for. I turned on debug but was not able to see anything different or better explanation from it.
>>>>>>>
>>>>>>>
>>>>>>>> On Mar 19, 2018, at 10:06 AM, Kevin Doran <kd...@apache.org> wrote:
>>>>>>>>
>>>>>>>> Ok, that use case should be fine.
>>>>>>>>
>>>>>>>> If it were an authorization issue you would see something in the logs saying that an authorization attempt failed and the server is responding with a 403. Just to be sure, can you enable debug logging if you haven't already, i.e., in your nifi-registry/conf/logback.xml file, change 'org.apache.nifif.registry' to debug:
>>>>>>>>
>>>>>>>> <!-- valid logging levels: TRACE, DEBUG, INFO, WARN, ERROR -->
>>>>>>>> <logger name="org.apache.nifi.registry" level="DEBUG"/>
>>>>>>>>
>>>>>>>> If there is nothing being written to nifi-registry-app.log, it points towards a connection issue, so I would double check your host, port, and TLS settings. You'll have to get an HTTPS cert from a root CA or configure your ELB to trust your company's self-signed cert (again, not sure if/how to do this, but I assume there should be some way to configure it. It might require settings not exposed in the AWS web console.)
>>>>>>>>
>>>>>>>> On 3/19/18, 10:51, "Scott Howell" <sc...@mobilgov.com> wrote:
>>>>>>>>
>>>>>>>> Thanks Kevin,
>>>>>>>>
>>>>>>>> I am just using the ELB to go from the public subnet to the private subnet. I will not have multiple instances running of registry.
>>>>>>>>
>>>>>>>> I will say on my authorizers.xml there is one difference between my nifi instance. On my nifi instance I am using file-provider for nifi.security.user.authorizer in my nifi.properties. I don’t think from reading the documents for nifi-registry that I can use that. If there is a way that might be my problem. I was running into some issues with my nifi instance when I was using managed-authorizers instead of file-provider.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> On Mar 19, 2018, at 9:35 AM, Kevin Doran <kd...@apache.org> wrote:
>>>>>>>>>
>>>>>>>>> Hey Scott,
>>>>>>>>>
>>>>>>>>> Assuming you are using two-way TLS with client certificates for authentication, I recommend configuring your ELB for TCP passthrough so that the TLS handshake is between the end-client and the NiFi Registry Server (in other words, no decryption/termination of the TLS connection happens in the ELB). If you are using some other form of authentication (e.g., LDAP), you will need to configure your ELB to trust the self-signed key NiFi Registry is using. I'm not sure how to do that as I've never run an ELB with that configuration before.
>>>>>>>>>
>>>>>>>>> Also, just a note about using an ELB with NiFi Registry:
>>>>>>>>>
>>>>>>>>> NiFi Registry is currently only supports single-instance use as persisted data and in-memory state is not synced between multiple instances. Are you hoping to use the ELB for actual load balancing, or is it just to take advantage of other ELB features, such as forwarding and security group rules? If the plan is to load balance multiple Registry instances, just be aware that you will probably run into some unexpected behavior. (As you mentioned using authorization, that is one case where I know the in-memory cache of the persisted data will not refresh across instances, so even if you were using some sort of shared network file system attached to multiple Registry instances, such as EFS, it would not work the way you hope.)
>>>>>>>>>
>>>>>>>>> Hope this helps,
>>>>>>>>> Kevin
>>>>>>>>>
>>>>>>>>> On 3/19/18, 10:20, "Scott Howell" <sc...@mobilgov.com> wrote:
>>>>>>>>>
>>>>>>>>> Thanks for the quick response.
>>>>>>>>>
>>>>>>>>> A couple of things I am seeing.
>>>>>>>>>
>>>>>>>>> 1. There is no error, I don’t see anything in the logs once the service comes up. This is because the health check is not even hitting the instance when secure.
>>>>>>>>>
>>>>>>>>> 2. Nothing interesting in the nifi-registry-app.logs. That was my concern because on my nifi instance I can see the health check hitting the instance from the ELB. This does not happen on the nifi-registry instance. I see the service startup and it tells me what domain and port I can access the UI but nothing else after that.
>>>>>>>>>
>>>>>>>>> 3. When I am on an instances in the same private subnet I am able to curl to the instance I get the TLS SSL which tells me the keystore is on the server. I am using a JKS keystore that is self-signed by the company I work for.
>>>>>>>>>
>>>>>>>>>> On Mar 19, 2018, at 9:10 AM, Bryan Bende <bb...@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>> What error are you getting when you cannot access the UI?
>>>>>>>>>>
>>>>>>>>>> Is there anything interesting in nifi-registry-app.log regarding
>>>>>>>>>> authentication/authorization when this happens?
>>>>>>>>>>
>>>>>>>>>> Can you access the UI securely without going through the ELB?
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>>
>>>>>>>>>> Bryan
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Mar 19, 2018 at 10:05 AM, Scott Howell <sc...@mobilgov.com> wrote:
>>>>>>>>>>> I was able to stand up nifi-registry behind an AWS ELB non-secure. Everything was working great and was able to access the UI anonymously. I set up the authorization just like on my nifi instances along with the authorizers and identity-provider. The service comes up without errors and everything looks good but the health check does not pass and I cannot access the UI to login. I was wondering if anyone else has ran into this issue using nifi-registry.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>
>
Re: Issue with AWS ELB on secure nifi-registry
Posted by Andy LoPresto <al...@apache.org>.
Scott,
The original exception is "nested exception is java.security.KeyStoreException: not found”. Can you verify that the keystore you’ve provided is valid using the “keytool” command? In addition, you will need a truststore as well. Try following Pierre's [1] or Bryan’s [2] instructions for setting up a secure cluster.
[1] https://pierrevillard.com/2016/11/29/apache-nifi-1-1-0-secured-cluster-setup/
[2] https://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-authorization-and-multi-tenancy <https://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-authorization-and-multi-tenancy>
Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Mar 20, 2018, at 11:05 AM, Scott Howell <sc...@mobilgov.com> wrote:
>
> Thanks for all of the help yesterday I was able to get a secure nifi and nifi-registry up and communicating. I am now trying to figure out how to create a secure cluster. I am currently getting this error when I start up nifi.
>
> tion; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'clusterCoordinationProtocolSenderListener' defined in class path resource [nifi-cluster-protocol-context.xml]: Cannot resolve reference to bean 'clusterCoordinationProtocolSender' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'clusterCoordinationProtocolSender' defined in class path resource [nifi-cluster-protocol-context.xml]: Cannot resolve reference to bean 'protocolSocketConfiguration' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'protocolSocketConfiguration': FactoryBean threw exception on object creation; nested exception is java.security.KeyStoreException: not found
> at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175)
> at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
> at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1634)
> at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317)
> at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
> at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
> ... 50 common frames omitted
> Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'clusterCoordinationProtocolSenderListener' defined in class path resource [nifi-cluster-protocol-context.xml]: Cannot resolve reference to bean 'clusterCoordinationProtocolSender' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'clusterCoordinationProtocolSender' defined in class path resource [nifi-cluster-protocol-context.xml]: Cannot resolve reference to bean 'protocolSocketConfiguration' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'protocolSocketConfiguration': FactoryBean threw exception on object creation; nested exception is java.security.KeyStoreException: not found
> at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:359)
> at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)
> at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:648)
> at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:145)
> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1193)
> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1095)
> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
> at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
> at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
> at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
> at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
> at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1084)
> at org.apache.nifi.cluster.spring.NodeClusterCoordinatorFactoryBean.getObject(NodeClusterCoordinatorFactoryBean.java:44)
> at org.apache.nifi.cluster.spring.NodeClusterCoordinatorFactoryBean.getObject(NodeClusterCoordinatorFactoryBean.java:34)
> at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
> ... 55 common frames omitted
> Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'clusterCoordinationProtocolSender' defined in class path resource [nifi-cluster-protocol-context.xml]: Cannot resolve reference to bean 'protocolSocketConfiguration' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'protocolSocketConfiguration': FactoryBean threw exception on object creation; nested exception is java.security.KeyStoreException: not found
> at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:359)
> at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)
> at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:648)
> at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:145)
> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1193)
> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1095)
> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
> at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
> at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
> at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
> at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
> at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
> ... 70 common frames omitted
> Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'protocolSocketConfiguration': FactoryBean threw exception on object creation; nested exception is java.security.KeyStoreException: not found
> at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175)
> at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
> at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1634)
> at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317)
> at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
> at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
> ... 82 common frames omitted
> Caused by: java.security.KeyStoreException: not found
> at java.security.KeyStore.getInstance(KeyStore.java:851)
> at org.apache.nifi.security.util.KeyStoreUtils.getKeyStore(KeyStoreUtils.java:66)
> at org.apache.nifi.security.util.KeyStoreUtils.getTrustStore(KeyStoreUtils.java:80)
> at org.apache.nifi.io.socket.SSLContextFactory.<init>(SSLContextFactory.java:73)
> at org.apache.nifi.cluster.protocol.spring.SocketConfigurationFactoryBean.getObject(SocketConfigurationFactoryBean.java:45)
> at org.apache.nifi.cluster.protocol.spring.SocketConfigurationFactoryBean.getObject(SocketConfigurationFactoryBean.java:30)
> at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
> ... 87 common frames omitted
> Caused by: java.security.NoSuchAlgorithmException: KeyStore not available
> at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
> at java.security.Security.getImpl(Security.java:695)
> at java.security.KeyStore.getInstance(KeyStore.java:848)
> ... 93 common frames omitted
>
> My nifi.properties file is.
>
> # Licensed to the Apache Software Foundation (ASF) under one or more
> # contributor license agreements. See the NOTICE file distributed with
> # this work for additional information regarding copyright ownership.
> # The ASF licenses this file to You under the Apache License, Version 2.0
> # (the "License"); you may not use this file except in compliance with
> # the License. You may obtain a copy of the License at
> #
> # http://www.apache.org/licenses/LICENSE-2.0
> #
> # Unless required by applicable law or agreed to in writing, software
> # distributed under the License is distributed on an "AS IS" BASIS,
> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> # See the License for the specific language governing permissions and
> # limitations under the License.
>
> # Core Properties #
> nifi.version={{nifi_version}}
> nifi.flow.configuration.file=/opt/config/flow.xml.gz
> nifi.flow.configuration.archive.enabled=true
> nifi.flow.configuration.archive.dir=/opt/config/archive/
> nifi.flow.configuration.archive.max.time=30 days
> nifi.flow.configuration.archive.max.storage=500 MB
> nifi.flowcontroller.autoResumeState=true
> nifi.flowcontroller.graceful.shutdown.period=10 sec
> nifi.flowservice.writedelay.interval=500 ms
> nifi.administrative.yield.duration=30 sec
> # If a component has no work to do (is "bored"), how long should we wait before checking again for work?
> nifi.bored.yield.duration=10 millis
>
>
> nifi.authorizer.configuration.file=/opt/config/authorizers.xml
> nifi.login.identity.provider.configuration.file=/opt/config/login-identity-providers.xml
> nifi.templates.directory=/opt/config/templates
> nifi.ui.banner.text=
> nifi.ui.autorefresh.interval=30 sec
> nifi.nar.library.directory=/opt/nifi/lib
> nifi.nar.library.directory.custom=/opt/config/processors
> nifi.nar.working.directory=/opt/nifi/work/nar/
> nifi.documentation.working.directory=./work/docs/components
>
> ####################
> # State Management #
> ####################
> nifi.state.management.configuration.file=/opt/config/state-management.xml
> # The ID of the local state provider
> nifi.state.management.provider.local=local-provider
> # The ID of the cluster-wide state provider. This will be ignored if NiFi is not clustered but must be populated if running in a cluster.
> nifi.state.management.provider.cluster=zk-provider
> # Specifies whether or not this instance of NiFi should run an embedded ZooKeeper server
> nifi.state.management.embedded.zookeeper.start=false
> # Properties file that provides the ZooKeeper properties to use if <nifi.state.management.embedded.zookeeper.start> is set to true
> nifi.state.management.embedded.zookeeper.properties=/opt/config/zookeeper.properties
>
>
> # H2 Settings
> nifi.database.directory=/opt/database_repository
> nifi.h2.url.append=;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE
>
> # FlowFile Repository
> nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository
> nifi.flowfile.repository.directory=/opt/flowfile_repository
> nifi.flowfile.repository.partitions=256
> nifi.flowfile.repository.checkpoint.interval=2 mins
> nifi.flowfile.repository.always.sync=false
>
> nifi.swap.manager.implementation=org.apache.nifi.controller.FileSystemSwapManager
> nifi.queue.swap.threshold=20000
> nifi.swap.in.period=5 sec
> nifi.swap.in.threads=1
> nifi.swap.out.period=5 sec
> nifi.swap.out.threads=4
>
> # Content Repository
> nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository
> nifi.content.claim.max.appendable.size=10 MB
> nifi.content.claim.max.flow.files=100
> nifi.content.repository.directory.default=/opt/content_repository
> nifi.content.repository.archive.max.retention.period=12 hours
> nifi.content.repository.archive.max.usage.percentage=50%
> nifi.content.repository.archive.enabled=true
> nifi.content.repository.always.sync=false
> nifi.content.viewer.url=/nifi-content-viewer/
>
> # Provenance Repository Properties
> nifi.provenance.repository.implementation=org.apache.nifi.provenance.PersistentProvenanceRepository
>
> # Persistent Provenance Repository Properties
> nifi.provenance.repository.directory.default=/opt/provenance_repository
> nifi.provenance.repository.max.storage.time=24 hours
> nifi.provenance.repository.max.storage.size=1 GB
> nifi.provenance.repository.rollover.time=30 secs
> nifi.provenance.repository.rollover.size=100 MB
> nifi.provenance.repository.query.threads=2
> nifi.provenance.repository.index.threads=1
> nifi.provenance.repository.compress.on.rollover=true
> nifi.provenance.repository.always.sync=false
> nifi.provenance.repository.journal.count=16
> # Comma-separated list of fields. Fields that are not indexed will not be searchable. Valid fields are:
> # EventType, FlowFileUUID, Filename, TransitURI, ProcessorID, AlternateIdentifierURI, Relationship, Details
> nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename, ProcessorID, Relationship
> # FlowFile Attributes that should be indexed and made searchable. Some examples to consider are filename, uuid, mime.type
> nifi.provenance.repository.indexed.attributes=
> # Large values for the shard size will result in more Java heap usage when searching the Provenance Repository
> # but should provide better performance
> nifi.provenance.repository.index.shard.size=500 MB
> # Indicates the maximum length that a FlowFile attribute can be when retrieving a Provenance Event from
> # the repository. If the length of any attribute exceeds this value, it will be truncated when the event is retrieved.
> nifi.provenance.repository.max.attribute.length=65536
>
> # Volatile Provenance Respository Properties
> nifi.provenance.repository.buffer.size=100000
>
> # Component Status Repository
> nifi.components.status.repository.implementation=org.apache.nifi.controller.status.history.VolatileComponentStatusRepository
> nifi.components.status.repository.buffer.size=1440
> nifi.components.status.snapshot.frequency=1 min
>
> # Site to Site properties
> nifi.remote.input.host=
> nifi.remote.input.secure=false
> nifi.remote.input.socket.port=9998
> nifi.remote.input.http.enabled=false
> nifi.remote.input.http.transaction.ttl=30 sec
>
> # web properties #
> nifi.web.war.directory=/opt/nifi/lib
> nifi.web.http.host=
> nifi.web.http.port=
> nifi.web.https.host={{redacted}}
> nifi.web.https.port=8443
> nifi.web.jetty.working.directory=/opt/nifi/work/jetty
> nifi.web.jetty.threads=200
>
> # security properties #
> nifi.sensitive.props.key=x0KDgO9L8lAhFGLdvu2VEjFVGc6Kg3V0R5I4bYwoqdgL47moo0wApKQtAVu1BvD
> nifi.sensitive.props.key.protected=
> nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
> nifi.sensitive.props.provider=BC
> nifi.sensitive.props.additional.keys=
>
> nifi.security.keystore=/opt/certs/payit_keystore
> nifi.security.keystoreType=JKS
> nifi.security.keystorePasswd={{keystore_password}}
> nifi.security.keyPasswd=
> nifi.security.truststore=
> nifi.security.truststoreType=
> nifi.security.truststorePasswd=
> nifi.security.needClientAuth=false
> nifi.security.user.authorizer=file-provider
> nifi.security.user.login.identity.provider=ldap-provider
> nifi.security.ocsp.responder.url=
> nifi.security.ocsp.responder.certificate=
>
> # Identity Mapping Properties #
> # These properties allow normalizing user identities such that identities coming from different identity providers
> # (certificates, LDAP, Kerberos) can be treated the same internally in NiFi. The following example demonstrates normalizing
> # DNs from certificates and principals from Kerberos into a common identity string:
> #
> #nifi.security.identity.mapping.pattern.dn=^cn=(.*?),ou=(.*?),dc=(.*?),dc=(.*?),dc=(.*?)$
> #nifi.security.identity.mapping.value.dn=$1
> # nifi.security.identity.mapping.pattern.kerb=^(.*?)/instance@(.*?)$
> # nifi.security.identity.mapping.value.kerb=$1@$2
>
> # cluster common properties (all nodes must have same values) #
> nifi.cluster.protocol.heartbeat.interval=5 sec
> nifi.cluster.protocol.is.secure=true
>
> # cluster node properties (only configure for cluster nodes) #
> nifi.cluster.is.node=true
> nifi.cluster.node.address=nifi-dev.mobilgov.com
> nifi.cluster.node.protocol.port=9999
> nifi.cluster.node.protocol.threads=10
> nifi.cluster.node.event.history.size=25
> nifi.cluster.node.connection.timeout=5 sec
> nifi.cluster.node.read.timeout=5 sec
> nifi.cluster.firewall.file=
>
>
> # zookeeper properties, used for cluster management #
> nifi.zookeeper.connect.string=internal-etcd-dev-EtcdLoadB-3RWA2WEFBBT8-2068560477.us-east-2.elb.amazonaws.com:2181,internal-etcd-dev-EtcdLoadB-3RWA2WEFBBT8-2068560477.us-east-2.elb.amazonaws.com2182,internal-etcd-dev-EtcdLoadB-3RWA2WEFBBT8-2068560477.us-east-2.elb.amazonaws.com:2183
> nifi.zookeeper.connect.timeout=3 secs
> nifi.zookeeper.session.timeout=3 secs
> nifi.zookeeper.root.node=/nifi
>
> # kerberos #
> nifi.kerberos.krb5.file=
>
> # kerberos service principle #
> nifi.kerberos.service.principal=
> nifi.kerberos.service.keytab.location=
>
> # kerberos spnego principle #
> nifi.kerberos.spnego.principal=
> nifi.kerberos.spnego.keytab.location=
> nifi.kerberos.spnego.authentication.expiration=12 hours
>
> # external properties files for variable registry
> # supports a comma delimited list of file locations
> nifi.variable.registry.properties=
>
> I think I have everything set correctly but I have not been able to start an instances up.
>
> Thanks,
>
> Scott
>
>> On Mar 19, 2018, at 4:35 PM, Bryan Bende <bb...@gmail.com> wrote:
>>
>> The base file is here for comparison:
>>
>> https://github.com/apache/nifi-registry/blob/master/nifi-registry-resources/src/main/resources/conf/identity-providers.xml#L23
>>
>> On Mon, Mar 19, 2018 at 5:34 PM, Bryan Bende <bb...@gmail.com> wrote:
>>> For your first file, is what you showed there actually wrapped in
>>> <identityProviders> </identityProviders> or is it exactly what you
>>> showed?
>>>
>>> It may just be that you only copied/pasted the one provider, but the
>>> root element is not <provider>, so as it is shown there it would not
>>> parse.
>>>
>>> On Mon, Mar 19, 2018 at 2:54 PM, Scott Howell <sc...@mobilgov.com> wrote:
>>>> Here is my file
>>>>
>>>> <provider>
>>>> <identifier>ldap-identity-provider</identifier>
>>>> <class>org.apache.nifi.registry.security.ldap.LdapProvider</class>
>>>> <property name="Authentication Strategy">SIMPLE</property>
>>>>
>>>> <property name="Manager DN">cn=Manager,dc=mobilgov,dc=com</property>
>>>> <property name="Manager Password”>redacted</property>
>>>>
>>>>
>>>> <property name="Referral Strategy">FOLLOW</property>
>>>> <property name="Connect Timeout">10 secs</property>
>>>> <property name="Read Timeout">10 secs</property>
>>>>
>>>> <property name="Url”>redacted</property>
>>>> <property name="User Search Base">ou=users,dc=mobilgov,dc=com</property>
>>>> <property name="User Search Filter">uid={0}</property>
>>>>
>>>> <property name="Identity Strategy">USE_DN</property>
>>>> <property name="Authentication Expiration">12 hours</property>
>>>> </provider>
>>>>
>>>> Here is my authorizers.xml
>>>>
>>>> <authorizers>
>>>>
>>>> <userGroupProvider>
>>>> <identifier>file-user-group-provider</identifier>
>>>> <class>org.apache.nifi.registry.security.authorization.file.FileUserGroupProvider</class>
>>>> <property name="Users File">conf/users.xml</property>
>>>> <property name="Legacy Authorized Users File"></property>
>>>> <property name="Initial User Identity 1”>redacted</property>
>>>> </userGroupProvider>
>>>>
>>>> <accessPolicyProvider>
>>>> <identifier>file-access-policy-provider</identifier>
>>>> <class>org.apache.nifi.registry.security.authorization.file.FileAccessPolicyProvider</class>
>>>> <property name="User Group Provider">file-user-group-provider</property>
>>>> <property name="Authorizations File">conf/authorizations.xml</property>
>>>> <property name="Initial Admin Identity”>redacted</property>
>>>> <property name="NiFi Identity 1"></property>
>>>> </accessPolicyProvider>
>>>>
>>>> <authorizer>
>>>> <identifier>managed-authorizer</identifier>
>>>> <class>org.apache.nifi.registry.security.authorization.StandardManagedAuthorizer</class>
>>>> <property name="Access Policy Provider">file-access-policy-provider</property>
>>>> </authorizer>
>>>> </authorizers>
>>>>
>>>>> On Mar 19, 2018, at 12:59 PM, Bryan Bende <bb...@gmail.com> wrote:
>>>>>
>>>>> It looks like that error would happen if your identity-providers.xml
>>>>> contained invalid XML.
>>>>>
>>>>> Did you start by modifying the identity-providers.xml file that was
>>>>> already there? Can you share the file, or the contents (removing
>>>>> anything sensitive)?
>>>>>
>>>>> On Mon, Mar 19, 2018 at 1:09 PM, Scott Howell <sc...@mobilgov.com> wrote:
>>>>>> So I was able to get the UI pulled up but now I am hitting a roadblock with my identity-provider.xml.
>>>>>>
>>>>>> I am getting a number of errors like this:
>>>>>>
>>>>>> Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'getIdentityProvider' defined in class path resource [org/apache/nifi/registry/security/authentication/IdentityProviderFactory.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.nifi.registry.security.authentication.IdentityProvider]: Factory method 'getIdentityProvider' threw exception; nested exception is java.lang.Exception: Unable to load the login identity provider configuration file at: /opt/nifi-registry-0.1.0/conf/identity-providers.xml
>>>>>> at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:587) ~[na:na]
>>>>>> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1250) ~[na:na]
>>>>>> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1099) ~[na:na]
>>>>>> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:545) ~[na:na]
>>>>>> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:502) ~[na:na]
>>>>>> at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:312) ~[na:na]
>>>>>> at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228) ~[na:na]
>>>>>> at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:310) ~[na:na]
>>>>>> at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200) ~[na:na]
>>>>>> at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:251) ~[na:na]
>>>>>> at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1135) ~[na:na]
>>>>>> at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1062) ~[na:na]
>>>>>> at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:815) ~[na:na]
>>>>>> at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:721) ~[na:na]
>>>>>> ... 43 common frames omitted
>>>>>>
>>>>>> I know it has to do with the identity-provider.xml but I have my setup just like the documentation ask for. I turned on debug but was not able to see anything different or better explanation from it.
>>>>>>
>>>>>>
>>>>>>> On Mar 19, 2018, at 10:06 AM, Kevin Doran <kd...@apache.org> wrote:
>>>>>>>
>>>>>>> Ok, that use case should be fine.
>>>>>>>
>>>>>>> If it were an authorization issue you would see something in the logs saying that an authorization attempt failed and the server is responding with a 403. Just to be sure, can you enable debug logging if you haven't already, i.e., in your nifi-registry/conf/logback.xml file, change 'org.apache.nifif.registry' to debug:
>>>>>>>
>>>>>>> <!-- valid logging levels: TRACE, DEBUG, INFO, WARN, ERROR -->
>>>>>>> <logger name="org.apache.nifi.registry" level="DEBUG"/>
>>>>>>>
>>>>>>> If there is nothing being written to nifi-registry-app.log, it points towards a connection issue, so I would double check your host, port, and TLS settings. You'll have to get an HTTPS cert from a root CA or configure your ELB to trust your company's self-signed cert (again, not sure if/how to do this, but I assume there should be some way to configure it. It might require settings not exposed in the AWS web console.)
>>>>>>>
>>>>>>> On 3/19/18, 10:51, "Scott Howell" <sc...@mobilgov.com> wrote:
>>>>>>>
>>>>>>> Thanks Kevin,
>>>>>>>
>>>>>>> I am just using the ELB to go from the public subnet to the private subnet. I will not have multiple instances running of registry.
>>>>>>>
>>>>>>> I will say on my authorizers.xml there is one difference between my nifi instance. On my nifi instance I am using file-provider for nifi.security.user.authorizer in my nifi.properties. I don’t think from reading the documents for nifi-registry that I can use that. If there is a way that might be my problem. I was running into some issues with my nifi instance when I was using managed-authorizers instead of file-provider.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> On Mar 19, 2018, at 9:35 AM, Kevin Doran <kd...@apache.org> wrote:
>>>>>>>>
>>>>>>>> Hey Scott,
>>>>>>>>
>>>>>>>> Assuming you are using two-way TLS with client certificates for authentication, I recommend configuring your ELB for TCP passthrough so that the TLS handshake is between the end-client and the NiFi Registry Server (in other words, no decryption/termination of the TLS connection happens in the ELB). If you are using some other form of authentication (e.g., LDAP), you will need to configure your ELB to trust the self-signed key NiFi Registry is using. I'm not sure how to do that as I've never run an ELB with that configuration before.
>>>>>>>>
>>>>>>>> Also, just a note about using an ELB with NiFi Registry:
>>>>>>>>
>>>>>>>> NiFi Registry is currently only supports single-instance use as persisted data and in-memory state is not synced between multiple instances. Are you hoping to use the ELB for actual load balancing, or is it just to take advantage of other ELB features, such as forwarding and security group rules? If the plan is to load balance multiple Registry instances, just be aware that you will probably run into some unexpected behavior. (As you mentioned using authorization, that is one case where I know the in-memory cache of the persisted data will not refresh across instances, so even if you were using some sort of shared network file system attached to multiple Registry instances, such as EFS, it would not work the way you hope.)
>>>>>>>>
>>>>>>>> Hope this helps,
>>>>>>>> Kevin
>>>>>>>>
>>>>>>>> On 3/19/18, 10:20, "Scott Howell" <sc...@mobilgov.com> wrote:
>>>>>>>>
>>>>>>>> Thanks for the quick response.
>>>>>>>>
>>>>>>>> A couple of things I am seeing.
>>>>>>>>
>>>>>>>> 1. There is no error, I don’t see anything in the logs once the service comes up. This is because the health check is not even hitting the instance when secure.
>>>>>>>>
>>>>>>>> 2. Nothing interesting in the nifi-registry-app.logs. That was my concern because on my nifi instance I can see the health check hitting the instance from the ELB. This does not happen on the nifi-registry instance. I see the service startup and it tells me what domain and port I can access the UI but nothing else after that.
>>>>>>>>
>>>>>>>> 3. When I am on an instances in the same private subnet I am able to curl to the instance I get the TLS SSL which tells me the keystore is on the server. I am using a JKS keystore that is self-signed by the company I work for.
>>>>>>>>
>>>>>>>>> On Mar 19, 2018, at 9:10 AM, Bryan Bende <bb...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> What error are you getting when you cannot access the UI?
>>>>>>>>>
>>>>>>>>> Is there anything interesting in nifi-registry-app.log regarding
>>>>>>>>> authentication/authorization when this happens?
>>>>>>>>>
>>>>>>>>> Can you access the UI securely without going through the ELB?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> Bryan
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Mar 19, 2018 at 10:05 AM, Scott Howell <sc...@mobilgov.com> wrote:
>>>>>>>>>> I was able to stand up nifi-registry behind an AWS ELB non-secure. Everything was working great and was able to access the UI anonymously. I set up the authorization just like on my nifi instances along with the authorizers and identity-provider. The service comes up without errors and everything looks good but the health check does not pass and I cannot access the UI to login. I was wondering if anyone else has ran into this issue using nifi-registry.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>
Re: Issue with AWS ELB on secure nifi-registry
Posted by Scott Howell <sc...@mobilgov.com>.
Thanks for all of the help yesterday I was able to get a secure nifi and nifi-registry up and communicating. I am now trying to figure out how to create a secure cluster. I am currently getting this error when I start up nifi.
tion; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'clusterCoordinationProtocolSenderListener' defined in class path resource [nifi-cluster-protocol-context.xml]: Cannot resolve reference to bean 'clusterCoordinationProtocolSender' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'clusterCoordinationProtocolSender' defined in class path resource [nifi-cluster-protocol-context.xml]: Cannot resolve reference to bean 'protocolSocketConfiguration' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'protocolSocketConfiguration': FactoryBean threw exception on object creation; nested exception is java.security.KeyStoreException: not found
at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175)
at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1634)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
... 50 common frames omitted
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'clusterCoordinationProtocolSenderListener' defined in class path resource [nifi-cluster-protocol-context.xml]: Cannot resolve reference to bean 'clusterCoordinationProtocolSender' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'clusterCoordinationProtocolSender' defined in class path resource [nifi-cluster-protocol-context.xml]: Cannot resolve reference to bean 'protocolSocketConfiguration' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'protocolSocketConfiguration': FactoryBean threw exception on object creation; nested exception is java.security.KeyStoreException: not found
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:359)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)
at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:648)
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:145)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1193)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1095)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1084)
at org.apache.nifi.cluster.spring.NodeClusterCoordinatorFactoryBean.getObject(NodeClusterCoordinatorFactoryBean.java:44)
at org.apache.nifi.cluster.spring.NodeClusterCoordinatorFactoryBean.getObject(NodeClusterCoordinatorFactoryBean.java:34)
at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
... 55 common frames omitted
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'clusterCoordinationProtocolSender' defined in class path resource [nifi-cluster-protocol-context.xml]: Cannot resolve reference to bean 'protocolSocketConfiguration' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'protocolSocketConfiguration': FactoryBean threw exception on object creation; nested exception is java.security.KeyStoreException: not found
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:359)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)
at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:648)
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:145)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1193)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1095)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
... 70 common frames omitted
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'protocolSocketConfiguration': FactoryBean threw exception on object creation; nested exception is java.security.KeyStoreException: not found
at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175)
at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1634)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
... 82 common frames omitted
Caused by: java.security.KeyStoreException: not found
at java.security.KeyStore.getInstance(KeyStore.java:851)
at org.apache.nifi.security.util.KeyStoreUtils.getKeyStore(KeyStoreUtils.java:66)
at org.apache.nifi.security.util.KeyStoreUtils.getTrustStore(KeyStoreUtils.java:80)
at org.apache.nifi.io.socket.SSLContextFactory.<init>(SSLContextFactory.java:73)
at org.apache.nifi.cluster.protocol.spring.SocketConfigurationFactoryBean.getObject(SocketConfigurationFactoryBean.java:45)
at org.apache.nifi.cluster.protocol.spring.SocketConfigurationFactoryBean.getObject(SocketConfigurationFactoryBean.java:30)
at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
... 87 common frames omitted
Caused by: java.security.NoSuchAlgorithmException: KeyStore not available
at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
at java.security.Security.getImpl(Security.java:695)
at java.security.KeyStore.getInstance(KeyStore.java:848)
... 93 common frames omitted
My nifi.properties file is.
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Core Properties #
nifi.version={{nifi_version}}
nifi.flow.configuration.file=/opt/config/flow.xml.gz
nifi.flow.configuration.archive.enabled=true
nifi.flow.configuration.archive.dir=/opt/config/archive/
nifi.flow.configuration.archive.max.time=30 days
nifi.flow.configuration.archive.max.storage=500 MB
nifi.flowcontroller.autoResumeState=true
nifi.flowcontroller.graceful.shutdown.period=10 sec
nifi.flowservice.writedelay.interval=500 ms
nifi.administrative.yield.duration=30 sec
# If a component has no work to do (is "bored"), how long should we wait before checking again for work?
nifi.bored.yield.duration=10 millis
nifi.authorizer.configuration.file=/opt/config/authorizers.xml
nifi.login.identity.provider.configuration.file=/opt/config/login-identity-providers.xml
nifi.templates.directory=/opt/config/templates
nifi.ui.banner.text=
nifi.ui.autorefresh.interval=30 sec
nifi.nar.library.directory=/opt/nifi/lib
nifi.nar.library.directory.custom=/opt/config/processors
nifi.nar.working.directory=/opt/nifi/work/nar/
nifi.documentation.working.directory=./work/docs/components
####################
# State Management #
####################
nifi.state.management.configuration.file=/opt/config/state-management.xml
# The ID of the local state provider
nifi.state.management.provider.local=local-provider
# The ID of the cluster-wide state provider. This will be ignored if NiFi is not clustered but must be populated if running in a cluster.
nifi.state.management.provider.cluster=zk-provider
# Specifies whether or not this instance of NiFi should run an embedded ZooKeeper server
nifi.state.management.embedded.zookeeper.start=false
# Properties file that provides the ZooKeeper properties to use if <nifi.state.management.embedded.zookeeper.start> is set to true
nifi.state.management.embedded.zookeeper.properties=/opt/config/zookeeper.properties
# H2 Settings
nifi.database.directory=/opt/database_repository
nifi.h2.url.append=;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE
# FlowFile Repository
nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository
nifi.flowfile.repository.directory=/opt/flowfile_repository
nifi.flowfile.repository.partitions=256
nifi.flowfile.repository.checkpoint.interval=2 mins
nifi.flowfile.repository.always.sync=false
nifi.swap.manager.implementation=org.apache.nifi.controller.FileSystemSwapManager
nifi.queue.swap.threshold=20000
nifi.swap.in.period=5 sec
nifi.swap.in.threads=1
nifi.swap.out.period=5 sec
nifi.swap.out.threads=4
# Content Repository
nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository
nifi.content.claim.max.appendable.size=10 MB
nifi.content.claim.max.flow.files=100
nifi.content.repository.directory.default=/opt/content_repository
nifi.content.repository.archive.max.retention.period=12 hours
nifi.content.repository.archive.max.usage.percentage=50%
nifi.content.repository.archive.enabled=true
nifi.content.repository.always.sync=false
nifi.content.viewer.url=/nifi-content-viewer/
# Provenance Repository Properties
nifi.provenance.repository.implementation=org.apache.nifi.provenance.PersistentProvenanceRepository
# Persistent Provenance Repository Properties
nifi.provenance.repository.directory.default=/opt/provenance_repository
nifi.provenance.repository.max.storage.time=24 hours
nifi.provenance.repository.max.storage.size=1 GB
nifi.provenance.repository.rollover.time=30 secs
nifi.provenance.repository.rollover.size=100 MB
nifi.provenance.repository.query.threads=2
nifi.provenance.repository.index.threads=1
nifi.provenance.repository.compress.on.rollover=true
nifi.provenance.repository.always.sync=false
nifi.provenance.repository.journal.count=16
# Comma-separated list of fields. Fields that are not indexed will not be searchable. Valid fields are:
# EventType, FlowFileUUID, Filename, TransitURI, ProcessorID, AlternateIdentifierURI, Relationship, Details
nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename, ProcessorID, Relationship
# FlowFile Attributes that should be indexed and made searchable. Some examples to consider are filename, uuid, mime.type
nifi.provenance.repository.indexed.attributes=
# Large values for the shard size will result in more Java heap usage when searching the Provenance Repository
# but should provide better performance
nifi.provenance.repository.index.shard.size=500 MB
# Indicates the maximum length that a FlowFile attribute can be when retrieving a Provenance Event from
# the repository. If the length of any attribute exceeds this value, it will be truncated when the event is retrieved.
nifi.provenance.repository.max.attribute.length=65536
# Volatile Provenance Respository Properties
nifi.provenance.repository.buffer.size=100000
# Component Status Repository
nifi.components.status.repository.implementation=org.apache.nifi.controller.status.history.VolatileComponentStatusRepository
nifi.components.status.repository.buffer.size=1440
nifi.components.status.snapshot.frequency=1 min
# Site to Site properties
nifi.remote.input.host=
nifi.remote.input.secure=false
nifi.remote.input.socket.port=9998
nifi.remote.input.http.enabled=false
nifi.remote.input.http.transaction.ttl=30 sec
# web properties #
nifi.web.war.directory=/opt/nifi/lib
nifi.web.http.host=
nifi.web.http.port=
nifi.web.https.host={{redacted}}
nifi.web.https.port=8443
nifi.web.jetty.working.directory=/opt/nifi/work/jetty
nifi.web.jetty.threads=200
# security properties #
nifi.sensitive.props.key=x0KDgO9L8lAhFGLdvu2VEjFVGc6Kg3V0R5I4bYwoqdgL47moo0wApKQtAVu1BvD
nifi.sensitive.props.key.protected=
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
nifi.sensitive.props.provider=BC
nifi.sensitive.props.additional.keys=
nifi.security.keystore=/opt/certs/payit_keystore
nifi.security.keystoreType=JKS
nifi.security.keystorePasswd={{keystore_password}}
nifi.security.keyPasswd=
nifi.security.truststore=
nifi.security.truststoreType=
nifi.security.truststorePasswd=
nifi.security.needClientAuth=false
nifi.security.user.authorizer=file-provider
nifi.security.user.login.identity.provider=ldap-provider
nifi.security.ocsp.responder.url=
nifi.security.ocsp.responder.certificate=
# Identity Mapping Properties #
# These properties allow normalizing user identities such that identities coming from different identity providers
# (certificates, LDAP, Kerberos) can be treated the same internally in NiFi. The following example demonstrates normalizing
# DNs from certificates and principals from Kerberos into a common identity string:
#
#nifi.security.identity.mapping.pattern.dn=^cn=(.*?),ou=(.*?),dc=(.*?),dc=(.*?),dc=(.*?)$
#nifi.security.identity.mapping.value.dn=$1
# nifi.security.identity.mapping.pattern.kerb=^(.*?)/instance@(.*?)$
# nifi.security.identity.mapping.value.kerb=$1@$2
# cluster common properties (all nodes must have same values) #
nifi.cluster.protocol.heartbeat.interval=5 sec
nifi.cluster.protocol.is.secure=true
# cluster node properties (only configure for cluster nodes) #
nifi.cluster.is.node=true
nifi.cluster.node.address=nifi-dev.mobilgov.com
nifi.cluster.node.protocol.port=9999
nifi.cluster.node.protocol.threads=10
nifi.cluster.node.event.history.size=25
nifi.cluster.node.connection.timeout=5 sec
nifi.cluster.node.read.timeout=5 sec
nifi.cluster.firewall.file=
# zookeeper properties, used for cluster management #
nifi.zookeeper.connect.string=internal-etcd-dev-EtcdLoadB-3RWA2WEFBBT8-2068560477.us-east-2.elb.amazonaws.com:2181,internal-etcd-dev-EtcdLoadB-3RWA2WEFBBT8-2068560477.us-east-2.elb.amazonaws.com2182,internal-etcd-dev-EtcdLoadB-3RWA2WEFBBT8-2068560477.us-east-2.elb.amazonaws.com:2183
nifi.zookeeper.connect.timeout=3 secs
nifi.zookeeper.session.timeout=3 secs
nifi.zookeeper.root.node=/nifi
# kerberos #
nifi.kerberos.krb5.file=
# kerberos service principle #
nifi.kerberos.service.principal=
nifi.kerberos.service.keytab.location=
# kerberos spnego principle #
nifi.kerberos.spnego.principal=
nifi.kerberos.spnego.keytab.location=
nifi.kerberos.spnego.authentication.expiration=12 hours
# external properties files for variable registry
# supports a comma delimited list of file locations
nifi.variable.registry.properties=
I think I have everything set correctly but I have not been able to start an instances up.
Thanks,
Scott
> On Mar 19, 2018, at 4:35 PM, Bryan Bende <bb...@gmail.com> wrote:
>
> The base file is here for comparison:
>
> https://github.com/apache/nifi-registry/blob/master/nifi-registry-resources/src/main/resources/conf/identity-providers.xml#L23
>
> On Mon, Mar 19, 2018 at 5:34 PM, Bryan Bende <bb...@gmail.com> wrote:
>> For your first file, is what you showed there actually wrapped in
>> <identityProviders> </identityProviders> or is it exactly what you
>> showed?
>>
>> It may just be that you only copied/pasted the one provider, but the
>> root element is not <provider>, so as it is shown there it would not
>> parse.
>>
>> On Mon, Mar 19, 2018 at 2:54 PM, Scott Howell <sc...@mobilgov.com> wrote:
>>> Here is my file
>>>
>>> <provider>
>>> <identifier>ldap-identity-provider</identifier>
>>> <class>org.apache.nifi.registry.security.ldap.LdapProvider</class>
>>> <property name="Authentication Strategy">SIMPLE</property>
>>>
>>> <property name="Manager DN">cn=Manager,dc=mobilgov,dc=com</property>
>>> <property name="Manager Password”>redacted</property>
>>>
>>>
>>> <property name="Referral Strategy">FOLLOW</property>
>>> <property name="Connect Timeout">10 secs</property>
>>> <property name="Read Timeout">10 secs</property>
>>>
>>> <property name="Url”>redacted</property>
>>> <property name="User Search Base">ou=users,dc=mobilgov,dc=com</property>
>>> <property name="User Search Filter">uid={0}</property>
>>>
>>> <property name="Identity Strategy">USE_DN</property>
>>> <property name="Authentication Expiration">12 hours</property>
>>> </provider>
>>>
>>> Here is my authorizers.xml
>>>
>>> <authorizers>
>>>
>>> <userGroupProvider>
>>> <identifier>file-user-group-provider</identifier>
>>> <class>org.apache.nifi.registry.security.authorization.file.FileUserGroupProvider</class>
>>> <property name="Users File">conf/users.xml</property>
>>> <property name="Legacy Authorized Users File"></property>
>>> <property name="Initial User Identity 1”>redacted</property>
>>> </userGroupProvider>
>>>
>>> <accessPolicyProvider>
>>> <identifier>file-access-policy-provider</identifier>
>>> <class>org.apache.nifi.registry.security.authorization.file.FileAccessPolicyProvider</class>
>>> <property name="User Group Provider">file-user-group-provider</property>
>>> <property name="Authorizations File">conf/authorizations.xml</property>
>>> <property name="Initial Admin Identity”>redacted</property>
>>> <property name="NiFi Identity 1"></property>
>>> </accessPolicyProvider>
>>>
>>> <authorizer>
>>> <identifier>managed-authorizer</identifier>
>>> <class>org.apache.nifi.registry.security.authorization.StandardManagedAuthorizer</class>
>>> <property name="Access Policy Provider">file-access-policy-provider</property>
>>> </authorizer>
>>> </authorizers>
>>>
>>>> On Mar 19, 2018, at 12:59 PM, Bryan Bende <bb...@gmail.com> wrote:
>>>>
>>>> It looks like that error would happen if your identity-providers.xml
>>>> contained invalid XML.
>>>>
>>>> Did you start by modifying the identity-providers.xml file that was
>>>> already there? Can you share the file, or the contents (removing
>>>> anything sensitive)?
>>>>
>>>> On Mon, Mar 19, 2018 at 1:09 PM, Scott Howell <sc...@mobilgov.com> wrote:
>>>>> So I was able to get the UI pulled up but now I am hitting a roadblock with my identity-provider.xml.
>>>>>
>>>>> I am getting a number of errors like this:
>>>>>
>>>>> Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'getIdentityProvider' defined in class path resource [org/apache/nifi/registry/security/authentication/IdentityProviderFactory.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.nifi.registry.security.authentication.IdentityProvider]: Factory method 'getIdentityProvider' threw exception; nested exception is java.lang.Exception: Unable to load the login identity provider configuration file at: /opt/nifi-registry-0.1.0/conf/identity-providers.xml
>>>>> at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:587) ~[na:na]
>>>>> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1250) ~[na:na]
>>>>> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1099) ~[na:na]
>>>>> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:545) ~[na:na]
>>>>> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:502) ~[na:na]
>>>>> at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:312) ~[na:na]
>>>>> at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228) ~[na:na]
>>>>> at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:310) ~[na:na]
>>>>> at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200) ~[na:na]
>>>>> at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:251) ~[na:na]
>>>>> at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1135) ~[na:na]
>>>>> at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1062) ~[na:na]
>>>>> at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:815) ~[na:na]
>>>>> at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:721) ~[na:na]
>>>>> ... 43 common frames omitted
>>>>>
>>>>> I know it has to do with the identity-provider.xml but I have my setup just like the documentation ask for. I turned on debug but was not able to see anything different or better explanation from it.
>>>>>
>>>>>
>>>>>> On Mar 19, 2018, at 10:06 AM, Kevin Doran <kd...@apache.org> wrote:
>>>>>>
>>>>>> Ok, that use case should be fine.
>>>>>>
>>>>>> If it were an authorization issue you would see something in the logs saying that an authorization attempt failed and the server is responding with a 403. Just to be sure, can you enable debug logging if you haven't already, i.e., in your nifi-registry/conf/logback.xml file, change 'org.apache.nifif.registry' to debug:
>>>>>>
>>>>>> <!-- valid logging levels: TRACE, DEBUG, INFO, WARN, ERROR -->
>>>>>> <logger name="org.apache.nifi.registry" level="DEBUG"/>
>>>>>>
>>>>>> If there is nothing being written to nifi-registry-app.log, it points towards a connection issue, so I would double check your host, port, and TLS settings. You'll have to get an HTTPS cert from a root CA or configure your ELB to trust your company's self-signed cert (again, not sure if/how to do this, but I assume there should be some way to configure it. It might require settings not exposed in the AWS web console.)
>>>>>>
>>>>>> On 3/19/18, 10:51, "Scott Howell" <sc...@mobilgov.com> wrote:
>>>>>>
>>>>>> Thanks Kevin,
>>>>>>
>>>>>> I am just using the ELB to go from the public subnet to the private subnet. I will not have multiple instances running of registry.
>>>>>>
>>>>>> I will say on my authorizers.xml there is one difference between my nifi instance. On my nifi instance I am using file-provider for nifi.security.user.authorizer in my nifi.properties. I don’t think from reading the documents for nifi-registry that I can use that. If there is a way that might be my problem. I was running into some issues with my nifi instance when I was using managed-authorizers instead of file-provider.
>>>>>>
>>>>>>
>>>>>>
>>>>>>> On Mar 19, 2018, at 9:35 AM, Kevin Doran <kd...@apache.org> wrote:
>>>>>>>
>>>>>>> Hey Scott,
>>>>>>>
>>>>>>> Assuming you are using two-way TLS with client certificates for authentication, I recommend configuring your ELB for TCP passthrough so that the TLS handshake is between the end-client and the NiFi Registry Server (in other words, no decryption/termination of the TLS connection happens in the ELB). If you are using some other form of authentication (e.g., LDAP), you will need to configure your ELB to trust the self-signed key NiFi Registry is using. I'm not sure how to do that as I've never run an ELB with that configuration before.
>>>>>>>
>>>>>>> Also, just a note about using an ELB with NiFi Registry:
>>>>>>>
>>>>>>> NiFi Registry is currently only supports single-instance use as persisted data and in-memory state is not synced between multiple instances. Are you hoping to use the ELB for actual load balancing, or is it just to take advantage of other ELB features, such as forwarding and security group rules? If the plan is to load balance multiple Registry instances, just be aware that you will probably run into some unexpected behavior. (As you mentioned using authorization, that is one case where I know the in-memory cache of the persisted data will not refresh across instances, so even if you were using some sort of shared network file system attached to multiple Registry instances, such as EFS, it would not work the way you hope.)
>>>>>>>
>>>>>>> Hope this helps,
>>>>>>> Kevin
>>>>>>>
>>>>>>> On 3/19/18, 10:20, "Scott Howell" <sc...@mobilgov.com> wrote:
>>>>>>>
>>>>>>> Thanks for the quick response.
>>>>>>>
>>>>>>> A couple of things I am seeing.
>>>>>>>
>>>>>>> 1. There is no error, I don’t see anything in the logs once the service comes up. This is because the health check is not even hitting the instance when secure.
>>>>>>>
>>>>>>> 2. Nothing interesting in the nifi-registry-app.logs. That was my concern because on my nifi instance I can see the health check hitting the instance from the ELB. This does not happen on the nifi-registry instance. I see the service startup and it tells me what domain and port I can access the UI but nothing else after that.
>>>>>>>
>>>>>>> 3. When I am on an instances in the same private subnet I am able to curl to the instance I get the TLS SSL which tells me the keystore is on the server. I am using a JKS keystore that is self-signed by the company I work for.
>>>>>>>
>>>>>>>> On Mar 19, 2018, at 9:10 AM, Bryan Bende <bb...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> What error are you getting when you cannot access the UI?
>>>>>>>>
>>>>>>>> Is there anything interesting in nifi-registry-app.log regarding
>>>>>>>> authentication/authorization when this happens?
>>>>>>>>
>>>>>>>> Can you access the UI securely without going through the ELB?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Bryan
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Mar 19, 2018 at 10:05 AM, Scott Howell <sc...@mobilgov.com> wrote:
>>>>>>>>> I was able to stand up nifi-registry behind an AWS ELB non-secure. Everything was working great and was able to access the UI anonymously. I set up the authorization just like on my nifi instances along with the authorizers and identity-provider. The service comes up without errors and everything looks good but the health check does not pass and I cannot access the UI to login. I was wondering if anyone else has ran into this issue using nifi-registry.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>
Re: Issue with AWS ELB on secure nifi-registry
Posted by Bryan Bende <bb...@gmail.com>.
The base file is here for comparison:
https://github.com/apache/nifi-registry/blob/master/nifi-registry-resources/src/main/resources/conf/identity-providers.xml#L23
On Mon, Mar 19, 2018 at 5:34 PM, Bryan Bende <bb...@gmail.com> wrote:
> For your first file, is what you showed there actually wrapped in
> <identityProviders> </identityProviders> or is it exactly what you
> showed?
>
> It may just be that you only copied/pasted the one provider, but the
> root element is not <provider>, so as it is shown there it would not
> parse.
>
> On Mon, Mar 19, 2018 at 2:54 PM, Scott Howell <sc...@mobilgov.com> wrote:
>> Here is my file
>>
>> <provider>
>> <identifier>ldap-identity-provider</identifier>
>> <class>org.apache.nifi.registry.security.ldap.LdapProvider</class>
>> <property name="Authentication Strategy">SIMPLE</property>
>>
>> <property name="Manager DN">cn=Manager,dc=mobilgov,dc=com</property>
>> <property name="Manager Password”>redacted</property>
>>
>>
>> <property name="Referral Strategy">FOLLOW</property>
>> <property name="Connect Timeout">10 secs</property>
>> <property name="Read Timeout">10 secs</property>
>>
>> <property name="Url”>redacted</property>
>> <property name="User Search Base">ou=users,dc=mobilgov,dc=com</property>
>> <property name="User Search Filter">uid={0}</property>
>>
>> <property name="Identity Strategy">USE_DN</property>
>> <property name="Authentication Expiration">12 hours</property>
>> </provider>
>>
>> Here is my authorizers.xml
>>
>> <authorizers>
>>
>> <userGroupProvider>
>> <identifier>file-user-group-provider</identifier>
>> <class>org.apache.nifi.registry.security.authorization.file.FileUserGroupProvider</class>
>> <property name="Users File">conf/users.xml</property>
>> <property name="Legacy Authorized Users File"></property>
>> <property name="Initial User Identity 1”>redacted</property>
>> </userGroupProvider>
>>
>> <accessPolicyProvider>
>> <identifier>file-access-policy-provider</identifier>
>> <class>org.apache.nifi.registry.security.authorization.file.FileAccessPolicyProvider</class>
>> <property name="User Group Provider">file-user-group-provider</property>
>> <property name="Authorizations File">conf/authorizations.xml</property>
>> <property name="Initial Admin Identity”>redacted</property>
>> <property name="NiFi Identity 1"></property>
>> </accessPolicyProvider>
>>
>> <authorizer>
>> <identifier>managed-authorizer</identifier>
>> <class>org.apache.nifi.registry.security.authorization.StandardManagedAuthorizer</class>
>> <property name="Access Policy Provider">file-access-policy-provider</property>
>> </authorizer>
>> </authorizers>
>>
>>> On Mar 19, 2018, at 12:59 PM, Bryan Bende <bb...@gmail.com> wrote:
>>>
>>> It looks like that error would happen if your identity-providers.xml
>>> contained invalid XML.
>>>
>>> Did you start by modifying the identity-providers.xml file that was
>>> already there? Can you share the file, or the contents (removing
>>> anything sensitive)?
>>>
>>> On Mon, Mar 19, 2018 at 1:09 PM, Scott Howell <sc...@mobilgov.com> wrote:
>>>> So I was able to get the UI pulled up but now I am hitting a roadblock with my identity-provider.xml.
>>>>
>>>> I am getting a number of errors like this:
>>>>
>>>> Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'getIdentityProvider' defined in class path resource [org/apache/nifi/registry/security/authentication/IdentityProviderFactory.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.nifi.registry.security.authentication.IdentityProvider]: Factory method 'getIdentityProvider' threw exception; nested exception is java.lang.Exception: Unable to load the login identity provider configuration file at: /opt/nifi-registry-0.1.0/conf/identity-providers.xml
>>>> at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:587) ~[na:na]
>>>> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1250) ~[na:na]
>>>> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1099) ~[na:na]
>>>> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:545) ~[na:na]
>>>> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:502) ~[na:na]
>>>> at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:312) ~[na:na]
>>>> at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228) ~[na:na]
>>>> at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:310) ~[na:na]
>>>> at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200) ~[na:na]
>>>> at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:251) ~[na:na]
>>>> at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1135) ~[na:na]
>>>> at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1062) ~[na:na]
>>>> at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:815) ~[na:na]
>>>> at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:721) ~[na:na]
>>>> ... 43 common frames omitted
>>>>
>>>> I know it has to do with the identity-provider.xml but I have my setup just like the documentation ask for. I turned on debug but was not able to see anything different or better explanation from it.
>>>>
>>>>
>>>>> On Mar 19, 2018, at 10:06 AM, Kevin Doran <kd...@apache.org> wrote:
>>>>>
>>>>> Ok, that use case should be fine.
>>>>>
>>>>> If it were an authorization issue you would see something in the logs saying that an authorization attempt failed and the server is responding with a 403. Just to be sure, can you enable debug logging if you haven't already, i.e., in your nifi-registry/conf/logback.xml file, change 'org.apache.nifif.registry' to debug:
>>>>>
>>>>> <!-- valid logging levels: TRACE, DEBUG, INFO, WARN, ERROR -->
>>>>> <logger name="org.apache.nifi.registry" level="DEBUG"/>
>>>>>
>>>>> If there is nothing being written to nifi-registry-app.log, it points towards a connection issue, so I would double check your host, port, and TLS settings. You'll have to get an HTTPS cert from a root CA or configure your ELB to trust your company's self-signed cert (again, not sure if/how to do this, but I assume there should be some way to configure it. It might require settings not exposed in the AWS web console.)
>>>>>
>>>>> On 3/19/18, 10:51, "Scott Howell" <sc...@mobilgov.com> wrote:
>>>>>
>>>>> Thanks Kevin,
>>>>>
>>>>> I am just using the ELB to go from the public subnet to the private subnet. I will not have multiple instances running of registry.
>>>>>
>>>>> I will say on my authorizers.xml there is one difference between my nifi instance. On my nifi instance I am using file-provider for nifi.security.user.authorizer in my nifi.properties. I don’t think from reading the documents for nifi-registry that I can use that. If there is a way that might be my problem. I was running into some issues with my nifi instance when I was using managed-authorizers instead of file-provider.
>>>>>
>>>>>
>>>>>
>>>>>> On Mar 19, 2018, at 9:35 AM, Kevin Doran <kd...@apache.org> wrote:
>>>>>>
>>>>>> Hey Scott,
>>>>>>
>>>>>> Assuming you are using two-way TLS with client certificates for authentication, I recommend configuring your ELB for TCP passthrough so that the TLS handshake is between the end-client and the NiFi Registry Server (in other words, no decryption/termination of the TLS connection happens in the ELB). If you are using some other form of authentication (e.g., LDAP), you will need to configure your ELB to trust the self-signed key NiFi Registry is using. I'm not sure how to do that as I've never run an ELB with that configuration before.
>>>>>>
>>>>>> Also, just a note about using an ELB with NiFi Registry:
>>>>>>
>>>>>> NiFi Registry is currently only supports single-instance use as persisted data and in-memory state is not synced between multiple instances. Are you hoping to use the ELB for actual load balancing, or is it just to take advantage of other ELB features, such as forwarding and security group rules? If the plan is to load balance multiple Registry instances, just be aware that you will probably run into some unexpected behavior. (As you mentioned using authorization, that is one case where I know the in-memory cache of the persisted data will not refresh across instances, so even if you were using some sort of shared network file system attached to multiple Registry instances, such as EFS, it would not work the way you hope.)
>>>>>>
>>>>>> Hope this helps,
>>>>>> Kevin
>>>>>>
>>>>>> On 3/19/18, 10:20, "Scott Howell" <sc...@mobilgov.com> wrote:
>>>>>>
>>>>>> Thanks for the quick response.
>>>>>>
>>>>>> A couple of things I am seeing.
>>>>>>
>>>>>> 1. There is no error, I don’t see anything in the logs once the service comes up. This is because the health check is not even hitting the instance when secure.
>>>>>>
>>>>>> 2. Nothing interesting in the nifi-registry-app.logs. That was my concern because on my nifi instance I can see the health check hitting the instance from the ELB. This does not happen on the nifi-registry instance. I see the service startup and it tells me what domain and port I can access the UI but nothing else after that.
>>>>>>
>>>>>> 3. When I am on an instances in the same private subnet I am able to curl to the instance I get the TLS SSL which tells me the keystore is on the server. I am using a JKS keystore that is self-signed by the company I work for.
>>>>>>
>>>>>>> On Mar 19, 2018, at 9:10 AM, Bryan Bende <bb...@gmail.com> wrote:
>>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> What error are you getting when you cannot access the UI?
>>>>>>>
>>>>>>> Is there anything interesting in nifi-registry-app.log regarding
>>>>>>> authentication/authorization when this happens?
>>>>>>>
>>>>>>> Can you access the UI securely without going through the ELB?
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Bryan
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Mar 19, 2018 at 10:05 AM, Scott Howell <sc...@mobilgov.com> wrote:
>>>>>>>> I was able to stand up nifi-registry behind an AWS ELB non-secure. Everything was working great and was able to access the UI anonymously. I set up the authorization just like on my nifi instances along with the authorizers and identity-provider. The service comes up without errors and everything looks good but the health check does not pass and I cannot access the UI to login. I was wondering if anyone else has ran into this issue using nifi-registry.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>
Re: Issue with AWS ELB on secure nifi-registry
Posted by Bryan Bende <bb...@gmail.com>.
For your first file, is what you showed there actually wrapped in
<identityProviders> </identityProviders> or is it exactly what you
showed?
It may just be that you only copied/pasted the one provider, but the
root element is not <provider>, so as it is shown there it would not
parse.
On Mon, Mar 19, 2018 at 2:54 PM, Scott Howell <sc...@mobilgov.com> wrote:
> Here is my file
>
> <provider>
> <identifier>ldap-identity-provider</identifier>
> <class>org.apache.nifi.registry.security.ldap.LdapProvider</class>
> <property name="Authentication Strategy">SIMPLE</property>
>
> <property name="Manager DN">cn=Manager,dc=mobilgov,dc=com</property>
> <property name="Manager Password”>redacted</property>
>
>
> <property name="Referral Strategy">FOLLOW</property>
> <property name="Connect Timeout">10 secs</property>
> <property name="Read Timeout">10 secs</property>
>
> <property name="Url”>redacted</property>
> <property name="User Search Base">ou=users,dc=mobilgov,dc=com</property>
> <property name="User Search Filter">uid={0}</property>
>
> <property name="Identity Strategy">USE_DN</property>
> <property name="Authentication Expiration">12 hours</property>
> </provider>
>
> Here is my authorizers.xml
>
> <authorizers>
>
> <userGroupProvider>
> <identifier>file-user-group-provider</identifier>
> <class>org.apache.nifi.registry.security.authorization.file.FileUserGroupProvider</class>
> <property name="Users File">conf/users.xml</property>
> <property name="Legacy Authorized Users File"></property>
> <property name="Initial User Identity 1”>redacted</property>
> </userGroupProvider>
>
> <accessPolicyProvider>
> <identifier>file-access-policy-provider</identifier>
> <class>org.apache.nifi.registry.security.authorization.file.FileAccessPolicyProvider</class>
> <property name="User Group Provider">file-user-group-provider</property>
> <property name="Authorizations File">conf/authorizations.xml</property>
> <property name="Initial Admin Identity”>redacted</property>
> <property name="NiFi Identity 1"></property>
> </accessPolicyProvider>
>
> <authorizer>
> <identifier>managed-authorizer</identifier>
> <class>org.apache.nifi.registry.security.authorization.StandardManagedAuthorizer</class>
> <property name="Access Policy Provider">file-access-policy-provider</property>
> </authorizer>
> </authorizers>
>
>> On Mar 19, 2018, at 12:59 PM, Bryan Bende <bb...@gmail.com> wrote:
>>
>> It looks like that error would happen if your identity-providers.xml
>> contained invalid XML.
>>
>> Did you start by modifying the identity-providers.xml file that was
>> already there? Can you share the file, or the contents (removing
>> anything sensitive)?
>>
>> On Mon, Mar 19, 2018 at 1:09 PM, Scott Howell <sc...@mobilgov.com> wrote:
>>> So I was able to get the UI pulled up but now I am hitting a roadblock with my identity-provider.xml.
>>>
>>> I am getting a number of errors like this:
>>>
>>> Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'getIdentityProvider' defined in class path resource [org/apache/nifi/registry/security/authentication/IdentityProviderFactory.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.nifi.registry.security.authentication.IdentityProvider]: Factory method 'getIdentityProvider' threw exception; nested exception is java.lang.Exception: Unable to load the login identity provider configuration file at: /opt/nifi-registry-0.1.0/conf/identity-providers.xml
>>> at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:587) ~[na:na]
>>> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1250) ~[na:na]
>>> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1099) ~[na:na]
>>> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:545) ~[na:na]
>>> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:502) ~[na:na]
>>> at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:312) ~[na:na]
>>> at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228) ~[na:na]
>>> at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:310) ~[na:na]
>>> at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200) ~[na:na]
>>> at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:251) ~[na:na]
>>> at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1135) ~[na:na]
>>> at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1062) ~[na:na]
>>> at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:815) ~[na:na]
>>> at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:721) ~[na:na]
>>> ... 43 common frames omitted
>>>
>>> I know it has to do with the identity-provider.xml but I have my setup just like the documentation ask for. I turned on debug but was not able to see anything different or better explanation from it.
>>>
>>>
>>>> On Mar 19, 2018, at 10:06 AM, Kevin Doran <kd...@apache.org> wrote:
>>>>
>>>> Ok, that use case should be fine.
>>>>
>>>> If it were an authorization issue you would see something in the logs saying that an authorization attempt failed and the server is responding with a 403. Just to be sure, can you enable debug logging if you haven't already, i.e., in your nifi-registry/conf/logback.xml file, change 'org.apache.nifif.registry' to debug:
>>>>
>>>> <!-- valid logging levels: TRACE, DEBUG, INFO, WARN, ERROR -->
>>>> <logger name="org.apache.nifi.registry" level="DEBUG"/>
>>>>
>>>> If there is nothing being written to nifi-registry-app.log, it points towards a connection issue, so I would double check your host, port, and TLS settings. You'll have to get an HTTPS cert from a root CA or configure your ELB to trust your company's self-signed cert (again, not sure if/how to do this, but I assume there should be some way to configure it. It might require settings not exposed in the AWS web console.)
>>>>
>>>> On 3/19/18, 10:51, "Scott Howell" <sc...@mobilgov.com> wrote:
>>>>
>>>> Thanks Kevin,
>>>>
>>>> I am just using the ELB to go from the public subnet to the private subnet. I will not have multiple instances running of registry.
>>>>
>>>> I will say on my authorizers.xml there is one difference between my nifi instance. On my nifi instance I am using file-provider for nifi.security.user.authorizer in my nifi.properties. I don’t think from reading the documents for nifi-registry that I can use that. If there is a way that might be my problem. I was running into some issues with my nifi instance when I was using managed-authorizers instead of file-provider.
>>>>
>>>>
>>>>
>>>>> On Mar 19, 2018, at 9:35 AM, Kevin Doran <kd...@apache.org> wrote:
>>>>>
>>>>> Hey Scott,
>>>>>
>>>>> Assuming you are using two-way TLS with client certificates for authentication, I recommend configuring your ELB for TCP passthrough so that the TLS handshake is between the end-client and the NiFi Registry Server (in other words, no decryption/termination of the TLS connection happens in the ELB). If you are using some other form of authentication (e.g., LDAP), you will need to configure your ELB to trust the self-signed key NiFi Registry is using. I'm not sure how to do that as I've never run an ELB with that configuration before.
>>>>>
>>>>> Also, just a note about using an ELB with NiFi Registry:
>>>>>
>>>>> NiFi Registry is currently only supports single-instance use as persisted data and in-memory state is not synced between multiple instances. Are you hoping to use the ELB for actual load balancing, or is it just to take advantage of other ELB features, such as forwarding and security group rules? If the plan is to load balance multiple Registry instances, just be aware that you will probably run into some unexpected behavior. (As you mentioned using authorization, that is one case where I know the in-memory cache of the persisted data will not refresh across instances, so even if you were using some sort of shared network file system attached to multiple Registry instances, such as EFS, it would not work the way you hope.)
>>>>>
>>>>> Hope this helps,
>>>>> Kevin
>>>>>
>>>>> On 3/19/18, 10:20, "Scott Howell" <sc...@mobilgov.com> wrote:
>>>>>
>>>>> Thanks for the quick response.
>>>>>
>>>>> A couple of things I am seeing.
>>>>>
>>>>> 1. There is no error, I don’t see anything in the logs once the service comes up. This is because the health check is not even hitting the instance when secure.
>>>>>
>>>>> 2. Nothing interesting in the nifi-registry-app.logs. That was my concern because on my nifi instance I can see the health check hitting the instance from the ELB. This does not happen on the nifi-registry instance. I see the service startup and it tells me what domain and port I can access the UI but nothing else after that.
>>>>>
>>>>> 3. When I am on an instances in the same private subnet I am able to curl to the instance I get the TLS SSL which tells me the keystore is on the server. I am using a JKS keystore that is self-signed by the company I work for.
>>>>>
>>>>>> On Mar 19, 2018, at 9:10 AM, Bryan Bende <bb...@gmail.com> wrote:
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> What error are you getting when you cannot access the UI?
>>>>>>
>>>>>> Is there anything interesting in nifi-registry-app.log regarding
>>>>>> authentication/authorization when this happens?
>>>>>>
>>>>>> Can you access the UI securely without going through the ELB?
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Bryan
>>>>>>
>>>>>>
>>>>>> On Mon, Mar 19, 2018 at 10:05 AM, Scott Howell <sc...@mobilgov.com> wrote:
>>>>>>> I was able to stand up nifi-registry behind an AWS ELB non-secure. Everything was working great and was able to access the UI anonymously. I set up the authorization just like on my nifi instances along with the authorizers and identity-provider. The service comes up without errors and everything looks good but the health check does not pass and I cannot access the UI to login. I was wondering if anyone else has ran into this issue using nifi-registry.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>
Re: Issue with AWS ELB on secure nifi-registry
Posted by Scott Howell <sc...@mobilgov.com>.
Here is my file
<provider>
<identifier>ldap-identity-provider</identifier>
<class>org.apache.nifi.registry.security.ldap.LdapProvider</class>
<property name="Authentication Strategy">SIMPLE</property>
<property name="Manager DN">cn=Manager,dc=mobilgov,dc=com</property>
<property name="Manager Password”>redacted</property>
<property name="Referral Strategy">FOLLOW</property>
<property name="Connect Timeout">10 secs</property>
<property name="Read Timeout">10 secs</property>
<property name="Url”>redacted</property>
<property name="User Search Base">ou=users,dc=mobilgov,dc=com</property>
<property name="User Search Filter">uid={0}</property>
<property name="Identity Strategy">USE_DN</property>
<property name="Authentication Expiration">12 hours</property>
</provider>
Here is my authorizers.xml
<authorizers>
<userGroupProvider>
<identifier>file-user-group-provider</identifier>
<class>org.apache.nifi.registry.security.authorization.file.FileUserGroupProvider</class>
<property name="Users File">conf/users.xml</property>
<property name="Legacy Authorized Users File"></property>
<property name="Initial User Identity 1”>redacted</property>
</userGroupProvider>
<accessPolicyProvider>
<identifier>file-access-policy-provider</identifier>
<class>org.apache.nifi.registry.security.authorization.file.FileAccessPolicyProvider</class>
<property name="User Group Provider">file-user-group-provider</property>
<property name="Authorizations File">conf/authorizations.xml</property>
<property name="Initial Admin Identity”>redacted</property>
<property name="NiFi Identity 1"></property>
</accessPolicyProvider>
<authorizer>
<identifier>managed-authorizer</identifier>
<class>org.apache.nifi.registry.security.authorization.StandardManagedAuthorizer</class>
<property name="Access Policy Provider">file-access-policy-provider</property>
</authorizer>
</authorizers>
> On Mar 19, 2018, at 12:59 PM, Bryan Bende <bb...@gmail.com> wrote:
>
> It looks like that error would happen if your identity-providers.xml
> contained invalid XML.
>
> Did you start by modifying the identity-providers.xml file that was
> already there? Can you share the file, or the contents (removing
> anything sensitive)?
>
> On Mon, Mar 19, 2018 at 1:09 PM, Scott Howell <sc...@mobilgov.com> wrote:
>> So I was able to get the UI pulled up but now I am hitting a roadblock with my identity-provider.xml.
>>
>> I am getting a number of errors like this:
>>
>> Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'getIdentityProvider' defined in class path resource [org/apache/nifi/registry/security/authentication/IdentityProviderFactory.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.nifi.registry.security.authentication.IdentityProvider]: Factory method 'getIdentityProvider' threw exception; nested exception is java.lang.Exception: Unable to load the login identity provider configuration file at: /opt/nifi-registry-0.1.0/conf/identity-providers.xml
>> at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:587) ~[na:na]
>> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1250) ~[na:na]
>> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1099) ~[na:na]
>> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:545) ~[na:na]
>> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:502) ~[na:na]
>> at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:312) ~[na:na]
>> at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228) ~[na:na]
>> at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:310) ~[na:na]
>> at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200) ~[na:na]
>> at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:251) ~[na:na]
>> at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1135) ~[na:na]
>> at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1062) ~[na:na]
>> at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:815) ~[na:na]
>> at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:721) ~[na:na]
>> ... 43 common frames omitted
>>
>> I know it has to do with the identity-provider.xml but I have my setup just like the documentation ask for. I turned on debug but was not able to see anything different or better explanation from it.
>>
>>
>>> On Mar 19, 2018, at 10:06 AM, Kevin Doran <kd...@apache.org> wrote:
>>>
>>> Ok, that use case should be fine.
>>>
>>> If it were an authorization issue you would see something in the logs saying that an authorization attempt failed and the server is responding with a 403. Just to be sure, can you enable debug logging if you haven't already, i.e., in your nifi-registry/conf/logback.xml file, change 'org.apache.nifif.registry' to debug:
>>>
>>> <!-- valid logging levels: TRACE, DEBUG, INFO, WARN, ERROR -->
>>> <logger name="org.apache.nifi.registry" level="DEBUG"/>
>>>
>>> If there is nothing being written to nifi-registry-app.log, it points towards a connection issue, so I would double check your host, port, and TLS settings. You'll have to get an HTTPS cert from a root CA or configure your ELB to trust your company's self-signed cert (again, not sure if/how to do this, but I assume there should be some way to configure it. It might require settings not exposed in the AWS web console.)
>>>
>>> On 3/19/18, 10:51, "Scott Howell" <sc...@mobilgov.com> wrote:
>>>
>>> Thanks Kevin,
>>>
>>> I am just using the ELB to go from the public subnet to the private subnet. I will not have multiple instances running of registry.
>>>
>>> I will say on my authorizers.xml there is one difference between my nifi instance. On my nifi instance I am using file-provider for nifi.security.user.authorizer in my nifi.properties. I don’t think from reading the documents for nifi-registry that I can use that. If there is a way that might be my problem. I was running into some issues with my nifi instance when I was using managed-authorizers instead of file-provider.
>>>
>>>
>>>
>>>> On Mar 19, 2018, at 9:35 AM, Kevin Doran <kd...@apache.org> wrote:
>>>>
>>>> Hey Scott,
>>>>
>>>> Assuming you are using two-way TLS with client certificates for authentication, I recommend configuring your ELB for TCP passthrough so that the TLS handshake is between the end-client and the NiFi Registry Server (in other words, no decryption/termination of the TLS connection happens in the ELB). If you are using some other form of authentication (e.g., LDAP), you will need to configure your ELB to trust the self-signed key NiFi Registry is using. I'm not sure how to do that as I've never run an ELB with that configuration before.
>>>>
>>>> Also, just a note about using an ELB with NiFi Registry:
>>>>
>>>> NiFi Registry is currently only supports single-instance use as persisted data and in-memory state is not synced between multiple instances. Are you hoping to use the ELB for actual load balancing, or is it just to take advantage of other ELB features, such as forwarding and security group rules? If the plan is to load balance multiple Registry instances, just be aware that you will probably run into some unexpected behavior. (As you mentioned using authorization, that is one case where I know the in-memory cache of the persisted data will not refresh across instances, so even if you were using some sort of shared network file system attached to multiple Registry instances, such as EFS, it would not work the way you hope.)
>>>>
>>>> Hope this helps,
>>>> Kevin
>>>>
>>>> On 3/19/18, 10:20, "Scott Howell" <sc...@mobilgov.com> wrote:
>>>>
>>>> Thanks for the quick response.
>>>>
>>>> A couple of things I am seeing.
>>>>
>>>> 1. There is no error, I don’t see anything in the logs once the service comes up. This is because the health check is not even hitting the instance when secure.
>>>>
>>>> 2. Nothing interesting in the nifi-registry-app.logs. That was my concern because on my nifi instance I can see the health check hitting the instance from the ELB. This does not happen on the nifi-registry instance. I see the service startup and it tells me what domain and port I can access the UI but nothing else after that.
>>>>
>>>> 3. When I am on an instances in the same private subnet I am able to curl to the instance I get the TLS SSL which tells me the keystore is on the server. I am using a JKS keystore that is self-signed by the company I work for.
>>>>
>>>>> On Mar 19, 2018, at 9:10 AM, Bryan Bende <bb...@gmail.com> wrote:
>>>>>
>>>>> Hello,
>>>>>
>>>>> What error are you getting when you cannot access the UI?
>>>>>
>>>>> Is there anything interesting in nifi-registry-app.log regarding
>>>>> authentication/authorization when this happens?
>>>>>
>>>>> Can you access the UI securely without going through the ELB?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Bryan
>>>>>
>>>>>
>>>>> On Mon, Mar 19, 2018 at 10:05 AM, Scott Howell <sc...@mobilgov.com> wrote:
>>>>>> I was able to stand up nifi-registry behind an AWS ELB non-secure. Everything was working great and was able to access the UI anonymously. I set up the authorization just like on my nifi instances along with the authorizers and identity-provider. The service comes up without errors and everything looks good but the health check does not pass and I cannot access the UI to login. I was wondering if anyone else has ran into this issue using nifi-registry.
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>>
>>
Re: Issue with AWS ELB on secure nifi-registry
Posted by Bryan Bende <bb...@gmail.com>.
It looks like that error would happen if your identity-providers.xml
contained invalid XML.
Did you start by modifying the identity-providers.xml file that was
already there? Can you share the file, or the contents (removing
anything sensitive)?
On Mon, Mar 19, 2018 at 1:09 PM, Scott Howell <sc...@mobilgov.com> wrote:
> So I was able to get the UI pulled up but now I am hitting a roadblock with my identity-provider.xml.
>
> I am getting a number of errors like this:
>
> Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'getIdentityProvider' defined in class path resource [org/apache/nifi/registry/security/authentication/IdentityProviderFactory.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.nifi.registry.security.authentication.IdentityProvider]: Factory method 'getIdentityProvider' threw exception; nested exception is java.lang.Exception: Unable to load the login identity provider configuration file at: /opt/nifi-registry-0.1.0/conf/identity-providers.xml
> at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:587) ~[na:na]
> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1250) ~[na:na]
> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1099) ~[na:na]
> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:545) ~[na:na]
> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:502) ~[na:na]
> at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:312) ~[na:na]
> at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228) ~[na:na]
> at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:310) ~[na:na]
> at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200) ~[na:na]
> at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:251) ~[na:na]
> at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1135) ~[na:na]
> at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1062) ~[na:na]
> at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:815) ~[na:na]
> at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:721) ~[na:na]
> ... 43 common frames omitted
>
> I know it has to do with the identity-provider.xml but I have my setup just like the documentation ask for. I turned on debug but was not able to see anything different or better explanation from it.
>
>
>> On Mar 19, 2018, at 10:06 AM, Kevin Doran <kd...@apache.org> wrote:
>>
>> Ok, that use case should be fine.
>>
>> If it were an authorization issue you would see something in the logs saying that an authorization attempt failed and the server is responding with a 403. Just to be sure, can you enable debug logging if you haven't already, i.e., in your nifi-registry/conf/logback.xml file, change 'org.apache.nifif.registry' to debug:
>>
>> <!-- valid logging levels: TRACE, DEBUG, INFO, WARN, ERROR -->
>> <logger name="org.apache.nifi.registry" level="DEBUG"/>
>>
>> If there is nothing being written to nifi-registry-app.log, it points towards a connection issue, so I would double check your host, port, and TLS settings. You'll have to get an HTTPS cert from a root CA or configure your ELB to trust your company's self-signed cert (again, not sure if/how to do this, but I assume there should be some way to configure it. It might require settings not exposed in the AWS web console.)
>>
>> On 3/19/18, 10:51, "Scott Howell" <sc...@mobilgov.com> wrote:
>>
>> Thanks Kevin,
>>
>> I am just using the ELB to go from the public subnet to the private subnet. I will not have multiple instances running of registry.
>>
>> I will say on my authorizers.xml there is one difference between my nifi instance. On my nifi instance I am using file-provider for nifi.security.user.authorizer in my nifi.properties. I don’t think from reading the documents for nifi-registry that I can use that. If there is a way that might be my problem. I was running into some issues with my nifi instance when I was using managed-authorizers instead of file-provider.
>>
>>
>>
>>> On Mar 19, 2018, at 9:35 AM, Kevin Doran <kd...@apache.org> wrote:
>>>
>>> Hey Scott,
>>>
>>> Assuming you are using two-way TLS with client certificates for authentication, I recommend configuring your ELB for TCP passthrough so that the TLS handshake is between the end-client and the NiFi Registry Server (in other words, no decryption/termination of the TLS connection happens in the ELB). If you are using some other form of authentication (e.g., LDAP), you will need to configure your ELB to trust the self-signed key NiFi Registry is using. I'm not sure how to do that as I've never run an ELB with that configuration before.
>>>
>>> Also, just a note about using an ELB with NiFi Registry:
>>>
>>> NiFi Registry is currently only supports single-instance use as persisted data and in-memory state is not synced between multiple instances. Are you hoping to use the ELB for actual load balancing, or is it just to take advantage of other ELB features, such as forwarding and security group rules? If the plan is to load balance multiple Registry instances, just be aware that you will probably run into some unexpected behavior. (As you mentioned using authorization, that is one case where I know the in-memory cache of the persisted data will not refresh across instances, so even if you were using some sort of shared network file system attached to multiple Registry instances, such as EFS, it would not work the way you hope.)
>>>
>>> Hope this helps,
>>> Kevin
>>>
>>> On 3/19/18, 10:20, "Scott Howell" <sc...@mobilgov.com> wrote:
>>>
>>> Thanks for the quick response.
>>>
>>> A couple of things I am seeing.
>>>
>>> 1. There is no error, I don’t see anything in the logs once the service comes up. This is because the health check is not even hitting the instance when secure.
>>>
>>> 2. Nothing interesting in the nifi-registry-app.logs. That was my concern because on my nifi instance I can see the health check hitting the instance from the ELB. This does not happen on the nifi-registry instance. I see the service startup and it tells me what domain and port I can access the UI but nothing else after that.
>>>
>>> 3. When I am on an instances in the same private subnet I am able to curl to the instance I get the TLS SSL which tells me the keystore is on the server. I am using a JKS keystore that is self-signed by the company I work for.
>>>
>>>> On Mar 19, 2018, at 9:10 AM, Bryan Bende <bb...@gmail.com> wrote:
>>>>
>>>> Hello,
>>>>
>>>> What error are you getting when you cannot access the UI?
>>>>
>>>> Is there anything interesting in nifi-registry-app.log regarding
>>>> authentication/authorization when this happens?
>>>>
>>>> Can you access the UI securely without going through the ELB?
>>>>
>>>> Thanks,
>>>>
>>>> Bryan
>>>>
>>>>
>>>> On Mon, Mar 19, 2018 at 10:05 AM, Scott Howell <sc...@mobilgov.com> wrote:
>>>>> I was able to stand up nifi-registry behind an AWS ELB non-secure. Everything was working great and was able to access the UI anonymously. I set up the authorization just like on my nifi instances along with the authorizers and identity-provider. The service comes up without errors and everything looks good but the health check does not pass and I cannot access the UI to login. I was wondering if anyone else has ran into this issue using nifi-registry.
>>>
>>>
>>>
>>>
>>
>>
>>
>>
>
Re: Issue with AWS ELB on secure nifi-registry
Posted by Scott Howell <sc...@mobilgov.com>.
So I was able to get the UI pulled up but now I am hitting a roadblock with my identity-provider.xml.
I am getting a number of errors like this:
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'getIdentityProvider' defined in class path resource [org/apache/nifi/registry/security/authentication/IdentityProviderFactory.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.nifi.registry.security.authentication.IdentityProvider]: Factory method 'getIdentityProvider' threw exception; nested exception is java.lang.Exception: Unable to load the login identity provider configuration file at: /opt/nifi-registry-0.1.0/conf/identity-providers.xml
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:587) ~[na:na]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1250) ~[na:na]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1099) ~[na:na]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:545) ~[na:na]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:502) ~[na:na]
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:312) ~[na:na]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228) ~[na:na]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:310) ~[na:na]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200) ~[na:na]
at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:251) ~[na:na]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1135) ~[na:na]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1062) ~[na:na]
at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:815) ~[na:na]
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:721) ~[na:na]
... 43 common frames omitted
I know it has to do with the identity-provider.xml but I have my setup just like the documentation ask for. I turned on debug but was not able to see anything different or better explanation from it.
> On Mar 19, 2018, at 10:06 AM, Kevin Doran <kd...@apache.org> wrote:
>
> Ok, that use case should be fine.
>
> If it were an authorization issue you would see something in the logs saying that an authorization attempt failed and the server is responding with a 403. Just to be sure, can you enable debug logging if you haven't already, i.e., in your nifi-registry/conf/logback.xml file, change 'org.apache.nifif.registry' to debug:
>
> <!-- valid logging levels: TRACE, DEBUG, INFO, WARN, ERROR -->
> <logger name="org.apache.nifi.registry" level="DEBUG"/>
>
> If there is nothing being written to nifi-registry-app.log, it points towards a connection issue, so I would double check your host, port, and TLS settings. You'll have to get an HTTPS cert from a root CA or configure your ELB to trust your company's self-signed cert (again, not sure if/how to do this, but I assume there should be some way to configure it. It might require settings not exposed in the AWS web console.)
>
> On 3/19/18, 10:51, "Scott Howell" <sc...@mobilgov.com> wrote:
>
> Thanks Kevin,
>
> I am just using the ELB to go from the public subnet to the private subnet. I will not have multiple instances running of registry.
>
> I will say on my authorizers.xml there is one difference between my nifi instance. On my nifi instance I am using file-provider for nifi.security.user.authorizer in my nifi.properties. I don’t think from reading the documents for nifi-registry that I can use that. If there is a way that might be my problem. I was running into some issues with my nifi instance when I was using managed-authorizers instead of file-provider.
>
>
>
>> On Mar 19, 2018, at 9:35 AM, Kevin Doran <kd...@apache.org> wrote:
>>
>> Hey Scott,
>>
>> Assuming you are using two-way TLS with client certificates for authentication, I recommend configuring your ELB for TCP passthrough so that the TLS handshake is between the end-client and the NiFi Registry Server (in other words, no decryption/termination of the TLS connection happens in the ELB). If you are using some other form of authentication (e.g., LDAP), you will need to configure your ELB to trust the self-signed key NiFi Registry is using. I'm not sure how to do that as I've never run an ELB with that configuration before.
>>
>> Also, just a note about using an ELB with NiFi Registry:
>>
>> NiFi Registry is currently only supports single-instance use as persisted data and in-memory state is not synced between multiple instances. Are you hoping to use the ELB for actual load balancing, or is it just to take advantage of other ELB features, such as forwarding and security group rules? If the plan is to load balance multiple Registry instances, just be aware that you will probably run into some unexpected behavior. (As you mentioned using authorization, that is one case where I know the in-memory cache of the persisted data will not refresh across instances, so even if you were using some sort of shared network file system attached to multiple Registry instances, such as EFS, it would not work the way you hope.)
>>
>> Hope this helps,
>> Kevin
>>
>> On 3/19/18, 10:20, "Scott Howell" <sc...@mobilgov.com> wrote:
>>
>> Thanks for the quick response.
>>
>> A couple of things I am seeing.
>>
>> 1. There is no error, I don’t see anything in the logs once the service comes up. This is because the health check is not even hitting the instance when secure.
>>
>> 2. Nothing interesting in the nifi-registry-app.logs. That was my concern because on my nifi instance I can see the health check hitting the instance from the ELB. This does not happen on the nifi-registry instance. I see the service startup and it tells me what domain and port I can access the UI but nothing else after that.
>>
>> 3. When I am on an instances in the same private subnet I am able to curl to the instance I get the TLS SSL which tells me the keystore is on the server. I am using a JKS keystore that is self-signed by the company I work for.
>>
>>> On Mar 19, 2018, at 9:10 AM, Bryan Bende <bb...@gmail.com> wrote:
>>>
>>> Hello,
>>>
>>> What error are you getting when you cannot access the UI?
>>>
>>> Is there anything interesting in nifi-registry-app.log regarding
>>> authentication/authorization when this happens?
>>>
>>> Can you access the UI securely without going through the ELB?
>>>
>>> Thanks,
>>>
>>> Bryan
>>>
>>>
>>> On Mon, Mar 19, 2018 at 10:05 AM, Scott Howell <sc...@mobilgov.com> wrote:
>>>> I was able to stand up nifi-registry behind an AWS ELB non-secure. Everything was working great and was able to access the UI anonymously. I set up the authorization just like on my nifi instances along with the authorizers and identity-provider. The service comes up without errors and everything looks good but the health check does not pass and I cannot access the UI to login. I was wondering if anyone else has ran into this issue using nifi-registry.
>>
>>
>>
>>
>
>
>
>
Re: Issue with AWS ELB on secure nifi-registry
Posted by Kevin Doran <kd...@apache.org>.
Ok, that use case should be fine.
If it were an authorization issue you would see something in the logs saying that an authorization attempt failed and the server is responding with a 403. Just to be sure, can you enable debug logging if you haven't already, i.e., in your nifi-registry/conf/logback.xml file, change 'org.apache.nifif.registry' to debug:
<!-- valid logging levels: TRACE, DEBUG, INFO, WARN, ERROR -->
<logger name="org.apache.nifi.registry" level="DEBUG"/>
If there is nothing being written to nifi-registry-app.log, it points towards a connection issue, so I would double check your host, port, and TLS settings. You'll have to get an HTTPS cert from a root CA or configure your ELB to trust your company's self-signed cert (again, not sure if/how to do this, but I assume there should be some way to configure it. It might require settings not exposed in the AWS web console.)
On 3/19/18, 10:51, "Scott Howell" <sc...@mobilgov.com> wrote:
Thanks Kevin,
I am just using the ELB to go from the public subnet to the private subnet. I will not have multiple instances running of registry.
I will say on my authorizers.xml there is one difference between my nifi instance. On my nifi instance I am using file-provider for nifi.security.user.authorizer in my nifi.properties. I don’t think from reading the documents for nifi-registry that I can use that. If there is a way that might be my problem. I was running into some issues with my nifi instance when I was using managed-authorizers instead of file-provider.
> On Mar 19, 2018, at 9:35 AM, Kevin Doran <kd...@apache.org> wrote:
>
> Hey Scott,
>
> Assuming you are using two-way TLS with client certificates for authentication, I recommend configuring your ELB for TCP passthrough so that the TLS handshake is between the end-client and the NiFi Registry Server (in other words, no decryption/termination of the TLS connection happens in the ELB). If you are using some other form of authentication (e.g., LDAP), you will need to configure your ELB to trust the self-signed key NiFi Registry is using. I'm not sure how to do that as I've never run an ELB with that configuration before.
>
> Also, just a note about using an ELB with NiFi Registry:
>
> NiFi Registry is currently only supports single-instance use as persisted data and in-memory state is not synced between multiple instances. Are you hoping to use the ELB for actual load balancing, or is it just to take advantage of other ELB features, such as forwarding and security group rules? If the plan is to load balance multiple Registry instances, just be aware that you will probably run into some unexpected behavior. (As you mentioned using authorization, that is one case where I know the in-memory cache of the persisted data will not refresh across instances, so even if you were using some sort of shared network file system attached to multiple Registry instances, such as EFS, it would not work the way you hope.)
>
> Hope this helps,
> Kevin
>
> On 3/19/18, 10:20, "Scott Howell" <sc...@mobilgov.com> wrote:
>
> Thanks for the quick response.
>
> A couple of things I am seeing.
>
> 1. There is no error, I don’t see anything in the logs once the service comes up. This is because the health check is not even hitting the instance when secure.
>
> 2. Nothing interesting in the nifi-registry-app.logs. That was my concern because on my nifi instance I can see the health check hitting the instance from the ELB. This does not happen on the nifi-registry instance. I see the service startup and it tells me what domain and port I can access the UI but nothing else after that.
>
> 3. When I am on an instances in the same private subnet I am able to curl to the instance I get the TLS SSL which tells me the keystore is on the server. I am using a JKS keystore that is self-signed by the company I work for.
>
>> On Mar 19, 2018, at 9:10 AM, Bryan Bende <bb...@gmail.com> wrote:
>>
>> Hello,
>>
>> What error are you getting when you cannot access the UI?
>>
>> Is there anything interesting in nifi-registry-app.log regarding
>> authentication/authorization when this happens?
>>
>> Can you access the UI securely without going through the ELB?
>>
>> Thanks,
>>
>> Bryan
>>
>>
>> On Mon, Mar 19, 2018 at 10:05 AM, Scott Howell <sc...@mobilgov.com> wrote:
>>> I was able to stand up nifi-registry behind an AWS ELB non-secure. Everything was working great and was able to access the UI anonymously. I set up the authorization just like on my nifi instances along with the authorizers and identity-provider. The service comes up without errors and everything looks good but the health check does not pass and I cannot access the UI to login. I was wondering if anyone else has ran into this issue using nifi-registry.
>
>
>
>
Re: Issue with AWS ELB on secure nifi-registry
Posted by Scott Howell <sc...@mobilgov.com>.
Thanks Kevin,
I am just using the ELB to go from the public subnet to the private subnet. I will not have multiple instances running of registry.
I will say on my authorizers.xml there is one difference between my nifi instance. On my nifi instance I am using file-provider for nifi.security.user.authorizer in my nifi.properties. I don’t think from reading the documents for nifi-registry that I can use that. If there is a way that might be my problem. I was running into some issues with my nifi instance when I was using managed-authorizers instead of file-provider.
> On Mar 19, 2018, at 9:35 AM, Kevin Doran <kd...@apache.org> wrote:
>
> Hey Scott,
>
> Assuming you are using two-way TLS with client certificates for authentication, I recommend configuring your ELB for TCP passthrough so that the TLS handshake is between the end-client and the NiFi Registry Server (in other words, no decryption/termination of the TLS connection happens in the ELB). If you are using some other form of authentication (e.g., LDAP), you will need to configure your ELB to trust the self-signed key NiFi Registry is using. I'm not sure how to do that as I've never run an ELB with that configuration before.
>
> Also, just a note about using an ELB with NiFi Registry:
>
> NiFi Registry is currently only supports single-instance use as persisted data and in-memory state is not synced between multiple instances. Are you hoping to use the ELB for actual load balancing, or is it just to take advantage of other ELB features, such as forwarding and security group rules? If the plan is to load balance multiple Registry instances, just be aware that you will probably run into some unexpected behavior. (As you mentioned using authorization, that is one case where I know the in-memory cache of the persisted data will not refresh across instances, so even if you were using some sort of shared network file system attached to multiple Registry instances, such as EFS, it would not work the way you hope.)
>
> Hope this helps,
> Kevin
>
> On 3/19/18, 10:20, "Scott Howell" <sc...@mobilgov.com> wrote:
>
> Thanks for the quick response.
>
> A couple of things I am seeing.
>
> 1. There is no error, I don’t see anything in the logs once the service comes up. This is because the health check is not even hitting the instance when secure.
>
> 2. Nothing interesting in the nifi-registry-app.logs. That was my concern because on my nifi instance I can see the health check hitting the instance from the ELB. This does not happen on the nifi-registry instance. I see the service startup and it tells me what domain and port I can access the UI but nothing else after that.
>
> 3. When I am on an instances in the same private subnet I am able to curl to the instance I get the TLS SSL which tells me the keystore is on the server. I am using a JKS keystore that is self-signed by the company I work for.
>
>> On Mar 19, 2018, at 9:10 AM, Bryan Bende <bb...@gmail.com> wrote:
>>
>> Hello,
>>
>> What error are you getting when you cannot access the UI?
>>
>> Is there anything interesting in nifi-registry-app.log regarding
>> authentication/authorization when this happens?
>>
>> Can you access the UI securely without going through the ELB?
>>
>> Thanks,
>>
>> Bryan
>>
>>
>> On Mon, Mar 19, 2018 at 10:05 AM, Scott Howell <sc...@mobilgov.com> wrote:
>>> I was able to stand up nifi-registry behind an AWS ELB non-secure. Everything was working great and was able to access the UI anonymously. I set up the authorization just like on my nifi instances along with the authorizers and identity-provider. The service comes up without errors and everything looks good but the health check does not pass and I cannot access the UI to login. I was wondering if anyone else has ran into this issue using nifi-registry.
>
>
>
>
Re: Issue with AWS ELB on secure nifi-registry
Posted by Kevin Doran <kd...@apache.org>.
Hey Scott,
Assuming you are using two-way TLS with client certificates for authentication, I recommend configuring your ELB for TCP passthrough so that the TLS handshake is between the end-client and the NiFi Registry Server (in other words, no decryption/termination of the TLS connection happens in the ELB). If you are using some other form of authentication (e.g., LDAP), you will need to configure your ELB to trust the self-signed key NiFi Registry is using. I'm not sure how to do that as I've never run an ELB with that configuration before.
Also, just a note about using an ELB with NiFi Registry:
NiFi Registry is currently only supports single-instance use as persisted data and in-memory state is not synced between multiple instances. Are you hoping to use the ELB for actual load balancing, or is it just to take advantage of other ELB features, such as forwarding and security group rules? If the plan is to load balance multiple Registry instances, just be aware that you will probably run into some unexpected behavior. (As you mentioned using authorization, that is one case where I know the in-memory cache of the persisted data will not refresh across instances, so even if you were using some sort of shared network file system attached to multiple Registry instances, such as EFS, it would not work the way you hope.)
Hope this helps,
Kevin
On 3/19/18, 10:20, "Scott Howell" <sc...@mobilgov.com> wrote:
Thanks for the quick response.
A couple of things I am seeing.
1. There is no error, I don’t see anything in the logs once the service comes up. This is because the health check is not even hitting the instance when secure.
2. Nothing interesting in the nifi-registry-app.logs. That was my concern because on my nifi instance I can see the health check hitting the instance from the ELB. This does not happen on the nifi-registry instance. I see the service startup and it tells me what domain and port I can access the UI but nothing else after that.
3. When I am on an instances in the same private subnet I am able to curl to the instance I get the TLS SSL which tells me the keystore is on the server. I am using a JKS keystore that is self-signed by the company I work for.
> On Mar 19, 2018, at 9:10 AM, Bryan Bende <bb...@gmail.com> wrote:
>
> Hello,
>
> What error are you getting when you cannot access the UI?
>
> Is there anything interesting in nifi-registry-app.log regarding
> authentication/authorization when this happens?
>
> Can you access the UI securely without going through the ELB?
>
> Thanks,
>
> Bryan
>
>
> On Mon, Mar 19, 2018 at 10:05 AM, Scott Howell <sc...@mobilgov.com> wrote:
>> I was able to stand up nifi-registry behind an AWS ELB non-secure. Everything was working great and was able to access the UI anonymously. I set up the authorization just like on my nifi instances along with the authorizers and identity-provider. The service comes up without errors and everything looks good but the health check does not pass and I cannot access the UI to login. I was wondering if anyone else has ran into this issue using nifi-registry.
Re: Issue with AWS ELB on secure nifi-registry
Posted by Scott Howell <sc...@mobilgov.com>.
Thanks for the quick response.
A couple of things I am seeing.
1. There is no error, I don’t see anything in the logs once the service comes up. This is because the health check is not even hitting the instance when secure.
2. Nothing interesting in the nifi-registry-app.logs. That was my concern because on my nifi instance I can see the health check hitting the instance from the ELB. This does not happen on the nifi-registry instance. I see the service startup and it tells me what domain and port I can access the UI but nothing else after that.
3. When I am on an instances in the same private subnet I am able to curl to the instance I get the TLS SSL which tells me the keystore is on the server. I am using a JKS keystore that is self-signed by the company I work for.
> On Mar 19, 2018, at 9:10 AM, Bryan Bende <bb...@gmail.com> wrote:
>
> Hello,
>
> What error are you getting when you cannot access the UI?
>
> Is there anything interesting in nifi-registry-app.log regarding
> authentication/authorization when this happens?
>
> Can you access the UI securely without going through the ELB?
>
> Thanks,
>
> Bryan
>
>
> On Mon, Mar 19, 2018 at 10:05 AM, Scott Howell <sc...@mobilgov.com> wrote:
>> I was able to stand up nifi-registry behind an AWS ELB non-secure. Everything was working great and was able to access the UI anonymously. I set up the authorization just like on my nifi instances along with the authorizers and identity-provider. The service comes up without errors and everything looks good but the health check does not pass and I cannot access the UI to login. I was wondering if anyone else has ran into this issue using nifi-registry.
Re: Issue with AWS ELB on secure nifi-registry
Posted by Bryan Bende <bb...@gmail.com>.
Hello,
What error are you getting when you cannot access the UI?
Is there anything interesting in nifi-registry-app.log regarding
authentication/authorization when this happens?
Can you access the UI securely without going through the ELB?
Thanks,
Bryan
On Mon, Mar 19, 2018 at 10:05 AM, Scott Howell <sc...@mobilgov.com> wrote:
> I was able to stand up nifi-registry behind an AWS ELB non-secure. Everything was working great and was able to access the UI anonymously. I set up the authorization just like on my nifi instances along with the authorizers and identity-provider. The service comes up without errors and everything looks good but the health check does not pass and I cannot access the UI to login. I was wondering if anyone else has ran into this issue using nifi-registry.