You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Sumanth Chinthagunta <xm...@gmail.com> on 2016/03/14 19:18:28 UTC
Re: NiFi error with embedded ZooKeeper for State Management
Thanks Mark.
I had java.arg.15 setting in bootstrap.conf from NiFi 0.4.x to make putHDFS processor work with my MapR cluster. If I remove it, putHDFS will fail.
Now with NiFi 0.5.1,
if I set nifi.state.management.embedded.zookeeper.start=false ,
can I point to dedicated external ZooKeeper in conf/state-management.xml without interfering with java.arg.15 setting that is used by putHDFS?
Thanks
Sumo
> On Mar 14, 2016, at 10:10 AM, Mark Payne <ma...@hotmail.com> wrote:
>
> Sumo,
>
> If your intent is to use an external ZooKeeper, you should not be starting the embedded ZooKeeper.
> You will also not want to set the java.arg.15 parameter there to point to a login config file, as that is
> necessary only when enabling Kerberos - not for use when security disabled.
>
> So you would want to change the following in your config:
> - Remove the java.arg.15 parameter from bootstrap.conf
> - Set the nifi.state.management.embedded.zookeeper.start property to false
> - Change the conf/state-management.xml to point to the external ZooKeeper via the Connect String property.
>
> Does all of this make sense?
>
> Thanks
> -Mark
>
>
>> On Mar 14, 2016, at 12:59 PM, Sumanth Chinthagunta <xm...@gmail.com> wrote:
>>
>> I am getting following error when NiFi Cluster started with embedded ZooKeeper enabled.
>> I would like to configure external ZooKeeper for NiFi State Management without interfering with java.security.auth.login.config setting for MapR HDFS. Can somebody provide me instructions how to configure external ZooKeeper for NiFi cluster State Management without security enabled?
>>
>> Env :
>> NiFi 0.5.1 cluster (NCM, Node1, Node2)
>> I have following line in bootstrap.conf
>> # ***For MapR HDFS***
>> java.arg.15=-Djava.security.auth.login.config=/opt/mapr/conf/mapr.login.conf
>>
>> /opt/mapr/conf/mapr.login.conf has
>> /**
>> * Used by Zookeeper
>> */
>> Server {
>> com.mapr.security.maprsasl.MaprSecurityLoginModule required
>> checkUGI=false
>> cldbkeylocation="/opt/mapr/conf/cldb.key"
>> debug=true;
>> };
>>
>> I have following in zookeeper.properties
>> server.2=myhost2:2888:3888
>>
>>
>> 2016-03-12 16:39:07,518 INFO [Framework Task Thread Thread-1] o.a.zookeeper.server.ZooKeeperServer Server environment:java.io.tmpdir=/tmp
>> 2016-03-12 16:39:07,518 INFO [Framework Task Thread Thread-1] o.a.zookeeper.server.ZooKeeperServer Server environment:java.compiler=<NA>
>> 2016-03-12 16:39:07,518 INFO [Framework Task Thread Thread-1] o.a.zookeeper.server.ZooKeeperServer Server environment:os.name=Linux
>> 2016-03-12 16:39:07,518 INFO [Framework Task Thread Thread-1] o.a.zookeeper.server.ZooKeeperServer Server environment:os.arch=amd64
>> 2016-03-12 16:39:07,518 INFO [Framework Task Thread Thread-1] o.a.zookeeper.server.ZooKeeperServer Server environment:os.version=2.6.32-573.3.1.el6.x86_64
>> 2016-03-12 16:39:07,518 INFO [Framework Task Thread Thread-1] o.a.zookeeper.server.ZooKeeperServer Server environment:user.name=sumo
>> 2016-03-12 16:39:07,518 INFO [Framework Task Thread Thread-1] o.a.zookeeper.server.ZooKeeperServer Server environment:user.home=/home/ sumo
>> 2016-03-12 16:39:07,518 INFO [Framework Task Thread Thread-1] o.a.zookeeper.server.ZooKeeperServer Server environment:user.dir=/app/runtime/nifi-node2
>> 2016-03-12 16:39:07,519 INFO [Framework Task Thread Thread-1] o.a.zookeeper.server.ZooKeeperServer tickTime set to 2000
>> 2016-03-12 16:39:07,519 INFO [Framework Task Thread Thread-1] o.a.zookeeper.server.ZooKeeperServer minSessionTimeout set to -1
>> 2016-03-12 16:39:07,519 INFO [Framework Task Thread Thread-1] o.a.zookeeper.server.ZooKeeperServer maxSessionTimeout set to -1
>> 2016-03-12 16:39:07,539 ERROR [Framework Task Thread Thread-1] o.apache.nifi.controller.FlowController NiFi was connected to the cluster but failed to start embedded ZooKeeper Server
>> java.io.IOException: Failed to start embedded ZooKeeper Server
>> at org.apache.nifi.controller.state.server.ZooKeeperStateServer.startStandalone(ZooKeeperStateServer.java:87) ~[na:na]
>> at org.apache.nifi.controller.state.server.ZooKeeperStateServer.start(ZooKeeperStateServer.java:60) ~[na:na]
>> at org.apache.nifi.controller.FlowController$5.run(FlowController.java:3145) ~[na:na]
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_65]
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_65]
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_65]
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_65]
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_65]
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_65]
>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
>> Caused by: java.io.IOException: Could not configure server because SASL configuration did not allow the ZooKeeper server to authenticate itself properly: javax.security.auth.login.LoginException: unable to find LoginModule class: com.mapr.security.maprsasl.MaprSecurityLoginModule
>> at org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:207) ~[na:na]
>> at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:87) ~[na:na]
>> at org.apache.nifi.controller.state.server.ZooKeeperStateServer.startStandalone(ZooKeeperStateServer.java:81) ~[na:na]
>> ... 9 common frames omitted
>>
>> Thanks
>> Sumo
>
Re: NiFi error with embedded ZooKeeper for State Management
Posted by Mark Payne <ma...@hotmail.com>.
Sumo,
I do believe that you should be okay leaving that java.arg.15 in, in that case. The ZooKeeper Client
will use the "Client" section of that file, if it is defined. In the example that you provided, that Client section
does not exist, so it is okay. What would not work in this case would be to use an Embedded ZooKeeper Server
without kerberos, but it sounds like that is not what you are after, so I believe you should be in
good shape with the setup that you described.
Of course, if you do run into specific issues running this way, please let us know!
Thanks
-Mark
> On Mar 14, 2016, at 2:18 PM, Sumanth Chinthagunta <xm...@gmail.com> wrote:
>
> Thanks Mark.
> I had java.arg.15 setting in bootstrap.conf from NiFi 0.4.x to make putHDFS processor work with my MapR cluster. If I remove it, putHDFS will fail.
>
> Now with NiFi 0.5.1,
> if I set nifi.state.management.embedded.zookeeper.start=false ,
> can I point to dedicated external ZooKeeper in conf/state-management.xml without interfering with java.arg.15 setting that is used by putHDFS?
> Thanks
> Sumo
>
>
>> On Mar 14, 2016, at 10:10 AM, Mark Payne <ma...@hotmail.com> wrote:
>>
>> Sumo,
>>
>> If your intent is to use an external ZooKeeper, you should not be starting the embedded ZooKeeper.
>> You will also not want to set the java.arg.15 parameter there to point to a login config file, as that is
>> necessary only when enabling Kerberos - not for use when security disabled.
>>
>> So you would want to change the following in your config:
>> - Remove the java.arg.15 parameter from bootstrap.conf
>> - Set the nifi.state.management.embedded.zookeeper.start property to false
>> - Change the conf/state-management.xml to point to the external ZooKeeper via the Connect String property.
>>
>> Does all of this make sense?
>>
>> Thanks
>> -Mark
>>
>>
>>> On Mar 14, 2016, at 12:59 PM, Sumanth Chinthagunta <xm...@gmail.com> wrote:
>>>
>>> I am getting following error when NiFi Cluster started with embedded ZooKeeper enabled.
>>> I would like to configure external ZooKeeper for NiFi State Management without interfering with java.security.auth.login.config setting for MapR HDFS. Can somebody provide me instructions how to configure external ZooKeeper for NiFi cluster State Management without security enabled?
>>>
>>> Env :
>>> NiFi 0.5.1 cluster (NCM, Node1, Node2)
>>> I have following line in bootstrap.conf
>>> # ***For MapR HDFS***
>>> java.arg.15=-Djava.security.auth.login.config=/opt/mapr/conf/mapr.login.conf
>>>
>>> /opt/mapr/conf/mapr.login.conf has
>>> /**
>>> * Used by Zookeeper
>>> */
>>> Server {
>>> com.mapr.security.maprsasl.MaprSecurityLoginModule required
>>> checkUGI=false
>>> cldbkeylocation="/opt/mapr/conf/cldb.key"
>>> debug=true;
>>> };
>>>
>>> I have following in zookeeper.properties
>>> server.2=myhost2:2888:3888
>>>
>>>
>>> 2016-03-12 16:39:07,518 INFO [Framework Task Thread Thread-1] o.a.zookeeper.server.ZooKeeperServer Server environment:java.io.tmpdir=/tmp
>>> 2016-03-12 16:39:07,518 INFO [Framework Task Thread Thread-1] o.a.zookeeper.server.ZooKeeperServer Server environment:java.compiler=<NA>
>>> 2016-03-12 16:39:07,518 INFO [Framework Task Thread Thread-1] o.a.zookeeper.server.ZooKeeperServer Server environment:os.name=Linux
>>> 2016-03-12 16:39:07,518 INFO [Framework Task Thread Thread-1] o.a.zookeeper.server.ZooKeeperServer Server environment:os.arch=amd64
>>> 2016-03-12 16:39:07,518 INFO [Framework Task Thread Thread-1] o.a.zookeeper.server.ZooKeeperServer Server environment:os.version=2.6.32-573.3.1.el6.x86_64
>>> 2016-03-12 16:39:07,518 INFO [Framework Task Thread Thread-1] o.a.zookeeper.server.ZooKeeperServer Server environment:user.name=sumo
>>> 2016-03-12 16:39:07,518 INFO [Framework Task Thread Thread-1] o.a.zookeeper.server.ZooKeeperServer Server environment:user.home=/home/ sumo
>>> 2016-03-12 16:39:07,518 INFO [Framework Task Thread Thread-1] o.a.zookeeper.server.ZooKeeperServer Server environment:user.dir=/app/runtime/nifi-node2
>>> 2016-03-12 16:39:07,519 INFO [Framework Task Thread Thread-1] o.a.zookeeper.server.ZooKeeperServer tickTime set to 2000
>>> 2016-03-12 16:39:07,519 INFO [Framework Task Thread Thread-1] o.a.zookeeper.server.ZooKeeperServer minSessionTimeout set to -1
>>> 2016-03-12 16:39:07,519 INFO [Framework Task Thread Thread-1] o.a.zookeeper.server.ZooKeeperServer maxSessionTimeout set to -1
>>> 2016-03-12 16:39:07,539 ERROR [Framework Task Thread Thread-1] o.apache.nifi.controller.FlowController NiFi was connected to the cluster but failed to start embedded ZooKeeper Server
>>> java.io.IOException: Failed to start embedded ZooKeeper Server
>>> at org.apache.nifi.controller.state.server.ZooKeeperStateServer.startStandalone(ZooKeeperStateServer.java:87) ~[na:na]
>>> at org.apache.nifi.controller.state.server.ZooKeeperStateServer.start(ZooKeeperStateServer.java:60) ~[na:na]
>>> at org.apache.nifi.controller.FlowController$5.run(FlowController.java:3145) ~[na:na]
>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_65]
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_65]
>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_65]
>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_65]
>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_65]
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_65]
>>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
>>> Caused by: java.io.IOException: Could not configure server because SASL configuration did not allow the ZooKeeper server to authenticate itself properly: javax.security.auth.login.LoginException: unable to find LoginModule class: com.mapr.security.maprsasl.MaprSecurityLoginModule
>>> at org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:207) ~[na:na]
>>> at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:87) ~[na:na]
>>> at org.apache.nifi.controller.state.server.ZooKeeperStateServer.startStandalone(ZooKeeperStateServer.java:81) ~[na:na]
>>> ... 9 common frames omitted
>>>
>>> Thanks
>>> Sumo
>>
>