You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "Pierre Villard (JIRA)" <ji...@apache.org> on 2017/09/05 20:12:00 UTC
[jira] [Resolved] (NIFI-4318) Processor cannot be stopped when
Kerberos authentication default to prompt
[ https://issues.apache.org/jira/browse/NIFI-4318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pierre Villard resolved NIFI-4318.
----------------------------------
Resolution: Not A Bug
Closing as not a bug.
For the Hive processor, it's been solved using the validate query parameter in the controller service.
For the HDFS processor, it was due to a relogin period time of 4 hours with a lifetime of 8 hours for generated tickets. The processor was not scheduled to run frequently and it was possible to have an execution of the processor without renewing the ticket. This is due to the unnecessary relogin period parameter that should be removed as it is already handled by the hadoop library (renewal will happen at 80% of ticket lifetime). Will raise a specific JIRA for that.
> Processor cannot be stopped when Kerberos authentication default to prompt
> --------------------------------------------------------------------------
>
> Key: NIFI-4318
> URL: https://issues.apache.org/jira/browse/NIFI-4318
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
> Affects Versions: 1.3.0
> Environment: 3-nodes cluster
> Reporter: Pierre Villard
> Attachments: image001.png, thread-2.txt, thread.txt
>
>
> I was unable to stop a PutHiveQL processor (it was showing running threads and remained in this state at least half an hour). I had to restart NiFi to solve the situation. It looks like the Kerberos authentication mechanism is falling back to manual user input and wait for some input (see below promptForName):
> {noformat}
> "Timer-Driven Process Thread-2" Id=139 RUNNABLE (in native code)
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:255)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - waiting on java.io.BufferedInputStream@2e2d3f92
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
> - waiting on java.io.InputStreamReader@64628fdf
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:161)
> at java.io.BufferedReader.readLine(BufferedReader.java:324)
> - waiting on java.io.InputStreamReader@64628fdf
> at java.io.BufferedReader.readLine(BufferedReader.java:389)
> at com.sun.security.auth.callback.TextCallbackHandler.readLine(TextCallbackHandler.java:153)
> at com.sun.security.auth.callback.TextCallbackHandler.handle(TextCallbackHandler.java:120)
> at com.sun.security.auth.module.Krb5LoginModule.promptForName(Krb5LoginModule.java:858)
> at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:704)
> at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
> at sun.reflect.GeneratedMethodAccessor597.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
> at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
> at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
> at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
> at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
> at sun.security.jgss.GSSUtil.login(GSSUtil.java:258)
> at sun.security.jgss.krb5.Krb5Util.getTicket(Krb5Util.java:158)
> at sun.security.jgss.krb5.Krb5InitCredential$1.run(Krb5InitCredential.java:335)
> at sun.security.jgss.krb5.Krb5InitCredential$1.run(Krb5InitCredential.java:331)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.security.jgss.krb5.Krb5InitCredential.getTgt(Krb5InitCredential.java:330)
> at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:145)
> at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
> at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
> at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
> at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
> at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
> at org.apache.hive.service.auth.HttpAuthUtils$HttpKerberosClientAction.run(HttpAuthUtils.java:183)
> at org.apache.hive.service.auth.HttpAuthUtils$HttpKerberosClientAction.run(HttpAuthUtils.java:151)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
> at org.apache.hive.service.auth.HttpAuthUtils.getKerberosServiceTicket(HttpAuthUtils.java:83)
> at org.apache.hive.jdbc.HttpKerberosRequestInterceptor.addHttpAuthHeader(HttpKerberosRequestInterceptor.java:62)
> at org.apache.hive.jdbc.HttpRequestInterceptorBase.process(HttpRequestInterceptorBase.java:74)
> at org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:132)
> at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:183)
> at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
> at org.apache.http.impl.execchain.ServiceUnavailableRetryExec.execute(ServiceUnavailableRetryExec.java:85)
> at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
> at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:118)
> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
> at org.apache.thrift.transport.THttpClient.flushUsingHttpClient(THttpClient.java:251)
> at org.apache.thrift.transport.THttpClient.flush(THttpClient.java:313)
> at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:73)
> at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
> at org.apache.hive.service.cli.thrift.TCLIService$Client.send_ExecuteStatement(TCLIService.java:223)
> at org.apache.hive.service.cli.thrift.TCLIService$Client.ExecuteStatement(TCLIService.java:215)
> at sun.reflect.GeneratedMethodAccessor504.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hive.jdbc.HiveConnection$SynchronizedHandler.invoke(HiveConnection.java:1374)
> at com.sun.proxy.$Proxy361.ExecuteStatement(Unknown Source)
> at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:299)
> at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:241)
> at org.apache.hive.jdbc.HivePreparedStatement.execute(HivePreparedStatement.java:98)
> at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172)
> at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172)
> at org.apache.nifi.processors.hive.PutHiveQL.lambda$null$3(PutHiveQL.java:218)
> at org.apache.nifi.processors.hive.PutHiveQL$$Lambda$507/743570245.apply(Unknown Source)
> at org.apache.nifi.processor.util.pattern.ExceptionHandler.execute(ExceptionHandler.java:127)
> at org.apache.nifi.processors.hive.PutHiveQL.lambda$new$4(PutHiveQL.java:199)
> at org.apache.nifi.processors.hive.PutHiveQL$$Lambda$76/1354314579.apply(Unknown Source)
> at org.apache.nifi.processor.util.pattern.Put.putFlowFiles(Put.java:59)
> at org.apache.nifi.processor.util.pattern.Put.onTrigger(Put.java:101)
> at org.apache.nifi.processors.hive.PutHiveQL.lambda$onTrigger$6(PutHiveQL.java:255)
> at org.apache.nifi.processors.hive.PutHiveQL$$Lambda$503/1913915475.execute(Unknown Source)
> at org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114)
> at org.apache.nifi.processor.util.pattern.RollbackOnFailure.onTrigger(RollbackOnFailure.java:184)
> at org.apache.nifi.processors.hive.PutHiveQL.onTrigger(PutHiveQL.java:255)
> at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1118)
> at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
> at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> Number of Locked Synchronizers: 3
> - java.util.concurrent.locks.ReentrantLock$FairSync@179f2932
> - java.util.concurrent.locks.ReentrantLock$FairSync@5417b82e
> - java.util.concurrent.ThreadPoolExecutor$Worker@30eaacc3
> {noformat}
> I faced the same issue today with ListHDFS processor. Here is the extract from the thread dump:
> {noformat}
> "Timer-Driven Process Thread-4" Id=160 RUNNABLE (in native code)
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:255)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - waiting on java.io.BufferedInputStream@36e17d2a
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
> - waiting on java.io.InputStreamReader@3ae79dc6
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:161)
> at java.io.BufferedReader.readLine(BufferedReader.java:324)
> - waiting on java.io.InputStreamReader@3ae79dc6
> at java.io.BufferedReader.readLine(BufferedReader.java:389)
> at com.sun.security.auth.callback.TextCallbackHandler.readLine(TextCallbackHandler.java:153)
> at com.sun.security.auth.callback.TextCallbackHandler.handle(TextCallbackHandler.java:120)
> at com.sun.security.auth.module.Krb5LoginModule.promptForName(Krb5LoginModule.java:858)
> at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:704)
> at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
> at sun.reflect.GeneratedMethodAccessor94.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
> at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
> at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
> at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
> at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
> at sun.security.jgss.GSSUtil.login(GSSUtil.java:258)
> at sun.security.jgss.krb5.Krb5Util.getTicket(Krb5Util.java:158)
> at sun.security.jgss.krb5.Krb5InitCredential$1.run(Krb5InitCredential.java:335)
> at sun.security.jgss.krb5.Krb5InitCredential$1.run(Krb5InitCredential.java:331)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.security.jgss.krb5.Krb5InitCredential.getTgt(Krb5InitCredential.java:330)
> at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:145)
> at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
> at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
> at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
> at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
> at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
> at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
> at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:414)
> at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:560)
> - waiting on org.apache.hadoop.ipc.Client$Connection@74a4e314
> at org.apache.hadoop.ipc.Client$Connection.access$1900(Client.java:375)
> at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:729)
> at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:725)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
> at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:725)
> - waiting on org.apache.hadoop.ipc.Client$Connection@74a4e314
> at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:375)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1528)
> at org.apache.hadoop.ipc.Client.call(Client.java:1451)
> at org.apache.hadoop.ipc.Client.call(Client.java:1412)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
> at com.sun.proxy.$Proxy527.getListing(Unknown Source)
> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getListing(ClientNamenodeProtocolTranslatorPB.java:573)
> at sun.reflect.GeneratedMethodAccessor834.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy528.getListing(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:2086)
> at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:2069)
> at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:791)
> at org.apache.hadoop.hdfs.DistributedFileSystem.access$700(DistributedFileSystem.java:106)
> at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:853)
> at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:849)
> at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:860)
> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1517)
> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1557)
> at org.apache.nifi.processors.hadoop.ListHDFS.getStatuses(ListHDFS.java:388)
> at org.apache.nifi.processors.hadoop.ListHDFS.onTrigger(ListHDFS.java:341)
> at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1118)
> at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
> at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> Number of Locked Synchronizers: 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@69a9f59d
> {noformat}
> Since NiFi won't answer the prompt, it could be interesting to default doNotPrompt to true so that authentication fails and can be retried ([source|http://docs.oracle.com/javase/7/docs/jre/api/security/jaas/spec/com/sun/security/auth/module/Krb5LoginModule.html]).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)