You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "ligou525 (via GitHub)" <gi...@apache.org> on 2024/02/27 02:16:37 UTC

[I] Hudi timelineserver doesn't support kerberos authenticated file system [hudi]

ligou525 opened a new issue, #10762:
URL: https://github.com/apache/hudi/issues/10762

   For hudi table using file system with kerberos authentication, the timeline server handler doesn't work. Handlers of timeline server doesn't consider the authenticated filesystem, authentication exception is thrown when  creating HoodieTableMetaClient as creating the filesystem:
    public SyncableFileSystemView getFileSystemView(String basePath) {
       return globalViewMap.computeIfAbsent(basePath, (path) -> {
         HoodieTableMetaClient metaClient = HoodieTableMetaClient.builder().setConf(conf.newCopy()).setBasePath(path).build();
         return viewCreator.apply(metaClient, viewStorageConfig);
       });
     }
   
   **Expected behavior**
   
   timeline server works well for kerberos-authenticated filesystem
   
   **Environment Description**
   
   * Hudi version : 0.11.0
   
   * Spark version :
   
   * Hive version : 
   
   * Hadoop version :
   
   * Storage (HDFS/S3/GCS..) : hdfs
   
   * Running on Docker? (yes/no) : no
   
   
   **Additional context**
   
   
   **Stacktrace**
   
   24/02/19 21:57:12 INFO view.RemoteHoodieTableFileSystemView: Sending request : (http://172.16.48.4:27329/v1/hoodie/view/compactions/pending/?basepath=hdfs%3A%2F%2Fhdfs_host%3A8020%2Fuser%2Fhdfs%2Ftest_location%2Ftest_decimal_hdfs&lastinstantts=20240219215711252&timeline
   hash=4a679c704ec969a1ad4b2f434ac012ff10005ba03948c375356913f52e6d8ed4)
   24/02/19 21:57:12 INFO table.HoodieTableMetaClient: Loading HoodieTableMetaClient from hdfs://172.16.48.6:8020/user/hdfs/test_location/test_decimal_hdfs
   24/02/19 21:57:12 ERROR service.RequestHandler: Got runtime exception servicing request basepath=hdfs%3A%2F%2F2Fhdfs_host%3A8020%2Fuser%2Fhdfs%2Ftest_location%2Ftest_decimal_hdfs&lastinstantts=20240219215711252&timelinehash=4a679c704ec969a1ad4b2f434ac012ff10005ba03948c375
   356913f52e6d8ed4
   org.apache.hudi.exception.HoodieIOException: Could not check if hdfs://2Fhdfs_host:8020/user/hdfs/test_location/test_decimal_hdfs is a valid table
           at org.apache.hudi.exception.TableNotFoundException.checkTableValidity(TableNotFoundException.java:59)
           at org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:124)
           at org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:78)
           at org.apache.hudi.common.table.HoodieTableMetaClient$Builder.build(HoodieTableMetaClient.java:670)
           at org.apache.hudi.common.table.view.FileSystemViewManager.lambda$getFileSystemView$0(FileSystemViewManager.java:98)
           at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
           at org.apache.hudi.common.table.view.FileSystemViewManager.getFileSystemView(FileSystemViewManager.java:97)
           at org.apache.hudi.timeline.service.RequestHandler.isLocalViewBehind(RequestHandler.java:125)
           at org.apache.hudi.timeline.service.RequestHandler.syncIfLocalViewBehind(RequestHandler.java:153)
           at org.apache.hudi.timeline.service.RequestHandler.access$100(RequestHandler.java:66)
           at org.apache.hudi.timeline.service.RequestHandler$ViewHandler.handle(RequestHandler.java:495)
           at io.javalin.security.SecurityUtil.noopAccessManager(SecurityUtil.kt:22)
           at io.javalin.Javalin.lambda$addHandler$0(Javalin.java:606)
           at io.javalin.core.JavalinServlet$service$2$1.invoke(JavalinServlet.kt:46)
           at io.javalin.core.JavalinServlet$service$2$1.invoke(JavalinServlet.kt:17)
           at io.javalin.core.JavalinServlet$service$1.invoke(JavalinServlet.kt:143)
           at io.javalin.core.JavalinServlet$service$2.invoke(JavalinServlet.kt:41)
           at io.javalin.core.JavalinServlet.service(JavalinServlet.kt:107)
           at io.javalin.core.util.JettyServerUtil$initialize$httpHandler$1.doHandle(JettyServerUtil.kt:72)
           at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
           at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
           at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1668)
           at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
           at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
           at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
           at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:61)
           at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:174)
           at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
           at org.eclipse.jetty.server.Server.handle(Server.java:502)
           at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)
           at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
           at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
           at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
           at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
           at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
           at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
           at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
           at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
           at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
           at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
           at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
           at java.lang.Thread.run(Thread.java:748)
   Caused by: java.io.IOException: DestHost:destPort 2Fhdfs_host:8020 , LocalHost:localPort core-25c178a-2/2Fclient_host:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
           at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
           at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
           at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
           at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
           at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:842)
           at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:817)
           at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1583)
           at org.apache.hadoop.ipc.Client.call(Client.java:1525)
           at org.apache.hadoop.ipc.Client.call(Client.java:1422)
           at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:231)
           at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Hudi timelineserver doesn't support kerberos authenticated file system [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #10762:
URL: https://github.com/apache/hudi/issues/10762#issuecomment-1967977809

   Hi, is the issue already resoved? Can you share your experience here?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Hudi timelineserver doesn't support kerberos authenticated file system [hudi]

Posted by "ligou525 (via GitHub)" <gi...@apache.org>.
ligou525 closed issue #10762: Hudi timelineserver doesn't support kerberos authenticated file system
URL: https://github.com/apache/hudi/issues/10762


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org