You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "weipengfei-sj (via GitHub)" <gi...@apache.org> on 2024/07/04 01:29:48 UTC

[I] [Bug] [seatunnel-engine-storage] map and checkpoint Writing HDFS Kerberos tickets with automatic 24-hour expiration [seatunnel]

weipengfei-sj opened a new issue, #7102:
URL: https://github.com/apache/seatunnel/issues/7102

   ### Search before asking
   
   - [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
   
   
   ### What happened
   
   采用seatunnel2.3.5版本,3个节点的集群模式
   hazelcast.yaml 配置如下:
     map:
       engine*:
          map-store:
            enabled: true
            initial-mode: EAGER
            factory-class-name: org.apache.seatunnel.engine.server.persistence.FileMapStoreFactory
            properties:
              type: hdfs
              namespace: /tmp/seatunnel/imap
              clusterName: seatunnel-cluster
              storage.type: hdfs
              fs.defaultFS: hdfs://fss:8020
              kerberosPrincipal: hdfs
              kerberosKeytabFilePath: /applinkis/ceph/share/hadoopcluster/fss/keytab/hdfs.keytab 
              krb5Path: /app/linkis/seatunnel/config/krb5.conf
              seatunnel.hadoop.dfs.nameservices: fss
              seatunnel.hadoop.dfs.ha.namenodes.fss: nn1,nn2
              seatunnel.hadoop.dfs.namenode.rpc-address.fss.nn1: nn1:8020
              seatunnel.hadoop.dfs.namenode.rpc-address.fss.nn2: nn2:8020
              seatunnel.hadoop.dfs.client.failover.proxy.provider.usdp-bing: org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
              seatunnel.hadoop.dfs.namenode.kerberos.principal: nn/_HOST@T1.COM
              seatunnel.hadoop.dfs.datanode.kerberos.principal: dn/_HOST@T1.COM
              seatunnel.hadoop.rpc.protection: authentication
              seatunnel.hadoop.security.authentication: kerberos
              hdfs_site_path: /applinkis/ceph/share/hadoopcluster/fss/hadoop/hdfs-site.xml
   配置map信息写入到hdfs上,当集群运行超过24h之后,观察服务日志,发现写hdfs存在kerberos票据过期问题
   
   分析源码如下:
   如果采用该方式认证hdfs写入hdfs,不自动刷新票据的逻辑话,必然存在票据过期的问题出现
   ![image](https://github.com/apache/seatunnel/assets/36789477/05d5dbc4-adce-4b3e-bfb4-c66628622196)
   
   尝试修改代码,增加认证后,启动定时任务自动刷新机制:
   ![image](https://github.com/apache/seatunnel/assets/36789477/000eaf6d-72b8-44f4-afd5-69469d66702a)
   ![image](https://github.com/apache/seatunnel/assets/36789477/74d22a24-c882-480e-b7a2-ea7c118c3713)
   
   但是增加上述自动刷新kerberos票据机制之后,24h后,服务写hdfs仍然报存在票据不可用的问题
   另外尝试了多个地方,比如在HdfsWriter类中也增加了票据自动刷新机制,但是均不生效,请社区的大佬帮忙指正一下,非常感谢
   
   
   
   ### SeaTunnel Version
   
   2.3.5
   
   ### SeaTunnel Config
   
   ```conf
   hazelcast.yaml 配置如下:
     map:
       engine*:
          map-store:
            enabled: true
            initial-mode: EAGER
            factory-class-name: org.apache.seatunnel.engine.server.persistence.FileMapStoreFactory
            properties:
              type: hdfs
              namespace: /tmp/seatunnel/imap
              clusterName: seatunnel-cluster
              storage.type: hdfs
              fs.defaultFS: hdfs://fss:8020
              kerberosPrincipal: hdfs
              kerberosKeytabFilePath: /applinkis/ceph/share/hadoopcluster/fss/keytab/hdfs.keytab 
              krb5Path: /app/linkis/seatunnel/config/krb5.conf
              seatunnel.hadoop.dfs.nameservices: fss
              seatunnel.hadoop.dfs.ha.namenodes.fss: nn1,nn2
              seatunnel.hadoop.dfs.namenode.rpc-address.fss.nn1: nn1:8020
              seatunnel.hadoop.dfs.namenode.rpc-address.fss.nn2: nn2:8020
              seatunnel.hadoop.dfs.client.failover.proxy.provider.usdp-bing: org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
              seatunnel.hadoop.dfs.namenode.kerberos.principal: nn/_HOST@T1.COM
              seatunnel.hadoop.dfs.datanode.kerberos.principal: dn/_HOST@T1.COM
              seatunnel.hadoop.rpc.protection: authentication
              seatunnel.hadoop.security.authentication: kerberos
              hdfs_site_path: /applinkis/ceph/share/hadoopcluster/fss/hadoop/hdfs-site.xml
   ```
   
   
   ### Running Command
   
   ```shell
   ./bin/seatunnel.sh  -c config/test-source-kerberos-kafka.yaml
   ```
   
   
   ### Error Exception
   
   ```log
   2024-07-03 15:12:50,607 WARN  [o.a.h.i.Client                ] [LeaseRenewer:hdfs@fsst1] - Exception encountered while connecting to the server
   javax.security.sasl.SaslException: GSS initiate failed
           at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211) ~[?:1.8.0_181]
           at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:408) ~[seatunnel-hadoop3-3.1.4-uber.jar:2.3.5]
           at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:622) ~[seatunnel-hadoop3-3.1.4-uber.jar:2.3.5]
           at org.apache.hadoop.ipc.Client$Connection.access$2300(Client.java:413) ~[seatunnel-hadoop3-3.1.4-uber.jar:2.3.5]
           at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:822) ~[seatunnel-hadoop3-3.1.4-uber.jar:2.3.5]
           at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:818) ~[seatunnel-hadoop3-3.1.4-uber.jar:2.3.5]
           at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_181]
           at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_181]
           at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) ~[seatunnel-hadoop3-3.1.4-uber.jar:2.3.5]
           at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:818) ~[seatunnel-hadoop3-3.1.4-uber.jar:2.3.5]
           at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:413) ~[seatunnel-hadoop3-3.1.4-uber.jar:2.3.5]
   /seatunnel-starter.jar
           at com.sun.proxy.$Proxy34.fsync(Unknown Source) ~[?:?]
           at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.fsync(ClientNamenodeProtocolTranslatorPB.java:984) ~[seatunnel-hadoop3-3.1.4-uber.jar:2.3.5]
           at sun.reflect.GeneratedMethodAccessor107.invoke(Unknown Source) ~[?:?]
           at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_181]
           at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
           at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) ~[seatunnel-hadoop3-3.1.4-uber.jar:2.3.5]
           at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) ~[seatunnel-hadoop3-3.1.4-uber.jar:2.3.5]
           at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) ~[seatunnel-hadoop3-3.1.4-uber.jar:2.3.5]
           at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) ~[seatunnel-hadoop3-3.1.4-uber.jar:2.3.5]
           at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) ~[seatunnel-hadoop3-3.1.4-uber.jar:2.3.5]
           at com.sun.proxy.$Proxy35.fsync(Unknown Source) ~[?:?]
           at org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java:706) ~[seatunnel-hadoop3-3.1.4-uber.jar:2.3.5]
           at org.apache.hadoop.hdfs.DFSOutputStream.hsync(DFSOutputStream.java:604) ~[seatunnel-hadoop3-3.1.4-uber.jar:2.3.5]
           at org.apache.hadoop.hdfs.client.HdfsDataOutputStream.hsync(HdfsDataOutputStream.java:96) ~[seatunnel-hadoop3-3.1.4-uber.jar:2.3.5]
           at org.apache.seatunnel.engine.imap.storage.file.wal.writer.HdfsWriter.flush(HdfsWriter.java:87) ~[seatunnel-starter.jar:2.3.5]
           at org.apache.seatunnel.engine.imap.storage.file.wal.writer.HdfsWriter.write(HdfsWriter.java:101) ~[seatunnel-starter.jar:2.3.5]
           at org.apache.seatunnel.engine.imap.storage.file.wal.writer.HdfsWriter.write(HdfsWriter.java:80) ~[seatunnel-starter.jar:2.3.5]
           at org.apache.seatunnel.engine.imap.storage.file.wal.writer.HdfsWriter.write(HdfsWriter.java:44) ~[seatunnel-starter.jar:2.3.5]
           at org.apache.seatunnel.engine.imap.storage.file.common.WALWriter.write(WALWriter.java:50) ~[seatunnel-starter.jar:2.3.5]
           at org.apache.seatunnel.engine.imap.storage.file.disruptor.WALWorkHandler.walEvent(WALWorkHandler.java:87) ~[seatunnel-starter.jar:2.3.5]
           at org.apache.seatunnel.engine.imap.storage.file.disruptor.WALWorkHandler.onEvent(WALWorkHandler.java:78) ~[seatunnel-starter.jar:2.3.5]
           at org.apache.seatunnel.engine.imap.storage.file.disruptor.WALWorkHandler.onEvent(WALWorkHandler.java:44) ~[seatunnel-starter.jar:2.3.5]
           at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) ~[seatunnel-starter.jar:2.3.5]
           at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
   Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
           at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) ~[?:1.8.0_181]
           at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122) ~[?:1.8.0_181]
           at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187) ~[?:1.8.0_181]
           at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224) ~[?:1.8.0_181]
           at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212) ~[?:1.8.0_181]
           at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) ~[?:1.8.0_181]
           at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192) ~[?:1.8.0_181]
           ... 39 more
   ```
   
   
   ### Zeta or Flink or Spark Version
   
   _No response_
   
   ### Java or Scala Version
   
   _No response_
   
   ### Screenshots
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [seatunnel-engine-storage] map and checkpoint Writing HDFS Kerberos tickets with automatic 24-hour expiration [seatunnel]

Posted by "liunaijie (via GitHub)" <gi...@apache.org>.
liunaijie commented on issue #7102:
URL: https://github.com/apache/seatunnel/issues/7102#issuecomment-2208274386

   hi, base your issue title and description, the exception is happend on write `IMAP` or `checkpoint` right, not on the data sync process, right?
   
   i guess is the `FileSystem` client not refresh, so even you refresh the config, the client still use old config then got issue.
   you can try to refresh the client to slove this issue.
   ![image](https://github.com/apache/seatunnel/assets/28497357/8f7fe369-3965-4941-bb37-dd5da6e52a6c)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [seatunnel-engine-storage] map and checkpoint Writing HDFS Kerberos tickets with automatic 24-hour expiration [seatunnel]

Posted by "shenzhy5 (via GitHub)" <gi...@apache.org>.
shenzhy5 commented on issue #7102:
URL: https://github.com/apache/seatunnel/issues/7102#issuecomment-2445948209

   @liunaijie refresh FileSystem cannot solve this problem


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [seatunnel-engine-storage] map and checkpoint Writing HDFS Kerberos tickets with automatic 24-hour expiration [seatunnel]

Posted by "weipengfei-sj (via GitHub)" <gi...@apache.org>.
weipengfei-sj commented on issue #7102:
URL: https://github.com/apache/seatunnel/issues/7102#issuecomment-2212689311

   ![image](https://github.com/apache/seatunnel/assets/36789477/d705d995-62f9-4ab4-9de5-5bba71a9478c)
   我尝试在客户端增加了定时刷新,24h后,仍然是报kerberos认证的错误


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [seatunnel-engine-storage] map and checkpoint Writing HDFS Kerberos tickets with automatic 24-hour expiration [seatunnel]

Posted by "liunaijie (via GitHub)" <gi...@apache.org>.
liunaijie commented on issue #7102:
URL: https://github.com/apache/seatunnel/issues/7102#issuecomment-2225295913

   > @liunaijie 请帮忙看看呢
   
   hi, please attach all the code you change, or give the repo link.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [seatunnel-engine-storage] map and checkpoint Writing HDFS Kerberos tickets with automatic 24-hour expiration [seatunnel]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #7102: [Bug] [seatunnel-engine-storage] map  and checkpoint Writing HDFS Kerberos tickets with automatic 24-hour expiration 
URL: https://github.com/apache/seatunnel/issues/7102


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [seatunnel-engine-storage] map and checkpoint Writing HDFS Kerberos tickets with automatic 24-hour expiration [seatunnel]

Posted by "shenzhy5 (via GitHub)" <gi...@apache.org>.
shenzhy5 commented on issue #7102:
URL: https://github.com/apache/seatunnel/issues/7102#issuecomment-2445614672

   i met the same problem


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [seatunnel-engine-storage] map and checkpoint Writing HDFS Kerberos tickets with automatic 24-hour expiration [seatunnel]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #7102:
URL: https://github.com/apache/seatunnel/issues/7102#issuecomment-3100147107

   This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [seatunnel-engine-storage] map and checkpoint Writing HDFS Kerberos tickets with automatic 24-hour expiration [seatunnel]

Posted by "yinbenrong (via GitHub)" <gi...@apache.org>.
yinbenrong commented on issue #7102:
URL: https://github.com/apache/seatunnel/issues/7102#issuecomment-3344710185

   @weipengfei-sj我尝试了#7995的fix没有得到解决, 老哥解决了没有,求解决方案


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [seatunnel-engine-storage] map and checkpoint Writing HDFS Kerberos tickets with automatic 24-hour expiration [seatunnel]

Posted by "liunaijie (via GitHub)" <gi...@apache.org>.
liunaijie commented on issue #7102:
URL: https://github.com/apache/seatunnel/issues/7102#issuecomment-2461925644

   @shenzhy5 @weipengfei-sj  I create a pr to fix this. can you help cherry-pick this commit and verify?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [seatunnel-engine-storage] map and checkpoint Writing HDFS Kerberos tickets with automatic 24-hour expiration [seatunnel]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #7102:
URL: https://github.com/apache/seatunnel/issues/7102#issuecomment-3064433429

   This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [seatunnel-engine-storage] map and checkpoint Writing HDFS Kerberos tickets with automatic 24-hour expiration [seatunnel]

Posted by "weipengfei-sj (via GitHub)" <gi...@apache.org>.
weipengfei-sj commented on issue #7102:
URL: https://github.com/apache/seatunnel/issues/7102#issuecomment-2216306115

   @liunaijie 请帮忙看看呢


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [seatunnel-engine-storage] map and checkpoint Writing HDFS Kerberos tickets with automatic 24-hour expiration [seatunnel]

Posted by "liunaijie (via GitHub)" <gi...@apache.org>.
liunaijie commented on issue #7102:
URL: https://github.com/apache/seatunnel/issues/7102#issuecomment-2461716184

   Looks this is an issue, I will take a look.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org