You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user-zh@flink.apache.org by xingoo <23...@qq.com> on 2021/02/20 09:27:16 UTC

Flink standalone模式如何区分各个任务的日志?

Dear All:
目前Flink部署主要采用standalone，想了解下如何在同一个taskmanager区分各个job的日志。



--
Sent from: http://apache-flink.147419.n8.nabble.com/

flink on yarn kerberos认证问题

Posted by "zjfplayer@hotmail.com" <zj...@hotmail.com>.
大家好，
     问题点：
       1.cdh中kerberos已经被cm托管的情况下，cm中修改kerberos配置，/var/kerberos/krb5kdc/kdc.conf和/etc/krb5.conf都不变，好像是存在其他位置，这个有没有人清楚？
        2.flink 1.8 on cdh5.14 yarn运行时，一天后报GSS initiate failed{caused by GSSException：No valid credentials provided}的报错，然后程序失败终止，怀疑是kerberos票据renew失效了，是不是我有哪里没配对，还是flink1.8还不支持renew？原先用spark-submit --keytab就会自动renew tgt

       服务器端的相关部分配置：
        1.flink-conf.yaml:
security.kerberos.login.use-ticket-cache: false
security.kerberos.login.keytab: /home/zjf/zjf.keytab
security.kerberos.login.principal: zjf@TEST.COM
security.kerberos.login.contexts: Client,KafkaClient
zookeeper.sasl.service-name: zookeeper
zookeeper.sasl.login-context-name: Client
        2./var/kerberos/krb5kdc/kdc.conf:
        
        3./etc/krb5.conf:
其中的max_renewable_life是后面手动加的，kdc已重启过还是无效。
    
    然后查看kerberos debug的日志， 发现有如下日志段：
Forwardable Ticket true
Forwarded Ticket false
Proxiable Ticket false
Proxy Ticket false 
Postdated Ticket false
Renewable Ticket false
Initial Ticket false
Auth Time =Fri Apr 30 14:38:36 CST 2021
Start Time =Fri Apr 30 14:38:36 CST 2021
End Time =Sat May 01 14:38:36 CST 2021      
Renew Till = null   




zjfplayer@hotmail.com

Re: Flink standalone模式如何区分各个任务的日志?

Posted by Yang Wang <da...@gmail.com>.
Flink的standalone application模式[1]是可以每个app都单独记录日志的

[1].
https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/kubernetes/#deploy-application-cluster

Best,
Yang

xingoo <23...@qq.com> 于2021年2月22日周一 下午12:01写道：

> Hi,
>
> 这样体验上还是不太友好，如果能做成spark那种每个Job独立记录日志就好了
>
>
>
> --
> Sent from: http://apache-flink.147419.n8.nabble.com/

Re: Flink standalone模式如何区分各个任务的日志?

Posted by xingoo <23...@qq.com>.
Hi, 

这样体验上还是不太友好，如果能做成spark那种每个Job独立记录日志就好了



--
Sent from: http://apache-flink.147419.n8.nabble.com/

Re: Flink standalone模式如何区分各个任务的日志?

Posted by Yang Wang <da...@gmail.com>.
我理解你说的应该是standalone session，这种模式下一个TM上面是会跑不同job的task的

TM里面的框架日志都是混在一起的，如果你的job class是在不同的package下面
可以用log4j2针对不同的package设置不同的logger以及appender来输出到不同路径


Best,
Yang

xingoo <23...@qq.com> 于2021年2月20日周六 下午5:31写道：

> Dear All:
> 目前Flink部署主要采用standalone，想了解下如何在同一个taskmanager区分各个job的日志。
>
>
>
> --
> Sent from: http://apache-flink.147419.n8.nabble.com/
>

flink on yarn kerberos认证问题

Posted by "zjfplayer@hotmail.com" <zj...@hotmail.com>.
Hi,
    My question is：
       1.When kerberos in cdh5.14 has been hosted by cm, modify the kerberos configuration in cm, but /var/kerberos/krb5kdc/kdc.conf and /etc/krb5.conf are the same with before. It seems the kerberos configuration is in another position. Does anyone know this?
        2.When flink 1.8 on cdh5.14 yarn is running, it reports GSS initiate failed {caused by GSSException: No valid credentials provided} after 24 hours, then the program fails and terminates. It is suspected that the kerberos ticket renew is failed. Is it because I configurated wrong? Or does flink1.8 still not support renew? When I originally used Spark, it would automatically renew.
       My configuration：
        1.flink-conf.yaml:
security.kerberos.login.use-ticket-cache: false
security.kerberos.login.keytab: /home/zjf/zjf.keytab
security.kerberos.login.principal: zjf@TEST.COM
security.kerberos.login.contexts: Client,KafkaClient
zookeeper.sasl.service-name: zookeeper
zookeeper.sasl.login-context-name: Client
        2./var/kerberos/krb5kdc/kdc.conf:
        
        3./etc/krb5.conf:
I have added max_renewable_life configuration，and run "systemctl restart krb5kdc", and it didn't work。
    
    then i check the kerberos debug log，see the following logs：
Forwardable Ticket true
Forwarded Ticket false
Proxiable Ticket false
Proxy Ticket false 
Postdated Ticket false
Renewable Ticket false
Initial Ticket false
Auth Time =Fri Apr 30 14:38:36 CST 2021
Start Time =Fri Apr 30 14:38:36 CST 2021
End Time =Sat May 01 14:38:36 CST 2021      
Renew Till = null   

    Can anyone help me? Thanks




zjfplayer@hotmail.com