You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by Ping PW Wang <wp...@cn.ibm.com> on 2016/05/17 07:36:36 UTC
In Kerberos, Thrift Sources/Sinks can only authenticate as principal "flume"?
Hi Flume team and experts,
I am using Thrift Sources/Sinks in Kerberos environment, found the agents
only work with principle "flume". If using other valid principle it will
hit ERROR of "GSS initiate failed". For some reason I can not use Flume
principle, can someone give me some guide on this? Thanks.
Here's my configuration file:
g1.sources.source1.type = spooldir
g1.sources.source1.spoolDir = /test
g1.sources.source1.fileHeader = false
g1.sinks.sink1.type = thrift
g1.sinks.sink1.hostname = localhost
g1.sinks.sink1.port = 55555
g1.channels.channel1.type = memory
g1.channels.channel1.capacity = 1000
g1.channels.channel1.transactionCapacity = 100
g1.sources.source1.channels = channel1
g1.sinks.sink1.channel = channel1
g2.sources = source2
g2.sinks = sink2
g2.channels = channel2
g2.sources.source2.type = thrift
g2.sources.source2.bind = localhost
g2.sources.source2.port = 55555
g2.sinks.sink2.type = hdfs
g2.sinks.sink2.hdfs.path = /tmp
g2.sinks.sink2.hdfs.filePrefix = thriftData
g2.sinks.sink2.hdfs.writeFormat = Text
g2.sinks.sink2.hdfs.fileType = DataStream
g2.channels.channel2.type = memory
g2.channels.channel2.capacity = 1000
g2.channels.channel2.transactionCapacity = 100
g2.sources.source2.channels = channel2
g2.sinks.sink2.channel = channel2
g1.sinks.sink1.kerberos = true
g1.sinks.sink1.client-principal = flume/hostname@XXX.COM
g1.sinks.sink1.client-keytab
= /etc/security/keytabs/flume-1563.server.keytab
g1.sinks.sink1.server-principal = flume/hostname@XXX.COM
g2.sources.source2.kerberos = true
g2.sources.source2.agent-principal = flume/hostname@XXX.COM
g2.sources.source2.agent-keytab
= /etc/security/keytabs/flume-1563.server.keytab
If using other valid principle, will hit error:
g1.sinks.sink1.kerberos = true
g1.sinks.sink1.client-principal = test@IBM.COM
g1.sinks.sink1.client-keytab = /home/test/test.keytab
g1.sinks.sink1.server-principal = test@IBM.COM
g2.sources.source2.kerberos = true
g2.sources.source2.agent-principal = test@IBM.COM
g2.sources.source2.agent-keytab = /home/test/test.keytab
Agent g1:
ERROR server.TThreadPoolServer: Error occurred during processing of
message.
java.lang.RuntimeException:
org.apache.thrift.transport.TTransportException: Peer indicated failure:
GSS initiate failed
at org.apache.thrift.transport.TSaslServerTransport
$Factory.getTransport(TSaslServerTransport.java:219)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run
(TThreadPoolServer.java:189)
at java.util.concurrent.ThreadPoolExecutor.runWorker
(ThreadPoolExecutor.java:1142)
Agent g2:
ERROR transport.TSaslTransport: SASL negotiation failure
javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: No valid credentials provided (Mechanism level: Server not
found in Kerberos database (7) - UNKNOWN_SERVER)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge
(GssKrb5Client.java:211)
Re: In Kerberos, Thrift Sources/Sinks can only authenticate as principal
"flume"?
Posted by Ping PW Wang <wp...@cn.ibm.com>.
After more test, I found thrift Sources/Sinks can only authenticate with
kerberos principal in format "name/_HOST@YOUR-REALM.COM".
If using other valid principle in the format "name@YOUR-REALM.COM" it will
hit ERROR of "GSS initiate failed".
I opened this JIRA for the strict limits on the principal.
https://issues.apache.org/jira/browse/FLUME-2912.
From: Ping PW Wang/China/IBM@IBMCN
To: dev@flume.apache.org, user@flume.apache.org
Date: 05/17/2016 05:00 PM
Subject: In Kerberos, Thrift Sources/Sinks can only authenticate as
principal "flume"?
Hi Flume team and experts,
I am using Thrift Sources/Sinks in Kerberos environment, found the agents
only work with principle "flume". If using other valid principle it will
hit ERROR of "GSS initiate failed". For some reason I can not use Flume
principle, can someone give me some guide on this? Thanks.
Here's my configuration file:
g1.sources.source1.type = spooldir
g1.sources.source1.spoolDir = /test
g1.sources.source1.fileHeader = false
g1.sinks.sink1.type = thrift
g1.sinks.sink1.hostname = localhost
g1.sinks.sink1.port = 55555
g1.channels.channel1.type = memory
g1.channels.channel1.capacity = 1000
g1.channels.channel1.transactionCapacity = 100
g1.sources.source1.channels = channel1
g1.sinks.sink1.channel = channel1
g2.sources = source2
g2.sinks = sink2
g2.channels = channel2
g2.sources.source2.type = thrift
g2.sources.source2.bind = localhost
g2.sources.source2.port = 55555
g2.sinks.sink2.type = hdfs
g2.sinks.sink2.hdfs.path = /tmp
g2.sinks.sink2.hdfs.filePrefix = thriftData
g2.sinks.sink2.hdfs.writeFormat = Text
g2.sinks.sink2.hdfs.fileType = DataStream
g2.channels.channel2.type = memory
g2.channels.channel2.capacity = 1000
g2.channels.channel2.transactionCapacity = 100
g2.sources.source2.channels = channel2
g2.sinks.sink2.channel = channel2
g1.sinks.sink1.kerberos = true
g1.sinks.sink1.client-principal = flume/hostname@XXX.COM
g1.sinks.sink1.client-keytab
= /etc/security/keytabs/flume-1563.server.keytab
g1.sinks.sink1.server-principal = flume/hostname@XXX.COM
g2.sources.source2.kerberos = true
g2.sources.source2.agent-principal = flume/hostname@XXX.COM
g2.sources.source2.agent-keytab
= /etc/security/keytabs/flume-1563.server.keytab
If using other valid principle, will hit error:
g1.sinks.sink1.kerberos = true
g1.sinks.sink1.client-principal = test@IBM.COM
g1.sinks.sink1.client-keytab = /home/test/test.keytab
g1.sinks.sink1.server-principal = test@IBM.COM
g2.sources.source2.kerberos = true
g2.sources.source2.agent-principal = test@IBM.COM
g2.sources.source2.agent-keytab = /home/test/test.keytab
Agent g1:
ERROR server.TThreadPoolServer: Error occurred during processing of
message.
java.lang.RuntimeException:
org.apache.thrift.transport.TTransportException: Peer indicated failure:
GSS initiate failed
at org.apache.thrift.transport.TSaslServerTransport
$Factory.getTransport(TSaslServerTransport.java:219)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run
(TThreadPoolServer.java:189)
at java.util.concurrent.ThreadPoolExecutor.runWorker
(ThreadPoolExecutor.java:1142)
Agent g2:
ERROR transport.TSaslTransport: SASL negotiation failure
javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: No valid credentials provided (Mechanism level: Server not
found in Kerberos database (7) - UNKNOWN_SERVER)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge
(GssKrb5Client.java:211)
Re: In Kerberos, Thrift Sources/Sinks can only authenticate as principal
"flume"?
Posted by Ping PW Wang <wp...@cn.ibm.com>.
After more test, I found thrift Sources/Sinks can only authenticate with
kerberos principal in format "name/_HOST@YOUR-REALM.COM".
If using other valid principle in the format "name@YOUR-REALM.COM" it will
hit ERROR of "GSS initiate failed".
I opened this JIRA for the strict limits on the principal.
https://issues.apache.org/jira/browse/FLUME-2912.
From: Ping PW Wang/China/IBM@IBMCN
To: dev@flume.apache.org, user@flume.apache.org
Date: 05/17/2016 05:00 PM
Subject: In Kerberos, Thrift Sources/Sinks can only authenticate as
principal "flume"?
Hi Flume team and experts,
I am using Thrift Sources/Sinks in Kerberos environment, found the agents
only work with principle "flume". If using other valid principle it will
hit ERROR of "GSS initiate failed". For some reason I can not use Flume
principle, can someone give me some guide on this? Thanks.
Here's my configuration file:
g1.sources.source1.type = spooldir
g1.sources.source1.spoolDir = /test
g1.sources.source1.fileHeader = false
g1.sinks.sink1.type = thrift
g1.sinks.sink1.hostname = localhost
g1.sinks.sink1.port = 55555
g1.channels.channel1.type = memory
g1.channels.channel1.capacity = 1000
g1.channels.channel1.transactionCapacity = 100
g1.sources.source1.channels = channel1
g1.sinks.sink1.channel = channel1
g2.sources = source2
g2.sinks = sink2
g2.channels = channel2
g2.sources.source2.type = thrift
g2.sources.source2.bind = localhost
g2.sources.source2.port = 55555
g2.sinks.sink2.type = hdfs
g2.sinks.sink2.hdfs.path = /tmp
g2.sinks.sink2.hdfs.filePrefix = thriftData
g2.sinks.sink2.hdfs.writeFormat = Text
g2.sinks.sink2.hdfs.fileType = DataStream
g2.channels.channel2.type = memory
g2.channels.channel2.capacity = 1000
g2.channels.channel2.transactionCapacity = 100
g2.sources.source2.channels = channel2
g2.sinks.sink2.channel = channel2
g1.sinks.sink1.kerberos = true
g1.sinks.sink1.client-principal = flume/hostname@XXX.COM
g1.sinks.sink1.client-keytab
= /etc/security/keytabs/flume-1563.server.keytab
g1.sinks.sink1.server-principal = flume/hostname@XXX.COM
g2.sources.source2.kerberos = true
g2.sources.source2.agent-principal = flume/hostname@XXX.COM
g2.sources.source2.agent-keytab
= /etc/security/keytabs/flume-1563.server.keytab
If using other valid principle, will hit error:
g1.sinks.sink1.kerberos = true
g1.sinks.sink1.client-principal = test@IBM.COM
g1.sinks.sink1.client-keytab = /home/test/test.keytab
g1.sinks.sink1.server-principal = test@IBM.COM
g2.sources.source2.kerberos = true
g2.sources.source2.agent-principal = test@IBM.COM
g2.sources.source2.agent-keytab = /home/test/test.keytab
Agent g1:
ERROR server.TThreadPoolServer: Error occurred during processing of
message.
java.lang.RuntimeException:
org.apache.thrift.transport.TTransportException: Peer indicated failure:
GSS initiate failed
at org.apache.thrift.transport.TSaslServerTransport
$Factory.getTransport(TSaslServerTransport.java:219)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run
(TThreadPoolServer.java:189)
at java.util.concurrent.ThreadPoolExecutor.runWorker
(ThreadPoolExecutor.java:1142)
Agent g2:
ERROR transport.TSaslTransport: SASL negotiation failure
javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: No valid credentials provided (Mechanism level: Server not
found in Kerberos database (7) - UNKNOWN_SERVER)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge
(GssKrb5Client.java:211)