[jira] [Commented] (GOBBLIN-1061) Kafka consumer Kerberos config

["Shirshanka Das (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/GOBBLIN-1061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17202703#comment-17202703 ] 

Shirshanka Das commented on GOBBLIN-1061:
-----------------------------------------

[~tichyj]: did you manage to resolve this?

> Kafka consumer Kerberos config
> ------------------------------
>
>                 Key: GOBBLIN-1061
>                 URL: https://issues.apache.org/jira/browse/GOBBLIN-1061
>             Project: Apache Gobblin
>          Issue Type: Bug
>    Affects Versions: 0.15.0
>         Environment: OS RHEL 7.5, Cloudera kafka 3.1.0 (kafka 1.0.1), CDH 5.16.2
>            Reporter: Jan Tichý
>            Priority: Blocker
>         Attachments: application.conf, sip_voice_raw.pull
>
>
> Does the gobblin support kerberos auth for kafka client consumer? I have right kafka consumer settings with sasl_plaintext configuration but if it tries to kinit it fails with exception below:
> {code:java}
> 2020-02-20 05:44:00 PST INFO [DefaultQuartzScheduler_Worker-1] org.apache.kafka.common.config.AbstractConfig - ConsumerConfig values:
> metric.reporters = []
> metadata.max.age.ms = 300000
> value.deserializer = class io.confluent.kafka.serializers.KafkaAvroDeserializer
> group.id = kafka09
> partition.assignment.strategy = [org.apache.kafka.clients.consumer.RangeAssignor]
> reconnect.backoff.ms = 50
> sasl.kerberos.ticket.renew.window.factor = 0.8
> max.partition.fetch.bytes = 1048576
> bootstrap.servers = [czrtim1hr.oskarmobil.cz:9092, czrtim2hr.oskarmobil.cz:9092]
> retry.backoff.ms = 100
> sasl.kerberos.kinit.cmd = /usr/bin/kinit
> sasl.kerberos.service.name = kafka
> sasl.kerberos.ticket.renew.jitter = 0.05
> ssl.keystore.type = JKS
> ssl.trustmanager.algorithm = PKIX
> enable.auto.commit = false
> ssl.key.password = null
> fetch.max.wait.ms = 500
> sasl.kerberos.min.time.before.relogin = 60000
> connections.max.idle.ms = 540000
> ssl.truststore.password = null
> session.timeout.ms = 30000
> metrics.num.samples = 2
> client.id =
> ssl.endpoint.identification.algorithm = null
> key.deserializer = class io.confluent.kafka.serializers.KafkaAvroDeserializer
> ssl.protocol = TLS
> check.crcs = true
> request.timeout.ms = 40000
> ssl.provider = null
> ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
> ssl.keystore.location = null
> heartbeat.interval.ms = 3000
> auto.commit.interval.ms = 5000
> receive.buffer.bytes = 32768
> ssl.cipher.suites = null
> ssl.truststore.type = JKS
> security.protocol = SASL_PLAINTEXT
> ssl.truststore.location = null
> ssl.keystore.password = null
> ssl.keymanager.algorithm = SunX509
> metrics.sample.window.ms = 30000
> fetch.min.bytes = 1
> send.buffer.bytes = 131072
> auto.offset.reset = latest2020-02-20 05:44:00 PST ERROR [DefaultQuartzScheduler_Worker-1] org.apache.gobblin.runtime.SourceDecorator - Failed to get work units for job job_SipVoiceRaw_1582206240042
> org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
> at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:648)
> at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:542)
> at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:524)
> at org.apache.gobblin.kafka.client.Kafka09ConsumerClient.<init>(Kafka09ConsumerClient.java:116)
> at org.apache.gobblin.kafka.client.Kafka09ConsumerClient.<init>(Kafka09ConsumerClient.java:69)
> at org.apache.gobblin.kafka.client.Kafka09ConsumerClient$Factory.create(Kafka09ConsumerClient.java:224)
> at org.apache.gobblin.source.extractor.extract.kafka.KafkaSource.getWorkunits(KafkaSource.java:210)
> at org.apache.gobblin.runtime.SourceDecorator.getWorkunitStream(SourceDecorator.java:81)
> at org.apache.gobblin.runtime.AbstractJobLauncher.launchJob(AbstractJobLauncher.java:411)
> at org.apache.gobblin.cluster.GobblinHelixJobLauncher.launchJob(GobblinHelixJobLauncher.java:378)
> at org.apache.gobblin.scheduler.JobScheduler.runJob(JobScheduler.java:487)
> at org.apache.gobblin.cluster.HelixRetriggeringJobCallable.runJobLauncherLoop(HelixRetriggeringJobCallable.java:203)
> at org.apache.gobblin.cluster.HelixRetriggeringJobCallable.call(HelixRetriggeringJobCallable.java:159)
> at org.apache.gobblin.cluster.GobblinHelixJobScheduler.runJob(GobblinHelixJobScheduler.java:228)
> at org.apache.gobblin.cluster.GobblinHelixJob.executeImpl(GobblinHelixJob.java:61)
> at org.apache.gobblin.scheduler.BaseGobblinJob.execute(BaseGobblinJob.java:58)
> at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
> at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
> Caused by: org.apache.kafka.common.KafkaException: java.lang.IllegalArgumentException: You must pass java.security.auth.login.config in secure mode.
> at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:74)
> at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:60){code}
> I have these conf options in my config for kerberos auth:
> gobblin.yarn.keytab.file.path="/home/morpheus2/.keytab"
> gobblin.yarn.keytab.principal.name=morpheus2
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)