You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@samza.apache.org by "Rodolfo Grave (JIRA)" <ji...@apache.org> on 2015/07/23 12:46:05 UTC
[jira] [Commented] (SAMZA-727) Support for Kerberos
[ https://issues.apache.org/jira/browse/SAMZA-727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14638634#comment-14638634 ]
Rodolfo Grave commented on SAMZA-727:
-------------------------------------
When running hello-samza on a CDH 5.4.0 Hadoop cluster secured with Kerberos, I get the following error:
java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "xxxx/xx.xx.xx.xx"; destination host is: "xxxxx":8020;
Running klist before and after running the Samza job returns:
-bash-4.1$ klist
Ticket cache: FILE:/tmp/krb5cc_3073905
Default principal: myuser@INTRANET.mycompany.COM
Valid starting Expires Service principal
07/23/15 10:00:22 07/23/15 20:00:22 krbtgt/INTRANET.mycompany.COM@INTRANET.mycompany.COM
The full stack trace is:
2015-07-23 10:00:57 SamzaAppMaster$ [INFO] got container id: container_e105_1437556818770_0108_01_000001
2015-07-23 10:00:57 SamzaAppMaster$ [INFO] got app attempt id: appattempt_1437556818770_0108_000001
2015-07-23 10:00:57 SamzaAppMaster$ [INFO] got node manager host: myserver2660.intranet.mycompany.com
2015-07-23 10:00:57 SamzaAppMaster$ [INFO] got node manager port: 8041
2015-07-23 10:00:57 SamzaAppMaster$ [INFO] got node manager http port: 8042
2015-07-23 10:00:57 SamzaAppMaster$ [INFO] got config: {task.inputs=kafka.wikipedia-raw, metrics.reporter.jmx.class=org.apache.samza.metrics.reporter.JmxReporterFactory, job.factory.class=org.apache.samza.job.yarn.YarnJobFactory, serializers.registry.metrics.class=org.apache.samza.serializers.MetricsSnapshotSerdeFactory, metrics.reporter.snapshot.class=org.apache.samza.metrics.reporter.MetricsSnapshotReporterFactory, task.checkpoint.replication.factor=1, job.name=wikipedia-parser, task.checkpoint.factory=org.apache.samza.checkpoint.kafka.KafkaCheckpointManagerFactory, systems.kafka.consumer.zookeeper.connect=myserver2649:2181/, systems.kafka.consumer.auto.offset.reset=largest, systems.kafka.samza.msg.serde=json, serializers.registry.json.class=org.apache.samza.serializers.JsonSerdeFactory, metrics.reporters=snapshot,jmx, task.checkpoint.system=kafka, systems.kafka.streams.metrics.samza.msg.serde=metrics, systems.kafka.producer.bootstrap.servers=myserver2663:9092,myserver2664:9092,myserver2665:9092, task.class=samza.examples.wikipedia.task.WikipediaParserStreamTask, yarn.package.path=hdfs://nameservice1/bigdata/projects/myprojectplayground/samza-hello-samza-master-0.9.1-dist.tar.gz, metrics.reporter.snapshot.stream=kafka.metrics, systems.kafka.samza.factory=org.apache.samza.system.kafka.KafkaSystemFactory}
2015-07-23 10:00:57 ClientHelper [INFO] trying to connect to RM 0.0.0.0:8032
2015-07-23 10:00:58 JmxServer [INFO] According to InetAddress.getLocalHost.getHostName we are myserver2660.intranet.mycompany.com
2015-07-23 10:00:58 JmxServer [INFO] Started JmxServer registry port=46994 server port=46995 url=service:jmx:rmi://localhost:46995/jndi/rmi://localhost:46994/jmxrmi
2015-07-23 10:00:58 JmxServer [INFO] If you are tunneling, you might want to try JmxServer registry port=46994 server port=46995 url=service:jmx:rmi://myserver2660.intranet.mycompany.com:46995/jndi/rmi://myserver2660.intranet.mycompany.com:46994/jmxrmi
2015-07-23 10:00:58 MetricsSnapshotReporterFactory [INFO] Creating new metrics snapshot reporter.
2015-07-23 10:00:58 MetricsSnapshotReporterFactory [WARN] Unable to find implementation version in jar's meta info. Defaulting to 0.0.1.
2015-07-23 10:00:58 MetricsSnapshotReporterFactory [INFO] Got system stream SystemStream [system=kafka, stream=metrics].
2015-07-23 10:00:58 MetricsSnapshotReporterFactory [INFO] Got system factory org.apache.samza.system.kafka.KafkaSystemFactory@2ad6b556.
2015-07-23 10:00:58 MetricsSnapshotReporterFactory [INFO] Got producer org.apache.samza.system.kafka.KafkaSystemProducer@32e3e90f.
2015-07-23 10:00:58 MetricsSnapshotReporterFactory [INFO] Got serde org.apache.samza.serializers.MetricsSnapshotSerde@2475986a.
2015-07-23 10:00:58 MetricsSnapshotReporter [INFO] got metrics snapshot reporter properties [job name: wikipedia-parser, job id: 1, containerName: ApplicationMaster, version: 0.0.1, samzaVersion: 0.9.1, host: myserver2660.intranet.mycompany.com]
2015-07-23 10:00:58 MetricsSnapshotReporter [INFO] Registering MetricsSnapshotReporterFactory with producer.
2015-07-23 10:00:58 MetricsSnapshotReporter [INFO] Registering ApplicationMaster with producer.
2015-07-23 10:00:58 JmxReporterFactory [INFO] Creating JMX reporter with name jmx.
2015-07-23 10:00:58 SamzaAppMasterTaskManager [INFO] No yarn.container.count specified. Defaulting to one container.
2015-07-23 10:00:58 VerifiableProperties [INFO] Verifying properties
2015-07-23 10:00:58 VerifiableProperties [INFO] Property auto.offset.reset is overridden to largest
2015-07-23 10:00:58 VerifiableProperties [INFO] Property client.id is overridden to samza_checkpoint_manager-wikipedia_parser-1-1437642058788-1
2015-07-23 10:00:58 VerifiableProperties [INFO] Property group.id is overridden to undefined-samza-consumer-group-ce2ba112-ecf1-4304-bf70-bc5dda3aea3b
2015-07-23 10:00:58 VerifiableProperties [INFO] Property zookeeper.connect is overridden to myserver2649:2181/
2015-07-23 10:00:58 KafkaCheckpointManager [INFO] Creating KafkaCheckpointManager with: clientId=samza_checkpoint_manager-wikipedia_parser-1-1437642058788-1, checkpointTopic=__samza_checkpoint_ver_1_for_wikipedia-parser_1, systemName=kafka
2015-07-23 10:00:58 VerifiableProperties [INFO] Verifying properties
2015-07-23 10:00:58 VerifiableProperties [INFO] Property auto.offset.reset is overridden to largest
2015-07-23 10:00:58 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-wikipedia_parser-1-1437642058864-2
2015-07-23 10:00:58 VerifiableProperties [INFO] Property group.id is overridden to undefined-samza-consumer-group-bb834e1a-1861-459e-a310-108667ccf2a0
2015-07-23 10:00:58 VerifiableProperties [INFO] Property zookeeper.connect is overridden to myserver2649:2181/
2015-07-23 10:00:58 VerifiableProperties [INFO] Verifying properties
2015-07-23 10:00:58 VerifiableProperties [INFO] Property client.id is overridden to samza_admin-wikipedia_parser-1-1437642058864-2
2015-07-23 10:00:58 VerifiableProperties [INFO] Property metadata.broker.list is overridden to myserver2663:9092,myserver2664:9092,myserver2665:9092
2015-07-23 10:00:58 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-07-23 10:00:58 ClientUtils$ [INFO] Fetching metadata from broker id:0,host:myserver2663,port:9092 with correlation id 0 for 1 topic(s) Set(wikipedia-raw)
2015-07-23 10:00:58 SyncProducer [INFO] Connected to myserver2663:9092 for producing
2015-07-23 10:00:59 SyncProducer [INFO] Disconnecting from myserver2663:9092
2015-07-23 10:00:59 KafkaSystemAdmin$ [INFO] Got metadata: Map(wikipedia-raw -> SystemStreamMetadata [streamName=wikipedia-raw, partitionMetadata={Partition [partition=0]=SystemStreamPartitionMetadata [oldestOffset=null, newestOffset=null, upcomingOffset=0]}])
2015-07-23 10:00:59 KafkaCheckpointManager [INFO] Attempting to create checkpoint topic __samza_checkpoint_ver_1_for_wikipedia-parser_1.
2015-07-23 10:00:59 ZkEventThread [INFO] Starting ZkClient event thread.
2015-07-23 10:00:59 ZooKeeper [INFO] Client environment:zookeeper.version=3.4.5-cdh5.4.0--1, built on 04/21/2015 19:12 GMT
2015-07-23 10:00:59 ZooKeeper [INFO] Client environment:host.name=myserver2660.intranet.mycompany.com
2015-07-23 10:00:59 ZooKeeper [INFO] Client environment:java.version=1.7.0_67
2015-07-23 10:00:59 ZooKeeper [INFO] Client environment:java.vendor=Oracle Corporation
2015-07-23 10:00:59 ZooKeeper [INFO] Client environment:java.home=/usr/java/jdk1.7.0_67-cloudera/jre
2015-07-23 10:00:59 ZooKeeper [INFO] Client environment:java.class.path=/var/run/cloudera-scm-agent/process/7124-yarn-NODEMANAGER:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/activation-1.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/akka-actor_2.10-2.1.2.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/aopalliance-1.0.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/apacheds-i18n-2.0.0-M15.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/api-asn1-api-1.0.0-M20.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/api-util-1.0.0-M20.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/asm-3.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/avro-1.7.6-cdh5.4.0.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/cglib-2.2.1-v20090111.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/commons-beanutils-1.7.0.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/commons-beanutils-core-1.8.0.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/commons-cli-1.2.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/commons-codec-1.6.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/commons-collections-3.2.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/commons-compress-1.4.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/commons-configuration-1.6.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/commons-daemon-1.0.13.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/commons-digester-1.8.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/commons-el-1.0.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/commons-httpclient-3.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/commons-io-2.4.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/commons-lang-2.6.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/commons-logging-1.1.3.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/commons-math3-3.1.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/commons-net-3.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/config-1.0.0.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/curator-client-2.7.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/curator-framework-2.7.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/curator-recipes-2.7.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/grizzled-slf4j_2.10-1.0.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/gson-2.2.4.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/guava-16.0.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/guice-3.0.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/guice-servlet-3.0.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/hadoop-annotations-2.6.0-cdh5.4.0.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/hadoop-auth-2.6.0-cdh5.4.0.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/hadoop-common-2.6.0-cdh5.4.0.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/hadoop-hdfs-2.6.0-cdh5.4.0.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/hadoop-yarn-api-2.6.0-cdh5.4.0.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/hadoop-yarn-client-2.6.0-cdh5.4.0.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/hadoop-yarn-common-2.6.0-cdh5.4.0.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/htrace-core-3.0.4.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/httpclient-4.2.5.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/httpcore-4.2.4.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/irclib-1.10.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jackson-core-asl-1.9.13.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jackson-jaxrs-1.8.8.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jackson-mapper-asl-1.9.13.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jackson-xc-1.8.8.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jasper-compiler-5.5.23.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jasper-runtime-5.5.23.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/java-xmlbuilder-0.4.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/javax.inject-1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/javax.servlet-3.0.0.v201112011016.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jaxb-api-2.2.2.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jaxb-impl-2.2.3-1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jersey-client-1.9.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jersey-core-1.9.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jersey-guice-1.9.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jersey-json-1.9.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jersey-server-1.9.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jets3t-0.9.0.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jettison-1.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jetty-6.1.26.cloudera.4.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jetty-continuation-8.1.8.v20121106.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jetty-http-8.1.8.v20121106.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jetty-io-8.1.8.v20121106.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jetty-security-8.1.8.v20121106.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jetty-server-8.1.8.v20121106.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jetty-servlet-8.1.8.v20121106.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jetty-util-6.1.26.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jetty-util-8.1.8.v20121106.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jetty-webapp-8.1.8.v20121106.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jetty-xml-8.1.8.v20121106.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jline-0.9.94.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/joda-convert-1.2.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/joda-time-2.2.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jopt-simple-3.2.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jsch-0.1.42.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jsp-api-2.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/jsr305-3.0.0.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/junit-3.8.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/juniversalchardet-1.0.3.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/kafka-clients-0.8.2.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/kafka_2.10-0.8.2.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/log4j-1.2.17.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/lz4-1.2.0.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/metrics-core-2.2.0.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/mime-util-2.1.3.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/netty-3.7.0.Final.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/paranamer-2.3.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/protobuf-java-2.5.0.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/rl_2.10-0.4.4.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/rocksdbjni-3.5.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/samza-api-0.9.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/samza-core_2.10-0.9.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/samza-hello-samza-master-0.9.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/samza-kafka_2.10-0.9.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/samza-kv-rocksdb_2.10-0.9.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/samza-kv_2.10-0.9.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/samza-log4j-0.9.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/samza-shell-0.9.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/samza-yarn_2.10-0.9.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/scala-compiler-2.10.4.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/scala-library-2.10.4.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/scala-reflect-2.10.4.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/scalate-core_2.10-1.6.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/scalate-util_2.10-1.6.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/scalatra-common_2.10-2.2.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/scalatra-scalate_2.10-2.2.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/scalatra_2.10-2.2.1.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/servlet-api-2.5.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/slf4j-api-1.7.7.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/slf4j-log4j12-1.7.7.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/snappy-java-1.1.1.6.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/stax-api-1.0-2.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/xmlenc-0.52.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/xz-1.0.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/zkclient-0.3.jar:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/lib/zookeeper-3.4.5-cdh5.4.0.jar
2015-07-23 10:00:59 ZooKeeper [INFO] Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2015-07-23 10:00:59 ZooKeeper [INFO] Client environment:java.io.tmpdir=/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/tmp
2015-07-23 10:00:59 ZooKeeper [INFO] Client environment:java.compiler=<NA>
2015-07-23 10:00:59 ZooKeeper [INFO] Client environment:os.name=Linux
2015-07-23 10:00:59 ZooKeeper [INFO] Client environment:os.arch=amd64
2015-07-23 10:00:59 ZooKeeper [INFO] Client environment:os.version=2.6.32-504.16.2.el6.x86_64
2015-07-23 10:00:59 ZooKeeper [INFO] Client environment:user.name=myuser
2015-07-23 10:00:59 ZooKeeper [INFO] Client environment:user.home=/home/myuser
2015-07-23 10:00:59 ZooKeeper [INFO] Client environment:user.dir=/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001
2015-07-23 10:00:59 ZooKeeper [INFO] Initiating client connection, connectString=myserver2649:2181/ sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@7dc94405
2015-07-23 10:00:59 ClientCnxn [INFO] Opening socket connection to server myserver2649.intranet.mycompany.com/10.xx.xx.1:2181. Will not attempt to authenticate using SASL (unknown error)
2015-07-23 10:00:59 ClientCnxn [INFO] Socket connection established to myserver2649.intranet.mycompany.com/10.xx.xx.1:2181, initiating session
2015-07-23 10:00:59 ClientCnxn [INFO] Session establishment complete on server myserver2649.intranet.mycompany.com/10.xx.xx.1:2181, sessionid = 0x34e4ea7b9cba938, negotiated timeout = 6000
2015-07-23 10:00:59 ZkClient [INFO] zookeeper state changed (SyncConnected)
2015-07-23 10:00:59 ZkEventThread [INFO] Terminate ZkClient event thread.
2015-07-23 10:00:59 ZooKeeper [INFO] Session: 0x34e4ea7b9cba938 closed
2015-07-23 10:00:59 ClientCnxn [INFO] EventThread shut down
2015-07-23 10:00:59 KafkaCheckpointManager [INFO] Checkpoint topic __samza_checkpoint_ver_1_for_wikipedia-parser_1 already exists.
2015-07-23 10:00:59 KafkaCheckpointManager [INFO] Validating checkpoint topic __samza_checkpoint_ver_1_for_wikipedia-parser_1.
2015-07-23 10:00:59 VerifiableProperties [INFO] Verifying properties
2015-07-23 10:00:59 VerifiableProperties [INFO] Property client.id is overridden to samza_checkpoint_manager-wikipedia_parser-1-1437642058788-1
2015-07-23 10:00:59 VerifiableProperties [INFO] Property metadata.broker.list is overridden to myserver2663:9092,myserver2664:9092,myserver2665:9092
2015-07-23 10:00:59 VerifiableProperties [INFO] Property request.timeout.ms is overridden to 30000
2015-07-23 10:00:59 ClientUtils$ [INFO] Fetching metadata from broker id:0,host:myserver2663,port:9092 with correlation id 0 for 1 topic(s) Set(__samza_checkpoint_ver_1_for_wikipedia-parser_1)
2015-07-23 10:00:59 SyncProducer [INFO] Connected to myserver2663:9092 for producing
2015-07-23 10:00:59 SyncProducer [INFO] Disconnecting from myserver2663:9092
2015-07-23 10:00:59 KafkaCheckpointManager [INFO] Successfully validated checkpoint topic __samza_checkpoint_ver_1_for_wikipedia-parser_1.
2015-07-23 10:00:59 KafkaCheckpointManager [INFO] Connecting to leader myserver2663.intranet.mycompany.com:9092 for topic __samza_checkpoint_ver_1_for_wikipedia-parser_1 and to fetch all checkpoint messages.
2015-07-23 10:00:59 KafkaCheckpointManager [INFO] Got offset 0 for topic __samza_checkpoint_ver_1_for_wikipedia-parser_1 and partition 0. Attempting to fetch messages for changelog partition mapping.
2015-07-23 10:00:59 KafkaCheckpointManager [INFO] Get latest offset 1 for topic __samza_checkpoint_ver_1_for_wikipedia-parser_1 and partition 0.
2015-07-23 10:00:59 JobCoordinator$ [INFO] SystemStreamPartitionGrouper org.apache.samza.container.grouper.stream.GroupByPartition@3022338b has grouped the SystemStreamPartitions into the following taskNames:
2015-07-23 10:00:59 JobCoordinator$ [INFO] Saving task-to-changelog partition mapping: Map(Partition 0 -> 0)
2015-07-23 10:00:59 ProducerConfig [INFO] ProducerConfig values:
value.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer
key.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer
block.on.buffer.full = true
retry.backoff.ms = 100
buffer.memory = 33554432
batch.size = 16384
metrics.sample.window.ms = 30000
metadata.max.age.ms = 300000
receive.buffer.bytes = 32768
timeout.ms = 30000
max.in.flight.requests.per.connection = 1
bootstrap.servers = [myserver2663:9092, myserver2664:9092, myserver2665:9092]
metric.reporters = []
client.id = samza_checkpoint_manager-wikipedia_parser-1-1437642058788-1
compression.type = none
retries = 2147483647
max.request.size = 1048576
send.buffer.bytes = 131072
acks = all
reconnect.backoff.ms = 10
linger.ms = 0
metrics.num.samples = 2
metadata.fetch.timeout.ms = 60000
2015-07-23 10:00:59 SamzaAppMasterService [INFO] Starting webapp at a random rpc and tracking port
2015-07-23 10:00:59 ClientHelper [INFO] trying to connect to RM 0.0.0.0:8032
2015-07-23 10:00:59 Server [INFO] jetty-8.1.8.v20121106
2015-07-23 10:00:59 ServletTemplateEngine [INFO] Scalate template engine using working directory: /data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/tmp/scalate-4241117790539801997-workdir
2015-07-23 10:00:59 AbstractConnector [INFO] Started SelectChannelConnector@0.0.0.0:58211
2015-07-23 10:00:59 HttpServer [INFO] Started HttpServer on: http://10.xx.xx.14:58211/
2015-07-23 10:00:59 Server [INFO] jetty-8.1.8.v20121106
2015-07-23 10:01:00 ServletTemplateEngine [INFO] Scalate template engine using working directory: /data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/container_e105_1437556818770_0108_01_000001/__package/tmp/scalate-4463253233581688239-workdir
2015-07-23 10:01:00 AbstractConnector [INFO] Started SelectChannelConnector@0.0.0.0:32842
2015-07-23 10:01:00 HttpServer [INFO] Started HttpServer on: http://10.xx.xx.14:32842/
2015-07-23 10:01:00 Server [INFO] jetty-8.1.8.v20121106
2015-07-23 10:01:00 AbstractConnector [INFO] Started SelectChannelConnector@0.0.0.0:42237
2015-07-23 10:01:00 HttpServer [INFO] Started HttpServer on: http://10.xx.xx.14:42237/
2015-07-23 10:01:00 JobCoordinator [INFO] Startd HTTP server: http://10.xx.xx.14:42237/
2015-07-23 10:01:00 SamzaAppMasterService [INFO] Webapp is started at (rpc http://10.xx.xx.14:58211/, tracking http://10.xx.xx.14:32842/, coordinator http://10.xx.xx.14:42237/)
2015-07-23 10:01:03 SamzaAppMasterLifecycle [INFO] Got AM register response. The YARN RM supports container requests with max-mem: 307200, max-cpu: 48
2015-07-23 10:01:03 MetricsSnapshotReporter [INFO] Starting producer.
2015-07-23 10:01:03 MetricsSnapshotReporter [INFO] Starting reporter timer.
2015-07-23 10:01:03 ContainerManagementProtocolProxy [INFO] yarn.client.max-cached-nodemanagers-proxies : 0
2015-07-23 10:01:03 SamzaAppMasterTaskManager [INFO] Requesting 1 containers
2015-07-23 10:01:03 SamzaAppMasterTaskManager [INFO] Requesting 1 container(s) with 1024mb of memory
2015-07-23 10:01:03 KafkaSystemProducer [INFO] Creating a new producer for system kafka.
2015-07-23 10:01:03 ProducerConfig [INFO] ProducerConfig values:
value.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer
key.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer
block.on.buffer.full = true
retry.backoff.ms = 100
buffer.memory = 33554432
batch.size = 16384
metrics.sample.window.ms = 30000
metadata.max.age.ms = 300000
receive.buffer.bytes = 32768
timeout.ms = 30000
max.in.flight.requests.per.connection = 1
bootstrap.servers = [myserver2663:9092, myserver2664:9092, myserver2665:9092]
metric.reporters = []
client.id = samza_producer-wikipedia_parser-1-1437642058692-0
compression.type = none
retries = 2147483647
max.request.size = 1048576
send.buffer.bytes = 131072
acks = 1
reconnect.backoff.ms = 10
linger.ms = 0
metrics.num.samples = 2
metadata.fetch.timeout.ms = 60000
2015-07-23 10:01:05 AMRMClientImpl [INFO] Received new token for : myserver2665.intranet.mycompany.com:8041
2015-07-23 10:01:05 SamzaAppMasterTaskManager [INFO] Got a container from YARN ResourceManager: Container: [ContainerId: container_e105_1437556818770_0108_01_000002, NodeId: myserver2665.intranet.mycompany.com:8041, NodeHttpAddress: myserver2665.intranet.mycompany.com:8042, Resource: <memory:17408, vCores:1>, Priority: 0, Token: Token { kind: ContainerToken, service: 10.xx.xx.19:8041 }, ]
2015-07-23 10:01:05 SamzaAppMasterTaskManager [INFO] Got available task id (0) for container: Container: [ContainerId: container_e105_1437556818770_0108_01_000002, NodeId: myserver2665.intranet.mycompany.com:8041, NodeHttpAddress: myserver2665.intranet.mycompany.com:8042, Resource: <memory:17408, vCores:1>, Priority: 0, Token: Token { kind: ContainerToken, service: 10.xx.xx.19:8041 }, ]
2015-07-23 10:01:05 SamzaAppMasterTaskManager [INFO] Claimed SSP taskNames ContainerModel [containerId=0, tasks={Partition 0=TaskModel [taskName=Partition 0, systemStreamPartitions=[SystemStreamPartition [kafka, wikipedia-raw, 0]], changeLogPartition=Partition [partition=0]]}] for container ID 0
2015-07-23 10:01:05 SamzaAppMasterTaskManager [INFO] Task ID 0 using command bin/run-container.sh
2015-07-23 10:01:05 SamzaAppMasterTaskManager [INFO] Task ID 0 using env Map(JAVA_OPTS -> , SAMZA_CONTAINER_ID -> 0, SAMZA_COORDINATOR_URL -> http://10.xx.xx.14:42237/)
2015-07-23 10:01:05 SamzaAppMasterTaskManager [INFO] Starting task ID 0 using package path hdfs://nameservice1/bigdata/projects/myprojectplayground/samza-hello-samza-master-0.9.1-dist.tar.gz
2015-07-23 10:01:05 SamzaAppMasterTaskManager [INFO] starting container hdfs://nameservice1/bigdata/projects/myprojectplayground/samza-hello-samza-master-0.9.1-dist.tar.gz Container: [ContainerId: container_e105_1437556818770_0108_01_000002, NodeId: myserver2665.intranet.mycompany.com:8041, NodeHttpAddress: myserver2665.intranet.mycompany.com:8042, Resource: <memory:17408, vCores:1>, Priority: 0, Token: Token { kind: ContainerToken, service: 10.xx.xx.19:8041 }, ] Map(JAVA_OPTS -> , SAMZA_CONTAINER_ID -> 0, SAMZA_COORDINATOR_URL -> http://10.xx.xx.14:42237/) WrappedArray(export SAMZA_LOG_DIR=<LOG_DIR> && ln -sfn <LOG_DIR> logs && exec ./__package/bin/run-container.sh 1>logs/stdout 2>logs/stderr)
2015-07-23 10:01:05 NativeCodeLoader [WARN] Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException as:myuser (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 Client [WARN] Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException as:myuser (auth:KERBEROS) cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException as:myuser (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 Client [WARN] Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException as:myuser (auth:KERBEROS) cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 RetryInvocationHandler [INFO] Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB over myserver2649.intranet.mycompany.com/10.xx.xx.1:8020 after 1 fail over attempts. Trying to fail over immediately.
java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "myserver2660.intranet.mycompany.com/10.xx.xx.14"; destination host is: "myserver2649.intranet.mycompany.com":8020;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
at org.apache.hadoop.ipc.Client.call(Client.java:1472)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy23.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy24.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1980)
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1128)
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1124)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1124)
at org.apache.samza.job.yarn.SamzaAppMasterTaskManager.startContainer(SamzaAppMasterTaskManager.scala:273)
at org.apache.samza.job.yarn.SamzaAppMasterTaskManager.onContainerAllocated(SamzaAppMasterTaskManager.scala:121)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
at scala.collection.immutable.List.foreach(List.scala:318)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at org.apache.samza.job.yarn.SamzaAppMaster$.onContainersAllocated(SamzaAppMaster.scala:137)
at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:292)
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:643)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:730)
at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
at org.apache.hadoop.ipc.Client.call(Client.java:1438)
... 29 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:553)
at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:368)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:722)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:718)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:717)
... 32 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
... 41 more
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException as:myuser (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 Client [WARN] Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException as:myuser (auth:KERBEROS) cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 RetryInvocationHandler [INFO] Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB over myserver2650.intranet.mycompany.com/10.xx.xx.2:8020 after 2 fail over attempts. Trying to fail over immediately.
java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "myserver2660.intranet.mycompany.com/10.xx.xx.14"; destination host is: "myserver2650.intranet.mycompany.com":8020;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
at org.apache.hadoop.ipc.Client.call(Client.java:1472)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy23.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy24.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1980)
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1128)
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1124)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1124)
at org.apache.samza.job.yarn.SamzaAppMasterTaskManager.startContainer(SamzaAppMasterTaskManager.scala:273)
at org.apache.samza.job.yarn.SamzaAppMasterTaskManager.onContainerAllocated(SamzaAppMasterTaskManager.scala:121)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
at scala.collection.immutable.List.foreach(List.scala:318)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at org.apache.samza.job.yarn.SamzaAppMaster$.onContainersAllocated(SamzaAppMaster.scala:137)
at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:292)
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:643)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:730)
at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
at org.apache.hadoop.ipc.Client.call(Client.java:1438)
... 29 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:553)
at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:368)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:722)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:718)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:717)
... 32 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
... 41 more
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException as:myuser (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 Client [WARN] Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException as:myuser (auth:KERBEROS) cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 RetryInvocationHandler [INFO] Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB over myserver2649.intranet.mycompany.com/10.xx.xx.1:8020 after 3 fail over attempts. Trying to fail over immediately.
java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "myserver2660.intranet.mycompany.com/10.xx.xx.14"; destination host is: "myserver2649.intranet.mycompany.com":8020;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
at org.apache.hadoop.ipc.Client.call(Client.java:1472)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy23.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy24.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1980)
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1128)
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1124)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1124)
at org.apache.samza.job.yarn.SamzaAppMasterTaskManager.startContainer(SamzaAppMasterTaskManager.scala:273)
at org.apache.samza.job.yarn.SamzaAppMasterTaskManager.onContainerAllocated(SamzaAppMasterTaskManager.scala:121)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
at scala.collection.immutable.List.foreach(List.scala:318)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at org.apache.samza.job.yarn.SamzaAppMaster$.onContainersAllocated(SamzaAppMaster.scala:137)
at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:292)
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:643)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:730)
at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
at org.apache.hadoop.ipc.Client.call(Client.java:1438)
... 29 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:553)
at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:368)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:722)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:718)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:717)
... 32 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
... 41 more
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException as:myuser (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 Client [WARN] Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException as:myuser (auth:KERBEROS) cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 RetryInvocationHandler [INFO] Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB over myserver2650.intranet.mycompany.com/10.xx.xx.2:8020 after 4 fail over attempts. Trying to fail over immediately.
java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "myserver2660.intranet.mycompany.com/10.xx.xx.14"; destination host is: "myserver2650.intranet.mycompany.com":8020;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
at org.apache.hadoop.ipc.Client.call(Client.java:1472)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy23.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy24.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1980)
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1128)
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1124)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1124)
at org.apache.samza.job.yarn.SamzaAppMasterTaskManager.startContainer(SamzaAppMasterTaskManager.scala:273)
at org.apache.samza.job.yarn.SamzaAppMasterTaskManager.onContainerAllocated(SamzaAppMasterTaskManager.scala:121)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
at scala.collection.immutable.List.foreach(List.scala:318)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at org.apache.samza.job.yarn.SamzaAppMaster$.onContainersAllocated(SamzaAppMaster.scala:137)
at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:292)
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:643)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:730)
at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
at org.apache.hadoop.ipc.Client.call(Client.java:1438)
... 29 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:553)
at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:368)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:722)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:718)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:717)
... 32 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
... 41 more
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException as:myuser (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 Client [WARN] Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException as:myuser (auth:KERBEROS) cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 RetryInvocationHandler [INFO] Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB over myserver2649.intranet.mycompany.com/10.xx.xx.1:8020 after 5 fail over attempts. Trying to fail over immediately.
java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "myserver2660.intranet.mycompany.com/10.xx.xx.14"; destination host is: "myserver2649.intranet.mycompany.com":8020;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
at org.apache.hadoop.ipc.Client.call(Client.java:1472)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy23.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy24.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1980)
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1128)
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1124)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1124)
at org.apache.samza.job.yarn.SamzaAppMasterTaskManager.startContainer(SamzaAppMasterTaskManager.scala:273)
at org.apache.samza.job.yarn.SamzaAppMasterTaskManager.onContainerAllocated(SamzaAppMasterTaskManager.scala:121)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
at scala.collection.immutable.List.foreach(List.scala:318)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at org.apache.samza.job.yarn.SamzaAppMaster$.onContainersAllocated(SamzaAppMaster.scala:137)
at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:292)
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:643)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:730)
at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
at org.apache.hadoop.ipc.Client.call(Client.java:1438)
... 29 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:553)
at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:368)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:722)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:718)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:717)
... 32 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
... 41 more
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException as:myuser (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 Client [WARN] Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException as:myuser (auth:KERBEROS) cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 RetryInvocationHandler [INFO] Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB over myserver2650.intranet.mycompany.com/10.xx.xx.2:8020 after 6 fail over attempts. Trying to fail over immediately.
java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "myserver2660.intranet.mycompany.com/10.xx.xx.14"; destination host is: "myserver2650.intranet.mycompany.com":8020;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
at org.apache.hadoop.ipc.Client.call(Client.java:1472)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy23.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy24.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1980)
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1128)
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1124)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1124)
at org.apache.samza.job.yarn.SamzaAppMasterTaskManager.startContainer(SamzaAppMasterTaskManager.scala:273)
at org.apache.samza.job.yarn.SamzaAppMasterTaskManager.onContainerAllocated(SamzaAppMasterTaskManager.scala:121)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
at scala.collection.immutable.List.foreach(List.scala:318)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at org.apache.samza.job.yarn.SamzaAppMaster$.onContainersAllocated(SamzaAppMaster.scala:137)
at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:292)
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:643)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:730)
at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
at org.apache.hadoop.ipc.Client.call(Client.java:1438)
... 29 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:553)
at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:368)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:722)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:718)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:717)
... 32 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
... 41 more
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException as:myuser (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 Client [WARN] Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException as:myuser (auth:KERBEROS) cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 RetryInvocationHandler [INFO] Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB over myserver2649.intranet.mycompany.com/10.xx.xx.1:8020 after 7 fail over attempts. Trying to fail over immediately.
java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "myserver2660.intranet.mycompany.com/10.xx.xx.14"; destination host is: "myserver2649.intranet.mycompany.com":8020;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
at org.apache.hadoop.ipc.Client.call(Client.java:1472)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy23.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy24.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1980)
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1128)
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1124)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1124)
at org.apache.samza.job.yarn.SamzaAppMasterTaskManager.startContainer(SamzaAppMasterTaskManager.scala:273)
at org.apache.samza.job.yarn.SamzaAppMasterTaskManager.onContainerAllocated(SamzaAppMasterTaskManager.scala:121)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
at scala.collection.immutable.List.foreach(List.scala:318)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at org.apache.samza.job.yarn.SamzaAppMaster$.onContainersAllocated(SamzaAppMaster.scala:137)
at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:292)
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:643)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:730)
at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
at org.apache.hadoop.ipc.Client.call(Client.java:1438)
... 29 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:553)
at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:368)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:722)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:718)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:717)
... 32 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
... 41 more
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException as:myuser (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 Client [WARN] Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException as:myuser (auth:KERBEROS) cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 RetryInvocationHandler [INFO] Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB over myserver2650.intranet.mycompany.com/10.xx.xx.2:8020 after 8 fail over attempts. Trying to fail over immediately.
java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "myserver2660.intranet.mycompany.com/10.xx.xx.14"; destination host is: "myserver2650.intranet.mycompany.com":8020;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
at org.apache.hadoop.ipc.Client.call(Client.java:1472)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy23.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy24.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1980)
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1128)
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1124)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1124)
at org.apache.samza.job.yarn.SamzaAppMasterTaskManager.startContainer(SamzaAppMasterTaskManager.scala:273)
at org.apache.samza.job.yarn.SamzaAppMasterTaskManager.onContainerAllocated(SamzaAppMasterTaskManager.scala:121)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
at scala.collection.immutable.List.foreach(List.scala:318)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at org.apache.samza.job.yarn.SamzaAppMaster$.onContainersAllocated(SamzaAppMaster.scala:137)
at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:292)
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:643)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:730)
at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
at org.apache.hadoop.ipc.Client.call(Client.java:1438)
... 29 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:553)
at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:368)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:722)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:718)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:717)
... 32 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
... 41 more
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException as:myuser (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 Client [WARN] Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException as:myuser (auth:KERBEROS) cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 RetryInvocationHandler [INFO] Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB over myserver2649.intranet.mycompany.com/10.xx.xx.1:8020 after 9 fail over attempts. Trying to fail over immediately.
java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "myserver2660.intranet.mycompany.com/10.xx.xx.14"; destination host is: "myserver2649.intranet.mycompany.com":8020;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
at org.apache.hadoop.ipc.Client.call(Client.java:1472)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy23.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy24.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1980)
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1128)
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1124)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1124)
at org.apache.samza.job.yarn.SamzaAppMasterTaskManager.startContainer(SamzaAppMasterTaskManager.scala:273)
at org.apache.samza.job.yarn.SamzaAppMasterTaskManager.onContainerAllocated(SamzaAppMasterTaskManager.scala:121)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
at scala.collection.immutable.List.foreach(List.scala:318)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at org.apache.samza.job.yarn.SamzaAppMaster$.onContainersAllocated(SamzaAppMaster.scala:137)
at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:292)
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:643)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:730)
at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
at org.apache.hadoop.ipc.Client.call(Client.java:1438)
... 29 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:553)
at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:368)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:722)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:718)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:717)
... 32 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
... 41 more
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException as:myuser (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 Client [WARN] Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException as:myuser (auth:KERBEROS) cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 RetryInvocationHandler [INFO] Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB over myserver2650.intranet.mycompany.com/10.xx.xx.2:8020 after 10 fail over attempts. Trying to fail over immediately.
java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "myserver2660.intranet.mycompany.com/10.xx.xx.14"; destination host is: "myserver2650.intranet.mycompany.com":8020;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
at org.apache.hadoop.ipc.Client.call(Client.java:1472)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy23.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy24.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1980)
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1128)
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1124)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1124)
at org.apache.samza.job.yarn.SamzaAppMasterTaskManager.startContainer(SamzaAppMasterTaskManager.scala:273)
at org.apache.samza.job.yarn.SamzaAppMasterTaskManager.onContainerAllocated(SamzaAppMasterTaskManager.scala:121)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
at scala.collection.immutable.List.foreach(List.scala:318)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at org.apache.samza.job.yarn.SamzaAppMaster$.onContainersAllocated(SamzaAppMaster.scala:137)
at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:292)
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:643)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:730)
at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
at org.apache.hadoop.ipc.Client.call(Client.java:1438)
... 29 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:553)
at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:368)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:722)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:718)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:717)
... 32 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
... 41 more
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException as:myuser (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 Client [WARN] Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException as:myuser (auth:KERBEROS) cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 RetryInvocationHandler [INFO] Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB over myserver2649.intranet.mycompany.com/10.xx.xx.1:8020 after 11 fail over attempts. Trying to fail over immediately.
java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "myserver2660.intranet.mycompany.com/10.xx.xx.14"; destination host is: "myserver2649.intranet.mycompany.com":8020;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
at org.apache.hadoop.ipc.Client.call(Client.java:1472)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy23.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy24.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1980)
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1128)
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1124)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1124)
at org.apache.samza.job.yarn.SamzaAppMasterTaskManager.startContainer(SamzaAppMasterTaskManager.scala:273)
at org.apache.samza.job.yarn.SamzaAppMasterTaskManager.onContainerAllocated(SamzaAppMasterTaskManager.scala:121)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
at scala.collection.immutable.List.foreach(List.scala:318)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at org.apache.samza.job.yarn.SamzaAppMaster$.onContainersAllocated(SamzaAppMaster.scala:137)
at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:292)
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:643)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:730)
at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
at org.apache.hadoop.ipc.Client.call(Client.java:1438)
... 29 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:553)
at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:368)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:722)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:718)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:717)
... 32 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
... 41 more
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException as:myuser (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 Client [WARN] Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException as:myuser (auth:KERBEROS) cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 RetryInvocationHandler [INFO] Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB over myserver2650.intranet.mycompany.com/10.xx.xx.2:8020 after 12 fail over attempts. Trying to fail over immediately.
java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "myserver2660.intranet.mycompany.com/10.xx.xx.14"; destination host is: "myserver2650.intranet.mycompany.com":8020;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
at org.apache.hadoop.ipc.Client.call(Client.java:1472)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy23.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy24.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1980)
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1128)
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1124)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1124)
at org.apache.samza.job.yarn.SamzaAppMasterTaskManager.startContainer(SamzaAppMasterTaskManager.scala:273)
at org.apache.samza.job.yarn.SamzaAppMasterTaskManager.onContainerAllocated(SamzaAppMasterTaskManager.scala:121)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
at scala.collection.immutable.List.foreach(List.scala:318)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at org.apache.samza.job.yarn.SamzaAppMaster$.onContainersAllocated(SamzaAppMaster.scala:137)
at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:292)
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:643)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:730)
at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
at org.apache.hadoop.ipc.Client.call(Client.java:1438)
... 29 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:553)
at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:368)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:722)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:718)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:717)
... 32 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
... 41 more
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException as:myuser (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 Client [WARN] Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException as:myuser (auth:KERBEROS) cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 RetryInvocationHandler [INFO] Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB over myserver2649.intranet.mycompany.com/10.xx.xx.1:8020 after 13 fail over attempts. Trying to fail over immediately.
java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "myserver2660.intranet.mycompany.com/10.xx.xx.14"; destination host is: "myserver2649.intranet.mycompany.com":8020;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
at org.apache.hadoop.ipc.Client.call(Client.java:1472)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy23.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy24.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1980)
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1128)
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1124)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1124)
at org.apache.samza.job.yarn.SamzaAppMasterTaskManager.startContainer(SamzaAppMasterTaskManager.scala:273)
at org.apache.samza.job.yarn.SamzaAppMasterTaskManager.onContainerAllocated(SamzaAppMasterTaskManager.scala:121)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
at scala.collection.immutable.List.foreach(List.scala:318)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at org.apache.samza.job.yarn.SamzaAppMaster$.onContainersAllocated(SamzaAppMaster.scala:137)
at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:292)
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:643)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:730)
at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
at org.apache.hadoop.ipc.Client.call(Client.java:1438)
... 29 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:553)
at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:368)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:722)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:718)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:717)
... 32 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
... 41 more
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException as:myuser (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 Client [WARN] Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException as:myuser (auth:KERBEROS) cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 RetryInvocationHandler [INFO] Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB over myserver2650.intranet.mycompany.com/10.xx.xx.2:8020 after 14 fail over attempts. Trying to fail over immediately.
java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "myserver2660.intranet.mycompany.com/10.xx.xx.14"; destination host is: "myserver2650.intranet.mycompany.com":8020;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
at org.apache.hadoop.ipc.Client.call(Client.java:1472)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy23.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy24.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1980)
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1128)
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1124)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1124)
at org.apache.samza.job.yarn.SamzaAppMasterTaskManager.startContainer(SamzaAppMasterTaskManager.scala:273)
at org.apache.samza.job.yarn.SamzaAppMasterTaskManager.onContainerAllocated(SamzaAppMasterTaskManager.scala:121)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
at scala.collection.immutable.List.foreach(List.scala:318)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at org.apache.samza.job.yarn.SamzaAppMaster$.onContainersAllocated(SamzaAppMaster.scala:137)
at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:292)
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:643)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:730)
at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
at org.apache.hadoop.ipc.Client.call(Client.java:1438)
... 29 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:553)
at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:368)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:722)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:718)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:717)
... 32 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
... 41 more
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException as:myuser (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 Client [WARN] Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 UserGroupInformation [WARN] PriviledgedActionException as:myuser (auth:KERBEROS) cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2015-07-23 10:01:05 RetryInvocationHandler [WARN] Exception while invoking class org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo over myserver2649.intranet.mycompany.com/10.xx.xx.1:8020. Not retrying because failovers (15) exceeded maximum allowed (15)
java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "myserver2660.intranet.mycompany.com/10.xx.xx.14"; destination host is: "myserver2649.intranet.mycompany.com":8020;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
at org.apache.hadoop.ipc.Client.call(Client.java:1472)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy23.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy24.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1980)
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1128)
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1124)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1124)
at org.apache.samza.job.yarn.SamzaAppMasterTaskManager.startContainer(SamzaAppMasterTaskManager.scala:273)
at org.apache.samza.job.yarn.SamzaAppMasterTaskManager.onContainerAllocated(SamzaAppMasterTaskManager.scala:121)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
at scala.collection.immutable.List.foreach(List.scala:318)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at org.apache.samza.job.yarn.SamzaAppMaster$.onContainersAllocated(SamzaAppMaster.scala:137)
at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:292)
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:643)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:730)
at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
at org.apache.hadoop.ipc.Client.call(Client.java:1438)
... 29 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:553)
at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:368)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:722)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:718)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:717)
... 32 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
... 41 more
2015-07-23 10:01:05 SamzaAppMaster$ [ERROR] Error occured in amClient's callback
java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "myserver2660.intranet.mycompany.com/10.xx.xx.14"; destination host is: "myserver2649.intranet.mycompany.com":8020;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
at org.apache.hadoop.ipc.Client.call(Client.java:1472)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy23.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy24.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1980)
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1128)
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1124)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1124)
at org.apache.samza.job.yarn.SamzaAppMasterTaskManager.startContainer(SamzaAppMasterTaskManager.scala:273)
at org.apache.samza.job.yarn.SamzaAppMasterTaskManager.onContainerAllocated(SamzaAppMasterTaskManager.scala:121)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1$$anonfun$apply$6.apply(SamzaAppMaster.scala:137)
at scala.collection.immutable.List.foreach(List.scala:318)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
at org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$onContainersAllocated$1.apply(SamzaAppMaster.scala:137)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at org.apache.samza.job.yarn.SamzaAppMaster$.onContainersAllocated(SamzaAppMaster.scala:137)
at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:292)
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:643)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:730)
at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
at org.apache.hadoop.ipc.Client.call(Client.java:1438)
... 29 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:553)
at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:368)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:722)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:718)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:717)
... 32 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
... 41 more
2015-07-23 10:01:06 ContextHandler [INFO] stopped o.e.j.s.ServletContextHandler{/,jar:file:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/filecache/10/samza-hello-samza-master-0.9.1-dist.tar.gz/lib/samza-yarn_2.10-0.9.1.jar!/scalate}
2015-07-23 10:01:07 HttpServer [INFO] Stopped server
2015-07-23 10:01:07 ContextHandler [INFO] stopped o.e.j.s.ServletContextHandler{/,jar:file:/data6/yarn/nm/usercache/myuser/appcache/application_1437556818770_0108/filecache/10/samza-hello-samza-master-0.9.1-dist.tar.gz/lib/samza-yarn_2.10-0.9.1.jar!/scalate}
2015-07-23 10:01:07 HttpServer [INFO] Stopped server
2015-07-23 10:01:07 ContextHandler [INFO] stopped o.e.j.s.ServletContextHandler{/,null}
2015-07-23 10:01:07 HttpServer [INFO] Stopped server
2015-07-23 10:01:07 JobCoordinator [INFO] Stopped HTTP server.
2015-07-23 10:01:07 SamzaAppMasterLifecycle [INFO] Shutting down.
2015-07-23 10:01:07 AMRMClientImpl [INFO] Waiting for application to be successfully unregistered.
2015-07-23 10:01:07 MetricsSnapshotReporter [INFO] Stopping producer.
2015-07-23 10:01:07 MetricsSnapshotReporter [INFO] Stopping reporter timer.
> Support for Kerberos
> --------------------
>
> Key: SAMZA-727
> URL: https://issues.apache.org/jira/browse/SAMZA-727
> Project: Samza
> Issue Type: New Feature
> Components: yarn
> Affects Versions: 0.9.0
> Environment: YARN with Kerberos
> Reporter: Qi FU
>
> Samza doesn't support Kerberos, which is very common for YARN cluster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)