You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sean Owen (JIRA)" <ji...@apache.org> on 2016/06/20 12:31:05 UTC
[jira] [Commented] (SPARK-16067) Spark overriding JAAS privilege
using keytab
[ https://issues.apache.org/jira/browse/SPARK-16067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15339414#comment-15339414 ]
Sean Owen commented on SPARK-16067:
-----------------------------------
I think this ends up being a JAAS question, not Spark.
> Spark overriding JAAS privilege using keytab
> ---------------------------------------------
>
> Key: SPARK-16067
> URL: https://issues.apache.org/jira/browse/SPARK-16067
> Project: Spark
> Issue Type: Bug
> Reporter: Partha Pratim Ghosh
>
> I am using a JAAS doAsPrivileged method with kerberos (with keytab) authenticated subject to invoke a Spark configuration but the spark conf is opening with the Kerberos authentication from system cache instead. I want that to use the authentication from JAAS.
> Following is my JAAS file -
> public void sparkJaas(){
> final String principal = "user2";
> final String keytab = "/app/user2.keytab";
> /*final String principal = "user1";
> final String keytab = "/app/user1.keytab";*/
> final Subject subject;
>
> subject = JaasKerbCall.getInstance().login(principal);
> Subject.doAsPrivileged(subject, new PrivilegedAction<Object>() {
> public Object run() {
> String classServerUri = "http://<server host>:<server port>";
> Set<Principal> principals = subject.getPrincipals();
> for (Principal principal : principals) {
> System.out.println("amlpoc : Subject principal" + principal.getName());
> }
> String sparkBasePath = "/app/spark-1.5.0-bin-hadoop2.6";
> File pysparkPath = new File(sparkBasePath, "python" + File.separator + "lib");
> File sparkPath = new File(sparkBasePath, "lib");
> String[] sparkLibs = new String[] { "spark-assembly-1.5.0-hadoop2.6.0.jar" };
> // Open Spark context
> SparkConf conf = new SparkConf().setMaster("yarn-client").setAppName("spark-test")
> .set("spark.repl.class.uri", classServerUri);
> conf.setSparkHome(sparkBasePath);
>
>
> conf.set("spark.app.name", "spark-test");
> conf.set("spark.executor.memory", "8g");
> conf.set("spark.scheduler.mode", "FAIR");
> conf.set("spark.yarn.principal", principal);
> conf.set("spark.yarn.keytab", keytab);
> // Only one of py4j-0.9-src.zip and py4j-0.8.2.1-src.zip should exist
> String[] pythonLibs = new String[] { "pyspark.zip", "py4j-0.9-src.zip", "py4j-0.8.2.1-src.zip" };
> ArrayList<String> pythonLibUris = new ArrayList<String>();
> for (String lib : pythonLibs) {
> File libFile = new File(pysparkPath, lib);
> if (libFile.exists()) {
> pythonLibUris.add(libFile.toURI().toString());
> }
> }
> for (String lib : sparkLibs) {
> File libFile = new File(sparkPath, lib);
> if (libFile.exists()) {
> pythonLibUris.add(libFile.toURI().toString());
> }
> }
> pythonLibUris.trimToSize();
> // Distribute two libraries(pyspark.zip and py4j-*.zip) to workers
> // when spark version is less than or equal to 1.4.1
> if (pythonLibUris.size() == 2) {
> try {
> String confValue = conf.get("spark.yarn.dist.files");
> conf.set("spark.yarn.dist.files", confValue + "," + Joiner.on(",").join(pythonLibUris));
> } catch (NoSuchElementException e) {
> conf.set("spark.yarn.dist.files", Joiner.on(",").join(pythonLibUris));
> }
> conf.set("spark.files", conf.get("spark.yarn.dist.files"));
> conf.set("spark.submit.pyArchives", Joiner.on(":").join(pythonLibs));
> }
> conf.set("spark.yarn.isPython", "true");
> SparkContext sparkContext = new SparkContext(conf);
> System.out.println("SparkContext created : AppId : " + sparkContext.getConf().getAppId());
> return sparkContext;
> }// End run()
> }, null);
> }
> Following is the kerberos log -
> [INFO]
> [INFO] --- exec-maven-plugin:1.4.0:java (default-cli) @ spark-connectivity ---
> >>> KeyTabInputStream, readName(): XX.XX.XX.XX
> >>> KeyTabInputStream, readName(): user2
> >>> KeyTab: load() entry length: 81; type: 18
> >>> KeyTabInputStream, readName(): XX.XX.XX.XX
> >>> KeyTabInputStream, readName(): user2
> >>> KeyTab: load() entry length: 65; type: 17
> >>> KeyTabInputStream, readName(): XX.XX.XX.XX
> >>> KeyTabInputStream, readName(): user2
> >>> KeyTab: load() entry length: 65; type: 17
> >>> KeyTabInputStream, readName(): XX.XX.XX.XX
> >>> KeyTabInputStream, readName(): user2
> >>> KeyTab: load() entry length: 65; type: 17
> Looking for keys for: user2@XX.XX.XX.XX
> Java config name: /app/java/spark-connectivity/src/main/resources/krb5.conf
> Loaded from Java config
> Added key: 17version: 1
> Added key: 17version: 1
> Added key: 17version: 1
> Added key: 18version: 1
> >>> KdcAccessibility: reset
> Looking for keys for: user2@XX.XX.XX.XX
> Added key: 17version: 1
> Added key: 17version: 1
> Added key: 17version: 1
> Added key: 18version: 1
> default etypes for default_tkt_enctypes: 17 16 23.
> >>> KrbAsReq creating message
> >>> KrbKdcReq send: kdc=kdcs2-yy.yy.yy.yy UDP:88, timeout=30000, number of retries =3, #bytes=160
> >>> KDCCommunication: kdc=kdcs2-yy.yy.yy.yy UDP:88, timeout=30000,Attempt =1, #bytes=160
> >>> KrbKdcReq send: #bytes read=255
> >>>Pre-Authentication Data:
> PA-DATA type = 2
> PA-ENC-TIMESTAMP
> >>>Pre-Authentication Data:
> PA-DATA type = 19
> PA-ETYPE-INFO2 etype = 17, salt = <salt>, s2kparams = null
> >>>Pre-Authentication Data:
> PA-DATA type = 13
> >>> KdcAccessibility: remove kdcs2-yy.yy.yy.yy:88
> >>> KDCRep: init() encoding tag is 126 req type is 11
> >>>KRBError:
> cTime is Wed Mar 30 13:18:20 EDT 2022 1648660700000
> sTime is Mon Jun 20 07:57:20 EDT 2016 1466423840000
> suSec is 762837
> error code is 25
> error Message is Additional pre-authentication required
> cname is user2@XX.XX.XX.XX
> sname is <server>/XX.XX.XX.XX@XX.XX.XX.XX
> eData provided.
> msgType is 30
> >>>Pre-Authentication Data:
> PA-DATA type = 2
> PA-ENC-TIMESTAMP
> >>>Pre-Authentication Data:
> PA-DATA type = 19
> PA-ETYPE-INFO2 etype = 17, salt = <salt>, s2kparams = null
> >>>Pre-Authentication Data:
> PA-DATA type = 13
> KRBError received: NEEDED_PREAUTH
> KrbAsReqBuilder: PREAUTH FAILED/REQ, re-send AS-REQ
> default etypes for default_tkt_enctypes: 17 16 23.
> Looking for keys for: user2@XX.XX.XX.XX
> Added key: 17version: 1
> Added key: 17version: 1
> Added key: 17version: 1
> Added key: 18version: 1
> Looking for keys for: user2@XX.XX.XX.XX
> Added key: 17version: 1
> Added key: 17version: 1
> Added key: 17version: 1
> Added key: 18version: 1
> default etypes for default_tkt_enctypes: 17 16 23.
> >>> EType: sun.security.krb5.internal.crypto.<CryptoType>
> >>> KrbAsReq creating message
> >>> KrbKdcReq send: kdc=kdcs2-yy.yy.yy.yy UDP:88, timeout=30000, number of retries =3, #bytes=247
> >>> KDCCommunication: kdc=kdcs2-yy.yy.yy.yy UDP:88, timeout=30000,Attempt =1, #bytes=247
> >>> KrbKdcReq send: #bytes read=632
> >>> KdcAccessibility: remove kdcs2-yy.yy.yy.yy:88
> Looking for keys for: user2@XX.XX.XX.XX
> Added key: 17version: 1
> Added key: 17version: 1
> Added key: 17version: 1
> Added key: 18version: 1
> >>> EType: sun.security.krb5.internal.crypto.<CryptoType>
> >>> KrbAsRep cons in KrbAsReq.getReply user2
> Authentication succeeded!
> <projName> : Subject principaluser2@XX.XX.XX.XX
> Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
> 16/06/20 07:57:21 INFO SparkContext: Running Spark version 1.5.0
> >>>KinitOptions cache name is /tmp/krb5cc_515
> >>>DEBUG <CCacheInputStream> client principal is user1@XX.XX.XX.XX
> Why is spark checking ticket cache when JAAS is providing keytab authentication?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org