[jira] [Created] (ZEPPELIN-5423) Running %spark.conf cell seem to break placeholder #{user} from config

["Theo Sardin (Jira)" <ji...@apache.org>]

Theo Sardin created ZEPPELIN-5423:
-------------------------------------

             Summary: Running %spark.conf cell seem to break placeholder #{user} from config
                 Key: ZEPPELIN-5423
                 URL: https://issues.apache.org/jira/browse/ZEPPELIN-5423
             Project: Zeppelin
          Issue Type: Bug
          Components: zeppelin-interpreter
    Affects Versions: 0.9.0
            Reporter: Theo Sardin


Hello,

First of all let me preface this by saying I'm by no means hyper well-versed in zeppelin/spark so please let me know if this behavior is intended or the issue doesn't make sense.

(Please note all the below is in a zeppelin + spark on *kubernetes* setup).

So the issue is as follow:

I have a config for the spark interpreter in zeppelin, in which I use several times the #\{user} "placeholder" (not sure about the correct term here) to set some config values to the LDAP user name of the current user. Most importantly I use a config "user.name" set to "#\{user}", that I then use for authentication stuff to other systems in a custom library that I have in my spark classpath. This works perfectly fine. 

Now, for some notebooks I am trying to tweak it with an additional %spark.conf cell on top of the note.  I just do something along these lines:

 
{code:scala|title=%spark.conf|borderStyle=solid}
spark.executor.instances 7

spark.executor.memory 3G

spark.executor.cores 2

{code}
This executes fine, but the issue comes up when I actually call my library and it tries to get the "user.name" value, I get the following output: 
{code:scala}
java.util.NoSuchElementException: user.name. {code}
 

To simplify the example, I just did the %spark.conf cell, and then just run:
{code:scala|title=%spark|borderStyle=solid}
sc.getConf.get("user.name")
{code}
Which yields the same output:
{code:scala}
java.util.NoSuchElementException: user.name. {code}
 

 Initially I thought that %spark.conf might reset the whole  config and you just have to explicitly write all the config values but obvisouly it is not the case as the rest of my config is here indeed. 

 

That's what leads me to believe that there is somehow a broken/unexpected interaction between spark.conf cells and #\{user} config value.

 

Thanks for any help,

Regards,

 

Theo.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)