You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by Patrik Iselind <pa...@gmail.com> on 2020/05/18 10:52:40 UTC
Version controlling the interpreter configurations
Hi everyone,
I'm using Zeppelin through a docker image. In this setup, I'd like to
version control my conf/interpreters.json file along with my Dockerfile and
other related files. This I can of course do already. There's a problem
related to this though.
When setting for example the AWS credentials in the interpreter
configuration, then I'm forced to write them out in clear text. I really
don't want to version control the credentials!
I've tried setting AWS_ACCESS_KEY_ID to "${AWS_ACCESS_KEY_ID_VAL}" but that
doesn't work. It will not expand. Setting AWS_ACCESS_KEY_ID to
"${AWS_ACCESS_KEY_ID}",
will not work as it will expand recursively.
For this reason I think it would be great if Zeppelin support
configurations that are of the type "expandable string" which would read
from the environment to expand the variable values once. That would allow
me to version control my interpreters.json without having to include my AWS
credentials.
Using the "password" type will only hide the password in the UI, it's still
cleartext in the interpreter.json file.
Do you see some other solution that might work here?
Best Regards,
Patrik Iselind
Re: Version controlling the interpreter configurations
Posted by Patrik Iselind <pa...@gmail.com>.
I managed to create the account now.
I've created
https://issues.apache.org/jira/browse/ZEPPELIN-4825
and
https://issues.apache.org/jira/browse/ZEPPELIN-4826
for the problems I've mentioned in this thread.
Patrik Iselind
On Tue, May 19, 2020 at 7:58 AM Patrik Iselind <pa...@gmail.com> wrote:
> Ok, how do I create that account then?
>
> Den mån 18 maj 2020 17:59Jeff Zhang <zj...@gmail.com> skrev:
>
>> No, the link is for committers, there's no prerequisites for creating new
>> apache account, just an email address.
>>
>> Patrik Iselind <pa...@gmail.com> 于2020年5月18日周一 下午11:51写道：
>>
>>> Thanks, for the hints.
>>>
>>> This part of Apache is new to me. Do I have to do everything on
>>> https://www.apache.org/dev/new-committers-guide.html to get that Apache
>>> account set up?
>>>
>>> Patrik Iselind
>>>
>>>
>>> On Mon, May 18, 2020 at 5:34 PM Jeff Zhang <zj...@gmail.com> wrote:
>>>
>>>> You need to create apache account first, then create ticket here
>>>> https://issues.apache.org/jira/browse/ZEPPELIN
>>>>
>>>>
>>>> Patrik Iselind <pa...@gmail.com> 于2020年5月18日周一 下午11:07写道：
>>>>
>>>>> I'm not allowed to create tickets in your Jira. What's the procedure
>>>>> to get a Jira account?
>>>>>
>>>>> I have other issues with Zeppelin as well, like I cannot give my Spark
>>>>> executer (local[*]) however much RAM I want. This means that I'm not using
>>>>> all my available RAM, which limits how many executors I can have chewing
>>>>> at the problem. Changing spark.driver.memory and spark.executor.memory
>>>>> doesn't seem to make any difference. There's something else that blocks how
>>>>> much RAM my executers can get.
>>>>>
>>>>> So it might make sense that I get my own Jira account, if you'll have
>>>>> me.
>>>>>
>>>>> Patrik Iselind
>>>>>
>>>>>
>>>>> On Mon, May 18, 2020 at 3:56 PM Jeff Zhang <zj...@gmail.com> wrote:
>>>>>
>>>>>> It is not supported to inject env into interpreter setting for now,
>>>>>> could you create a ticket for it ?
>>>>>>
>>>>>> Patrik Iselind <pa...@gmail.com> 于2020年5月18日周一 下午6:52写道：
>>>>>>
>>>>>>> Hi everyone,
>>>>>>>
>>>>>>> I'm using Zeppelin through a docker image. In this setup, I'd like
>>>>>>> to version control my conf/interpreters.json file along with my Dockerfile
>>>>>>> and other related files. This I can of course do already. There's a problem
>>>>>>> related to this though.
>>>>>>>
>>>>>>> When setting for example the AWS credentials in the interpreter
>>>>>>> configuration, then I'm forced to write them out in clear text. I really
>>>>>>> don't want to version control the credentials!
>>>>>>>
>>>>>>> I've tried setting AWS_ACCESS_KEY_ID to "${AWS_ACCESS_KEY_ID_VAL}"
>>>>>>> but that doesn't work. It will not expand. Setting AWS_ACCESS_KEY_ID
>>>>>>> to "${AWS_ACCESS_KEY_ID}", will not work as it will expand
>>>>>>> recursively.
>>>>>>>
>>>>>>> For this reason I think it would be great if Zeppelin support
>>>>>>> configurations that are of the type "expandable string" which would read
>>>>>>> from the environment to expand the variable values once. That would allow
>>>>>>> me to version control my interpreters.json without having to include my AWS
>>>>>>> credentials.
>>>>>>>
>>>>>>> Using the "password" type will only hide the password in the UI,
>>>>>>> it's still cleartext in the interpreter.json file.
>>>>>>>
>>>>>>> Do you see some other solution that might work here?
>>>>>>>
>>>>>>> Best Regards,
>>>>>>> Patrik Iselind
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best Regards
>>>>>>
>>>>>> Jeff Zhang
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Best Regards
>>>>
>>>> Jeff Zhang
>>>>
>>>
>>
>> --
>> Best Regards
>>
>> Jeff Zhang
>>
>
Re: Version controlling the interpreter configurations
Posted by Patrik Iselind <pa...@gmail.com>.
Ok, how do I create that account then?
Den mån 18 maj 2020 17:59Jeff Zhang <zj...@gmail.com> skrev:
> No, the link is for committers, there's no prerequisites for creating new
> apache account, just an email address.
>
> Patrik Iselind <pa...@gmail.com> 于2020年5月18日周一 下午11:51写道：
>
>> Thanks, for the hints.
>>
>> This part of Apache is new to me. Do I have to do everything on
>> https://www.apache.org/dev/new-committers-guide.html to get that Apache
>> account set up?
>>
>> Patrik Iselind
>>
>>
>> On Mon, May 18, 2020 at 5:34 PM Jeff Zhang <zj...@gmail.com> wrote:
>>
>>> You need to create apache account first, then create ticket here
>>> https://issues.apache.org/jira/browse/ZEPPELIN
>>>
>>>
>>> Patrik Iselind <pa...@gmail.com> 于2020年5月18日周一 下午11:07写道：
>>>
>>>> I'm not allowed to create tickets in your Jira. What's the procedure to
>>>> get a Jira account?
>>>>
>>>> I have other issues with Zeppelin as well, like I cannot give my Spark
>>>> executer (local[*]) however much RAM I want. This means that I'm not using
>>>> all my available RAM, which limits how many executors I can have chewing
>>>> at the problem. Changing spark.driver.memory and spark.executor.memory
>>>> doesn't seem to make any difference. There's something else that blocks how
>>>> much RAM my executers can get.
>>>>
>>>> So it might make sense that I get my own Jira account, if you'll have
>>>> me.
>>>>
>>>> Patrik Iselind
>>>>
>>>>
>>>> On Mon, May 18, 2020 at 3:56 PM Jeff Zhang <zj...@gmail.com> wrote:
>>>>
>>>>> It is not supported to inject env into interpreter setting for now,
>>>>> could you create a ticket for it ?
>>>>>
>>>>> Patrik Iselind <pa...@gmail.com> 于2020年5月18日周一 下午6:52写道：
>>>>>
>>>>>> Hi everyone,
>>>>>>
>>>>>> I'm using Zeppelin through a docker image. In this setup, I'd like to
>>>>>> version control my conf/interpreters.json file along with my Dockerfile and
>>>>>> other related files. This I can of course do already. There's a problem
>>>>>> related to this though.
>>>>>>
>>>>>> When setting for example the AWS credentials in the interpreter
>>>>>> configuration, then I'm forced to write them out in clear text. I really
>>>>>> don't want to version control the credentials!
>>>>>>
>>>>>> I've tried setting AWS_ACCESS_KEY_ID to "${AWS_ACCESS_KEY_ID_VAL}"
>>>>>> but that doesn't work. It will not expand. Setting AWS_ACCESS_KEY_ID
>>>>>> to "${AWS_ACCESS_KEY_ID}", will not work as it will expand
>>>>>> recursively.
>>>>>>
>>>>>> For this reason I think it would be great if Zeppelin support
>>>>>> configurations that are of the type "expandable string" which would read
>>>>>> from the environment to expand the variable values once. That would allow
>>>>>> me to version control my interpreters.json without having to include my AWS
>>>>>> credentials.
>>>>>>
>>>>>> Using the "password" type will only hide the password in the UI, it's
>>>>>> still cleartext in the interpreter.json file.
>>>>>>
>>>>>> Do you see some other solution that might work here?
>>>>>>
>>>>>> Best Regards,
>>>>>> Patrik Iselind
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best Regards
>>>>>
>>>>> Jeff Zhang
>>>>>
>>>>
>>>
>>> --
>>> Best Regards
>>>
>>> Jeff Zhang
>>>
>>
>
> --
> Best Regards
>
> Jeff Zhang
>
Re: Version controlling the interpreter configurations
Posted by Jeff Zhang <zj...@gmail.com>.
No, the link is for committers, there's no prerequisites for creating new
apache account, just an email address.
Patrik Iselind <pa...@gmail.com> 于2020年5月18日周一 下午11:51写道：
> Thanks, for the hints.
>
> This part of Apache is new to me. Do I have to do everything on
> https://www.apache.org/dev/new-committers-guide.html to get that Apache
> account set up?
>
> Patrik Iselind
>
>
> On Mon, May 18, 2020 at 5:34 PM Jeff Zhang <zj...@gmail.com> wrote:
>
>> You need to create apache account first, then create ticket here
>> https://issues.apache.org/jira/browse/ZEPPELIN
>>
>>
>> Patrik Iselind <pa...@gmail.com> 于2020年5月18日周一 下午11:07写道：
>>
>>> I'm not allowed to create tickets in your Jira. What's the procedure to
>>> get a Jira account?
>>>
>>> I have other issues with Zeppelin as well, like I cannot give my Spark
>>> executer (local[*]) however much RAM I want. This means that I'm not using
>>> all my available RAM, which limits how many executors I can have chewing
>>> at the problem. Changing spark.driver.memory and spark.executor.memory
>>> doesn't seem to make any difference. There's something else that blocks how
>>> much RAM my executers can get.
>>>
>>> So it might make sense that I get my own Jira account, if you'll have me.
>>>
>>> Patrik Iselind
>>>
>>>
>>> On Mon, May 18, 2020 at 3:56 PM Jeff Zhang <zj...@gmail.com> wrote:
>>>
>>>> It is not supported to inject env into interpreter setting for now,
>>>> could you create a ticket for it ?
>>>>
>>>> Patrik Iselind <pa...@gmail.com> 于2020年5月18日周一 下午6:52写道：
>>>>
>>>>> Hi everyone,
>>>>>
>>>>> I'm using Zeppelin through a docker image. In this setup, I'd like to
>>>>> version control my conf/interpreters.json file along with my Dockerfile and
>>>>> other related files. This I can of course do already. There's a problem
>>>>> related to this though.
>>>>>
>>>>> When setting for example the AWS credentials in the interpreter
>>>>> configuration, then I'm forced to write them out in clear text. I really
>>>>> don't want to version control the credentials!
>>>>>
>>>>> I've tried setting AWS_ACCESS_KEY_ID to "${AWS_ACCESS_KEY_ID_VAL}"
>>>>> but that doesn't work. It will not expand. Setting AWS_ACCESS_KEY_ID
>>>>> to "${AWS_ACCESS_KEY_ID}", will not work as it will expand
>>>>> recursively.
>>>>>
>>>>> For this reason I think it would be great if Zeppelin support
>>>>> configurations that are of the type "expandable string" which would read
>>>>> from the environment to expand the variable values once. That would allow
>>>>> me to version control my interpreters.json without having to include my AWS
>>>>> credentials.
>>>>>
>>>>> Using the "password" type will only hide the password in the UI, it's
>>>>> still cleartext in the interpreter.json file.
>>>>>
>>>>> Do you see some other solution that might work here?
>>>>>
>>>>> Best Regards,
>>>>> Patrik Iselind
>>>>>
>>>>
>>>>
>>>> --
>>>> Best Regards
>>>>
>>>> Jeff Zhang
>>>>
>>>
>>
>> --
>> Best Regards
>>
>> Jeff Zhang
>>
>
--
Best Regards
Jeff Zhang
Re: Version controlling the interpreter configurations
Posted by Patrik Iselind <pa...@gmail.com>.
Thanks, for the hints.
This part of Apache is new to me. Do I have to do everything on
https://www.apache.org/dev/new-committers-guide.html to get that Apache
account set up?
Patrik Iselind
On Mon, May 18, 2020 at 5:34 PM Jeff Zhang <zj...@gmail.com> wrote:
> You need to create apache account first, then create ticket here
> https://issues.apache.org/jira/browse/ZEPPELIN
>
>
> Patrik Iselind <pa...@gmail.com> 于2020年5月18日周一 下午11:07写道：
>
>> I'm not allowed to create tickets in your Jira. What's the procedure to
>> get a Jira account?
>>
>> I have other issues with Zeppelin as well, like I cannot give my Spark
>> executer (local[*]) however much RAM I want. This means that I'm not using
>> all my available RAM, which limits how many executors I can have chewing
>> at the problem. Changing spark.driver.memory and spark.executor.memory
>> doesn't seem to make any difference. There's something else that blocks how
>> much RAM my executers can get.
>>
>> So it might make sense that I get my own Jira account, if you'll have me.
>>
>> Patrik Iselind
>>
>>
>> On Mon, May 18, 2020 at 3:56 PM Jeff Zhang <zj...@gmail.com> wrote:
>>
>>> It is not supported to inject env into interpreter setting for now,
>>> could you create a ticket for it ?
>>>
>>> Patrik Iselind <pa...@gmail.com> 于2020年5月18日周一 下午6:52写道：
>>>
>>>> Hi everyone,
>>>>
>>>> I'm using Zeppelin through a docker image. In this setup, I'd like to
>>>> version control my conf/interpreters.json file along with my Dockerfile and
>>>> other related files. This I can of course do already. There's a problem
>>>> related to this though.
>>>>
>>>> When setting for example the AWS credentials in the interpreter
>>>> configuration, then I'm forced to write them out in clear text. I really
>>>> don't want to version control the credentials!
>>>>
>>>> I've tried setting AWS_ACCESS_KEY_ID to "${AWS_ACCESS_KEY_ID_VAL}" but
>>>> that doesn't work. It will not expand. Setting AWS_ACCESS_KEY_ID to "${AWS_ACCESS_KEY_ID}",
>>>> will not work as it will expand recursively.
>>>>
>>>> For this reason I think it would be great if Zeppelin support
>>>> configurations that are of the type "expandable string" which would read
>>>> from the environment to expand the variable values once. That would allow
>>>> me to version control my interpreters.json without having to include my AWS
>>>> credentials.
>>>>
>>>> Using the "password" type will only hide the password in the UI, it's
>>>> still cleartext in the interpreter.json file.
>>>>
>>>> Do you see some other solution that might work here?
>>>>
>>>> Best Regards,
>>>> Patrik Iselind
>>>>
>>>
>>>
>>> --
>>> Best Regards
>>>
>>> Jeff Zhang
>>>
>>
>
> --
> Best Regards
>
> Jeff Zhang
>
Re: Version controlling the interpreter configurations
Posted by Jeff Zhang <zj...@gmail.com>.
You need to create apache account first, then create ticket here
https://issues.apache.org/jira/browse/ZEPPELIN
Patrik Iselind <pa...@gmail.com> 于2020年5月18日周一 下午11:07写道：
> I'm not allowed to create tickets in your Jira. What's the procedure to
> get a Jira account?
>
> I have other issues with Zeppelin as well, like I cannot give my Spark
> executer (local[*]) however much RAM I want. This means that I'm not using
> all my available RAM, which limits how many executors I can have chewing
> at the problem. Changing spark.driver.memory and spark.executor.memory
> doesn't seem to make any difference. There's something else that blocks how
> much RAM my executers can get.
>
> So it might make sense that I get my own Jira account, if you'll have me.
>
> Patrik Iselind
>
>
> On Mon, May 18, 2020 at 3:56 PM Jeff Zhang <zj...@gmail.com> wrote:
>
>> It is not supported to inject env into interpreter setting for now, could
>> you create a ticket for it ?
>>
>> Patrik Iselind <pa...@gmail.com> 于2020年5月18日周一 下午6:52写道：
>>
>>> Hi everyone,
>>>
>>> I'm using Zeppelin through a docker image. In this setup, I'd like to
>>> version control my conf/interpreters.json file along with my Dockerfile and
>>> other related files. This I can of course do already. There's a problem
>>> related to this though.
>>>
>>> When setting for example the AWS credentials in the interpreter
>>> configuration, then I'm forced to write them out in clear text. I really
>>> don't want to version control the credentials!
>>>
>>> I've tried setting AWS_ACCESS_KEY_ID to "${AWS_ACCESS_KEY_ID_VAL}" but
>>> that doesn't work. It will not expand. Setting AWS_ACCESS_KEY_ID to "${AWS_ACCESS_KEY_ID}",
>>> will not work as it will expand recursively.
>>>
>>> For this reason I think it would be great if Zeppelin support
>>> configurations that are of the type "expandable string" which would read
>>> from the environment to expand the variable values once. That would allow
>>> me to version control my interpreters.json without having to include my AWS
>>> credentials.
>>>
>>> Using the "password" type will only hide the password in the UI, it's
>>> still cleartext in the interpreter.json file.
>>>
>>> Do you see some other solution that might work here?
>>>
>>> Best Regards,
>>> Patrik Iselind
>>>
>>
>>
>> --
>> Best Regards
>>
>> Jeff Zhang
>>
>
--
Best Regards
Jeff Zhang
Re: Version controlling the interpreter configurations
Posted by Patrik Iselind <pa...@gmail.com>.
I'm not allowed to create tickets in your Jira. What's the procedure to get
a Jira account?
I have other issues with Zeppelin as well, like I cannot give my Spark
executer (local[*]) however much RAM I want. This means that I'm not using
all my available RAM, which limits how many executors I can have chewing
at the problem. Changing spark.driver.memory and spark.executor.memory
doesn't seem to make any difference. There's something else that blocks how
much RAM my executers can get.
So it might make sense that I get my own Jira account, if you'll have me.
Patrik Iselind
On Mon, May 18, 2020 at 3:56 PM Jeff Zhang <zj...@gmail.com> wrote:
> It is not supported to inject env into interpreter setting for now, could
> you create a ticket for it ?
>
> Patrik Iselind <pa...@gmail.com> 于2020年5月18日周一 下午6:52写道：
>
>> Hi everyone,
>>
>> I'm using Zeppelin through a docker image. In this setup, I'd like to
>> version control my conf/interpreters.json file along with my Dockerfile and
>> other related files. This I can of course do already. There's a problem
>> related to this though.
>>
>> When setting for example the AWS credentials in the interpreter
>> configuration, then I'm forced to write them out in clear text. I really
>> don't want to version control the credentials!
>>
>> I've tried setting AWS_ACCESS_KEY_ID to "${AWS_ACCESS_KEY_ID_VAL}" but
>> that doesn't work. It will not expand. Setting AWS_ACCESS_KEY_ID to "${AWS_ACCESS_KEY_ID}",
>> will not work as it will expand recursively.
>>
>> For this reason I think it would be great if Zeppelin support
>> configurations that are of the type "expandable string" which would read
>> from the environment to expand the variable values once. That would allow
>> me to version control my interpreters.json without having to include my AWS
>> credentials.
>>
>> Using the "password" type will only hide the password in the UI, it's
>> still cleartext in the interpreter.json file.
>>
>> Do you see some other solution that might work here?
>>
>> Best Regards,
>> Patrik Iselind
>>
>
>
> --
> Best Regards
>
> Jeff Zhang
>
Re: Version controlling the interpreter configurations
Posted by Jeff Zhang <zj...@gmail.com>.
It is not supported to inject env into interpreter setting for now, could
you create a ticket for it ?
Patrik Iselind <pa...@gmail.com> 于2020年5月18日周一 下午6:52写道：
> Hi everyone,
>
> I'm using Zeppelin through a docker image. In this setup, I'd like to
> version control my conf/interpreters.json file along with my Dockerfile and
> other related files. This I can of course do already. There's a problem
> related to this though.
>
> When setting for example the AWS credentials in the interpreter
> configuration, then I'm forced to write them out in clear text. I really
> don't want to version control the credentials!
>
> I've tried setting AWS_ACCESS_KEY_ID to "${AWS_ACCESS_KEY_ID_VAL}" but
> that doesn't work. It will not expand. Setting AWS_ACCESS_KEY_ID to "${AWS_ACCESS_KEY_ID}",
> will not work as it will expand recursively.
>
> For this reason I think it would be great if Zeppelin support
> configurations that are of the type "expandable string" which would read
> from the environment to expand the variable values once. That would allow
> me to version control my interpreters.json without having to include my AWS
> credentials.
>
> Using the "password" type will only hide the password in the UI, it's
> still cleartext in the interpreter.json file.
>
> Do you see some other solution that might work here?
>
> Best Regards,
> Patrik Iselind
>
--
Best Regards
Jeff Zhang