Anyway to use hdp_version in component configuration file ?

[Jeff Zhang <zj...@gmail.com>]

I am trying to create configuration file (spark-defaults.xml) for my
component

This configuration is very similar to hadoop configuration file.  It seems
support variable substitution. and can I use hdp_version variable (or any
other variable represent hdp_version) in this configuration file ? Thanks



-- 
Best Regards

Jeff Zhang

Re: Anyway to use hdp_version in component configuration file ?

[Bhuvnesh Chaudhary <bc...@pivotal.io>]

Variables in param.py are visible in the template files which can be xml or
any plain text files. (Case 1 suggested in last email)
These templates files are placed under folder
<path_to_your_service>/x.x.x/package/templates with .j2 suffix

Re: Anyway to use hdp_version in component configuration file ?

[Jeff Zhang <zj...@gmail.com>]

Hi Bhuvnesh,

Thanks for your reply. Does it mean any variable in param.py
is available for the xml configuration file ?

On Tue, Jan 5, 2016 at 2:43 PM, Bhuvnesh Chaudhary <bc...@pivotal.io>
wrote:

> Technically, you can have the value of hdp_version in your
> spark-defaults.xml. But I hope you might have already considered the
> impacts due to upgrade, etc.
>
> Case1:
> if you are not exposing it on UI then you can use template with variable
> substitution..
> step1:  put a template for spark-default.xml under package/templates with
> the required label.
> ex spark-default.xml.j2
> <property>hdp_version</property>
> <value>{{hdp_version_derived}}</value>
>
> step2: derive the value of hdp_version_derived in params.py. (let's say
> after derivation)
> hdp_version_derived = 2.3
>
> step3: Write it to the filesystem using File Resource.
> ex: File(target_file, template(your_template), owner=required_owner,
> group=required_group)
>
> Case2:
> if you are exposing it UI, template's with variable substitution cannot be
> used. You will have to perform the below
> step1. expose a configuration file under configurations directory (with
> some default for hdp_version)
> step2. read the properties when service is loaded / started
> step3. Propagate the value to filesystem. (XmlConfig can be used to write
> Xml files with required values)
>
> Be advised for case2, if you update the value at backend by some logic in
> your code, your UI config display may not be in sync with the actual value
> written on the filesystem and you will need to update the correct value on
> UI if you want them to be in sync. This is kind of hack and not
> recommended.
>
> Let me know if there are any questions.
>
>
>
>
>
> Thanks,
> Bhuvnesh Chaudhary
> Email: bchau <bc...@gopivotal.com>dhary@pivotal.io
> Desk: +1-650-846-1696 | Mobile: +1-973-906-6976
>
> On Mon, Jan 4, 2016 at 8:11 PM, Jeff Zhang <zj...@gmail.com> wrote:
>
> > I am trying to create configuration file (spark-defaults.xml) for my
> > component
> >
> > This configuration is very similar to hadoop configuration file.  It
> seems
> > support variable substitution. and can I use hdp_version variable (or any
> > other variable represent hdp_version) in this configuration file ? Thanks
> >
> >
> >
> > --
> > Best Regards
> >
> > Jeff Zhang
> >
>



-- 
Best Regards

Jeff Zhang

Re: Anyway to use hdp_version in component configuration file ?

[Bhuvnesh Chaudhary <bc...@pivotal.io>]

Technically, you can have the value of hdp_version in your
spark-defaults.xml. But I hope you might have already considered the
impacts due to upgrade, etc.

Case1:
if you are not exposing it on UI then you can use template with variable
substitution..
step1:  put a template for spark-default.xml under package/templates with
the required label.
ex spark-default.xml.j2
<property>hdp_version</property>
<value>{{hdp_version_derived}}</value>

step2: derive the value of hdp_version_derived in params.py. (let's say
after derivation)
hdp_version_derived = 2.3

step3: Write it to the filesystem using File Resource.
ex: File(target_file, template(your_template), owner=required_owner,
group=required_group)

Case2:
if you are exposing it UI, template's with variable substitution cannot be
used. You will have to perform the below
step1. expose a configuration file under configurations directory (with
some default for hdp_version)
step2. read the properties when service is loaded / started
step3. Propagate the value to filesystem. (XmlConfig can be used to write
Xml files with required values)

Be advised for case2, if you update the value at backend by some logic in
your code, your UI config display may not be in sync with the actual value
written on the filesystem and you will need to update the correct value on
UI if you want them to be in sync. This is kind of hack and not recommended.

Let me know if there are any questions.





Thanks,
Bhuvnesh Chaudhary
Email: bchau <bc...@gopivotal.com>dhary@pivotal.io
Desk: +1-650-846-1696 | Mobile: +1-973-906-6976

On Mon, Jan 4, 2016 at 8:11 PM, Jeff Zhang <zj...@gmail.com> wrote:

> I am trying to create configuration file (spark-defaults.xml) for my
> component
>
> This configuration is very similar to hadoop configuration file.  It seems
> support variable substitution. and can I use hdp_version variable (or any
> other variable represent hdp_version) in this configuration file ? Thanks
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>