You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oodt.apache.org by Imesha Sudasingha <im...@cse.mrt.ac.lk> on 2017/06/10 07:25:22 UTC
[GSoC 2017] Distributed Configuration Management - Discussion
Hi Tom,
I have started going through the code in deep in order to continue
distributed configuration manager development. Following are some
approaches that came to my mind and few questions I need to get clarified.
As I understand, all 3 major components (Resource Manager, File Manager and
the Workflow Manager) *use several other files (ex: xml files located at
policy/*)* which are used for additional configuration (ex: for mime type
repository). However, all those file locations are found through properties
in the *.properties* file at runtime. When implementing distributed
configuration management, I have two options.
1. *Assuming that all such files are the same for all nodes*, ask users
to copy the same OODT distribution files to all nodes when setting up the
cluster. This assumes only the *.properties* file will have node
specific configuration.
2. Otherwise, If I am allowed to write to */policy/** directory, I can
download those files from zookeeper and make them available for each node
at runtime.
The problems I have at the moment are,
1. Whether my assumption in *suggestion 1 above* is correct?
2. Are there any node specific configurations in *.properties* files of
any component?
Can you kindly review my suggestions and help me clarify ambiguities I
have? Your suggestions are also welcome.
Thank you!
Kind Regards,
*Imesha Sudasingha*
Undergraduate of Department of Computer Science and Engineering,
University of Moratuwa,
Sri Lanka.
<https://lk.linkedin.com/in/imeshasudasingha> <https://github.com/IMS94>
<http://stackoverflow.com/users/4012073/imesha-sudasingha>
<https://twitter.com/Imesha94>
Re: [GSoC 2017] Distributed Configuration Management - Discussion
Posted by Imesha Sudasingha <im...@cse.mrt.ac.lk>.
Hi Tom,
Since all the other configuration files (*ex: XML files*) are found through
properties in the properties files, I can read those properties files and
then identify the files in it by "*file:"* prefix. Then I can store/fetch
them through zookeeper.
What do you think on that?
Thank you!
Kind Regards,
*Imesha Sudasingha*
Undergraduate of Department of Computer Science and Engineering,
University of Moratuwa,
Sri Lanka.
<https://lk.linkedin.com/in/imeshasudasingha> <https://github.com/IMS94>
<http://stackoverflow.com/users/4012073/imesha-sudasingha>
<https://twitter.com/Imesha94>
On 12 June 2017 at 15:46, Imesha Sudasingha <im...@cse.mrt.ac.lk> wrote:
> Hi Tom,
>
> Thanks for the details. Yes, we can store the XML file content in
> zookeeper and fetch those XML content in the runtime (probably when
> starting up).
> I will implement and test it that way so that we can discuss further on
> the pros and cons. I will update you soon.
>
>
> Kind Regards,
> *Imesha Sudasingha*
> Undergraduate of Department of Computer Science and Engineering,
> University of Moratuwa,
> Sri Lanka.
>
> <https://lk.linkedin.com/in/imeshasudasingha> <https://github.com/IMS94>
> <http://stackoverflow.com/users/4012073/imesha-sudasingha>
> <https://twitter.com/Imesha94>
>
> On 12 June 2017 at 14:57, Tom Barber <to...@meteorite.bi> wrote:
>
>> Hi Imesha
>>
>> Thanks for the questions:
>>
>> I think for multiplicities sake you can assume that all configs will be
>> identical on all nodes, for now at least, we can fine tune that in the
>> future.
>>
>> The XML files are the type of thing that would be very useful to have in
>> a centralised configuration setup. Take elements.xml,
>> product-type-element-map.xml and product-types.xml. These do some of the
>> heavy definition work in the file manager. If they actually lived in ZK
>> they would be far more flexible over a multinode setup, instead of reading
>> them from XML, can we just read the XML file content direct from ZK?
>>
>> Tom
>>
>> On Sat, Jun 10, 2017 at 8:25 AM, Imesha Sudasingha <
>> imesha.13@cse.mrt.ac.lk> wrote:
>>
>>> Hi Tom,
>>>
>>> I have started going through the code in deep in order to continue
>>> distributed configuration manager development. Following are some
>>> approaches that came to my mind and few questions I need to get clarified.
>>>
>>> As I understand, all 3 major components (Resource Manager, File Manager
>>> and the Workflow Manager) *use several other files (ex: xml files
>>> located at policy/*)* which are used for additional configuration (ex:
>>> for mime type repository). However, all those file locations are found
>>> through properties in the *.properties* file at runtime. When
>>> implementing distributed configuration management, I have two options.
>>>
>>> 1. *Assuming that all such files are the same for all nodes*, ask
>>> users to copy the same OODT distribution files to all nodes when setting up
>>> the cluster. This assumes only the *.properties* file will have node
>>> specific configuration.
>>> 2. Otherwise, If I am allowed to write to */policy/** directory, I
>>> can download those files from zookeeper and make them available for each
>>> node at runtime.
>>>
>>> The problems I have at the moment are,
>>>
>>> 1. Whether my assumption in *suggestion 1 above* is correct?
>>> 2. Are there any node specific configurations in *.properties* files
>>> of any component?
>>>
>>> Can you kindly review my suggestions and help me clarify ambiguities I
>>> have? Your suggestions are also welcome.
>>>
>>> Thank you!
>>>
>>> Kind Regards,
>>> *Imesha Sudasingha*
>>> Undergraduate of Department of Computer Science and Engineering,
>>> University of Moratuwa,
>>> Sri Lanka.
>>>
>>> <https://lk.linkedin.com/in/imeshasudasingha>
>>> <https://github.com/IMS94>
>>> <http://stackoverflow.com/users/4012073/imesha-sudasingha>
>>> <https://twitter.com/Imesha94>
>>>
>>
>>
>
Re: [GSoC 2017] Distributed Configuration Management - Discussion
Posted by Imesha Sudasingha <im...@cse.mrt.ac.lk>.
Hi Tom,
Thanks for the details. Yes, we can store the XML file content in zookeeper
and fetch those XML content in the runtime (probably when starting up).
I will implement and test it that way so that we can discuss further on the
pros and cons. I will update you soon.
Kind Regards,
*Imesha Sudasingha*
Undergraduate of Department of Computer Science and Engineering,
University of Moratuwa,
Sri Lanka.
<https://lk.linkedin.com/in/imeshasudasingha> <https://github.com/IMS94>
<http://stackoverflow.com/users/4012073/imesha-sudasingha>
<https://twitter.com/Imesha94>
On 12 June 2017 at 14:57, Tom Barber <to...@meteorite.bi> wrote:
> Hi Imesha
>
> Thanks for the questions:
>
> I think for multiplicities sake you can assume that all configs will be
> identical on all nodes, for now at least, we can fine tune that in the
> future.
>
> The XML files are the type of thing that would be very useful to have in a
> centralised configuration setup. Take elements.xml,
> product-type-element-map.xml and product-types.xml. These do some of the
> heavy definition work in the file manager. If they actually lived in ZK
> they would be far more flexible over a multinode setup, instead of reading
> them from XML, can we just read the XML file content direct from ZK?
>
> Tom
>
> On Sat, Jun 10, 2017 at 8:25 AM, Imesha Sudasingha <
> imesha.13@cse.mrt.ac.lk> wrote:
>
>> Hi Tom,
>>
>> I have started going through the code in deep in order to continue
>> distributed configuration manager development. Following are some
>> approaches that came to my mind and few questions I need to get clarified.
>>
>> As I understand, all 3 major components (Resource Manager, File Manager
>> and the Workflow Manager) *use several other files (ex: xml files
>> located at policy/*)* which are used for additional configuration (ex:
>> for mime type repository). However, all those file locations are found
>> through properties in the *.properties* file at runtime. When
>> implementing distributed configuration management, I have two options.
>>
>> 1. *Assuming that all such files are the same for all nodes*, ask
>> users to copy the same OODT distribution files to all nodes when setting up
>> the cluster. This assumes only the *.properties* file will have node
>> specific configuration.
>> 2. Otherwise, If I am allowed to write to */policy/** directory, I
>> can download those files from zookeeper and make them available for each
>> node at runtime.
>>
>> The problems I have at the moment are,
>>
>> 1. Whether my assumption in *suggestion 1 above* is correct?
>> 2. Are there any node specific configurations in *.properties* files
>> of any component?
>>
>> Can you kindly review my suggestions and help me clarify ambiguities I
>> have? Your suggestions are also welcome.
>>
>> Thank you!
>>
>> Kind Regards,
>> *Imesha Sudasingha*
>> Undergraduate of Department of Computer Science and Engineering,
>> University of Moratuwa,
>> Sri Lanka.
>>
>> <https://lk.linkedin.com/in/imeshasudasingha> <https://github.com/IMS94>
>> <http://stackoverflow.com/users/4012073/imesha-sudasingha>
>> <https://twitter.com/Imesha94>
>>
>
>
Re: [GSoC 2017] Distributed Configuration Management - Discussion
Posted by Tom Barber <to...@meteorite.bi>.
Hi Imesha
Thanks for the questions:
I think for multiplicities sake you can assume that all configs will be
identical on all nodes, for now at least, we can fine tune that in the
future.
The XML files are the type of thing that would be very useful to have in a
centralised configuration setup. Take elements.xml,
product-type-element-map.xml and product-types.xml. These do some of the
heavy definition work in the file manager. If they actually lived in ZK
they would be far more flexible over a multinode setup, instead of reading
them from XML, can we just read the XML file content direct from ZK?
Tom
On Sat, Jun 10, 2017 at 8:25 AM, Imesha Sudasingha <im...@cse.mrt.ac.lk>
wrote:
> Hi Tom,
>
> I have started going through the code in deep in order to continue
> distributed configuration manager development. Following are some
> approaches that came to my mind and few questions I need to get clarified.
>
> As I understand, all 3 major components (Resource Manager, File Manager
> and the Workflow Manager) *use several other files (ex: xml files located
> at policy/*)* which are used for additional configuration (ex: for mime
> type repository). However, all those file locations are found through
> properties in the *.properties* file at runtime. When implementing
> distributed configuration management, I have two options.
>
> 1. *Assuming that all such files are the same for all nodes*, ask
> users to copy the same OODT distribution files to all nodes when setting up
> the cluster. This assumes only the *.properties* file will have node
> specific configuration.
> 2. Otherwise, If I am allowed to write to */policy/** directory, I can
> download those files from zookeeper and make them available for each node
> at runtime.
>
> The problems I have at the moment are,
>
> 1. Whether my assumption in *suggestion 1 above* is correct?
> 2. Are there any node specific configurations in *.properties* files
> of any component?
>
> Can you kindly review my suggestions and help me clarify ambiguities I
> have? Your suggestions are also welcome.
>
> Thank you!
>
> Kind Regards,
> *Imesha Sudasingha*
> Undergraduate of Department of Computer Science and Engineering,
> University of Moratuwa,
> Sri Lanka.
>
> <https://lk.linkedin.com/in/imeshasudasingha> <https://github.com/IMS94>
> <http://stackoverflow.com/users/4012073/imesha-sudasingha>
> <https://twitter.com/Imesha94>
>