What to override when extending a stack in 1.5?

[Chris Mildebrandt <ch...@woodenrhino.com>]

Hello,

I'd like to know how the "extends" mechanism works. I want to extend an
existing stack to use as a base for my stack. How would the following
scenarios be accomplished:

- I'd like to extend a stack under HDP, but from a directory outside of HDP
(e.g. stacks/COMPANY/2.2/). Is that possible, is that recommended, and how
does one do that? I understand I'd have to re-test when updates are made to
the base stack and any other underneath it.
- When I extend a stack, how do the python modules get loaded? Let's take
Ganglia as an example. There's a ganglia.py and ganglia_monitor.py (which
imports ganglia.py) in the package/scripts directory. If I'd like to change
the functionality of a method in ganglia.py, how is that done? Do I:
  - create a ganglia.py in my stack with only that method implemented, and
import the rest from the base?
  - have my own copy of the full ganglia.py?
    - If yes to either of the above, do I need to have my own copy of
ganglia_monitor.py so that it imports the ganglia.py from my stack instead
of the base?
  - have my own copy of the whole package/scripts directory?
  - have my own copy of the whole services/GANGLIA directory?
  - something else entirely?

Thanks for any answers/pointers,
-Chris

Re: What to override when extending a stack in 1.5?

[Chris Mildebrandt <ch...@woodenrhino.com>]

Hi Sid and Dmitry,

Thank you for the very detailed answers, that helps a lot.

Thanks again,
-Chris


On Wed, Apr 2, 2014 at 9:57 AM, Siddharth Wagle <sw...@hortonworks.com>wrote:

> Hello Chris,
>
> Based on conversation with Dmitry L, responses posted inline.
>
> I'd like to know how the "extends" mechanism works. I want to extend an
> existing stack to use as a base for my stack. How would the following
> scenarios be accomplished:
>
> - I'd like to extend a stack under HDP, but from a directory outside of
> HDP (e.g. stacks/COMPANY/2.2/). Is that possible, is that recommended, and
> how does one do that? I understand I'd have to re-test when updates are
> made to the base stack and any other underneath it.
> It should be recommended, and probably works (I've done a quick look at
> responsible code). The only issue I see is that stack extension seems to
> work only inside the same stack (when we extend stack, we specify only
> stack version and not a stack name). Something like:
> <extends>2.0.6</extends>
> That's why for now it makes sense to put your new stack into HDP directory
> if you are going to use stack extension. An example of such approach is
> 2.0.6.GlusterFS directory.
>
> - When I extend a stack, how do the python modules get loaded? Let's take
> Ganglia as an example. There's a ganglia.py and ganglia_monitor.py (which
> imports ganglia.py) in the package/scripts directory. If I'd like to change
> the functionality of a method in ganglia.py, how is that done? Do I  -
> create a ganglia.py in my stack with only that method implemented, and
> import the rest from the base?
> - have my own copy of the full ganglia.py?
> - If yes to either of the above, do I need to have my own copy of
> ganglia_monitor.py so that it imports the ganglia.py from my stack instead
> of the base?
> - have my own copy of the whole package/scripts directory?
>
> If you are going to change some python files, you have to copy entire
> "scripts" folder to a child stack. Per-file overrides of python files are
> not supported, and are not expected to be supported in future because
> resolving/understanding imports would be too complicated in that case. So
> you either copy entire "scripts" folder to a child stack (and change
> whatever you want), or do not create "scripts" folder at a child stack at
> all and it gets inherited from the parent stack.
>  Also, current paradigm requires that python files never access any stack
> folders outside their own stack folder. The reason is that agent
> downloads only stacks required for current operation (current stack and
> it's parent stacks), and any other stack folders at cache may be
> outdated/missing.
>
>
> - have my own copy of the whole services/GANGLIA directory?
> https://issues.apache.org/jira/browse/AMBARI-4358 jira description is a
> good starting point, that describes what may be inherited/overridden.
> Also, https://issues.apache.org/jira/browse/AMBARI-2819 is a bit older
> and a more basic one.
> Also, as of now, role_command_order.json files are not inherited yet
> (have to copy file into a new stack), and (I'm not sure about current
> status) configuration-dependencies might not be inherited. Both issues are
> planned for implementation.
>
> Best Regards,
> Sid
>
>
> On Tue, Apr 1, 2014 at 12:36 PM, Chris Mildebrandt <ch...@woodenrhino.com>wrote:
>
>> Hello,
>>
>> I'd like to know how the "extends" mechanism works. I want to extend an
>> existing stack to use as a base for my stack. How would the following
>> scenarios be accomplished:
>>
>> - I'd like to extend a stack under HDP, but from a directory outside of
>> HDP (e.g. stacks/COMPANY/2.2/). Is that possible, is that recommended, and
>> how does one do that? I understand I'd have to re-test when updates are
>> made to the base stack and any other underneath it.
>> - When I extend a stack, how do the python modules get loaded? Let's take
>> Ganglia as an example. There's a ganglia.py and ganglia_monitor.py (which
>> imports ganglia.py) in the package/scripts directory. If I'd like to change
>> the functionality of a method in ganglia.py, how is that done? Do I:
>>   - create a ganglia.py in my stack with only that method implemented,
>> and import the rest from the base?
>>   - have my own copy of the full ganglia.py?
>>     - If yes to either of the above, do I need to have my own copy of
>> ganglia_monitor.py so that it imports the ganglia.py from my stack instead
>> of the base?
>>   - have my own copy of the whole package/scripts directory?
>>   - have my own copy of the whole services/GANGLIA directory?
>>   - something else entirely?
>>
>> Thanks for any answers/pointers,
>> -Chris
>>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.

Re: What to override when extending a stack in 1.5?

[Siddharth Wagle <sw...@hortonworks.com>]

Hello Chris,

Based on conversation with Dmitry L, responses posted inline.

I'd like to know how the "extends" mechanism works. I want to extend an
existing stack to use as a base for my stack. How would the following
scenarios be accomplished:

- I'd like to extend a stack under HDP, but from a directory outside of HDP
(e.g. stacks/COMPANY/2.2/). Is that possible, is that recommended, and how
does one do that? I understand I'd have to re-test when updates are made to
the base stack and any other underneath it.
It should be recommended, and probably works (I've done a quick look at
responsible code). The only issue I see is that stack extension seems to
work only inside the same stack (when we extend stack, we specify only
stack version and not a stack name). Something like:
<extends>2.0.6</extends>
That's why for now it makes sense to put your new stack into HDP directory
if you are going to use stack extension. An example of such approach is
2.0.6.GlusterFS directory.

- When I extend a stack, how do the python modules get loaded? Let's take
Ganglia as an example. There's a ganglia.py and ganglia_monitor.py (which
imports ganglia.py) in the package/scripts directory. If I'd like to change
the functionality of a method in ganglia.py, how is that done? Do I  -
create a ganglia.py in my stack with only that method implemented, and
import the rest from the base?
- have my own copy of the full ganglia.py?
- If yes to either of the above, do I need to have my own copy of
ganglia_monitor.py so that it imports the ganglia.py from my stack instead
of the base?
- have my own copy of the whole package/scripts directory?

If you are going to change some python files, you have to copy entire
"scripts" folder to a child stack. Per-file overrides of python files are
not supported, and are not expected to be supported in future because
resolving/understanding imports would be too complicated in that case. So
you either copy entire "scripts" folder to a child stack (and change
whatever you want), or do not create "scripts" folder at a child stack at
all and it gets inherited from the parent stack.
 Also, current paradigm requires that python files never access any stack
folders outside their own stack folder. The reason is that agent downloads
only stacks required for current operation (current stack and it's parent
stacks), and any other stack folders at cache may be outdated/missing.

- have my own copy of the whole services/GANGLIA directory?
https://issues.apache.org/jira/browse/AMBARI-4358 jira description is a
good starting point, that describes what may be inherited/overridden.
Also, https://issues.apache.org/jira/browse/AMBARI-2819 is a bit older and
a more basic one.
Also, as of now, role_command_order.json files are not inherited yet (have
to copy file into a new stack), and (I'm not sure about current status)
configuration-dependencies might not be inherited. Both issues are planned
for implementation.

Best Regards,
Sid


On Tue, Apr 1, 2014 at 12:36 PM, Chris Mildebrandt <ch...@woodenrhino.com>wrote:

> Hello,
>
> I'd like to know how the "extends" mechanism works. I want to extend an
> existing stack to use as a base for my stack. How would the following
> scenarios be accomplished:
>
> - I'd like to extend a stack under HDP, but from a directory outside of
> HDP (e.g. stacks/COMPANY/2.2/). Is that possible, is that recommended, and
> how does one do that? I understand I'd have to re-test when updates are
> made to the base stack and any other underneath it.
> - When I extend a stack, how do the python modules get loaded? Let's take
> Ganglia as an example. There's a ganglia.py and ganglia_monitor.py (which
> imports ganglia.py) in the package/scripts directory. If I'd like to change
> the functionality of a method in ganglia.py, how is that done? Do I:
>   - create a ganglia.py in my stack with only that method implemented, and
> import the rest from the base?
>   - have my own copy of the full ganglia.py?
>     - If yes to either of the above, do I need to have my own copy of
> ganglia_monitor.py so that it imports the ganglia.py from my stack instead
> of the base?
>   - have my own copy of the whole package/scripts directory?
>   - have my own copy of the whole services/GANGLIA directory?
>   - something else entirely?
>
> Thanks for any answers/pointers,
> -Chris
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.