You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@falcon.apache.org by Pallavi Rao <pa...@inmobi.com> on 2015/11/16 05:32:57 UTC
[DISCUSS] Namespace-ing properties and FALCON-1573
Folks,
With FALCON-1573, Daniel came up with a very good suggestion of allowing
user-supplied properties at schedule time. He proposed 2 approaches
<https://issues.apache.org/jira/browse/FALCON-1573?focusedCommentId=14982386&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14982386>
-
1. Allow user to specify free form name-value pairs. Since user-supplied
property names can clash with Falcon property names, give Falcon properties
(generated when the bundle is built) priority over user-supplied ones, in
case of clash.
2. Namespace user-supplied properties, so we can distinguish user and
system properties. This will allow user to supply properties without any
clash with system properties.
Daniel has already uploaded patch for approach (1). Regarding this, wish to
solicit your input on 2 questions:
1. Should we pursue approach (2) in FALCON-1573 itself or take it up
separately?
2. When we eventually namespace the properties, what should the
namespace format be?
My 2 cents:
Question (1) : I think we should pursue namespace approach in FALCON-1573
itself. Two reasons for the same:
- When user supplies a name-value pair and we silently ignore (we only
log) it when it clashes with system property, it is not very intuitive to
the user.
- If we don't introduce it now, but, bring it in later, we'll have to
deal with 2 kinds of properties, one that is namespace-d and one that isn't.
Question (2) : I suggest we use falcon.system.* (or just falcon.*) for
system properties and falcon.user.* for user-supplied properties.
Apologies for the tad lengthy mail. Request you all to provide your inputs.
Thanks and Regards,
Pallavi
--
_____________________________________________________________
The information contained in this communication is intended solely for the
use of the individual or entity to whom it is addressed and others
authorized to receive it. It may contain confidential or legally privileged
information. If you are not the intended recipient you are hereby notified
that any disclosure, copying, distribution or taking any action in reliance
on the contents of this information is strictly prohibited and may be
unlawful. If you have received this communication in error, please notify
us immediately by responding to this email and then delete it from your
system. The firm is neither liable for the proper and complete transmission
of the information contained in this communication nor for any delay in its
receipt.
Re: [DISCUSS] Namespace-ing properties and FALCON-1573
Posted by Ajay Yadav <aj...@gmail.com>.
Here is some more context
FALCON-1573 is an extension of FALCON-1434. A JIRA for namespace was
created even before FALCON-1434 as FALCON-1308. FALCON-1434 didn't deal
with the issue of namespaces either and namespaces were never discussed(It
also doesn't follow the original suggestion
<https://issues.apache.org/jira/browse/FALCON-1573?focusedCommentId=14991127&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14991127>
in
JIRA to have falcon.system.* for system properties). The issue FALCON-1573
has been blocked for 11 days in the pretext that the namespace issue must
be addressed as part of the *same* *JIRA* and suggestion to tackle
namespaces holistically as part of a separate JIRA has to my surprise been
found unacceptable.
As for namespaces
1. Any old system properties can not be namespaced as it will cause
backward compatibility issues.
2. We have never discussed or agreed upon a convention for namespaces, so
there are not 2 types of properties but only one, without namespaces. If we
decide with a namespace of falcon.system.* for system properties then we
have to fix changes done in FALCON-1434 as well.
3. I am still perplexed as why addressing this in another JIRA makes it a
problem. We can revisit it in a separate JIRA in few days after discussing
namespace issue holistically. This will not cause any issues at all,
including backward incompatibility.
4. I find the idea of namespacing user properties with falcon.user.* quite
intrusive and verbose. I prefer a shorter, without namespace version, only
for user properties as done in FALCON-1573. This will avoid conflicts as
the user properties are without namespace and system properties will be
with namespace. Apart from being concise this has an added benefit of being
backward compatible with older user properties as well.
However, I am not worried about the code changes as much as community
dynamics. Apache is not about code but about community and people. So the
bigger issue for me is not namespace or it's format but this bizarre stand
that all enhancements must be addressed in same JIRA. There are enough
precedents in the community to show that all contributors have at some
point created a JIRA to address some review comments and unblocked the
existing JIRA. This allows developers to get some or large part of
functionality checked in and focus only on smaller subset of enhancements.
Not to mention the psychological boost of a change being accepted for new
contributors. I am quite sad by this sudden shift and find it quite
detrimental for community.
I request the community to unblock FALCON-1573 and tackle namespace
holistically in a separate JIRA.
Thanks
Ajay Yadava
On Mon, Nov 16, 2015 at 10:02 AM, Pallavi Rao <pa...@inmobi.com>
wrote:
> Folks,
> With FALCON-1573, Daniel came up with a very good suggestion of allowing
> user-supplied properties at schedule time. He proposed 2 approaches
> <
> https://issues.apache.org/jira/browse/FALCON-1573?focusedCommentId=14982386&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14982386
> >
> -
>
> 1. Allow user to specify free form name-value pairs. Since user-supplied
> property names can clash with Falcon property names, give Falcon
> properties
> (generated when the bundle is built) priority over user-supplied ones,
> in
> case of clash.
> 2. Namespace user-supplied properties, so we can distinguish user and
> system properties. This will allow user to supply properties without any
> clash with system properties.
>
> Daniel has already uploaded patch for approach (1). Regarding this, wish to
> solicit your input on 2 questions:
>
> 1. Should we pursue approach (2) in FALCON-1573 itself or take it up
> separately?
> 2. When we eventually namespace the properties, what should the
> namespace format be?
>
> My 2 cents:
>
> Question (1) : I think we should pursue namespace approach in FALCON-1573
> itself. Two reasons for the same:
>
> - When user supplies a name-value pair and we silently ignore (we only
> log) it when it clashes with system property, it is not very intuitive
> to
> the user.
> - If we don't introduce it now, but, bring it in later, we'll have to
> deal with 2 kinds of properties, one that is namespace-d and one that
> isn't.
>
> Question (2) : I suggest we use falcon.system.* (or just falcon.*) for
> system properties and falcon.user.* for user-supplied properties.
>
> Apologies for the tad lengthy mail. Request you all to provide your inputs.
>
> Thanks and Regards,
> Pallavi
>
> --
> _____________________________________________________________
> The information contained in this communication is intended solely for the
> use of the individual or entity to whom it is addressed and others
> authorized to receive it. It may contain confidential or legally privileged
> information. If you are not the intended recipient you are hereby notified
> that any disclosure, copying, distribution or taking any action in reliance
> on the contents of this information is strictly prohibited and may be
> unlawful. If you have received this communication in error, please notify
> us immediately by responding to this email and then delete it from your
> system. The firm is neither liable for the proper and complete transmission
> of the information contained in this communication nor for any delay in its
> receipt.
>
RE: [DISCUSS] Namespace-ing properties and FALCON-1573
Posted by Srikanth Sundarrajan <sr...@hotmail.com>.
Thanks Pallavi for bring this up. I have been wanting namespace isolation for properties for a fair while and this allows to build consensus and move forward.
>From what I can gather from the discussions on 1573, multiple issues are hinted at and proposed to be solved.
1. Allowing for properties being overidden at schedule time - directly being passed to underlying scheduler, essentially a pass through for falcon
2. Allowing for default properties in falcon configured in entity definition to be overriden at schedule time
3. Namespacing of properties such that there is no accidental clash in property name and there is general improvement in maintainability.
To pass properties that override falcon entity “properties” and have them passed through as is to the underlying scheduler would be a good start.
My personal preference would be for the following overlay scheme, where user overrides gets preference and the resolution order is as follows.
——> FALCON SYSTEM <— —
(USER PROPERTIES)
> (PROCESS PROPERTIES)
> (FEED PROPERTIES)
> (CLUSTER PROPERTIES)
——> SCHEDULER SYSTEM (OOZIE) <— —
@Schedule / Instance Management
(RESOLVED PROPERTIES FROM ABOVE)
> (SCHEDULER DEFAULT PROPERTIES)
The properties in the entity definition today aren’t totally opaque (for ex. queue-name, priority etc). We ought to segregate properties into various classes such as System level Internal properties, Entity level internal properties, Entity level properties, Entity level user properties.
System level internal properties are used by falcon and is application to the entire instance and as such isn’t available for override. However these properties can be re-read by the server if it is deemed to be a runtime property. Tucking them under a namespace is desirable (for. ex falcon.system). Properties with this namespace if sent for any entity or instance operation should be ignored and appropriately warned.
Entity level internal property is what is used by falcon internally to exchange information between different systems and these shouldn’t overridable either. Namespace for these could be say falcon.internal. Properties with this namespace if sent for any entity or instance operation should be ignored and appropriately warned.
Entity level properties can be overriden by users. Seems very important to put this in an exclusive namespace (for ex. falcon.entity.override). This is well defined list of properties and by definition is not a first class property element in the property bag of entity. For ex. priority, process-parallelism, scheduler etc
Entity level user properties are the ones typically enlisted in the property bag of the entity definition. Anything supplied by the user overrides or is additionally added to the entity or instance operation. This I would prefer to keep it without any namespace, as these property names make it ways into workflow definition and possibly into user code and it would be preferable not to carry the prefix all the way and muddy user code.
Optionally if we particularly feel the need to pass something to the scheduler and scheduler only, we can set aside a namespace for it.
Regards
Srikanth Sundarrajan
> Date: Mon, 16 Nov 2015 10:02:57 +0530
> Subject: [DISCUSS] Namespace-ing properties and FALCON-1573
> From: pallavi.rao@inmobi.com
> To: dev@falcon.apache.org
> CC: dadelcas@hotmail.com
>
> Folks,
> With FALCON-1573, Daniel came up with a very good suggestion of allowing
> user-supplied properties at schedule time. He proposed 2 approaches
> <https://issues.apache.org/jira/browse/FALCON-1573?focusedCommentId=14982386&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14982386>
> -
>
> 1. Allow user to specify free form name-value pairs. Since user-supplied
> property names can clash with Falcon property names, give Falcon properties
> (generated when the bundle is built) priority over user-supplied ones, in
> case of clash.
> 2. Namespace user-supplied properties, so we can distinguish user and
> system properties. This will allow user to supply properties without any
> clash with system properties.
>
> Daniel has already uploaded patch for approach (1). Regarding this, wish to
> solicit your input on 2 questions:
>
> 1. Should we pursue approach (2) in FALCON-1573 itself or take it up
> separately?
> 2. When we eventually namespace the properties, what should the
> namespace format be?
>
> My 2 cents:
>
> Question (1) : I think we should pursue namespace approach in FALCON-1573
> itself. Two reasons for the same:
>
> - When user supplies a name-value pair and we silently ignore (we only
> log) it when it clashes with system property, it is not very intuitive to
> the user.
> - If we don't introduce it now, but, bring it in later, we'll have to
> deal with 2 kinds of properties, one that is namespace-d and one that isn't.
>
> Question (2) : I suggest we use falcon.system.* (or just falcon.*) for
> system properties and falcon.user.* for user-supplied properties.
>
> Apologies for the tad lengthy mail. Request you all to provide your inputs.
>
> Thanks and Regards,
> Pallavi
>
> --
> _____________________________________________________________
> The information contained in this communication is intended solely for the
> use of the individual or entity to whom it is addressed and others
> authorized to receive it. It may contain confidential or legally privileged
> information. If you are not the intended recipient you are hereby notified
> that any disclosure, copying, distribution or taking any action in reliance
> on the contents of this information is strictly prohibited and may be
> unlawful. If you have received this communication in error, please notify
> us immediately by responding to this email and then delete it from your
> system. The firm is neither liable for the proper and complete transmission
> of the information contained in this communication nor for any delay in its
> receipt.