You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tamaya.apache.org by Anatole Tresch <at...@gmail.com> on 2016/01/24 12:13:59 UTC
Request for a small change for PropertyFilter SPI
Hi all
working on the concept of key filtering I encountered a problem that was
not really solvable given the current SPI. I try to explain my use case:
- following discussions we also had around metadata entries (it was
related to collection mapping at that time mainly), I also looked what
other store provide. Actually coreos/etcd does it quite fairly simply: it
hides all entries starting with a '_'.
- So I wanted to hide all entries starting with an '_' similarly, BUT
still these entries should be accessible.
The problem ist that I can workaround that by filtering everything AFTER
having accessed my keys something like
Map<String,String> config =
config.getProperties().filter(...);
But this is cumbersome and it should be possible to cover this that *BY
DEFAULT* hidden entries should not be returned as part of the overall
payload.
Now the problem is that the current filter API only gives me the key and
value of the entry. I dont have the additional meta-information if the
value is filtered as part of a call to getProperties() or an explicit call
such as getOrDefault(String key, String defautlVal).
Another use case would be masquerading of entries. E.g. accesses to
getProperties() may return password values as mysecret=*****. Since the
access is generalized security is sufficient in most cases, but you see
that a value is present. Whereas accessing the value explicitly, it can
undergo authorization and decryption using the same filter. Also in this
use case I have to see if the property is accessed as part of a properties
extract or an explicit access.
Given that I propose to extend the interface PropertyFilter from
public interface PropertyFilter {
String filterProperty(String key, String valueToBeFiltered);
}
to
public interface PropertyFilter {
String filterProperty(String key, String valueToBeFiltered,
boolean singleAccess);
}
If no one has any important objection I will commit later today this
change targeting *02-incubating *(it is is already ready and tested and has
proven to work).
Anatole
--
*Anatole Tresch*
Java Engineer & Architect, JSR Spec Lead
Glärnischweg 10
CH - 8620 Wetzikon
*Switzerland, Europe Zurich, GMT+1*
*Twitter: @atsticks*
*Blogs: **http://javaremarkables.blogspot.ch/
<http://javaremarkables.blogspot.ch/>*
*Google: atsticksMobile +41-76 344 62 79*
Re: Request for a small change for PropertyFilter SPI
Posted by Romain Manni-Bucau <rm...@gmail.com>.
2016-01-25 8:53 GMT+01:00 Anatole Tresch <at...@gmail.com>:
> So I summsrize:
>
> - we have 2 ways of accessing properties: as single property or as map. No
> room for discussions, just a fact.
>
that's an issue exposed in the API, the map access should be an internal
optimization only otherwise you cant guarantee any behavior of tamaya -
just a fact.
> - I have identified a use case where I think this additional context info
> is useful for filter implementations and did a proposal.
>
>
we already discuss the contextual data and decided it was easy enough to
keep it where it belongs to - ie not the API.
> I expect you either approove the proposal or come up with a better way how
> to solve it (meaning your own proposal). I did not ask for any theoretical
> discussion on anything else not strongly related to my request.
>
>
That's what I did saying it is super easy and fast to impl it in your
backend instead of pushing it to the API - which is already too complicated
IMHO.
> Thanks.helping saving my time
> Anatole
> Am 25.01.2016 7:44 AM schrieb "Romain Manni-Bucau" <rmannibucau@gmail.com
> >:
>
> > Le 25 janv. 2016 00:44, "Anatole Tresch" <at...@gmail.com> a écrit :
> > >
> > > Hi Romain
> > >
> > > we have already discussed this point. Stream/Client side filtering is
> > > definitively one of the things we support as well. That is not the
> > issue...
> > >
> > >
> > > Also as deltaspike showed, iterating over properties is not always
> > possible
> > > > and not a base feature for a config framework do should be done
> lazily
> > at
> > > > evaluation level so not sure the collection need is that relevant as
> > this.
> > >
> > > We already have discussed that. That's why we added the isScannable
> and
> > > other things. HAving only direct access is definitively not approriate
> as
> > > well, regardless of the capabilities of some backends providing data...
> > >
> >
> > This means we are broken by design - literally it means you ignore the
> api
> > and go the the impl behavior - or that you need to filter in your own
> > backend/source which also means it is not an api issue.
> >
> > >
> > > >
> > > > Finally the filtering should be by access: logging, usage/injection.
> > > >
> > >
> > > You access things as well when you access all properties, since we
> have
> > > the getProperites() We cannot ignore this access path IMO regarding
> > > filtering. Especially when accessing subareas this is the basic
> > > functionality behind everything. And as mentioned in my use
> masquerading
> > > case, there is NOT one filtering, effectively we have two slightly
> > > different filtering use cases, one for single accessed values and one
> for
> > > multiple or Map-accessed values. Not having this distinction in the SPI
> > > makes my use case not work, unless I add the default filtering to the
> > core
> > > implementation logic, which I think is not a valid path as well,
> because
> > it
> > > would render my logic being implementation dependent...
> > >
> >
> > Having this distinction breaks the api. isScannable is only an
> optimization
> > thing.
> >
> > > More opinions welcome ;)
> > >
> > >
> >
> > If you put such an impl in place you will quickly realize the API is only
> > used to access/inject, potentially decrypt, data but all other use cases
> > are backend specific. Making them in the API is making the api+impl hard
> > compared to sticking to it as an impl/backend.
> >
> > Thinking more about it your convention belongs to an impl for me, ie the
> > convertion between read and provided values so doesnt bring anything to
> the
> > api.
> >
> > >
> > > > Le 24 janv. 2016 12:14, "Anatole Tresch" <at...@gmail.com> a
> écrit
> > :
> > > >
> > > > > Hi all
> > > > >
> > > > > working on the concept of key filtering I encountered a problem
> that
> > was
> > > > > not really solvable given the current SPI. I try to explain my use
> > case:
> > > > >
> > > > > - following discussions we also had around metadata entries (it
> > was
> > > > > related to collection mapping at that time mainly), I also
> looked
> > what
> > > > > other store provide. Actually coreos/etcd does it quite fairly
> > simply:
> > > > > it
> > > > > hides all entries starting with a '_'.
> > > > > - So I wanted to hide all entries starting with an '_'
> similarly,
> > BUT
> > > > > still these entries should be accessible.
> > > > >
> > > > > The problem ist that I can workaround that by filtering everything
> > AFTER
> > > > > having accessed my keys something like
> > > > >
> > > > > Map<String,String> config =
> > > > > config.getProperties().filter(...);
> > > > >
> > > > > But this is cumbersome and it should be possible to cover this that
> > *BY
> > > > > DEFAULT* hidden entries should not be returned as part of the
> overall
> > > > > payload.
> > > > > Now the problem is that the current filter API only gives me the
> key
> > and
> > > > > value of the entry. I dont have the additional meta-information if
> > the
> > > > > value is filtered as part of a call to getProperties() or an
> explicit
> > > > call
> > > > > such as getOrDefault(String key, String defautlVal).
> > > > >
> > > > > Another use case would be masquerading of entries. E.g. accesses to
> > > > > getProperties() may return password values as mysecret=*****. Since
> > the
> > > > > access is generalized security is sufficient in most cases, but you
> > see
> > > > > that a value is present. Whereas accessing the value explicitly, it
> > can
> > > > > undergo authorization and decryption using the same filter. Also in
> > this
> > > > > use case I have to see if the property is accessed as part of a
> > > > properties
> > > > > extract or an explicit access.
> > > > >
> > > > > Given that I propose to extend the interface PropertyFilter from
> > > > >
> > > > > public interface PropertyFilter {
> > > > > String filterProperty(String key, String valueToBeFiltered);
> > > > >
> > > > > }
> > > > >
> > > > > to
> > > > >
> > > > > public interface PropertyFilter {
> > > > > String filterProperty(String key, String valueToBeFiltered,
> > > > > boolean singleAccess);
> > > > >
> > > > > }
> > > > >
> > > > > If no one has any important objection I will commit later today
> > this
> > > > > change targeting *02-incubating *(it is is already ready and tested
> > and
> > > > has
> > > > > proven to work).
> > > > >
> > > > > Anatole
> > > > >
> > > > >
> > > > > --
> > > > > *Anatole Tresch*
> > > > > Java Engineer & Architect, JSR Spec Lead
> > > > > Glärnischweg 10
> > > > > CH - 8620 Wetzikon
> > > > >
> > > > > *Switzerland, Europe Zurich, GMT+1*
> > > > > *Twitter: @atsticks*
> > > > > *Blogs: **http://javaremarkables.blogspot.ch/
> > > > > <http://javaremarkables.blogspot.ch/>*
> > > > >
> > > > > *Google: atsticksMobile +41-76 344 62 79*
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > *Anatole Tresch*
> > > Java Engineer & Architect, JSR Spec Lead
> > > Glärnischweg 10
> > > CH - 8620 Wetzikon
> > >
> > > *Switzerland, Europe Zurich, GMT+1*
> > > *Twitter: @atsticks*
> > > *Blogs: **http://javaremarkables.blogspot.ch/
> > > <http://javaremarkables.blogspot.ch/>*
> > >
> > > *Google: atsticksMobile +41-76 344 62 79*
> >
>
Re: Request for a small change for PropertyFilter SPI
Posted by "P. Ottlinger" <po...@apache.org>.
Am 26.01.2016 um 01:04 schrieb Romain Manni-Bucau:
> Do we have a @Experimental? Going 1 with @Experimental for some getters can
> be a compromise, wdyt?
+1 :-)
Sounds a like a compromise to me.
Cheers,
Phil
Re: Request for a small change for PropertyFilter SPI
Posted by Anatole Tresch <at...@gmail.com>.
Definitevly! We dont have yet an @Experimental, but we can take this and
create one...
2016-01-26 1:04 GMT+01:00 Romain Manni-Bucau <rm...@gmail.com>:
> Do we have a @Experimental? Going 1 with @Experimental for some getters can
> be a compromise, wdyt?
> Le 26 janv. 2016 00:53, "Anatole Tresch" <at...@gmail.com> a écrit :
>
> > Hi Romain/all
> >
> > only briefly since it is already late...
> >
> > 2016-01-25 23:22 GMT+01:00 Romain Manni-Bucau <rm...@gmail.com>:
> >
> > > issue with
> > >
> > > public interface PropertyFilter {
> > > String filterProperty(String key, String valueToBeFiltered,
> > > boolean singleAccess);
> > >
> > > }
> > >
> > > is that next step is "I miss which source provided the value", then "I
> > miss
> > > the filtered value", then "I miss which usage it is" (model can be
> > > different for injection/binding/... depending the company standards)...
> > >
> >
> > I tried to keep the change minimalistic. We had a similar scenario with
> > the PropertyConverter, but the converter case is simpler, because we
> have a
> > finished key/value evaluation (filtering is already done when convertes
> > come into action). When filtering we have only one evaluated "raw" value
> > (single access) or the full map (map access), so it is useless to try to
> > provide anything more than the single value already calculated. Once
> > filtering is done config evaluation is finished. Everything from then
> must
> > happen outside of Tamaya's core evaluation routine. Basically also
> > conversion could be something added on top. Summarizing I dont see reason
> > to add more than this little switch as of now, also having the experience
> > of implemented a myriad of different functionality such as formats,
> > collection support (not finished), remoting, serialization and more. So I
> > think there is a good chance we will not have to introduce another change
> > here... BTW I agree completely that effective filtering can be quite
> > different based on what's desired by the companies ;)
> >
> >
> > > Making a real Context will make it heavy for simple cases and can
> create
> > > cycle if we want it complete where it is actually a company thing so it
> > has
> > > to be hidden in a source/provider by design. That said it is better
> than
> > > adding single values to the signature for backward compatibility.
> > >
> >
> > As outlined above I also dont see reason or use to add a full context
> > here.
> >
> > And regarding cycles: this could already happen now, e.g. with dynamic
> > placeholders. The core implementation has logic to detect if there was a
> > change during a filter run. If not filtering is obviously finished.
> > Additionally it is possible that filters do change things vice versa,
> > ending up in a non ending loop as well. This is also prevented by a
> maximal
> > number of filter cycles done (by default 1o AFAIK).
> >
> > >
> > > Having a solid read API is the main point of a configuration solution.
> > The
> > > next one is to be extensible enough to fit any enterprise backend
> without
> > > changing the API - te logic goes there not in another API layer,
> > > password/security being the commonly accepted exception. The
> > configuration
> > > is really the link between ops and dev (and not the opposite ;)) so dev
> > > part should be minimalistic IMHO.
> > >
> > Agreed. I struggled a long time trying to figure out alternatives first
> > for my (I would say more Ops/security related) use case.
> >
> > @Anatole: let's step back and try to see it differently: where does come
> > > your use case - ie dev will or ops convention? No way to model it
> closer
> > to
> > > Tamaya to not require this change?
> >
> >
> > Sounds good ;)
> >
> > My considerations so far were:
> >
> > 1) as said adding a full context like with the converters is quite heavy,
> > but would be the most complete solution:
> >
> > String filter(FilterContext context);
> >
> > class FilterContext{
> >
> > String
> > key,
> > String value;
> >
> > Map<String,String> context
> > ,
> > boolean singleAccess);
> >
> > 2) if we could register filters differently, so I can handle the single
> and
> > map case separately I would not require this change as well. But for
> this I
> > added even more complexity in other parts of the SPI.
> >
> > 3) Another option would be 2 add another filter method but without Java 8
> > we cannot make it a default method routing to our previous one:
> >
> > String filter(String key, String valueToBeFilered);
> > String filter(String key, String valueToBeFilered,
> > FilterContext context);
> >
> > 4) Or we would change signature like:
> >
> > String filter(String key, String valueToBeFilered);
> > String filterSingle(String key, String valueToBeFilered
> > );
> >
> > The problem with this is, as Romain said, that the meta-entries context
> is
> > missing, which could be an important part for filtering (e.g. filtering
> per
> > role or something similar).
> >
> > 5) Adding some context info elsewhere in the SPI also adds up complexity
> > ans also requires thread local handling, so also not useful IMO.
> >
> > So looking at all these variants option 1 is the most complete one,
> similar
> > to the one used for convertes. From an implementation perspective it is
> > save IMO (the context Map will be defined/passed as immutable). I would
> say
> > I stop here and lets see if somebody else has a winning idea...
> >
> > Anatole
> >
> >
> >
> >
> >
> > >
> > >
> > > Romain Manni-Bucau
> > > @rmannibucau <https://twitter.com/rmannibucau> | Blog
> > > <http://rmannibucau.wordpress.com> | Github <
> > > https://github.com/rmannibucau> |
> > > LinkedIn <https://www.linkedin.com/in/rmannibucau> | Tomitriber
> > > <http://www.tomitribe.com>
> > >
> > > 2016-01-25 22:46 GMT+01:00 P. Ottlinger <po...@apache.org>:
> > >
> > > > Hi guys,
> > > >
> > > > following your brief discussion I just want to remind you that
> > sometimes
> > > > it's good two just take some steps away from the stuff ....
> > > > please be patient with each other.
> > > >
> > > >
> > > > Am 25.01.2016 um 08:53 schrieb Anatole Tresch:
> > > > > I expect you either approove the proposal or come up with a better
> > way
> > > > how
> > > > > to solve it (meaning your own proposal). I did not ask for any
> > > > theoretical
> > > > > discussion on anything else not strongly related to my request.
> > > >
> > > > Since I can understand Romain's fears and Anatole's need for a
> > pragmatic
> > > > solution I think it's good to add Anatole's proposal and give a hint
> in
> > > > the API's javadoc that this method of accessing the values might
> change
> > > > and still is volatile.
> > > >
> > > > The whole project is in 0.2-SNAPSHOT-incubating - no need to start
> > > > architecture wars.
> > > >
> > > > Let's implement and propose a solution and see what kind of feedback
> we
> > > > get - if the API is too complicated we may fix it.
> > > >
> > > > Cheers,
> > > > Phil
> > > >
> > > >
> > >
> >
> >
> >
> > --
> > *Anatole Tresch*
> > Java Engineer & Architect, JSR Spec Lead
> > Glärnischweg 10
> > CH - 8620 Wetzikon
> >
> > *Switzerland, Europe Zurich, GMT+1*
> > *Twitter: @atsticks*
> > *Blogs: **http://javaremarkables.blogspot.ch/
> > <http://javaremarkables.blogspot.ch/>*
> >
> > *Google: atsticksMobile +41-76 344 62 79 <%2B41-76%20344%2062%2079>*
> >
>
--
*Anatole Tresch*
Java Engineer & Architect, JSR Spec Lead
Glärnischweg 10
CH - 8620 Wetzikon
*Switzerland, Europe Zurich, GMT+1*
*Twitter: @atsticks*
*Blogs: **http://javaremarkables.blogspot.ch/
<http://javaremarkables.blogspot.ch/>*
*Google: atsticksMobile +41-76 344 62 79*
Re: Request for a small change for PropertyFilter SPI
Posted by Romain Manni-Bucau <rm...@gmail.com>.
Do we have a @Experimental? Going 1 with @Experimental for some getters can
be a compromise, wdyt?
Le 26 janv. 2016 00:53, "Anatole Tresch" <at...@gmail.com> a écrit :
> Hi Romain/all
>
> only briefly since it is already late...
>
> 2016-01-25 23:22 GMT+01:00 Romain Manni-Bucau <rm...@gmail.com>:
>
> > issue with
> >
> > public interface PropertyFilter {
> > String filterProperty(String key, String valueToBeFiltered,
> > boolean singleAccess);
> >
> > }
> >
> > is that next step is "I miss which source provided the value", then "I
> miss
> > the filtered value", then "I miss which usage it is" (model can be
> > different for injection/binding/... depending the company standards)...
> >
>
> I tried to keep the change minimalistic. We had a similar scenario with
> the PropertyConverter, but the converter case is simpler, because we have a
> finished key/value evaluation (filtering is already done when convertes
> come into action). When filtering we have only one evaluated "raw" value
> (single access) or the full map (map access), so it is useless to try to
> provide anything more than the single value already calculated. Once
> filtering is done config evaluation is finished. Everything from then must
> happen outside of Tamaya's core evaluation routine. Basically also
> conversion could be something added on top. Summarizing I dont see reason
> to add more than this little switch as of now, also having the experience
> of implemented a myriad of different functionality such as formats,
> collection support (not finished), remoting, serialization and more. So I
> think there is a good chance we will not have to introduce another change
> here... BTW I agree completely that effective filtering can be quite
> different based on what's desired by the companies ;)
>
>
> > Making a real Context will make it heavy for simple cases and can create
> > cycle if we want it complete where it is actually a company thing so it
> has
> > to be hidden in a source/provider by design. That said it is better than
> > adding single values to the signature for backward compatibility.
> >
>
> As outlined above I also dont see reason or use to add a full context
> here.
>
> And regarding cycles: this could already happen now, e.g. with dynamic
> placeholders. The core implementation has logic to detect if there was a
> change during a filter run. If not filtering is obviously finished.
> Additionally it is possible that filters do change things vice versa,
> ending up in a non ending loop as well. This is also prevented by a maximal
> number of filter cycles done (by default 1o AFAIK).
>
> >
> > Having a solid read API is the main point of a configuration solution.
> The
> > next one is to be extensible enough to fit any enterprise backend without
> > changing the API - te logic goes there not in another API layer,
> > password/security being the commonly accepted exception. The
> configuration
> > is really the link between ops and dev (and not the opposite ;)) so dev
> > part should be minimalistic IMHO.
> >
> Agreed. I struggled a long time trying to figure out alternatives first
> for my (I would say more Ops/security related) use case.
>
> @Anatole: let's step back and try to see it differently: where does come
> > your use case - ie dev will or ops convention? No way to model it closer
> to
> > Tamaya to not require this change?
>
>
> Sounds good ;)
>
> My considerations so far were:
>
> 1) as said adding a full context like with the converters is quite heavy,
> but would be the most complete solution:
>
> String filter(FilterContext context);
>
> class FilterContext{
>
> String
> key,
> String value;
>
> Map<String,String> context
> ,
> boolean singleAccess);
>
> 2) if we could register filters differently, so I can handle the single and
> map case separately I would not require this change as well. But for this I
> added even more complexity in other parts of the SPI.
>
> 3) Another option would be 2 add another filter method but without Java 8
> we cannot make it a default method routing to our previous one:
>
> String filter(String key, String valueToBeFilered);
> String filter(String key, String valueToBeFilered,
> FilterContext context);
>
> 4) Or we would change signature like:
>
> String filter(String key, String valueToBeFilered);
> String filterSingle(String key, String valueToBeFilered
> );
>
> The problem with this is, as Romain said, that the meta-entries context is
> missing, which could be an important part for filtering (e.g. filtering per
> role or something similar).
>
> 5) Adding some context info elsewhere in the SPI also adds up complexity
> ans also requires thread local handling, so also not useful IMO.
>
> So looking at all these variants option 1 is the most complete one, similar
> to the one used for convertes. From an implementation perspective it is
> save IMO (the context Map will be defined/passed as immutable). I would say
> I stop here and lets see if somebody else has a winning idea...
>
> Anatole
>
>
>
>
>
> >
> >
> > Romain Manni-Bucau
> > @rmannibucau <https://twitter.com/rmannibucau> | Blog
> > <http://rmannibucau.wordpress.com> | Github <
> > https://github.com/rmannibucau> |
> > LinkedIn <https://www.linkedin.com/in/rmannibucau> | Tomitriber
> > <http://www.tomitribe.com>
> >
> > 2016-01-25 22:46 GMT+01:00 P. Ottlinger <po...@apache.org>:
> >
> > > Hi guys,
> > >
> > > following your brief discussion I just want to remind you that
> sometimes
> > > it's good two just take some steps away from the stuff ....
> > > please be patient with each other.
> > >
> > >
> > > Am 25.01.2016 um 08:53 schrieb Anatole Tresch:
> > > > I expect you either approove the proposal or come up with a better
> way
> > > how
> > > > to solve it (meaning your own proposal). I did not ask for any
> > > theoretical
> > > > discussion on anything else not strongly related to my request.
> > >
> > > Since I can understand Romain's fears and Anatole's need for a
> pragmatic
> > > solution I think it's good to add Anatole's proposal and give a hint in
> > > the API's javadoc that this method of accessing the values might change
> > > and still is volatile.
> > >
> > > The whole project is in 0.2-SNAPSHOT-incubating - no need to start
> > > architecture wars.
> > >
> > > Let's implement and propose a solution and see what kind of feedback we
> > > get - if the API is too complicated we may fix it.
> > >
> > > Cheers,
> > > Phil
> > >
> > >
> >
>
>
>
> --
> *Anatole Tresch*
> Java Engineer & Architect, JSR Spec Lead
> Glärnischweg 10
> CH - 8620 Wetzikon
>
> *Switzerland, Europe Zurich, GMT+1*
> *Twitter: @atsticks*
> *Blogs: **http://javaremarkables.blogspot.ch/
> <http://javaremarkables.blogspot.ch/>*
>
> *Google: atsticksMobile +41-76 344 62 79 <%2B41-76%20344%2062%2079>*
>
Re: Request for a small change for PropertyFilter SPI
Posted by Anatole Tresch <at...@gmail.com>.
Hi Romain/all
only briefly since it is already late...
2016-01-25 23:22 GMT+01:00 Romain Manni-Bucau <rm...@gmail.com>:
> issue with
>
> public interface PropertyFilter {
> String filterProperty(String key, String valueToBeFiltered,
> boolean singleAccess);
>
> }
>
> is that next step is "I miss which source provided the value", then "I miss
> the filtered value", then "I miss which usage it is" (model can be
> different for injection/binding/... depending the company standards)...
>
I tried to keep the change minimalistic. We had a similar scenario with
the PropertyConverter, but the converter case is simpler, because we have a
finished key/value evaluation (filtering is already done when convertes
come into action). When filtering we have only one evaluated "raw" value
(single access) or the full map (map access), so it is useless to try to
provide anything more than the single value already calculated. Once
filtering is done config evaluation is finished. Everything from then must
happen outside of Tamaya's core evaluation routine. Basically also
conversion could be something added on top. Summarizing I dont see reason
to add more than this little switch as of now, also having the experience
of implemented a myriad of different functionality such as formats,
collection support (not finished), remoting, serialization and more. So I
think there is a good chance we will not have to introduce another change
here... BTW I agree completely that effective filtering can be quite
different based on what's desired by the companies ;)
> Making a real Context will make it heavy for simple cases and can create
> cycle if we want it complete where it is actually a company thing so it has
> to be hidden in a source/provider by design. That said it is better than
> adding single values to the signature for backward compatibility.
>
As outlined above I also dont see reason or use to add a full context
here.
And regarding cycles: this could already happen now, e.g. with dynamic
placeholders. The core implementation has logic to detect if there was a
change during a filter run. If not filtering is obviously finished.
Additionally it is possible that filters do change things vice versa,
ending up in a non ending loop as well. This is also prevented by a maximal
number of filter cycles done (by default 1o AFAIK).
>
> Having a solid read API is the main point of a configuration solution. The
> next one is to be extensible enough to fit any enterprise backend without
> changing the API - te logic goes there not in another API layer,
> password/security being the commonly accepted exception. The configuration
> is really the link between ops and dev (and not the opposite ;)) so dev
> part should be minimalistic IMHO.
>
Agreed. I struggled a long time trying to figure out alternatives first
for my (I would say more Ops/security related) use case.
@Anatole: let's step back and try to see it differently: where does come
> your use case - ie dev will or ops convention? No way to model it closer to
> Tamaya to not require this change?
Sounds good ;)
My considerations so far were:
1) as said adding a full context like with the converters is quite heavy,
but would be the most complete solution:
String filter(FilterContext context);
class FilterContext{
String
key,
String value;
Map<String,String> context
,
boolean singleAccess);
2) if we could register filters differently, so I can handle the single and
map case separately I would not require this change as well. But for this I
added even more complexity in other parts of the SPI.
3) Another option would be 2 add another filter method but without Java 8
we cannot make it a default method routing to our previous one:
String filter(String key, String valueToBeFilered);
String filter(String key, String valueToBeFilered,
FilterContext context);
4) Or we would change signature like:
String filter(String key, String valueToBeFilered);
String filterSingle(String key, String valueToBeFilered
);
The problem with this is, as Romain said, that the meta-entries context is
missing, which could be an important part for filtering (e.g. filtering per
role or something similar).
5) Adding some context info elsewhere in the SPI also adds up complexity
ans also requires thread local handling, so also not useful IMO.
So looking at all these variants option 1 is the most complete one, similar
to the one used for convertes. From an implementation perspective it is
save IMO (the context Map will be defined/passed as immutable). I would say
I stop here and lets see if somebody else has a winning idea...
Anatole
>
>
> Romain Manni-Bucau
> @rmannibucau <https://twitter.com/rmannibucau> | Blog
> <http://rmannibucau.wordpress.com> | Github <
> https://github.com/rmannibucau> |
> LinkedIn <https://www.linkedin.com/in/rmannibucau> | Tomitriber
> <http://www.tomitribe.com>
>
> 2016-01-25 22:46 GMT+01:00 P. Ottlinger <po...@apache.org>:
>
> > Hi guys,
> >
> > following your brief discussion I just want to remind you that sometimes
> > it's good two just take some steps away from the stuff ....
> > please be patient with each other.
> >
> >
> > Am 25.01.2016 um 08:53 schrieb Anatole Tresch:
> > > I expect you either approove the proposal or come up with a better way
> > how
> > > to solve it (meaning your own proposal). I did not ask for any
> > theoretical
> > > discussion on anything else not strongly related to my request.
> >
> > Since I can understand Romain's fears and Anatole's need for a pragmatic
> > solution I think it's good to add Anatole's proposal and give a hint in
> > the API's javadoc that this method of accessing the values might change
> > and still is volatile.
> >
> > The whole project is in 0.2-SNAPSHOT-incubating - no need to start
> > architecture wars.
> >
> > Let's implement and propose a solution and see what kind of feedback we
> > get - if the API is too complicated we may fix it.
> >
> > Cheers,
> > Phil
> >
> >
>
--
*Anatole Tresch*
Java Engineer & Architect, JSR Spec Lead
Glärnischweg 10
CH - 8620 Wetzikon
*Switzerland, Europe Zurich, GMT+1*
*Twitter: @atsticks*
*Blogs: **http://javaremarkables.blogspot.ch/
<http://javaremarkables.blogspot.ch/>*
*Google: atsticksMobile +41-76 344 62 79 <%2B41-76%20344%2062%2079>*
Re: Request for a small change for PropertyFilter SPI
Posted by Romain Manni-Bucau <rm...@gmail.com>.
issue with
public interface PropertyFilter {
String filterProperty(String key, String valueToBeFiltered,
boolean singleAccess);
}
is that next step is "I miss which source provided the value", then "I miss
the filtered value", then "I miss which usage it is" (model can be
different for injection/binding/... depending the company standards)...
Making a real Context will make it heavy for simple cases and can create
cycle if we want it complete where it is actually a company thing so it has
to be hidden in a source/provider by design. That said it is better than
adding single values to the signature for backward compatibility.
Having a solid read API is the main point of a configuration solution. The
next one is to be extensible enough to fit any enterprise backend without
changing the API - te logic goes there not in another API layer,
password/security being the commonly accepted exception. The configuration
is really the link between ops and dev (and not the opposite ;)) so dev
part should be minimalistic IMHO.
@Anatole: let's step back and try to see it differently: where does come
your use case - ie dev will or ops convention? No way to model it closer to
Tamaya to not require this change?
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Tomitriber
<http://www.tomitribe.com>
2016-01-25 22:46 GMT+01:00 P. Ottlinger <po...@apache.org>:
> Hi guys,
>
> following your brief discussion I just want to remind you that sometimes
> it's good two just take some steps away from the stuff ....
> please be patient with each other.
>
>
> Am 25.01.2016 um 08:53 schrieb Anatole Tresch:
> > I expect you either approove the proposal or come up with a better way
> how
> > to solve it (meaning your own proposal). I did not ask for any
> theoretical
> > discussion on anything else not strongly related to my request.
>
> Since I can understand Romain's fears and Anatole's need for a pragmatic
> solution I think it's good to add Anatole's proposal and give a hint in
> the API's javadoc that this method of accessing the values might change
> and still is volatile.
>
> The whole project is in 0.2-SNAPSHOT-incubating - no need to start
> architecture wars.
>
> Let's implement and propose a solution and see what kind of feedback we
> get - if the API is too complicated we may fix it.
>
> Cheers,
> Phil
>
>
Re: Request for a small change for PropertyFilter SPI
Posted by "P. Ottlinger" <po...@apache.org>.
Hi guys,
following your brief discussion I just want to remind you that sometimes
it's good two just take some steps away from the stuff ....
please be patient with each other.
Am 25.01.2016 um 08:53 schrieb Anatole Tresch:
> I expect you either approove the proposal or come up with a better way how
> to solve it (meaning your own proposal). I did not ask for any theoretical
> discussion on anything else not strongly related to my request.
Since I can understand Romain's fears and Anatole's need for a pragmatic
solution I think it's good to add Anatole's proposal and give a hint in
the API's javadoc that this method of accessing the values might change
and still is volatile.
The whole project is in 0.2-SNAPSHOT-incubating - no need to start
architecture wars.
Let's implement and propose a solution and see what kind of feedback we
get - if the API is too complicated we may fix it.
Cheers,
Phil
Re: Request for a small change for PropertyFilter SPI
Posted by Anatole Tresch <at...@gmail.com>.
So I summsrize:
- we have 2 ways of accessing properties: as single property or as map. No
room for discussions, just a fact.
- I have identified a use case where I think this additional context info
is useful for filter implementations and did a proposal.
I expect you either approove the proposal or come up with a better way how
to solve it (meaning your own proposal). I did not ask for any theoretical
discussion on anything else not strongly related to my request.
Thanks.helping saving my time
Anatole
Am 25.01.2016 7:44 AM schrieb "Romain Manni-Bucau" <rm...@gmail.com>:
> Le 25 janv. 2016 00:44, "Anatole Tresch" <at...@gmail.com> a écrit :
> >
> > Hi Romain
> >
> > we have already discussed this point. Stream/Client side filtering is
> > definitively one of the things we support as well. That is not the
> issue...
> >
> >
> > Also as deltaspike showed, iterating over properties is not always
> possible
> > > and not a base feature for a config framework do should be done lazily
> at
> > > evaluation level so not sure the collection need is that relevant as
> this.
> >
> > We already have discussed that. That's why we added the isScannable and
> > other things. HAving only direct access is definitively not approriate as
> > well, regardless of the capabilities of some backends providing data...
> >
>
> This means we are broken by design - literally it means you ignore the api
> and go the the impl behavior - or that you need to filter in your own
> backend/source which also means it is not an api issue.
>
> >
> > >
> > > Finally the filtering should be by access: logging, usage/injection.
> > >
> >
> > You access things as well when you access all properties, since we have
> > the getProperites() We cannot ignore this access path IMO regarding
> > filtering. Especially when accessing subareas this is the basic
> > functionality behind everything. And as mentioned in my use masquerading
> > case, there is NOT one filtering, effectively we have two slightly
> > different filtering use cases, one for single accessed values and one for
> > multiple or Map-accessed values. Not having this distinction in the SPI
> > makes my use case not work, unless I add the default filtering to the
> core
> > implementation logic, which I think is not a valid path as well, because
> it
> > would render my logic being implementation dependent...
> >
>
> Having this distinction breaks the api. isScannable is only an optimization
> thing.
>
> > More opinions welcome ;)
> >
> >
>
> If you put such an impl in place you will quickly realize the API is only
> used to access/inject, potentially decrypt, data but all other use cases
> are backend specific. Making them in the API is making the api+impl hard
> compared to sticking to it as an impl/backend.
>
> Thinking more about it your convention belongs to an impl for me, ie the
> convertion between read and provided values so doesnt bring anything to the
> api.
>
> >
> > > Le 24 janv. 2016 12:14, "Anatole Tresch" <at...@gmail.com> a écrit
> :
> > >
> > > > Hi all
> > > >
> > > > working on the concept of key filtering I encountered a problem that
> was
> > > > not really solvable given the current SPI. I try to explain my use
> case:
> > > >
> > > > - following discussions we also had around metadata entries (it
> was
> > > > related to collection mapping at that time mainly), I also looked
> what
> > > > other store provide. Actually coreos/etcd does it quite fairly
> simply:
> > > > it
> > > > hides all entries starting with a '_'.
> > > > - So I wanted to hide all entries starting with an '_' similarly,
> BUT
> > > > still these entries should be accessible.
> > > >
> > > > The problem ist that I can workaround that by filtering everything
> AFTER
> > > > having accessed my keys something like
> > > >
> > > > Map<String,String> config =
> > > > config.getProperties().filter(...);
> > > >
> > > > But this is cumbersome and it should be possible to cover this that
> *BY
> > > > DEFAULT* hidden entries should not be returned as part of the overall
> > > > payload.
> > > > Now the problem is that the current filter API only gives me the key
> and
> > > > value of the entry. I dont have the additional meta-information if
> the
> > > > value is filtered as part of a call to getProperties() or an explicit
> > > call
> > > > such as getOrDefault(String key, String defautlVal).
> > > >
> > > > Another use case would be masquerading of entries. E.g. accesses to
> > > > getProperties() may return password values as mysecret=*****. Since
> the
> > > > access is generalized security is sufficient in most cases, but you
> see
> > > > that a value is present. Whereas accessing the value explicitly, it
> can
> > > > undergo authorization and decryption using the same filter. Also in
> this
> > > > use case I have to see if the property is accessed as part of a
> > > properties
> > > > extract or an explicit access.
> > > >
> > > > Given that I propose to extend the interface PropertyFilter from
> > > >
> > > > public interface PropertyFilter {
> > > > String filterProperty(String key, String valueToBeFiltered);
> > > >
> > > > }
> > > >
> > > > to
> > > >
> > > > public interface PropertyFilter {
> > > > String filterProperty(String key, String valueToBeFiltered,
> > > > boolean singleAccess);
> > > >
> > > > }
> > > >
> > > > If no one has any important objection I will commit later today
> this
> > > > change targeting *02-incubating *(it is is already ready and tested
> and
> > > has
> > > > proven to work).
> > > >
> > > > Anatole
> > > >
> > > >
> > > > --
> > > > *Anatole Tresch*
> > > > Java Engineer & Architect, JSR Spec Lead
> > > > Glärnischweg 10
> > > > CH - 8620 Wetzikon
> > > >
> > > > *Switzerland, Europe Zurich, GMT+1*
> > > > *Twitter: @atsticks*
> > > > *Blogs: **http://javaremarkables.blogspot.ch/
> > > > <http://javaremarkables.blogspot.ch/>*
> > > >
> > > > *Google: atsticksMobile +41-76 344 62 79*
> > > >
> > >
> >
> >
> >
> > --
> > *Anatole Tresch*
> > Java Engineer & Architect, JSR Spec Lead
> > Glärnischweg 10
> > CH - 8620 Wetzikon
> >
> > *Switzerland, Europe Zurich, GMT+1*
> > *Twitter: @atsticks*
> > *Blogs: **http://javaremarkables.blogspot.ch/
> > <http://javaremarkables.blogspot.ch/>*
> >
> > *Google: atsticksMobile +41-76 344 62 79*
>
Re: Request for a small change for PropertyFilter SPI
Posted by Romain Manni-Bucau <rm...@gmail.com>.
Le 25 janv. 2016 00:44, "Anatole Tresch" <at...@gmail.com> a écrit :
>
> Hi Romain
>
> we have already discussed this point. Stream/Client side filtering is
> definitively one of the things we support as well. That is not the
issue...
>
>
> Also as deltaspike showed, iterating over properties is not always
possible
> > and not a base feature for a config framework do should be done lazily
at
> > evaluation level so not sure the collection need is that relevant as
this.
>
> We already have discussed that. That's why we added the isScannable and
> other things. HAving only direct access is definitively not approriate as
> well, regardless of the capabilities of some backends providing data...
>
This means we are broken by design - literally it means you ignore the api
and go the the impl behavior - or that you need to filter in your own
backend/source which also means it is not an api issue.
>
> >
> > Finally the filtering should be by access: logging, usage/injection.
> >
>
> You access things as well when you access all properties, since we have
> the getProperites() We cannot ignore this access path IMO regarding
> filtering. Especially when accessing subareas this is the basic
> functionality behind everything. And as mentioned in my use masquerading
> case, there is NOT one filtering, effectively we have two slightly
> different filtering use cases, one for single accessed values and one for
> multiple or Map-accessed values. Not having this distinction in the SPI
> makes my use case not work, unless I add the default filtering to the core
> implementation logic, which I think is not a valid path as well, because
it
> would render my logic being implementation dependent...
>
Having this distinction breaks the api. isScannable is only an optimization
thing.
> More opinions welcome ;)
>
>
If you put such an impl in place you will quickly realize the API is only
used to access/inject, potentially decrypt, data but all other use cases
are backend specific. Making them in the API is making the api+impl hard
compared to sticking to it as an impl/backend.
Thinking more about it your convention belongs to an impl for me, ie the
convertion between read and provided values so doesnt bring anything to the
api.
>
> > Le 24 janv. 2016 12:14, "Anatole Tresch" <at...@gmail.com> a écrit :
> >
> > > Hi all
> > >
> > > working on the concept of key filtering I encountered a problem that
was
> > > not really solvable given the current SPI. I try to explain my use
case:
> > >
> > > - following discussions we also had around metadata entries (it was
> > > related to collection mapping at that time mainly), I also looked
what
> > > other store provide. Actually coreos/etcd does it quite fairly
simply:
> > > it
> > > hides all entries starting with a '_'.
> > > - So I wanted to hide all entries starting with an '_' similarly,
BUT
> > > still these entries should be accessible.
> > >
> > > The problem ist that I can workaround that by filtering everything
AFTER
> > > having accessed my keys something like
> > >
> > > Map<String,String> config =
> > > config.getProperties().filter(...);
> > >
> > > But this is cumbersome and it should be possible to cover this that
*BY
> > > DEFAULT* hidden entries should not be returned as part of the overall
> > > payload.
> > > Now the problem is that the current filter API only gives me the key
and
> > > value of the entry. I dont have the additional meta-information if the
> > > value is filtered as part of a call to getProperties() or an explicit
> > call
> > > such as getOrDefault(String key, String defautlVal).
> > >
> > > Another use case would be masquerading of entries. E.g. accesses to
> > > getProperties() may return password values as mysecret=*****. Since
the
> > > access is generalized security is sufficient in most cases, but you
see
> > > that a value is present. Whereas accessing the value explicitly, it
can
> > > undergo authorization and decryption using the same filter. Also in
this
> > > use case I have to see if the property is accessed as part of a
> > properties
> > > extract or an explicit access.
> > >
> > > Given that I propose to extend the interface PropertyFilter from
> > >
> > > public interface PropertyFilter {
> > > String filterProperty(String key, String valueToBeFiltered);
> > >
> > > }
> > >
> > > to
> > >
> > > public interface PropertyFilter {
> > > String filterProperty(String key, String valueToBeFiltered,
> > > boolean singleAccess);
> > >
> > > }
> > >
> > > If no one has any important objection I will commit later today this
> > > change targeting *02-incubating *(it is is already ready and tested
and
> > has
> > > proven to work).
> > >
> > > Anatole
> > >
> > >
> > > --
> > > *Anatole Tresch*
> > > Java Engineer & Architect, JSR Spec Lead
> > > Glärnischweg 10
> > > CH - 8620 Wetzikon
> > >
> > > *Switzerland, Europe Zurich, GMT+1*
> > > *Twitter: @atsticks*
> > > *Blogs: **http://javaremarkables.blogspot.ch/
> > > <http://javaremarkables.blogspot.ch/>*
> > >
> > > *Google: atsticksMobile +41-76 344 62 79*
> > >
> >
>
>
>
> --
> *Anatole Tresch*
> Java Engineer & Architect, JSR Spec Lead
> Glärnischweg 10
> CH - 8620 Wetzikon
>
> *Switzerland, Europe Zurich, GMT+1*
> *Twitter: @atsticks*
> *Blogs: **http://javaremarkables.blogspot.ch/
> <http://javaremarkables.blogspot.ch/>*
>
> *Google: atsticksMobile +41-76 344 62 79*
Re: Request for a small change for PropertyFilter SPI
Posted by Anatole Tresch <at...@gmail.com>.
Hi Romain
we have already discussed this point. Stream/Client side filtering is
definitively one of the things we support as well. That is not the issue...
Also as deltaspike showed, iterating over properties is not always possible
> and not a base feature for a config framework do should be done lazily at
> evaluation level so not sure the collection need is that relevant as this.
We already have discussed that. That's why we added the isScannable and
other things. HAving only direct access is definitively not approriate as
well, regardless of the capabilities of some backends providing data...
>
> Finally the filtering should be by access: logging, usage/injection.
>
You access things as well when you access all properties, since we have
the getProperites() We cannot ignore this access path IMO regarding
filtering. Especially when accessing subareas this is the basic
functionality behind everything. And as mentioned in my use masquerading
case, there is NOT one filtering, effectively we have two slightly
different filtering use cases, one for single accessed values and one for
multiple or Map-accessed values. Not having this distinction in the SPI
makes my use case not work, unless I add the default filtering to the core
implementation logic, which I think is not a valid path as well, because it
would render my logic being implementation dependent...
More opinions welcome ;)
> Le 24 janv. 2016 12:14, "Anatole Tresch" <at...@gmail.com> a écrit :
>
> > Hi all
> >
> > working on the concept of key filtering I encountered a problem that was
> > not really solvable given the current SPI. I try to explain my use case:
> >
> > - following discussions we also had around metadata entries (it was
> > related to collection mapping at that time mainly), I also looked what
> > other store provide. Actually coreos/etcd does it quite fairly simply:
> > it
> > hides all entries starting with a '_'.
> > - So I wanted to hide all entries starting with an '_' similarly, BUT
> > still these entries should be accessible.
> >
> > The problem ist that I can workaround that by filtering everything AFTER
> > having accessed my keys something like
> >
> > Map<String,String> config =
> > config.getProperties().filter(...);
> >
> > But this is cumbersome and it should be possible to cover this that *BY
> > DEFAULT* hidden entries should not be returned as part of the overall
> > payload.
> > Now the problem is that the current filter API only gives me the key and
> > value of the entry. I dont have the additional meta-information if the
> > value is filtered as part of a call to getProperties() or an explicit
> call
> > such as getOrDefault(String key, String defautlVal).
> >
> > Another use case would be masquerading of entries. E.g. accesses to
> > getProperties() may return password values as mysecret=*****. Since the
> > access is generalized security is sufficient in most cases, but you see
> > that a value is present. Whereas accessing the value explicitly, it can
> > undergo authorization and decryption using the same filter. Also in this
> > use case I have to see if the property is accessed as part of a
> properties
> > extract or an explicit access.
> >
> > Given that I propose to extend the interface PropertyFilter from
> >
> > public interface PropertyFilter {
> > String filterProperty(String key, String valueToBeFiltered);
> >
> > }
> >
> > to
> >
> > public interface PropertyFilter {
> > String filterProperty(String key, String valueToBeFiltered,
> > boolean singleAccess);
> >
> > }
> >
> > If no one has any important objection I will commit later today this
> > change targeting *02-incubating *(it is is already ready and tested and
> has
> > proven to work).
> >
> > Anatole
> >
> >
> > --
> > *Anatole Tresch*
> > Java Engineer & Architect, JSR Spec Lead
> > Glärnischweg 10
> > CH - 8620 Wetzikon
> >
> > *Switzerland, Europe Zurich, GMT+1*
> > *Twitter: @atsticks*
> > *Blogs: **http://javaremarkables.blogspot.ch/
> > <http://javaremarkables.blogspot.ch/>*
> >
> > *Google: atsticksMobile +41-76 344 62 79*
> >
>
--
*Anatole Tresch*
Java Engineer & Architect, JSR Spec Lead
Glärnischweg 10
CH - 8620 Wetzikon
*Switzerland, Europe Zurich, GMT+1*
*Twitter: @atsticks*
*Blogs: **http://javaremarkables.blogspot.ch/
<http://javaremarkables.blogspot.ch/>*
*Google: atsticksMobile +41-76 344 62 79*
Re: Request for a small change for PropertyFilter SPI
Posted by Romain Manni-Bucau <rm...@gmail.com>.
Hi Anatole,
No way to just use Stream there? Sounds easier and lighter.
Also as deltaspike showed, iterating over properties is not always possible
and not a base feature for a config framework do should be done lazily at
evaluation level so not sure the collection need is that relevant as this.
Finally the filtering should be by access: logging, usage/injection.
Le 24 janv. 2016 12:14, "Anatole Tresch" <at...@gmail.com> a écrit :
> Hi all
>
> working on the concept of key filtering I encountered a problem that was
> not really solvable given the current SPI. I try to explain my use case:
>
> - following discussions we also had around metadata entries (it was
> related to collection mapping at that time mainly), I also looked what
> other store provide. Actually coreos/etcd does it quite fairly simply:
> it
> hides all entries starting with a '_'.
> - So I wanted to hide all entries starting with an '_' similarly, BUT
> still these entries should be accessible.
>
> The problem ist that I can workaround that by filtering everything AFTER
> having accessed my keys something like
>
> Map<String,String> config =
> config.getProperties().filter(...);
>
> But this is cumbersome and it should be possible to cover this that *BY
> DEFAULT* hidden entries should not be returned as part of the overall
> payload.
> Now the problem is that the current filter API only gives me the key and
> value of the entry. I dont have the additional meta-information if the
> value is filtered as part of a call to getProperties() or an explicit call
> such as getOrDefault(String key, String defautlVal).
>
> Another use case would be masquerading of entries. E.g. accesses to
> getProperties() may return password values as mysecret=*****. Since the
> access is generalized security is sufficient in most cases, but you see
> that a value is present. Whereas accessing the value explicitly, it can
> undergo authorization and decryption using the same filter. Also in this
> use case I have to see if the property is accessed as part of a properties
> extract or an explicit access.
>
> Given that I propose to extend the interface PropertyFilter from
>
> public interface PropertyFilter {
> String filterProperty(String key, String valueToBeFiltered);
>
> }
>
> to
>
> public interface PropertyFilter {
> String filterProperty(String key, String valueToBeFiltered,
> boolean singleAccess);
>
> }
>
> If no one has any important objection I will commit later today this
> change targeting *02-incubating *(it is is already ready and tested and has
> proven to work).
>
> Anatole
>
>
> --
> *Anatole Tresch*
> Java Engineer & Architect, JSR Spec Lead
> Glärnischweg 10
> CH - 8620 Wetzikon
>
> *Switzerland, Europe Zurich, GMT+1*
> *Twitter: @atsticks*
> *Blogs: **http://javaremarkables.blogspot.ch/
> <http://javaremarkables.blogspot.ch/>*
>
> *Google: atsticksMobile +41-76 344 62 79*
>