BlobFactory (was Re: svn commit: r1401571 - in /jackrabbit/oak/trunk/oak-core/src/main/java/org/apache/jackrabbit/oak/api: BlobFactory.java Root.java)

[Michael Dürig <md...@apache.org>]

On 24.10.12 8:33, angela@apache.org wrote:
> Author: angela
> Date: Wed Oct 24 07:33:02 2012
> New Revision: 1401571
>
> URL: http://svn.apache.org/viewvc?rev=1401571&view=rev
> Log:
> OAK-392 : Review ContentSession#createBlob
> - * BlobFactory... TODO
> + * BlobFactory...
> + * TODO review again if we really need/want to expose that in the OAK API

The consequences of not doing so is, that there is no way to create 
blobs through the Oak API which are immediately streamed down to the 
Microkernel.

> + * TODO in particular exposing this interface (and Blob) requires additional thoughts on
> + * TODO - lifecycle of the factory,
> + * TODO - lifecycle of the Blob,
> + * TODO - access restrictions and how permissions are enforced on blob creation
> + * TODO - searchability, versioning and so forth

All these TODOs also apply to the blobs which get created through the 
ValueFactory in oak-jcr and this was already the case before the 
refactorings of OAK-350. What where the considerations there?

Michael

>    */
>   public interface BlobFactory {

Re: BlobFactory (was Re: svn commit: r1401571 - in /jackrabbit/oak/trunk/oak-core/src/main/java/org/apache/jackrabbit/oak/api: BlobFactory.java Root.java)

[Jukka Zitting <ju...@gmail.com>]

Hi,

On Wed, Oct 24, 2012 at 12:30 PM, Angela Schreiber <an...@adobe.com> wrote:
> - is the content ID of those binaries exposed somewhere?

IMO it shouldn't be exposed to clients.

> - is there an alternative way of accessing such an binary except
>   for regular property-access which was controlled by access control?

No, not beyond the session that created the binary.

> - how is the http-binding of such an binary?

A remote client should only be able to read a binary through the
containing property. The access controls on that property limit who
can access the binary.

> - how do we deal with binaries that are not associated with an
>   existing property? are they accessible somewhere/somehow? and if
>   so by whom?

Such binaries should only be accessible by the session that created them.

> - is there a way of searching the binary-store by other means that
>   executing a regular jcr/oak search on the content tree?

No.

> - can we prevent some sessions from writing to the binary-store
>   altogehter? if we can't what are the implications?

I guess we could, though I think the easiest way to do that would be
to have some special "upload binary" permission that's granted by
default but can be explicitly denied for a particular user or a
session.

> - is there way of reverting change made to the binary store without
>   having to wait for garbage collection? who would be in charge
>   of doing so? what was the interface to do that?

One could always trigger a new garbage collection run. Alternatively,
if we modify the MicroKernel API accordingly, it could be possible for
the backend implementation to tie new binaries to the sessions that
created them so that when the session is closed any binaries that
didn't get added to the content tree could be automatically released.

BR,

Jukka Zitting

Re: BlobFactory (was Re: svn commit: r1401571 - in /jackrabbit/oak/trunk/oak-core/src/main/java/org/apache/jackrabbit/oak/api: BlobFactory.java Root.java)

[Michael Dürig <mi...@gmail.com>]

On 24.10.12 11:30, Angela Schreiber wrote:

> - is the content ID of those binaries exposed somewhere?
No.

> - is there an alternative way of accessing such an binary except
>    for regular property-access which was controlled by access control?
No.

> - how is the http-binding of such an binary?
Only through a property.

> - how do we deal with binaries that are not associated with an
>    existing property? are they accessible somewhere/somehow? and if
>    so by whom?
Currently they will get garbage collected at some point. Until then they 
are only accessible when you knwo the corresponding binary ID. The 
binary ID lives and dies with the corresponding binary property.

> - is there a way of searching the binary-store by other means that
>    executing a regular jcr/oak search on the content tree?
No.

> - can we prevent some sessions from writing to the binary-store
>    altogehter? if we can't what are the implications?
I guess we could. The only implication I see is that without this 
everyone with access to a Root instance can fill the binary store with 
garbage while without such an access control mechanism in place you 
could only do that when you also possess the necessary permission (which 
will be quite regularly the case since everyone who wants to create 
binary properties will have to have that permission).

> - is there way of reverting change made to the binary store without
>    having to wait for garbage collection? who would be in charge
>    of doing so? what was the interface to do that?
Currently not. This would require support from the Microkernel.

Michael

>
> it's this kind of questions that i have with respect to the binary
> store and which i would prefer us to have thought about before we
> have to deal with customer escalation.
>
> regards
> angela
>
>

Re: BlobFactory (was Re: svn commit: r1401571 - in /jackrabbit/oak/trunk/oak-core/src/main/java/org/apache/jackrabbit/oak/api: BlobFactory.java Root.java)

[Angela Schreiber <an...@adobe.com>]

hi jukka

> On Wed, Oct 24, 2012 at 11:01 AM, Angela Schreiber<an...@adobe.com>  wrote:
>> basically i don't care how we implement it. but i think it's
>> crucial that we have the ability to control it... immediately
>> persisting binary-values without being able to enforce any kind
>> of validation or permission checks and ultimately reverting back
>> the changes is IMO not acceptable.
>
> What kind of validation or permission checks should/could be applied
> at that point?

it's not only about permissions... i just prevent us from
running into similar security issues we are having with the
version store in jackrabbit.
that's why i feel it's important that we define a life-cycle
for the items stored in the separate binary-store. this includes
questions like:

- is the content ID of those binaries exposed somewhere?
- is there an alternative way of accessing such an binary except
   for regular property-access which was controlled by access control?
- how is the http-binding of such an binary?
- how do we deal with binaries that are not associated with an
   existing property? are they accessible somewhere/somehow? and if
   so by whom?
- is there a way of searching the binary-store by other means that
   executing a regular jcr/oak search on the content tree?
- can we prevent some sessions from writing to the binary-store
   altogehter? if we can't what are the implications?
- is there way of reverting change made to the binary store without
   having to wait for garbage collection? who would be in charge
   of doing so? what was the interface to do that?

it's this kind of questions that i have with respect to the binary
store and which i would prefer us to have thought about before we
have to deal with customer escalation.

regards
angela

Re: BlobFactory (was Re: svn commit: r1401571 - in /jackrabbit/oak/trunk/oak-core/src/main/java/org/apache/jackrabbit/oak/api: BlobFactory.java Root.java)

[Jukka Zitting <ju...@gmail.com>]

Hi,

On Wed, Oct 24, 2012 at 11:01 AM, Angela Schreiber <an...@adobe.com> wrote:
> basically i don't care how we implement it. but i think it's
> crucial that we have the ability to control it... immediately
> persisting binary-values without being able to enforce any kind
> of validation or permission checks and ultimately reverting back
> the changes is IMO not acceptable.

What kind of validation or permission checks should/could be applied
at that point?

Trying to prevent denial-of-service attacks beyond things like normal
garbage collection at this level is an effort in vain. There's no
reliable way for the repository to know which higher level access
patterns are valid and which are not.

BR,

Jukka Zitting

Re: BlobFactory (was Re: svn commit: r1401571 - in /jackrabbit/oak/trunk/oak-core/src/main/java/org/apache/jackrabbit/oak/api: BlobFactory.java Root.java)

[Angela Schreiber <an...@adobe.com>]

hi michael

>>> The consequences of not doing so is, that there is no way to create
>>> blobs through the Oak API which are immediately streamed down to the
>>> Microkernel.
>>
>> yes... i know :)
>> don't get me wrong, i am not opposed to that in general.
>> but there are some open issues associated with that, which i
>> would like us to clarify before we finalize the API.
>
> Since we have to address these issues anyway because we have the same
> problem with ValueFactory.createBinary(), I'd rather also keep a means
> for creating Blobs in the Oak API. Otherwise the API wouldn't be
> complete and other bindings would have to come up with their own plugins
> just to write binaries.

well... valuefactory.createBinary uses the same blob-factory as
it used the contentsession#createBlob before. i just wanted to
reflect the fact that binary-creation is totally separate from
creation of regular content in the API. in particular i don't
see why a valuefactory instance should have access to the
complete feature-set exposed by the Root when it only needs
to create binaries. but that's obviously a matter of taste.

>> yes, that's correct... but it doesn't mean that we shouldn't
>> think about the consequences of having such a feature.
>
> Right. I just wanted to check whether there where considerations re.
> this earlier and what these where. But maybe the whole thing only came
> up through having the BlobFactory that explicitly in the API.

no, that's not the case.
i already asked for being very conscious/careful about security issues
with respect to that binary-store a year ago. i would have
added the same comment/TODO to any other createBlob method...

>> what happens if the adding/changing a binary property fails due
>> to lack of permissions or due to conflicts or value constraint
>> violations or some other custom value-validation? i think that we
>> need the ability to revert the changes made to the binary-store
>> in such cases...
>> if we don't have that ability everyone having a Root/ContentSession
>> at hand (even if it was a read-only session) could dump any kind
>> of binaries into the repository. that doesn't make sense to me
>> and looks like a critical security issue... remember o.html? IMO
>> it's that kind of troubles we are looking for when not having any
>> means to control who and what is written to the binary store and
>> not having the ability to revert changes if the overall commit
>> fails.
>
> Currently the data store garbage collector of the Microkernel cleans
> unreferenced binaries out after some time. If we want to have more
> explicit control of the life cycle for binaries, we'd need to have
> support for this from the Microkernel.

basically i don't care how we implement it. but i think it's
crucial that we have the ability to control it... immediately
persisting binary-values without being able to enforce any kind
of validation or permission checks and ultimately reverting back
the changes is IMO not acceptable.

kind regards
angela



> Michael
>
>>
>> kind regards
>> angela

Re: BlobFactory (was Re: svn commit: r1401571 - in /jackrabbit/oak/trunk/oak-core/src/main/java/org/apache/jackrabbit/oak/api: BlobFactory.java Root.java)

[Michael Dürig <md...@apache.org>]

On 24.10.12 9:24, Angela Schreiber wrote:
> hi michael
>
>>> + * TODO review again if we really need/want to expose that in the
>>> OAK API
>>
>> The consequences of not doing so is, that there is no way to create
>> blobs through the Oak API which are immediately streamed down to the
>> Microkernel.
>
> yes... i know :)
> don't get me wrong, i am not opposed to that in general.
> but there are some open issues associated with that, which i
> would like us to clarify before we finalize the API.

Since we have to address these issues anyway because we have the same 
problem with ValueFactory.createBinary(), I'd rather also keep a means 
for creating Blobs in the Oak API. Otherwise the API wouldn't be 
complete and other bindings would have to come up with their own plugins 
just to write binaries.

>
>>> + * TODO in particular exposing this interface (and Blob) requires
>>> additional thoughts on
>>> + * TODO - lifecycle of the factory,
>>> + * TODO - lifecycle of the Blob,
>>> + * TODO - access restrictions and how permissions are enforced on
>>> blob creation
>>> + * TODO - searchability, versioning and so forth
>>
>> All these TODOs also apply to the blobs which get created through the
>> ValueFactory in oak-jcr and this was already the case before the
>> refactorings of OAK-350.
>
> yes, that's correct... but it doesn't mean that we shouldn't
> think about the consequences of having such a feature.

Right. I just wanted to check whether there where considerations re. 
this earlier and what these where. But maybe the whole thing only came 
up through having the BlobFactory that explicitly in the API.


> what happens if the adding/changing a binary property fails due
> to lack of permissions or due to conflicts or value constraint
> violations or some other custom value-validation? i think that we
> need the ability to revert the changes made to the binary-store
> in such cases...
> if we don't have that ability everyone having a Root/ContentSession
> at hand (even if it was a read-only session) could dump any kind
> of binaries into the repository. that doesn't make sense to me
> and looks like a critical security issue... remember o.html? IMO
> it's that kind of troubles we are looking for when not having any
> means to control who and what is written to the binary store and
> not having the ability to revert changes if the overall commit
> fails.

Currently the data store garbage collector of the Microkernel cleans 
unreferenced binaries out after some time. If we want to have more 
explicit control of the life cycle for binaries, we'd need to have 
support for this from the Microkernel.

Michael

>
> kind regards
> angela

Re: BlobFactory (was Re: svn commit: r1401571 - in /jackrabbit/oak/trunk/oak-core/src/main/java/org/apache/jackrabbit/oak/api: BlobFactory.java Root.java)

[Angela Schreiber <an...@adobe.com>]

hi michael

>> + * TODO review again if we really need/want to expose that in the OAK API
>
> The consequences of not doing so is, that there is no way to create
> blobs through the Oak API which are immediately streamed down to the
> Microkernel.

yes... i know :)
don't get me wrong, i am not opposed to that in general.
but there are some open issues associated with that, which i
would like us to clarify before we finalize the API.

>> + * TODO in particular exposing this interface (and Blob) requires additional thoughts on
>> + * TODO - lifecycle of the factory,
>> + * TODO - lifecycle of the Blob,
>> + * TODO - access restrictions and how permissions are enforced on blob creation
>> + * TODO - searchability, versioning and so forth
>
> All these TODOs also apply to the blobs which get created through the
> ValueFactory in oak-jcr and this was already the case before the
> refactorings of OAK-350.

yes, that's correct... but it doesn't mean that we shouldn't
think about the consequences of having such a feature.

what happens if the adding/changing a binary property fails due
to lack of permissions or due to conflicts or value constraint
violations or some other custom value-validation? i think that we
need the ability to revert the changes made to the binary-store
in such cases...
if we don't have that ability everyone having a Root/ContentSession
at hand (even if it was a read-only session) could dump any kind
of binaries into the repository. that doesn't make sense to me
and looks like a critical security issue... remember o.html? IMO
it's that kind of troubles we are looking for when not having any
means to control who and what is written to the binary store and
not having the ability to revert changes if the overall commit
fails.

kind regards
angela