Going beyond path-based access in MK (Was: Mongo property limit)

[Jukka Zitting <ju...@gmail.com>]

Hi,

On Mon, Sep 24, 2012 at 12:03 PM, Mete Atamel <ma...@adobe.com> wrote:
> In MongoMK, we have a "path" property which stores the full path of a
> node.

This actually raises a bigger question about the MicroKernel design.
Currently all content access through the MicroKernel is keyed by the
revision and full path of the node being accessed.

In practice this means that the MicroKernel needs to do a
revision/path lookup for each access, which for example in this case
means that the node records in the MongoMK need to store the full path
of the node.

However, when accessing a node, Oak will follow the path and access
all the ancestor nodes first (in practice they're already cached) and
thus we may well have additional information that the MicroKernel
implementation could use for accessing a node.

For example, in MongoMK, instead of storing just a set of child node
names in a parent node and then do an index lookup based on the
combined path of a child, an alternative could be to store a name ->
oid map in the parent node and thus be able to directly lookup a child
node based on its object id.

Would it make sense to consider adjusting the MicroKernel interface to
allow such optimizations?

BR,

Jukka Zitting

Re: Going beyond path-based access in MK (Was: Mongo property limit)

[Mete Atamel <ma...@adobe.com>]

I think we all agree that we need to handle the path limit in MongoMK
sooner rather than later but the open question is whether we can fix this
with or without MicroKernel API changes. My feeling is that we can
probably handle this as a MongoMK implementation detail (i.e. no
MicroKernal API changes) but I haven't spent too much time thinking about
possible solutions yet. Once I do that, I'll be able to comment further.

-Mete

On 10/1/12 6:46 PM, "Michael Dürig" <md...@apache.org> wrote:

>
>
>On 25.9.12 10:20, Stefan Guggisberg wrote:
>> On Mon, Sep 24, 2012 at 2:23 PM, Jukka Zitting
>><ju...@gmail.com> wrote:
>>> Hi,
>>>
>>>
>>> Would it make sense to consider adjusting the MicroKernel interface to
>>> allow such optimizations?
>>
>> IIUC storing the path in every record is an implementation detail of the
>> current MongoMK design. i am rather reluctant to make major changes
>> to the MicroKernel API at this late stage and based on limitations of a
>> specific MK implementation approach.
>
>I'd rather make this change at this early stage if it turns out to be
>necessary than having to fix it when we are in production.
>
>Michael
>
>>
>> cheers
>> stefan
>>
>>>
>>> BR,
>>>
>>> Jukka Zitting

Re: Going beyond path-based access in MK (Was: Mongo property limit)

[Michael Dürig <md...@apache.org>]

On 25.9.12 10:20, Stefan Guggisberg wrote:
> On Mon, Sep 24, 2012 at 2:23 PM, Jukka Zitting <ju...@gmail.com> wrote:
>> Hi,
>>
>>
>> Would it make sense to consider adjusting the MicroKernel interface to
>> allow such optimizations?
>
> IIUC storing the path in every record is an implementation detail of the
> current MongoMK design. i am rather reluctant to make major changes
> to the MicroKernel API at this late stage and based on limitations of a
> specific MK implementation approach.

I'd rather make this change at this early stage if it turns out to be 
necessary than having to fix it when we are in production.

Michael

>
> cheers
> stefan
>
>>
>> BR,
>>
>> Jukka Zitting

Re: Going beyond path-based access in MK (Was: Mongo property limit)

[Stefan Guggisberg <st...@gmail.com>]

On Mon, Sep 24, 2012 at 2:23 PM, Jukka Zitting <ju...@gmail.com> wrote:
> Hi,
>
> On Mon, Sep 24, 2012 at 12:03 PM, Mete Atamel <ma...@adobe.com> wrote:
>> In MongoMK, we have a "path" property which stores the full path of a
>> node.
>
> This actually raises a bigger question about the MicroKernel design.
> Currently all content access through the MicroKernel is keyed by the
> revision and full path of the node being accessed.
>
> In practice this means that the MicroKernel needs to do a
> revision/path lookup for each access, which for example in this case
> means that the node records in the MongoMK need to store the full path
> of the node.
>
> However, when accessing a node, Oak will follow the path and access
> all the ancestor nodes first (in practice they're already cached) and
> thus we may well have additional information that the MicroKernel
> implementation could use for accessing a node.
>
> For example, in MongoMK, instead of storing just a set of child node
> names in a parent node and then do an index lookup based on the
> combined path of a child, an alternative could be to store a name ->
> oid map in the parent node and thus be able to directly lookup a child
> node based on its object id.
>
> Would it make sense to consider adjusting the MicroKernel interface to
> allow such optimizations?

IIUC storing the path in every record is an implementation detail of the
current MongoMK design. i am rather reluctant to make major changes
to the MicroKernel API at this late stage and based on limitations of a
specific MK implementation approach.

cheers
stefan

>
> BR,
>
> Jukka Zitting

Re: Going beyond path-based access in MK (Was: Mongo property limit)

[Mete Atamel <ma...@adobe.com>]

Hi,


That's an interesting idea. On one hand, we'll save space by not storing
the full path of every node, but on the other hand, we'll need to maintain
this additional node name -> oid map in every parent node. Additionally,
for every node retrieval, we'll have to traverse from root all the way to
the leaf child and at every step, we'll need to check the name -> oid map
to figure out the next step, right? So, instead of one Mongo query with
full path, we'll be doing N queries with oids where N is the depth of the
full path being retrieved. So, at this point, I'm wondering if this is
really an optimization?

-Mete

On 9/24/12 2:23 PM, "Jukka Zitting" <ju...@gmail.com> wrote:

>Hi,
>
>On Mon, Sep 24, 2012 at 12:03 PM, Mete Atamel <ma...@adobe.com> wrote:
>> In MongoMK, we have a "path" property which stores the full path of a
>> node.
>
>This actually raises a bigger question about the MicroKernel design.
>Currently all content access through the MicroKernel is keyed by the
>revision and full path of the node being accessed.
>
>In practice this means that the MicroKernel needs to do a
>revision/path lookup for each access, which for example in this case
>means that the node records in the MongoMK need to store the full path
>of the node.
>
>However, when accessing a node, Oak will follow the path and access
>all the ancestor nodes first (in practice they're already cached) and
>thus we may well have additional information that the MicroKernel
>implementation could use for accessing a node.
>
>For example, in MongoMK, instead of storing just a set of child node
>names in a parent node and then do an index lookup based on the
>combined path of a child, an alternative could be to store a name ->
>oid map in the parent node and thus be able to directly lookup a child
>node based on its object id.
>
>Would it make sense to consider adjusting the MicroKernel interface to
>allow such optimizations?
>
>BR,
>
>Jukka Zitting