Re: [ops] Unify NodeStore/DataStore configurations by using nstab

[Michael Dürig <md...@apache.org>]

Hi Arek,

My mistake, my initial message (below) was intended for oak-dev.

Michael

On 10.05.17 11:01, Arek Kita wrote:
> Hi Michael,
> 
> sorry, but I noticed right now, that the email was directed *only* to
> me (not sure if on purpose).
> 
> Yes, I guess the approach should be not in the way how to create and
> load such file, but starting from the other way round, how we can glue
> all NodeStores (DataStores) together first based on what we have now
> using one Builder/Factory interfaces that will create and link
> everything. Then we can think about higher-level configuration format
> (human configurable, not only dev configurable).
> 
> I guess this would be a good topic for an Oakathon because I'm not
> involved in Oak heavily right now at any particular story. I guess
> however that it would be easier for me to be focused and involved once
> for that particular improvement.
> 
> Thanks,
> Arek
> 
> 2017-05-02 10:42 GMT+02:00 Michael Dürig <md...@apache.org>:
>>
>> Hi Arek,
>>
>> I agree that Oak would benefit from being simpler and more uniform to
>> configure and run. The current approach with the Oak and Jcr builder classes
>> has somehow outgrown its initial purpose.
>>
>> I am somewhat sceptic about (starting with) configurations files thought.
>> Configuration files tend to suffer from weak semantics that is not well
>> understood and poorly documented. Going forward it can be difficult to
>> evolve the format in a consistent way while keeping backward compatibility.
>>
>> I would therefore rather suggest to start experimenting with simpler ways to
>> setup Oak in code through an internal DSL. From there we can start a
>> discussion whether it is worth to introduce external bindings and how these
>> should look. Maybe such bindings would just expose the most common subset of
>> features and you would need to turn to the internal configuration DSL for
>> the full expressive power.
>>
>> Michael
>>
>>
>> On 28.04.17 12:56, Arek Kita wrote:
>>>
>>> Hi,
>>>
>>> I've noticed recently that with many different NodeStore
>>> implementation (Segment, Document, Multiplexing) but also DataStore
>>> implementation (File, S3, Azure) and some composite ones like
>>> (Hierarchical, Federated - that was already mentioned in [0]) it
>>> becomes more and more difficult to set up everything correctly and be
>>> able to know the current persistence state of repository (especially
>>> with pretty aged repos).
>>>
>>> Moreover, the configuration pattern that is based on individual PID of
>>> one service becomes problematic (i.e. recent change for
>>> SegmentNodeStoreService).
>>>
>>>   From the operations and user perspective everything should be treated
>>> as a whole IMHO no matter which service handles which fragment of
>>> persistence layout. Oak should know itself how to "autowire" different
>>> parts, obviously with some hints and pointers from users as they want
>>> to run Oak in their own preferred layout.
>>>
>>> My proposal would be to integrate everything together to a pretty old
>>> concept called "fstab". For our purposes I would call it "nstab".
>>>
>>> This could look like [1] for the most simple case (with internal
>>> blobs), [2] for typical SegmentMK + FDS, [3] for SegmentMK + S3DS, [4]
>>> for MultiplexingNodeStore with some areas of repo set as read only. I
>>> think we could also model Hierarchical and Federated DataStores as
>>> well in the future.
>>>
>>> Examples are for illustration purposes but I guess such setup will
>>> help changing layout without a need to inspect many OSGi
>>> configurations in a current setup and making sure some conflicting
>>> ones aren't active.
>>>
>>> The schema is also similar to an UNIX-way of configuring filesystem so
>>> it will help Oak users to understand the layout (at least better than
>>> it is now). I see also advantage for automated tooling like
>>> oak-upgrade for complex cases in the future - user just provides
>>> source nstab and target nstab in order to migrate repository.
>>>
>>> The config should be also simpler avoiding things like customBlobStore
>>> (it will be inferred from context).
>>>
>>> WDYT? I have some thoughts how could this be implemented but first I
>>> would like to know your opinions on that.
>>>
>>> Thanks in advance for feedback!
>>> Arek
>>>
>>>
>>> [0] http://oak.markmail.org/thread/22dvuo6b7ab5ib7m
>>> [1]
>>> https://gist.githubusercontent.com/kitarek/f755dab6e889d1dfc5a1c595727f0171/raw/53d41ac7f935886783afd6c85d60e38e565a9259/nstab.1
>>> [2]
>>> https://gist.githubusercontent.com/kitarek/f755dab6e889d1dfc5a1c595727f0171/raw/53d41ac7f935886783afd6c85d60e38e565a9259/nstab.2
>>> [3]
>>> https://gist.githubusercontent.com/kitarek/f755dab6e889d1dfc5a1c595727f0171/raw/53d41ac7f935886783afd6c85d60e38e565a9259/nstab.3
>>> [4]
>>> https://gist.githubusercontent.com/kitarek/f755dab6e889d1dfc5a1c595727f0171/raw/53d41ac7f935886783afd6c85d60e38e565a9259/nstab.4
>>>
>>