You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cocoon.apache.org by Alan <al...@engrm.com> on 2004/02/16 20:32:34 UTC
Announce: Momento
Momento is a native XML persistence engine. Is supports XSLT 2.0 and
XQuery 1.0 (via Saxon), it supports XUpdate.
It is transactional and ACID.
It is designed with Cocoon in mind.
I am considering a open source path for future development.
http://engrm.com/project/com.agtrz.momento/
Thoughts?
--
Alan / alan@engrm.com / http://engrm.com/
Re: Announce: Momento
Posted by Alan <al...@engrm.com>.
* Stefano Mazzocchi <st...@apache.org> [2004-02-18 14:02]:
> Alan wrote:
>
> >* Stefano Mazzocchi <st...@apache.org> [2004-02-17 04:22]:
> >
> >>Alan wrote:
> >>
> >>
> >>>Momento is a native XML persistence engine. Is supports XSLT 2.0 and
> >>> XQuery 1.0 (via Saxon), it supports XUpdate.
> >>Very interesting. Can you tell us more on how it works? have any numbers
> >>on how fast/scalable it gets? what's the difference between Momento and
> >>a native xml database like XIndice?
> > 3) I can't compare Momento to Xindice at the moment. Last I looked
> > at Xindice was last November. I'm announcing to get such insight.
> >
> > (I am willing to say that Momento isn't wedded to a specific
> > API, such as XML::DB. It works with Saxon to provide
> > XQuery and XSLT.
> What is the interfacing API then? JAXP? or Saxon's?
JAXP. Standard APIs where possible.
XUpdate is implemented as a SAX filter. Wrap the filter around a
Momento document, put the filter in an XMLReader, feed XUpdate
to the parse method of an XMLReader.
> > I'm implementing XUpdate currently. A read-only W3 DOM is a
> > simple matter, if there is call for it. A read-write W3 DOM is
> > not so simple, or desireable, but entirely feasable.
> >
> > I could be way off base. Let me know.
> I'm not sure I get what you mean here with read/write DOM being desirable.
If there is a desire for a W3 DOM implementation that is read
and write, it is feasable. It does not strike me as a good use
of Momento. There is no interface to support transactions.
The W3 DOM event model does make sense, however.
> > 2) No numbers. I've not designed any benchmarks. Momento is to the
> > point where I need an example application to focus my energy to
> > get Momento to beta. I was going to use Linotype, actually, and
> > use Momento to store my blog.
> Uh, awesome!
A good initial application of Momento, I think. Brian McCallister
has implemented an Atom feed. I say we put the three together.
> > Towards scalablity Momento supports concurrent reads, and, with
> > some educated decisions by the application developer, concurrent
> > updates.
> uh, this sentence is worrysome. Can you elaborate more?
What worries you? Momento asks the application developer to choose
her clusters just as PostgreSQL asks her to chose her indicies.
> > 1) Momento has three concepts that rise to the surface when I
> > consider it.
> >
> > * Zero, I always forget that I spent a month writing a
> > journaling file data structure. It just hums along quietly
> > in the background now. It splits a random access file into
> > pages. Reads those pages in and out of memory. It uses
> > weak references to implement a page cache. It's pretty
> > cool, but its pretty much done, so...
>
> uh, sounds useful. We are having problems with JISP. Care to tell us
> more about this?
It is a journaled paged file structure. I built a record storage
interface and a string storage interface using it. I built this
out far enough to support Momento. It can do more.
The key here is journalling. You can state that writes happen as
an atomic transaction or not at all.
No indicies though. Once Momento has an indicies, someone could
use it as a basis for to replace JISP. Is JSIP ACID?
> > * First, Momento maintains a version axis.
> >
> > The version-axis allows for however many concurrent
> > queries, they will not have to wait for updates.
>
> Interesting approach. I don't see how you can do rollback if you garbage
> collect a node, though.
Rollback means to rollback the current transcation. The state of the
document prior to the current transaction is maintained on disk.
I don't think I said "garbage" collect. Collection is in
reference to collecting previous versions of the document on
disk.
> > It makes sense to to cluster the issue nodes themselves
> > together since the most likely axis of traversal is the the
> > next-sibling axis, used to find a specific issue by name.
> >
> > Clusters are akin to files on the file system, really.
>
> What is the strategy you use to clusterize nodes?
That is for the application developer to decide. If I were
implementing said bug tracking application, I would choose to
cluster the issue elements themselves together, and I'd put
the children on their own cluter.
> > I do hope to foster discussion of this project. I'll check in at
> > the airport. More questions, please.
> >
> > There is a mailing list too. Please join to participate or observe.
> >
> > momento-subscribe@engrm.com
> I won't subscribe to a new mail list until I see there is a reason for
> it, and for sure not before there is any code to take a look at.
Tried to put time in on a Linux bug that keeps me from posting the
distribution (I develop on OS X, so Linux/W2K issues sneek up on
me.) It is, however, a challenge to work with two laptops
on an airplane.
I will have that zip distribution at end of day PST.
I hope that once you see the code, you will help me to build a
quorum on the Momento list.
> >Thanks again for asking.
> You are welcome.
I am very interested in Cocoon community adoption. Please don't
hesitate to ask questions.
--
Alan / alan@engrm.com / http://engrm.com/
aim/yim: alanengrm - icq: 228631855 - msn: alanengrm@hotmail.com
Re: Announce: Momento
Posted by Stefano Mazzocchi <st...@apache.org>.
Alan wrote:
> * Stefano Mazzocchi <st...@apache.org> [2004-02-17 04:22]:
>
>>Alan wrote:
>>
>>
>>>Momento is a native XML persistence engine. Is supports XSLT 2.0 and
>>> XQuery 1.0 (via Saxon), it supports XUpdate.
>>>
>>> It is transactional and ACID.
>>>
>>> It is designed with Cocoon in mind.
>>>
>>> I am considering a open source path for future development.
>>>
>>> http://engrm.com/project/com.agtrz.momento/
>>>
>>>Thoughts?
>>
>>Very interesting. Can you tell us more on how it works? have any numbers
>>on how fast/scalable it gets? what's the difference between Momento and
>>a native xml database like XIndice?
>
>
> Stefano
>
> Thanks for asking.
>
> 3) I can't compare Momento to Xindice at the moment. Last I looked
> at Xindice was last November. I'm announcing to get such insight.
>
> (I am willing to say that Momento isn't wedded to a specific
> API, such as XML::DB. It works with Saxon to provide
> XQuery and XSLT.
What is the interfacing API then? JAXP? or Saxon's?
I'm implementing XUpdate currently. A
> read-only W3 DOM is a simple matter, if there is call
> for it. A read-write W3 DOM is not so simple, or
> desireable, but entirely feasable.
>
> I could be way off base. Let me know.
I'm not sure I get what you mean here with read/write DOM being desirable.
> )
>
> 2) No numbers. I've not designed any benchmarks. Momento is to the
> point where I need an example application to focus my energy to
> get Momento to beta. I was going to use Linotype, actually, and
> use Momento to store my blog.
Uh, awesome!
>
>
> (Linotype is a good application since I would want to backup
> Momento after an update for the time being. That extra
> step is acceptable for a blog, provided it is a single
> user blog.)
>
>
> Towards scalablity Momento supports concurrent reads, and, with
> some educated decisions by the application developer, concurrent
> updates.
uh, this sentence is worrysome. Can you elaborate more?
> Momento works as a data store for a multi-threaded
> server. It would work nicely as a servlet, or in a Cocoon pipeline.
>
> For further scaleability, I plan on supporting multi-process
> operation, and indicies. (That's right, no indicies yet. They
> are not at the core of Momento.)
ok
>
> 1) Momento has three concepts that rise to the surface when I
> consider it.
>
> * Zero, I always forget that I spent a month writing a
> journaling file data structure. It just hums along quietly
> in the background now. It splits a random access file into
> pages. Reads those pages in and out of memory. It uses
> weak references to implement a page cache. It's pretty
> cool, but its pretty much done, so...
uh, sounds useful. We are having problems with JISP. Care to tell us
more about this?
> * First, Momento maintains a version axis.
>
> Rather than updating a node, Momento links a new version
> of that node to its version axis. An XSLT transform
> navigates a Momento document with a version number in
> hand. When you get the first child, say, you check to see
> if there is a new version of that child, and iterate, but
> never past the maximum version.
>
> The older versions are kept around until any queries
> referencing them terminate. At that point, the older
> versions of the nodes can be collected.
>
> Thus a newer version of the document can be assembled
> while the existing version is queried. That newer version
> can even be discarded and it will be ignored (iterate
> beyond to the next good version, or stick with the last
> good version). Volia: commit and rollback.
>
> The version-axis allows for however many concurrent
> queries, they will not have to wait for updates.
Interesting approach. I don't see how you can do rollback if you garbage
collect a node, though.
>
> * Second, Momento organizes its nodes in clusters.
>
> An application developer tunes their application for
> performance by specifying which nodes will to be clustered
> on the same pages. This ought to be a obvious decision for
> the most part. Consdier a bug database.
>
> <bug-document xmlns="http://engrm.com/bugs">
> <project name="Momento">
> <issue name="Won't do this">
> <comment />
> <comment />
> .
> .
> </issue>
> <issue name="Doesn't do that">
> <comment />
> <comment />
> .
> .
> </issue>
> .
> .
> .
> .
> </project>
> </project>
>
> In the above document, it is likely that most of the data
> manipulation will occur within an issue. If the nodes are
> clustered by issue, then Momento can place an read lock
> at the issue node, allowing other issue nodes to be updated
> concurrently.
>
> There will be aggreagte queries, but most queries
> (transforms) will manipulate an issue. It makes sense then
> to cluster the nodes in an issue on the same pages.
>
> It makes sense to to cluster the issue nodes themselves
> together since the most likely axis of traversal is the the
> next-sibling axis, used to find a specific issue by name.
>
> Clusters are akin to files on the file system, really.
What is the strategy you use to clusterize nodes?
>
> * Third, Momento maintains performance through node proximity
> by reorganizing clusters, and performance of updates by
> roorganinzing tclusters as a spearate step apart from
> updating them.
>
> This is tricky to explain, not that its a complicated
> concept, I just haven't tried to document it yet, bare
> with..
>
> In its organized state, a cluster contains it's nodes in
> document order. This puts first children on the same page
> a parent and a next sibling on the same page a previous
> sibling, that is, not too far away.
>
> When a cluster mutates, new nodes are allocated from a
> scrap page of nodes and linked to the version axis. Now the
> document order is wonky. The newer version will cause
> queries to iterate into the scrap pages, proximity starts
> to suffer.
>
> Therefore, as a second step to updating the document,
> Momento must organize itself by copying the last commited
> version of a cluster to a new set of pages, retaining only
> the latest version of each node.
>
> This can occur in a separate thread, or more likely at the
> prompting of the application developer. If a user is
> pecking away at an interactive form, it may make sense for
> the user to press Okay before going to the trouble to
> organize the cluster.
>
> In most XML applications, there are going to be natural
> candidates for a Momento cluster. Often this will map
> directly to a file on the file system, in an existing
> application.
>
>
> My communication skills are getting streched by all the
> announcing. Please let me know if this is a good explaintion.
> I can use it to create a better overview document.
What you outlines sounds like an interesting approach but it's kinda
foggy. A better outline document would be very useful (to me, at least)
to understand if your approach could be used as a persistent native XML
database that could scale.
> I'll try not to be so prolix in the follow up. They were broad
> questions. With these concepts, everything else in Momento is
> obvious. It is pretty simple.
>
> I was planning on announcing all week, but I'm traveling instead.
Announcing all week? one announcement is enough :-) the rest can be a
regular email exchange don't you think?
I
> do hope to foster discussion of this project. I'll check in at
> the airport. More questions, please.
>
> There is a mailing list too. Please join to participate or observe.
>
> momento-subscribe@engrm.com
I won't subscribe to a new mail list until I see there is a reason for
it, and for sure not before there is any code to take a look at.
For the licensing issue, keep in mind that we wouldn't be able to
distribute your software (due to ASF policies) if you choose a license
of the GPL family.
>
> Thanks again for asking.
You are welcome.
--
Stefano.
Re: Announce: Momento
Posted by Vadim Gritsenko <va...@reverycodes.com>.
Alan wrote:
> I'd like to see a Momento Cocoon block. I'd like to have Saxon
> for that. Otherwise, I'd have to look into making Momento
> available via Xerces.
>
> Michael Kay, author of Saxon, is participating on the Momento
> mailing list. It would be a good place to discuss licensing
> issues and hopefully resolve them.
>
> momento-subscribe@engrm.com
>
> Otherwise, I am going to have to shuttle between lists on this
> matter especially.
>
>
Here are some more links:
http://marc.theaimsgroup.com/?l=xindice-dev&m=107664320809629&w=2
http://marc.theaimsgroup.com/?t=101055403400041&r=1&w=2
Vadim
Re: Announce: Momento
Posted by Alan <al...@engrm.com>.
* Vadim Gritsenko <va...@reverycodes.com> [2004-02-17 11:55]:
> Alan wrote:
>
> >* Bertrand Delacretaz <bd...@apache.org> [2004-02-17 06:55]:
> >
> >
> >>Le Mardi, 17 f?v 2004, ? 07:42 Europe/Zurich, Alan a ?crit :
> >>
> >>
> >>>... My communication skills are getting streched by all the
> >>> announcing. Please let me know if this is a good explaintion.
> >>> I can use it to create a better overview document....
> >>>
> >>>
> >>hmmm..this sounds very interesting but for me the next step would be to
> >>have code that I can read and play with.
> >>
> >>Can we download Momento somewhere? I didn't find download links on your
> >>web site.
> >>
> >>
> >
> >Bertrand
> >
> > I don't have a license yet. I am asking people about licensing.
> > Apache, Mozilla, GPL, LGPL.
> >
> AFAIU, there is a licensing issue with Saxon. It uses MPL 1.0, which is
> not acceptable to ASF, but MPL 1.1 is ok (any updates on this issue? was
> this decision "final"?)
Curious.
I'd like to see a Momento Cocoon block. I'd like to have Saxon
for that. Otherwise, I'd have to look into making Momento
available via Xerces.
Michael Kay, author of Saxon, is participating on the Momento
mailing list. It would be a good place to discuss licensing
issues and hopefully resolve them.
momento-subscribe@engrm.com
Otherwise, I am going to have to shuttle between lists on this
matter especially.
> As for your code, take a look at ALv2, Apache License version 2.
Let me do so. There may be a steep learning curve, for me. I would
have hoped that BSD style licenses could play together. I
thought I'd just do as Saxon does, but if it means Momento runs
afoul of Apache XML, that won't do.
Thank you for the link Steven.
--
Alan / alan@engrm.com / http://engrm.com/
aim/yim: alanengrm - icq: 228631855 - msn: alanengrm@hotmail.com
Re: Announce: Momento
Posted by Steven Noels <st...@outerthought.org>.
On 17 Feb 2004, at 12:55, Vadim Gritsenko wrote:
> As for your code, take a look at ALv2, Apache License version 2.
http://www.apache.org/licenses/LICENSE-2.0.html
</Steven>
--
Steven Noels http://outerthought.org/
Outerthought - Open Source Java & XML An Orixo Member
Read my weblog at http://blogs.cocoondev.org/stevenn/
stevenn at outerthought.org stevenn at apache.org
Re: Announce: Momento
Posted by Vadim Gritsenko <va...@reverycodes.com>.
Alan wrote:
>* Bertrand Delacretaz <bd...@apache.org> [2004-02-17 06:55]:
>
>
>>Le Mardi, 17 f?v 2004, ? 07:42 Europe/Zurich, Alan a ?crit :
>>
>>
>>>... My communication skills are getting streched by all the
>>> announcing. Please let me know if this is a good explaintion.
>>> I can use it to create a better overview document....
>>>
>>>
>>hmmm..this sounds very interesting but for me the next step would be to
>>have code that I can read and play with.
>>
>>Can we download Momento somewhere? I didn't find download links on your
>>web site.
>>
>>
>
>Bertrand
>
> I don't have a license yet. I am asking people about licensing.
> Apache, Mozilla, GPL, LGPL.
>
AFAIU, there is a licensing issue with Saxon. It uses MPL 1.0, which is
not acceptable to ASF, but MPL 1.1 is ok (any updates on this issue? was
this decision "final"?)
As for your code, take a look at ALv2, Apache License version 2.
Vadim
Re: Announce: Momento
Posted by Alan <al...@engrm.com>.
* Bertrand Delacretaz <bd...@apache.org> [2004-02-17 06:55]:
> Le Mardi, 17 f?v 2004, ? 07:42 Europe/Zurich, Alan a ?crit :
> >... My communication skills are getting streched by all the
> > announcing. Please let me know if this is a good explaintion.
> > I can use it to create a better overview document....
>
> hmmm..this sounds very interesting but for me the next step would be to
> have code that I can read and play with.
>
> Can we download Momento somewhere? I didn't find download links on your
> web site.
Bertrand
I don't have a license yet. I am asking people about licensing.
Apache, Mozilla, GPL, LGPL. Many much. If no one was interested
at all, I wasn't going to bother with the decision.
I will have to assemble a zip distribution. It builds no
problem on OS X, Linux, and W2K, but requires Ant 1.6. I'm
developing in Eclipse. Let me make a tarball of Momento,
XFixture, and SAX Strategy.
Did you see the cross-referenced source?
http://engrm.com/project/com.agtrz.momento/
To see how one executes XUpdate and XSLT:
http://engrm.com/project/com.agtrz.momento/var/xref/xfixture/com/agtrz/momento/XFixture.html
Nothing beats step execution, I know. I'm on it.
--
Alan / alan@engrm.com / http://engrm.com/
aim/yim: alanengrm - icq: 228631855 - msn: alanengrm@hotmail.com
Re: Announce: Momento
Posted by Bertrand Delacretaz <bd...@apache.org>.
Le Mardi, 17 fév 2004, à 07:42 Europe/Zurich, Alan a écrit :
> ... My communication skills are getting streched by all the
> announcing. Please let me know if this is a good explaintion.
> I can use it to create a better overview document....
hmmm..this sounds very interesting but for me the next step would be to
have code that I can read and play with.
Can we download Momento somewhere? I didn't find download links on your
web site.
-Bertrand
Re: Announce: Momento
Posted by Alan <al...@engrm.com>.
* Stefano Mazzocchi <st...@apache.org> [2004-02-17 04:22]:
> Alan wrote:
>
> >Momento is a native XML persistence engine. Is supports XSLT 2.0 and
> > XQuery 1.0 (via Saxon), it supports XUpdate.
> >
> > It is transactional and ACID.
> >
> > It is designed with Cocoon in mind.
> >
> > I am considering a open source path for future development.
> >
> > http://engrm.com/project/com.agtrz.momento/
> >
> >Thoughts?
>
> Very interesting. Can you tell us more on how it works? have any numbers
> on how fast/scalable it gets? what's the difference between Momento and
> a native xml database like XIndice?
Stefano
Thanks for asking.
3) I can't compare Momento to Xindice at the moment. Last I looked
at Xindice was last November. I'm announcing to get such insight.
(I am willing to say that Momento isn't wedded to a specific
API, such as XML::DB. It works with Saxon to provide
XQuery and XSLT. I'm implementing XUpdate currently. A
read-only W3 DOM is a simple matter, if there is call
for it. A read-write W3 DOM is not so simple, or
desireable, but entirely feasable.
I could be way off base. Let me know.
)
2) No numbers. I've not designed any benchmarks. Momento is to the
point where I need an example application to focus my energy to
get Momento to beta. I was going to use Linotype, actually, and
use Momento to store my blog.
(Linotype is a good application since I would want to backup
Momento after an update for the time being. That extra
step is acceptable for a blog, provided it is a single
user blog.)
Towards scalablity Momento supports concurrent reads, and, with
some educated decisions by the application developer, concurrent
updates. Momento works as a data store for a multi-threaded
server. It would work nicely as a servlet, or in a Cocoon pipeline.
For further scaleability, I plan on supporting multi-process
operation, and indicies. (That's right, no indicies yet. They
are not at the core of Momento.)
1) Momento has three concepts that rise to the surface when I
consider it.
* Zero, I always forget that I spent a month writing a
journaling file data structure. It just hums along quietly
in the background now. It splits a random access file into
pages. Reads those pages in and out of memory. It uses
weak references to implement a page cache. It's pretty
cool, but its pretty much done, so...
* First, Momento maintains a version axis.
Rather than updating a node, Momento links a new version
of that node to its version axis. An XSLT transform
navigates a Momento document with a version number in
hand. When you get the first child, say, you check to see
if there is a new version of that child, and iterate, but
never past the maximum version.
The older versions are kept around until any queries
referencing them terminate. At that point, the older
versions of the nodes can be collected.
Thus a newer version of the document can be assembled
while the existing version is queried. That newer version
can even be discarded and it will be ignored (iterate
beyond to the next good version, or stick with the last
good version). Volia: commit and rollback.
The version-axis allows for however many concurrent
queries, they will not have to wait for updates.
* Second, Momento organizes its nodes in clusters.
An application developer tunes their application for
performance by specifying which nodes will to be clustered
on the same pages. This ought to be a obvious decision for
the most part. Consdier a bug database.
<bug-document xmlns="http://engrm.com/bugs">
<project name="Momento">
<issue name="Won't do this">
<comment />
<comment />
.
.
</issue>
<issue name="Doesn't do that">
<comment />
<comment />
.
.
</issue>
.
.
.
.
</project>
</project>
In the above document, it is likely that most of the data
manipulation will occur within an issue. If the nodes are
clustered by issue, then Momento can place an read lock
at the issue node, allowing other issue nodes to be updated
concurrently.
There will be aggreagte queries, but most queries
(transforms) will manipulate an issue. It makes sense then
to cluster the nodes in an issue on the same pages.
It makes sense to to cluster the issue nodes themselves
together since the most likely axis of traversal is the the
next-sibling axis, used to find a specific issue by name.
Clusters are akin to files on the file system, really.
* Third, Momento maintains performance through node proximity
by reorganizing clusters, and performance of updates by
roorganinzing tclusters as a spearate step apart from
updating them.
This is tricky to explain, not that its a complicated
concept, I just haven't tried to document it yet, bare
with..
In its organized state, a cluster contains it's nodes in
document order. This puts first children on the same page
a parent and a next sibling on the same page a previous
sibling, that is, not too far away.
When a cluster mutates, new nodes are allocated from a
scrap page of nodes and linked to the version axis. Now the
document order is wonky. The newer version will cause
queries to iterate into the scrap pages, proximity starts
to suffer.
Therefore, as a second step to updating the document,
Momento must organize itself by copying the last commited
version of a cluster to a new set of pages, retaining only
the latest version of each node.
This can occur in a separate thread, or more likely at the
prompting of the application developer. If a user is
pecking away at an interactive form, it may make sense for
the user to press Okay before going to the trouble to
organize the cluster.
In most XML applications, there are going to be natural
candidates for a Momento cluster. Often this will map
directly to a file on the file system, in an existing
application.
My communication skills are getting streched by all the
announcing. Please let me know if this is a good explaintion.
I can use it to create a better overview document.
I'll try not to be so prolix in the follow up. They were broad
questions. With these concepts, everything else in Momento is
obvious. It is pretty simple.
I was planning on announcing all week, but I'm traveling instead. I
do hope to foster discussion of this project. I'll check in at
the airport. More questions, please.
There is a mailing list too. Please join to participate or observe.
momento-subscribe@engrm.com
Thanks again for asking.
--
Alan / alan@engrm.com / http://engrm.com/
aim/yim: alanengrm - icq: 228631855 - msn: alanengrm@hotmail.com
Re: Announce: Momento
Posted by Stefano Mazzocchi <st...@apache.org>.
Alan wrote:
> Momento is a native XML persistence engine. Is supports XSLT 2.0 and
> XQuery 1.0 (via Saxon), it supports XUpdate.
>
> It is transactional and ACID.
>
> It is designed with Cocoon in mind.
>
> I am considering a open source path for future development.
>
> http://engrm.com/project/com.agtrz.momento/
>
> Thoughts?
Very interesting. Can you tell us more on how it works? have any numbers
on how fast/scalable it gets? what's the difference between Momento and
a native xml database like XIndice?
TIA
--
Stefano.