Re[2]: Metadata (was: Planning the future)

[Dawid Weiss <Da...@cs.put.poznan.pl>]

KS> What if it was something that was stupid simple like automatically
KS> creating two collections, one for documents and one for metadata? Each

We have adopted such a 'stupid simple' approach. It works well. We'd rather
think of the approach as simple than stupid of course... Any term used, the
separation  of metadata from the contents based on a prefixed subcollection
(prefix   reserved)   has  several  advantages  over  storing  them  inside
documents.

KS> The metadata doc
KS> would have database managed portions and user managed portions,

Amazing  coincidence  in  thinking. It only proves the design is simple and
intuitive...

KS> The meta-data collection name would just have a reserved prefix/suffix and
KS> could be ignored in collection listings.

Exactly :)

KS> Maybe It could be an option so that if you don't want the overhead you

Exactly :)

KS> access the data like any other collection. It wouldn't even require any
KS> changes to the XML:DB APi if you layered on top.

Exactly. Amazing, I could have been an author of your e-mail...

KS> It's kind of hacky, but it's simple and in reality it might work decently 
KS> well.

It does. We have such an API.

KS> I don't know, just a thought.

A  very  good  one.  Thanks,  it  to  some extent proves we chose the right
approach to the problem we faced (i.e. lack of versioning/ metadata).

Dawid

Re: Metadata (was: Planning the future)

[Kimbro Staken <ks...@xmldatabases.org>]

On Monday, February 18, 2002, at 07:16 AM, Gianugo Rabellino wrote:

> Kimbro Staken wrote:
>> On Monday, February 18, 2002, at 06:05 AM, Dawid Weiss wrote:
>>>
>>> KS> What if it was something that was stupid simple like automatically
>>> KS> creating two collections, one for documents and one for metadata?
>
> OK, this is the cleanest hack I can think of, but still it's a hack. :)
>

Never claimed otherwise, "It's kind of hacky". :-)

> It seems to me that we are playing different levels, let's try to make 
> some points clear: you're talking about practical solutions to achieve 
> the result of having metadata, I'm stuck to a generic metadata support, i.
> e. if should metadata be offered to Joe User.  This is *vital*, and we 
> all agree on this, at least for basic "stat-like" metadata. Now:
>

You're right, this is a completely different issue. The problem with the 
XML:DB API is that it's going to be colored by what other databases 
support. I don't know exactly what they support though, so I have to do 
research in order to figure it out. From what I remember most don't have 
very rich support, if any at all.

> * Supporting this means IMHO extending the XML:DB APIs somehow (I'm not 
> in favor of proprietary extensions and the like).
>

You're probably right, I just don't want to tackle that yet. The XML:DB 
API is probably going to change quite a bit before too much longer. I want 
to focus on Xindice right now. Let's figure out what we want to support 
and how we want to expose it through our network API. Then we can look at 
seeing how the XML:DB API shapes up. I'd also prefer to have a little more 
real world experience with this before tackling the real API changes. My 
opinion is that we just add a proprietary service until then

In addition, I'm kind of driving both projects, among others, and right 
now I want to make sure we get Xindice moving nicely first. We've been 
kind of stalled as far as development goes. Once we're rolling along 
nicely, then I'll focus a little more on the XML:DB API project which is 
also kind of stalled. I also need to factor in SixDML which is becoming an 
XML:DB project too.

The major problem with XML:DB is that the XML database industry overall is 
really struggling. We need to get SiXDML going as there is a lot more 
vendor interest in that then the raw API. The good thing is it builds on 
the raw API so it will drive that too. It's just going to take some time 
and I don't want it to affect this project too much. It's more important 
that we get solid support in Xindice first. After all if the XML database 
industry doesn't pick up soon, Xindice may be the only one left and then 
we won't have to worry about anything else. :-(

> * If we are to undertake an effort to change the XML:DB APIs IMHO it's 
> worth to have some investigation and understand what kind of metadata we 
> are to offer. I'm all for having a set of "system level" metadata plus a 
> hook for user ones, but I might be stuck by FS at its best, so please don'
> t hesitate to try my asbestos underwear on this issue. :)
>
> * Given this, it doesn't really matter, at this point of the discussion, 
> whether metadata should be stored inside the DOM, in a parallel document,
>  in a parallel Collection or in my closet: this is just an implementation 
> detail.
>
> I hope I made myself clear now :) Yet the more I think about it the more 
> I'm convinced that this discussion really belongs to xapi-dev...
>

Yes it does.

> Ciao,
>
> -- Gianugo Rabellino
>
>
Kimbro Staken
XML Database Software, Consulting and Writing
http://www.xmldatabases.org/

Re: Metadata (was: Planning the future)

[Gianugo Rabellino <gi...@apache.org>]

Kimbro Staken wrote:
> 
> On Monday, February 18, 2002, at 06:05 AM, Dawid Weiss wrote:
> 
>>
>> KS> What if it was something that was stupid simple like automatically
>> KS> creating two collections, one for documents and one for metadata? 

OK, this is the cleanest hack I can think of, but still it's a hack. :)

It seems to me that we are playing different levels, let's try to make 
some points clear: you're talking about practical solutions to achieve 
the result of having metadata, I'm stuck to a generic metadata support, 
i.e. if should metadata be offered to Joe User.  This is *vital*, and we 
all agree on this, at least for basic "stat-like" metadata. Now:

* Supporting this means IMHO extending the XML:DB APIs somehow (I'm not 
in favor of proprietary extensions and the like).

* If we are to undertake an effort to change the XML:DB APIs IMHO it's 
worth to have some investigation and understand what kind of metadata we 
are to offer. I'm all for having a set of "system level" metadata plus a 
hook for user ones, but I might be stuck by FS at its best, so please 
don't hesitate to try my asbestos underwear on this issue. :)

* Given this, it doesn't really matter, at this point of the discussion, 
whether metadata should be stored inside the DOM, in a parallel 
document, in a parallel Collection or in my closet: this is just an 
implementation detail.

I hope I made myself clear now :) Yet the more I think about it the more 
I'm convinced that this discussion really belongs to xapi-dev...

Ciao,

-- 
Gianugo Rabellino

Re[6]: Metadata (was: Planning the future)

[Dawid Weiss <Da...@cs.put.poznan.pl>]

KS> Are you still planning to contribute this code? It sounds like it would be
KS> a really good starting place.

I  think it would be no problem. We have JUST finished the prototype though
and  JUnit  shows it works ok. My good guess would be to hold on a month or
two  -  we'll  gain  experience  on  the  interface, its pros and cons, and
contribute  a  more  reliable, stable version. Another side of this coin is
that   we   have  it  implemented  in a very structural-like approach, i.e.
there  has  been no class-responsibility analysis etc. The reason for it is
that we intentionally wanted the interface to be so simple.

We  may change it in the future - time will tell.

KS> That's what we're trying to figure out. I don't actually use Xindice for
KS> anything, so I rely on the real users to tell us what we need.

I'm glad to hear it.
Dawid

Re: Re[4]: Metadata (was: Planning the future)

[Kimbro Staken <ks...@xmldatabases.org>]

On Monday, February 18, 2002, at 06:21 AM, Dawid Weiss wrote:

>
> KS> Yeah, sorry I just realized Krzysztof posted a description of that 
> very
> KS> thing right before I sent that mail. Heh, heh, see what I get for 
> being
> KS> too busy writing and not reading. So Kudos to you guys for not only 
> coming
> KS> up with it, but also implementing it.
>
> That's  all right, it was just funny to see you came out with the same 
> idea
> :)  We  actually spent some time investigating various concepts and the 
> one
> we  chose  seemed  to  have least drawbacks: the database remains "clean"
>  -
> xpath,  document  retrieval  etc - these are all transparent, which was 
> out
> goal.
>

Are you still planning to contribute this code? It sounds like it would be 
a really good starting place.

> Nonetheless,  I  would  love  to  see  a  standard  API  for versioning 
> and
> metadata, and it seems many other people seem to crave for it, not only 
> me.
> It  may  be quite easy to walk around the problem, but since the project 
> is
> open  source, it should perhaps be taken into account what the actual 
> users
> need,  not  only  what  is the 'ideal path', which Xindice should follow 
> ;)
> Just my remark, nothing personal.
>

That's what we're trying to figure out. I don't actually use Xindice for 
anything, so I rely on the real users to tell us what we need. Now, I'm 
definitely hearing that metadata is an important issue, even more then I 
already thought. This is what I need to hear, and I'm listening.


> Dawid
>
>
Kimbro Staken
XML Database Software, Consulting and Writing
http://www.xmldatabases.org/

Re[4]: Metadata (was: Planning the future)

[Dawid Weiss <Da...@cs.put.poznan.pl>]

KS> Yeah, sorry I just realized Krzysztof posted a description of that very 
KS> thing right before I sent that mail. Heh, heh, see what I get for being 
KS> too busy writing and not reading. So Kudos to you guys for not only coming 
KS> up with it, but also implementing it.

That's  all right, it was just funny to see you came out with the same idea
:)  We  actually spent some time investigating various concepts and the one
we  chose  seemed  to  have least drawbacks: the database remains "clean" -
xpath,  document  retrieval  etc - these are all transparent, which was out
goal.

Nonetheless,  I  would  love  to  see  a  standard  API  for versioning and
metadata, and it seems many other people seem to crave for it, not only me.
It  may  be quite easy to walk around the problem, but since the project is
open  source, it should perhaps be taken into account what the actual users
need,  not  only  what  is the 'ideal path', which Xindice should follow ;)
Just my remark, nothing personal.

Dawid

Re: Re[2]: Metadata (was: Planning the future)

[Kimbro Staken <ks...@xmldatabases.org>]

On Monday, February 18, 2002, at 06:05 AM, Dawid Weiss wrote:

>
> KS> What if it was something that was stupid simple like automatically
> KS> creating two collections, one for documents and one for metadata? Each
>
> We have adopted such a 'stupid simple' approach. It works well. We'd 
> rather
> think of the approach as simple than stupid of course... Any term used, 
> the
> separation  of metadata from the contents based on a prefixed 
> subcollection
> (prefix   reserved)   has  several  advantages  over  storing  them  
> inside
> documents.
>

Yeah, sorry I just realized Krzysztof posted a description of that very 
thing right before I sent that mail. Heh, heh, see what I get for being 
too busy writing and not reading. So Kudos to you guys for not only coming 
up with it, but also implementing it.

> KS> The metadata doc
> KS> would have database managed portions and user managed portions,
>
> Amazing  coincidence  in  thinking. It only proves the design is simple 
> and
> intuitive...
>
> KS> The meta-data collection name would just have a reserved 
> prefix/suffix and
> KS> could be ignored in collection listings.
>
> Exactly :)
>
> KS> Maybe It could be an option so that if you don't want the overhead you
>
> Exactly :)
>
> KS> access the data like any other collection. It wouldn't even require 
> any
> KS> changes to the XML:DB APi if you layered on top.
>
> Exactly. Amazing, I could have been an author of your e-mail...
>
> KS> It's kind of hacky, but it's simple and in reality it might work 
> decently
> KS> well.
>
> It does. We have such an API.
>
> KS> I don't know, just a thought.
>
> A  very  good  one.  Thanks,  it  to  some extent proves we chose the 
> right
> approach to the problem we faced (i.e. lack of versioning/ metadata).

>
> Dawid
>
>
Kimbro Staken
XML Database Software, Consulting and Writing
http://www.xmldatabases.org/