You are viewing a plain text version of this content. The canonical link for it is here.
Posted to xindice-users@xml.apache.org by Gudmundur Arni Thorisson <mu...@cshl.org> on 2002/12/20 16:12:18 UTC
Xindice scalability: using in a large bio
Hey, folks. This is my first message to this mailing list. I work in a
non-profit biological research lab (Cold Spring Harbor Laboratory
http://www.cshl.org). Our lab recently became a part of a rather large,
international collaboration, the Haplotype Map project, to product
enormous amounts of biological data (genotypes). Our role will be to
synchronize data handling and build the database to hold the stuff (see
http://www.genome.gov/page.cfm?pageID=10001688 if you're interested in
this thing), plus related tasks.
We'd originally planned to use a fairly XML-centric approach from the
ground up, for a multitude of reasons. One very strong reason was that we
could use the allegedly powerful XML-capabilities of Oracle 9i XMLDB to
produce a XML-relational schema from our own data handling/exchange XML
Schema definitions.
To cut this story short, we got our funding cut down quite a bit and
will now not be able to afford the big-buch Oracle licenses we'd need
(3x$15.000). Open source is now pretty much the only option for us
database-wise, which is in fact a blessing in disguise because it will
make the endevour entirely open source (Oracle would have been the only
proprietary component otherwise).
As of now, we are investigating open source alternatives in either the
XML-on-top-of-RDMS arena or the native XML database arena. There are some
commercial offerings (Tamino, Ipedo), but as I said above, the preference
is always open source. Our lab is very fond of the Apache/mod_perl world
(I work for Dr. Lincoln Stein, a longtime guru in the Perl world) and
we'll likely use some of the Apache XML project components (Xalan, Xerces)
for XML-processing in this project.
As a part of this investigation, Xindice appeared on the horizon. After
looking at some simple commandline examples etc. of how one goes about
handling XML-documents and collections in Xindice, it looks we just might
be able to use the thing. There is only one thing that we have concerns
about: scalability. It says on the Xindice website that the db is designed
for many, small documents. The XML dataset that we will be handling will
contain fairly small documents but VERY many of them; up to 400 million
instances of the most populous record class.
My question is therefore this: has anyone used/tested Xindice with
datasets of this size (hundreds of millions) with decent performance as
well? This will be mainly import + query work, hardly any heavy updating
load, if that would make a difference as far as performance goes.
Thanks in advance of your reply. Regards,
Mummi, Cold Spring Harbor Laboratory
PS I have attached key XML Schema draft component files for the portion
of our total schema that has been nailed down mostly so far, plus one
(hapmap.xsd) that ties them all together. There will be maybe 2 or 3 times
this many types of objects in the total schema. The file genotype.xsd
defines the <400M record class.
<File attached: batch_submission.xsd><File attached: genotype.xsd><File attached: hapmap.xsd><File attached: snp.xsd><File attached: element_groups.xsd><File attached: simple_types.xsd>
--
----------------------------------------------------
Gudmundur Arni Thorisson, bioinformatics researcher, B.Sc.
Steinlab, Cold Spring Harbor Laboratory
w-phone#: 516-367-6904
w-fax#: 516-367-8389
1 Bungtown Road
Cold Spring Harbor
11724 New York
USA
Re: Xindice scalability: using in a large bio
Posted by Jeff Greif <jg...@alumni.princeton.edu>.
Disclaimer: I'm just an Xindice user, not a developer.
Perhaps it would be helpful if you looked in more detail at the reasons why
Xindice is better for large numbers of relatively small documents.
Essentially, the issue is in indexing for queries. Indexes (at least when I
last looked at the source code) were on collections (not on the documents),
and mapped from values of elements or attributes to the subset of docs which
contained those elements or attributes. If you issue a queries for certain
subtrees of the documents satisfying certain conditions, first the
appropriate subset is found using whatever indexes you provide, and then
XPath is used to extract the subtrees from the selected documents. The
index, being some kind of Btree or hash, is reasonably efficient for large
collections. XPath is reasonably efficient for small documents, unless it
contains expressions that cause scans of the entire document, e.g. //x (all
the elements named x, at whatever position in the document). A query which
retrieves an entire document by id will be quite fast, regardless of how
large the collection is (probably logarithmic in collection size at worst).
I don't believe there is any inherent limit in the size of a collection
except that the internal compressed form of the collection and its indexes
must fit in a single file (so you'll need a big disk and appropriate file
system), at least in the current implementation (I think).
All that being said, it is likely that 400 million non-trivial docs is
larger than has been used in Xindice before. If you were to attempt to use
it for this project, you would likely hit some problems and drive
improvements in the software. If I were you thinking of embarking on such
an experiment, I'd want to have a good picture of the pace of development of
Xindice, the responsiveness of the developers to reported problems, since
you're likely to hit some showstoppers, and the plan for future development
(to see if its aims overlap largely with yours, or whether there is a
different focus).
Jeff
Re: Xindice scalability: using in a large bio
Posted by Gudmundur Arni Thorisson <mu...@cshl.org>.
Thanks for the info, Devrim. Seems like Tamino is unfortunately out of our
leage $$$-wise, same as with Oracle.
Mummi
On Monday, December 23, 2002, at 01:17 PM, Devrim Ergel wrote:
> Hello,
>
> Software AG Tamino XML Database might be the right answer. They are
> offering
> as a robust native xml database for mission critical applications. The
> only
> problem its price was 45000$ last year!
> (In our case, we are Software AG Partner Software Company in Turkey so we
> can bundle with reasonable prices)
>
> Devrim
> Parsera IT
>
> ----- Original Message -----
> From: "Gudmundur Arni Thorisson" <mu...@cshl.org>
> To: <xi...@xml.apache.org>
> Sent: Friday, December 20, 2002 11:07 PM
> Subject: Re: Xindice scalability: using in a large bio
>
>
>> Ximbro, I have not estimated the total storage requirements for the
>> project yet, as we have not yet finalized more than a part of the schema,
>> including the 400M instance class. But it is certain that the size of
>> each record in that class will be quite small, in an XML-sense, something
>> like this:
>>
>> <genotype
> lsid="urn:LSID:genome.wi.mit.edu:HapMap/Genotype:883423434:1"
>>>
>> <snp_assay
>> lsid="urn:LSID:genome.wi.mit.edu:HapMap/SNPAssay:300004343:1"/>
>> <sample lsid="urn:LSID:hapmap.org:HapMap/Sample:1004:1"/>
>> <genotyping_protocol
>> lsid="urn:LSID:genome.wi.mit.edu:HapMap/Protocol:0034:1:1/>
>> <alleles>
>> <allele base="G"/>
>> <allele base="T"/>
>> </alleles>
>> </genotype>
>>
>> Let's see, this less than 1/2 Kb in size for a single file, times 400
>> million records equals a whole bunch of space, you're right! I suppose I'
>> d
>> better look into the filesize limit thing and make sure that candidate
>> db'
>> s and OS platform(s) (Linux preferable) can in fact handle this size of
>> files. Thanks for the tip, Kimbro.
>>
>>
>> Mummi
>>
>>
>> On Friday, December 20, 2002, at 06:19 PM, Kimbro Staken wrote:
>>
>>>
>>> On Friday, December 20, 2002, at 09:06 AM, Gudmundur Arni Thorisson
>>> wrote:
>>>
>>>> Thanks for rapid reply, Murray. There could be some division there,
>>>> articifical if need be (divide by e.g. laboratory that produced
>>>> genotypes)
>>>> . But after looking briefly at the Xindice quick tutorial, it seemed
>>>> to
>>>> me that it would be natural to put each document type in its own
>>>> collection:
>>>>
>>>> /db/genotype/
>>>> /db/snp/
>>>> /db/haplotype/
>>>> /db/sample/
>>>> /db/individual/
>>>> /db/pedigree/
>>>> ..and so on.
>>>>
>>>> Where the genotype collection would be by far the biggest one (one
>>>> genotype per sample per SNP, where the number of samples will be in
>>>> the
>>>> range 180-270 and SNPs from 500 thousand up to 1.5 million). So, yes,
>>>> unless someone can suggest otherwise, I'd think that a single
> collection
>>>> would need to contain those 400M records.
>>>> Also, ince if one wants to retrieve a genotype by its unique (within
>>>> that type class) identifier, it would go something like this, using
>>>> LSIDs (Life Science Identifiers):
> /db/genotype/@lsid='urn:LSID:washu.edu:
>>>> HapMap/
>>>> Genotype:23423432434:1
>>>> (I'm no good at XPath, I know!)
>>>> But if there is a per-laboratory division, one would actually have to
>>>> know which lab the genotype came from, in addition to its identifier.
>>>> Not a Good Thing. This would probably also affect other, more complex
>>>> queries,
>>>> I don't know.
>>>>
>>>> Is there a hard limit on the number of documents per Xindice
>>>> collection?
>>>
>>> There is no hard limit.
>>>
>>>> Max number of files per directory or whatever, something outside
>>>> Xindice'
>>>> s control?
>>>
>>> The first external limit you'll run into will be file size. Xindice
> can't
>>> span a collection across files yet, so if your file system limits file
>>> size to 4GB or something that will be all you can store. This of course
>>> varies by platform.
>>>
>>> Really though 400 million is a pretty big number. The most I've ever
>>> tested with was a little over 1 million. The server could handle more,
>>> but it was really pushing the limits of the current system. So until
>>> Xindice matures quite a bit more, I'd have to recommend against it as a
>>> solution.
>>>
>>> It will be tough to find an open source solution that can easily handle
>>> that many documents with acceptable performance. As far as I know eXist
>>> won't be any better in this area. Honestly, I'd really have to question
>>> whether Oracle can even handle that much XML. Obviously, for relational
>>> data it's up to the task, but XML is quite a bit different and there's
>>> still some pretty inefficient aspects to what they're doing. Of course
>>> I
>>> do think Oracle is better then anything else currently available.
>>>
>>> Beside number of documents, have you estimated document size and storage
>>> space required? Even if you're looking at only 1k per document, I
> believe
>>> once you throw in indexes and overhead, you're pushing 1TB in data size.
>>> That's a pretty big chunk of data, it's not going to be easy to manage
> no
>>> matter which route you take.
>>>
>>>>
>>>>
>>>> Mummi, CSHL
>>>>
>>>> On Friday, December 20, 2002, at 03:43 PM, Murray Altheim wrote:
>>>>
>>>>> Gudmundur Arni Thorisson wrote:
>>>>>
>>>>> [...]
>>>>>
>>>>>> It says on the Xindice website that the db is designed for many,
>>>>>> small documents. The XML dataset that we will be handling will
>>>>>> contain fairly small documents but VERY many of them; up to 400
>>>>>> million instances of the most populous record class.
>>>>>> My question is therefore this: has anyone used/tested Xindice with
>>>>>> datasets of this size (hundreds of millions) with decent performance
>>>>>> as well? This will be mainly import + query work, hardly any heavy
>>>>>> updating load, if that would make a difference as far as performance
>>>>>> goes.
>>>>>
>>>>>
>>>>> One question that may help answer this: would 400 million records
>>>>> be in *one* Xindice Collection, or could these be organized according
>>>>> to some hierarchy, such that there would be a smaller limit at the
>>>>> Collection level?
>>>>>
>>>>>
>>>>> Murray
>>>>>
>>>>> .....................................................................
>>>>> .
>>>>> Murray Altheim <http://kmi.open.ac.uk/people/murray/
>>>>> >
>>>>> Knowledge Media Institute
>>>>> The Open University, Milton Keynes, Bucks, MK7 6AA, UK
>>>>>
>>>>> If you're the first person in a new territory,
>>>>> you're likely to get shot at.
>>>>> -- ma
>>>>>
>>>>>
>>>>
>>>>
>>> Kimbro Staken
>>> Java and XML Software, Consulting and Writing
> http://www.xmldatabases.org/
>>> Apache Xindice native XML database http://xml.apache.org/xindice
>>> XML:DB Initiative http://www.xmldb.org
>>>
>>>
>>
>>
>>
>
>
>
Re: Xindice scalability: using in a large bio
Posted by Devrim Ergel <de...@parsera.com>.
Hello,
Software AG Tamino XML Database might be the right answer. They are offering
as a robust native xml database for mission critical applications. The only
problem its price was 45000$ last year!
(In our case, we are Software AG Partner Software Company in Turkey so we
can bundle with reasonable prices)
Devrim
Parsera IT
----- Original Message -----
From: "Gudmundur Arni Thorisson" <mu...@cshl.org>
To: <xi...@xml.apache.org>
Sent: Friday, December 20, 2002 11:07 PM
Subject: Re: Xindice scalability: using in a large bio
> Ximbro, I have not estimated the total storage requirements for the
> project yet, as we have not yet finalized more than a part of the schema,
> including the 400M instance class. But it is certain that the size of
> each record in that class will be quite small, in an XML-sense, something
> like this:
>
> <genotype
lsid="urn:LSID:genome.wi.mit.edu:HapMap/Genotype:883423434:1"
> >
> <snp_assay
> lsid="urn:LSID:genome.wi.mit.edu:HapMap/SNPAssay:300004343:1"/>
> <sample lsid="urn:LSID:hapmap.org:HapMap/Sample:1004:1"/>
> <genotyping_protocol
> lsid="urn:LSID:genome.wi.mit.edu:HapMap/Protocol:0034:1:1/>
> <alleles>
> <allele base="G"/>
> <allele base="T"/>
> </alleles>
> </genotype>
>
> Let's see, this less than 1/2 Kb in size for a single file, times 400
> million records equals a whole bunch of space, you're right! I suppose I'd
> better look into the filesize limit thing and make sure that candidate db'
> s and OS platform(s) (Linux preferable) can in fact handle this size of
> files. Thanks for the tip, Kimbro.
>
>
> Mummi
>
>
> On Friday, December 20, 2002, at 06:19 PM, Kimbro Staken wrote:
>
> >
> > On Friday, December 20, 2002, at 09:06 AM, Gudmundur Arni Thorisson
> > wrote:
> >
> >> Thanks for rapid reply, Murray. There could be some division there,
> >> articifical if need be (divide by e.g. laboratory that produced
> >> genotypes)
> >> . But after looking briefly at the Xindice quick tutorial, it seemed to
> >> me that it would be natural to put each document type in its own
> >> collection:
> >>
> >> /db/genotype/
> >> /db/snp/
> >> /db/haplotype/
> >> /db/sample/
> >> /db/individual/
> >> /db/pedigree/
> >> ..and so on.
> >>
> >> Where the genotype collection would be by far the biggest one (one
> >> genotype per sample per SNP, where the number of samples will be in the
> >> range 180-270 and SNPs from 500 thousand up to 1.5 million). So, yes,
> >> unless someone can suggest otherwise, I'd think that a single
collection
> >> would need to contain those 400M records.
> >> Also, ince if one wants to retrieve a genotype by its unique (within
> >> that type class) identifier, it would go something like this, using
> >> LSIDs (Life Science Identifiers):
/db/genotype/@lsid='urn:LSID:washu.edu:
> >> HapMap/
> >> Genotype:23423432434:1
> >> (I'm no good at XPath, I know!)
> >> But if there is a per-laboratory division, one would actually have to
> >> know which lab the genotype came from, in addition to its identifier.
> >> Not a Good Thing. This would probably also affect other, more complex
> >> queries,
> >> I don't know.
> >>
> >> Is there a hard limit on the number of documents per Xindice
> >> collection?
> >
> > There is no hard limit.
> >
> >> Max number of files per directory or whatever, something outside
> >> Xindice'
> >> s control?
> >
> > The first external limit you'll run into will be file size. Xindice
can't
> > span a collection across files yet, so if your file system limits file
> > size to 4GB or something that will be all you can store. This of course
> > varies by platform.
> >
> > Really though 400 million is a pretty big number. The most I've ever
> > tested with was a little over 1 million. The server could handle more,
> > but it was really pushing the limits of the current system. So until
> > Xindice matures quite a bit more, I'd have to recommend against it as a
> > solution.
> >
> > It will be tough to find an open source solution that can easily handle
> > that many documents with acceptable performance. As far as I know eXist
> > won't be any better in this area. Honestly, I'd really have to question
> > whether Oracle can even handle that much XML. Obviously, for relational
> > data it's up to the task, but XML is quite a bit different and there's
> > still some pretty inefficient aspects to what they're doing. Of course I
> > do think Oracle is better then anything else currently available.
> >
> > Beside number of documents, have you estimated document size and storage
> > space required? Even if you're looking at only 1k per document, I
believe
> > once you throw in indexes and overhead, you're pushing 1TB in data size.
> > That's a pretty big chunk of data, it's not going to be easy to manage
no
> > matter which route you take.
> >
> >>
> >>
> >> Mummi, CSHL
> >>
> >> On Friday, December 20, 2002, at 03:43 PM, Murray Altheim wrote:
> >>
> >>> Gudmundur Arni Thorisson wrote:
> >>>
> >>> [...]
> >>>
> >>>> It says on the Xindice website that the db is designed for many,
> >>>> small documents. The XML dataset that we will be handling will
> >>>> contain fairly small documents but VERY many of them; up to 400
> >>>> million instances of the most populous record class.
> >>>> My question is therefore this: has anyone used/tested Xindice with
> >>>> datasets of this size (hundreds of millions) with decent performance
> >>>> as well? This will be mainly import + query work, hardly any heavy
> >>>> updating load, if that would make a difference as far as performance
> >>>> goes.
> >>>
> >>>
> >>> One question that may help answer this: would 400 million records
> >>> be in *one* Xindice Collection, or could these be organized according
> >>> to some hierarchy, such that there would be a smaller limit at the
> >>> Collection level?
> >>>
> >>>
> >>> Murray
> >>>
> >>> ......................................................................
> >>> Murray Altheim <http://kmi.open.ac.uk/people/murray/>
> >>> Knowledge Media Institute
> >>> The Open University, Milton Keynes, Bucks, MK7 6AA, UK
> >>>
> >>> If you're the first person in a new territory,
> >>> you're likely to get shot at.
> >>> -- ma
> >>>
> >>>
> >>
> >>
> > Kimbro Staken
> > Java and XML Software, Consulting and Writing
http://www.xmldatabases.org/
> > Apache Xindice native XML database http://xml.apache.org/xindice
> > XML:DB Initiative http://www.xmldb.org
> >
> >
>
>
>
Re: Xindice scalability: using in a large bio
Posted by Gudmundur Arni Thorisson <mu...@cshl.org>.
Ximbro, I have not estimated the total storage requirements for the
project yet, as we have not yet finalized more than a part of the schema,
including the 400M instance class. But it is certain that the size of
each record in that class will be quite small, in an XML-sense, something
like this:
<genotype lsid="urn:LSID:genome.wi.mit.edu:HapMap/Genotype:883423434:1"
>
<snp_assay
lsid="urn:LSID:genome.wi.mit.edu:HapMap/SNPAssay:300004343:1"/>
<sample lsid="urn:LSID:hapmap.org:HapMap/Sample:1004:1"/>
<genotyping_protocol
lsid="urn:LSID:genome.wi.mit.edu:HapMap/Protocol:0034:1:1/>
<alleles>
<allele base="G"/>
<allele base="T"/>
</alleles>
</genotype>
Let's see, this less than 1/2 Kb in size for a single file, times 400
million records equals a whole bunch of space, you're right! I suppose I'd
better look into the filesize limit thing and make sure that candidate db'
s and OS platform(s) (Linux preferable) can in fact handle this size of
files. Thanks for the tip, Kimbro.
Mummi
On Friday, December 20, 2002, at 06:19 PM, Kimbro Staken wrote:
>
> On Friday, December 20, 2002, at 09:06 AM, Gudmundur Arni Thorisson
> wrote:
>
>> Thanks for rapid reply, Murray. There could be some division there,
>> articifical if need be (divide by e.g. laboratory that produced
>> genotypes)
>> . But after looking briefly at the Xindice quick tutorial, it seemed to
>> me that it would be natural to put each document type in its own
>> collection:
>>
>> /db/genotype/
>> /db/snp/
>> /db/haplotype/
>> /db/sample/
>> /db/individual/
>> /db/pedigree/
>> ..and so on.
>>
>> Where the genotype collection would be by far the biggest one (one
>> genotype per sample per SNP, where the number of samples will be in the
>> range 180-270 and SNPs from 500 thousand up to 1.5 million). So, yes,
>> unless someone can suggest otherwise, I'd think that a single collection
>> would need to contain those 400M records.
>> Also, ince if one wants to retrieve a genotype by its unique (within
>> that type class) identifier, it would go something like this, using
>> LSIDs (Life Science Identifiers): /db/genotype/@lsid='urn:LSID:washu.edu:
>> HapMap/
>> Genotype:23423432434:1
>> (I'm no good at XPath, I know!)
>> But if there is a per-laboratory division, one would actually have to
>> know which lab the genotype came from, in addition to its identifier.
>> Not a Good Thing. This would probably also affect other, more complex
>> queries,
>> I don't know.
>>
>> Is there a hard limit on the number of documents per Xindice
>> collection?
>
> There is no hard limit.
>
>> Max number of files per directory or whatever, something outside
>> Xindice'
>> s control?
>
> The first external limit you'll run into will be file size. Xindice can't
> span a collection across files yet, so if your file system limits file
> size to 4GB or something that will be all you can store. This of course
> varies by platform.
>
> Really though 400 million is a pretty big number. The most I've ever
> tested with was a little over 1 million. The server could handle more,
> but it was really pushing the limits of the current system. So until
> Xindice matures quite a bit more, I'd have to recommend against it as a
> solution.
>
> It will be tough to find an open source solution that can easily handle
> that many documents with acceptable performance. As far as I know eXist
> won't be any better in this area. Honestly, I'd really have to question
> whether Oracle can even handle that much XML. Obviously, for relational
> data it's up to the task, but XML is quite a bit different and there's
> still some pretty inefficient aspects to what they're doing. Of course I
> do think Oracle is better then anything else currently available.
>
> Beside number of documents, have you estimated document size and storage
> space required? Even if you're looking at only 1k per document, I believe
> once you throw in indexes and overhead, you're pushing 1TB in data size.
> That's a pretty big chunk of data, it's not going to be easy to manage no
> matter which route you take.
>
>>
>>
>> Mummi, CSHL
>>
>> On Friday, December 20, 2002, at 03:43 PM, Murray Altheim wrote:
>>
>>> Gudmundur Arni Thorisson wrote:
>>>
>>> [...]
>>>
>>>> It says on the Xindice website that the db is designed for many,
>>>> small documents. The XML dataset that we will be handling will
>>>> contain fairly small documents but VERY many of them; up to 400
>>>> million instances of the most populous record class.
>>>> My question is therefore this: has anyone used/tested Xindice with
>>>> datasets of this size (hundreds of millions) with decent performance
>>>> as well? This will be mainly import + query work, hardly any heavy
>>>> updating load, if that would make a difference as far as performance
>>>> goes.
>>>
>>>
>>> One question that may help answer this: would 400 million records
>>> be in *one* Xindice Collection, or could these be organized according
>>> to some hierarchy, such that there would be a smaller limit at the
>>> Collection level?
>>>
>>>
>>> Murray
>>>
>>> ......................................................................
>>> Murray Altheim <http://kmi.open.ac.uk/people/murray/>
>>> Knowledge Media Institute
>>> The Open University, Milton Keynes, Bucks, MK7 6AA, UK
>>>
>>> If you're the first person in a new territory,
>>> you're likely to get shot at.
>>> -- ma
>>>
>>>
>>
>>
> Kimbro Staken
> Java and XML Software, Consulting and Writing http://www.xmldatabases.org/
> Apache Xindice native XML database http://xml.apache.org/xindice
> XML:DB Initiative http://www.xmldb.org
>
>
Re: Xindice scalability: using in a large bio
Posted by Kimbro Staken <ks...@xmldatabases.org>.
On Friday, December 20, 2002, at 09:06 AM, Gudmundur Arni Thorisson
wrote:
> Thanks for rapid reply, Murray. There could be some division there,
> articifical if need be (divide by e.g. laboratory that produced
> genotypes)
> . But after looking briefly at the Xindice quick tutorial, it seemed
> to me that it would be natural to put each document type in its own
> collection:
>
> /db/genotype/
> /db/snp/
> /db/haplotype/
> /db/sample/
> /db/individual/
> /db/pedigree/
> ..and so on.
>
> Where the genotype collection would be by far the biggest one (one
> genotype per sample per SNP, where the number of samples will be in
> the range 180-270 and SNPs from 500 thousand up to 1.5 million). So,
> yes, unless someone can suggest otherwise, I'd think that a single
> collection would need to contain those 400M records.
> Also, ince if one wants to retrieve a genotype by its unique (within
> that type class) identifier, it would go something like this, using
> LSIDs (Life Science Identifiers):
> /db/genotype/@lsid='urn:LSID:washu.edu:HapMap/
> Genotype:23423432434:1
> (I'm no good at XPath, I know!)
> But if there is a per-laboratory division, one would actually have to
> know which lab the genotype came from, in addition to its identifier.
> Not a Good Thing. This would probably also affect other, more complex
> queries,
> I don't know.
>
> Is there a hard limit on the number of documents per Xindice
> collection?
There is no hard limit.
> Max number of files per directory or whatever, something outside
> Xindice'
> s control?
The first external limit you'll run into will be file size. Xindice
can't span a collection across files yet, so if your file system limits
file size to 4GB or something that will be all you can store. This of
course varies by platform.
Really though 400 million is a pretty big number. The most I've ever
tested with was a little over 1 million. The server could handle more,
but it was really pushing the limits of the current system. So until
Xindice matures quite a bit more, I'd have to recommend against it as a
solution.
It will be tough to find an open source solution that can easily handle
that many documents with acceptable performance. As far as I know eXist
won't be any better in this area. Honestly, I'd really have to question
whether Oracle can even handle that much XML. Obviously, for relational
data it's up to the task, but XML is quite a bit different and there's
still some pretty inefficient aspects to what they're doing. Of course
I do think Oracle is better then anything else currently available.
Beside number of documents, have you estimated document size and
storage space required? Even if you're looking at only 1k per document,
I believe once you throw in indexes and overhead, you're pushing 1TB in
data size. That's a pretty big chunk of data, it's not going to be easy
to manage no matter which route you take.
>
>
> Mummi, CSHL
>
> On Friday, December 20, 2002, at 03:43 PM, Murray Altheim wrote:
>
>> Gudmundur Arni Thorisson wrote:
>>
>> [...]
>>
>>> It says on the Xindice website that the db is designed for many,
>>> small documents. The XML dataset that we will be handling will
>>> contain fairly small documents but VERY many of them; up to 400
>>> million instances of the most populous record class.
>>> My question is therefore this: has anyone used/tested Xindice with
>>> datasets of this size (hundreds of millions) with decent performance
>>> as well? This will be mainly import + query work, hardly any heavy
>>> updating load, if that would make a difference as far as performance
>>> goes.
>>
>>
>> One question that may help answer this: would 400 million records
>> be in *one* Xindice Collection, or could these be organized according
>> to some hierarchy, such that there would be a smaller limit at the
>> Collection level?
>>
>>
>> Murray
>>
>> ......................................................................
>> Murray Altheim <http://kmi.open.ac.uk/people/murray/>
>> Knowledge Media Institute
>> The Open University, Milton Keynes, Bucks, MK7 6AA, UK
>>
>> If you're the first person in a new territory,
>> you're likely to get shot at.
>> -- ma
>>
>>
>
>
Kimbro Staken
Java and XML Software, Consulting and Writing
http://www.xmldatabases.org/
Apache Xindice native XML database http://xml.apache.org/xindice
XML:DB Initiative http://www.xmldb.org
Re: Xindice scalability: using in a large bio
Posted by Gudmundur Arni Thorisson <mu...@cshl.org>.
Thanks for rapid reply, Murray. There could be some division there,
articifical if need be (divide by e.g. laboratory that produced genotypes)
. But after looking briefly at the Xindice quick tutorial, it seemed to me
that it would be natural to put each document type in its own collection:
/db/genotype/
/db/snp/
/db/haplotype/
/db/sample/
/db/individual/
/db/pedigree/
..and so on.
Where the genotype collection would be by far the biggest one (one
genotype per sample per SNP, where the number of samples will be in the
range 180-270 and SNPs from 500 thousand up to 1.5 million). So, yes,
unless someone can suggest otherwise, I'd think that a single collection
would need to contain those 400M records.
Also, ince if one wants to retrieve a genotype by its unique (within
that type class) identifier, it would go something like this, using LSIDs
(Life Science Identifiers): /db/genotype/@lsid='urn:LSID:washu.edu:HapMap/
Genotype:23423432434:1
(I'm no good at XPath, I know!)
But if there is a per-laboratory division, one would actually have to
know which lab the genotype came from, in addition to its identifier. Not
a Good Thing. This would probably also affect other, more complex queries,
I don't know.
Is there a hard limit on the number of documents per Xindice collection?
Max number of files per directory or whatever, something outside Xindice'
s control?
Mummi, CSHL
On Friday, December 20, 2002, at 03:43 PM, Murray Altheim wrote:
> Gudmundur Arni Thorisson wrote:
>
> [...]
>
>> It says on the Xindice website that the db is designed for many, small
>> documents. The XML dataset that we will be handling will contain fairly
>> small documents but VERY many of them; up to 400 million instances of
>> the most populous record class.
>> My question is therefore this: has anyone used/tested Xindice with
>> datasets of this size (hundreds of millions) with decent performance as
>> well? This will be mainly import + query work, hardly any heavy updating
>> load, if that would make a difference as far as performance goes.
>
>
> One question that may help answer this: would 400 million records
> be in *one* Xindice Collection, or could these be organized according
> to some hierarchy, such that there would be a smaller limit at the
> Collection level?
>
>
> Murray
>
> ......................................................................
> Murray Altheim <http://kmi.open.ac.uk/people/murray/>
> Knowledge Media Institute
> The Open University, Milton Keynes, Bucks, MK7 6AA, UK
>
> If you're the first person in a new territory,
> you're likely to get shot at.
> -- ma
>
>
Re: Xindice scalability: using in a large bio
Posted by Murray Altheim <m....@open.ac.uk>.
Gudmundur Arni Thorisson wrote:
[...]
> It says on the Xindice website that the db
> is designed for many, small documents. The XML dataset that we will be
> handling will contain fairly small documents but VERY many of them; up
> to 400 million instances of the most populous record class.
> My question is therefore this: has anyone used/tested Xindice with
> datasets of this size (hundreds of millions) with decent performance as
> well? This will be mainly import + query work, hardly any heavy updating
> load, if that would make a difference as far as performance goes.
One question that may help answer this: would 400 million records
be in *one* Xindice Collection, or could these be organized according
to some hierarchy, such that there would be a smaller limit at the
Collection level?
Murray
......................................................................
Murray Altheim <http://kmi.open.ac.uk/people/murray/>
Knowledge Media Institute
The Open University, Milton Keynes, Bucks, MK7 6AA, UK
If you're the first person in a new territory,
you're likely to get shot at.
-- ma