xml getting corrupted

[Sankalp Jain <sa...@yahoo.com>]

Hi,
I am with the SAM-Grid team at Fermilab,IL.
We use Xindice for monitoring jobs. We are running Xindice
( v1.1b4 ) in a stand alone mode under tomcat. The database
is updated remotely from various nodes using xmlrpc. 

   There can be any number of jobs between 1 - 200 that
concurrently update a single document. Also the document
size can vary from a few kbs to a few Mbs (between 4-5 Mbs)

   We have noticed that whenever there is a large amount of
concurrent activity the document is corrupted.

There are a few questions that I would like to ask :

1. Is there a limit on document size that Xindice can
handle?

2. Also How many concurrent writes to the same document are
OK? Are there any known problems regarding this issue?

Thanks in advance
Sankalp




________________________________________________________________________
Yahoo! India Matrimony: Find your partner online. http://yahoo.shaadi.com/india-matrimony/

Re: xml getting corrupted

[Vadim Gritsenko <va...@reverycodes.com>]

Sankalp Jain wrote:

>Hi Vadim,
>Thanks for your reply. Few more questions for you.
>
>Are there any limitations on having large number of
>documents (say thousands) in a single collection? 
>  
>

Hundreds of thousands documents are Ok; it all depends on what kind of 
access and indexing you do. If you want each time to pull all documents 
- it (obviously) will be slow. If you use proper index to fetch handful 
of documents - it will be really fast.


>Do you foresee any problems in doing so given that many
>(possibly hundreds) of these documents will be individually
>updated (using Xupdate) at the same time.
>  
>

I don't have much first-hand XUpdate experience. Update of hundreds of 
documents xindice should be able to perform, but it can be slow - 
depending on documents size and query efficiency.

Vadim

Re: xml getting corrupted

[Sankalp Jain <sa...@yahoo.com>]

Hi Vadim,
Thanks for your reply. Few more questions for you.

Are there any limitations on having large number of
documents (say thousands) in a single collection? 

Do you foresee any problems in doing so given that many
(possibly hundreds) of these documents will be individually
updated (using Xupdate) at the same time.

Thanks
- Sankalp Jain


 --- Vadim Gritsenko <va...@reverycodes.com> wrote: >
Sankalp Jain wrote:
> ...
> 
> >   There can be any number of jobs between 1 - 200 that
> >concurrently update a single document. Also the document
> >size can vary from a few kbs to a few Mbs (between 4-5
> Mbs)
> >
> >   We have noticed that whenever there is a large amount
> of
> >concurrent activity the document is corrupted.
> >
> >There are a few questions that I would like to ask :
> >
> >1. Is there a limit on document size that Xindice can
> >handle?
> >  
> >
> 
> Xindice is not suitable for large documents, it was
> designed to work 
> with smaller documents. There it no hard limit for
> document size, but 
> with larger documents performance and memory consuption
> will increase. 
> 4-5 Mb per document is too large file, IMHO, for Xindice.
> 
> 
> >2. Also How many concurrent writes to the same document
> are
> >OK? Are there any known problems regarding this issue?
> >  
> >
> 
> What do you mean under "concurrent writes"? Are you using
> xupdate, or 
> overwriting whole document each time? For XUpdate, see:
>    
> http://nagoya.apache.org/bugzilla/show_bug.cgi?id=13745
> 
> 
> If you come up with Unit test exposing your issue, it
> will be easier to 
> find and fix the problem. See
>   
>
http://cvs.apache.org/viewcvs.cgi/xml-xindice/java/tests/src/org/apache/xindice/core/filer/FilerTestBase.java?rev=1.3&view=auto
> method testConcurrentInsert() for an example of the
> concurrency test.
> 
> Vadim
>  

________________________________________________________________________
Yahoo! India Matrimony: Find your partner online. http://yahoo.shaadi.com/india-matrimony/

Re: xml getting corrupted

[Vadim Gritsenko <va...@reverycodes.com>]

Sankalp Jain wrote:
...

>   There can be any number of jobs between 1 - 200 that
>concurrently update a single document. Also the document
>size can vary from a few kbs to a few Mbs (between 4-5 Mbs)
>
>   We have noticed that whenever there is a large amount of
>concurrent activity the document is corrupted.
>
>There are a few questions that I would like to ask :
>
>1. Is there a limit on document size that Xindice can
>handle?
>  
>

Xindice is not suitable for large documents, it was designed to work 
with smaller documents. There it no hard limit for document size, but 
with larger documents performance and memory consuption will increase. 
4-5 Mb per document is too large file, IMHO, for Xindice.


>2. Also How many concurrent writes to the same document are
>OK? Are there any known problems regarding this issue?
>  
>

What do you mean under "concurrent writes"? Are you using xupdate, or 
overwriting whole document each time? For XUpdate, see:
    http://nagoya.apache.org/bugzilla/show_bug.cgi?id=13745


If you come up with Unit test exposing your issue, it will be easier to 
find and fix the problem. See
  
http://cvs.apache.org/viewcvs.cgi/xml-xindice/java/tests/src/org/apache/xindice/core/filer/FilerTestBase.java?rev=1.3&view=auto
method testConcurrentInsert() for an example of the concurrency test.

Vadim