You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@jackrabbit.apache.org by Bénigot Yves <yv...@ginerativ.fr> on 2010/07/19 18:14:46 UTC
Is there a way to store JackRabbit documents in two different datastores for one repository and yet index them with Lucene
I have two categories of documents in one repository :
- some files are stored by the users, they would go in a file datastore
- other files are generated, and already stored in an SQL table in a BLOB column
I would like to create a specific DataStore or PersistenceManager to be able to leave the files stored in the SQL table where they are,
yet define them as JackRabbit documents, and let Lucene index them
Is it possible to have such a hybrid setup with two different storage seen as one repository in JackRabbit ?
yves
Re: Is there a way to store JackRabbit documents in two different
datastores for one repository and yet index them with Lucene [SEC=UNCLASSIFIED]
Posted by Ro...@ipaustralia.gov.au.
Isn't this the kind of thing that the JBOSS JCR implementation is supposed
to handle?
Ross.
Bénigot Yves <yv...@ginerativ.fr> wrote on 20/07/2010 02:14:46 AM:
> From: Bénigot Yves <yv...@ginerativ.fr>
> To: <us...@jackrabbit.apache.org>
> Date: 20/07/2010 02:15 AM
> Subject: Is there a way to store JackRabbit documents in two
> different datastores for one repository and yet index them with Lucene
>
> I have two categories of documents in one repository :
>
> - some files are stored by the users, they would go in a file
datastore
>
> - other files are generated, and already stored in an SQL table
> in a BLOB column
>
>
>
> I would like to create a specific DataStore or PersistenceManager to
> be able to leave the files stored in the SQL table where they are,
>
> yet define them as JackRabbit documents, and let Lucene index them
>
>
>
> Is it possible to have such a hybrid setup with two different
> storage seen as one repository in JackRabbit ?
>
>
>
> yves
>
Re: Is there a way to store JackRabbit documents in two different
datastores for one repository and yet index them with Lucene
Posted by Alexander Klimetschek <ak...@day.com>.
On Tue, Jul 20, 2010 at 09:15, Thomas Müller <th...@day.com> wrote:
>> https://issues.apache.org/jira/browse/JCR-2682
>
> For me, it sounds like this is a slightly different problem: "Allow
> the FileDataStore to scale over several millions of files" - it sounds
> like it's just a scalability issue (where you don't care too much
> which document is stored where; but you need a solution that can store
> and backup a very large number of documents), not so much a mounting
> issue (where you do care where exactly which document is stored).
I basically agree, I was just thinking that the same mechanism could
be used to solve two different problems ;-)
Regards,
Alex
--
Alexander Klimetschek
alexander.klimetschek@day.com
Re: Is there a way to store JackRabbit documents in two different
datastores for one repository and yet index them with Lucene
Posted by Thomas Müller <th...@day.com>.
Hi,
Actually I wouldn't consider this feature for Jackrabbit 3, instead I
would try to solve this problem using a "virtual repository" feature.
I think the storage of one repository should be at one place. If you
want to distribute / split storage, then use multiple repositories,
and logically combine those into one, using a "virtual repository"
feature (mount points). (This might be done automatically at some
point, but I think it's much more important to be able to split
manually like in Linux, because as a developer / user you know what
data goes together - software can't efficiently decide that.)
> I agree that such a configuration would be useful. I could imagine
> this being per-path, for example, one could say that /var or /tmp
> folders (as in Unix file systems) are served by in-memory persistence
> managers.
You can do that with virtual repositories as well. I think a virtual
repository is something like a mount point / device.
> https://issues.apache.org/jira/browse/JCR-2682
For me, it sounds like this is a slightly different problem: "Allow
the FileDataStore to scale over several millions of files" - it sounds
like it's just a scalability issue (where you don't care too much
which document is stored where; but you need a solution that can store
and backup a very large number of documents), not so much a mounting
issue (where you do care where exactly which document is stored).
Regards,
Thomas
Re: Is there a way to store JackRabbit documents in two different
datastores for one repository and yet index them with Lucene
Posted by Alexander Klimetschek <ak...@day.com>.
On Mon, Jul 19, 2010 at 21:38, Alexander Klimetschek <ak...@day.com> wrote:
> 2010/7/19 Bénigot Yves <yv...@ginerativ.fr>:
>> I have two categories of documents in one repository :
>>
>> - some files are stored by the users, they would go in a file datastore
>>
>> - other files are generated, and already stored in an SQL table in a BLOB column
>>
>>
>>
>> I would like to create a specific DataStore or PersistenceManager to be able to leave the files stored in the SQL table where they are,
>>
>> yet define them as JackRabbit documents, and let Lucene index them
>>
>>
>>
>> Is it possible to have such a hybrid setup with two different storage seen as one repository in JackRabbit ?
>
> The only way to do that would be by using two different workspaces for
> the two use cases, and have each workspace configured differently. But
> that would only apply to the persistence manager, not for the
> DataStore, which is a single one for the entire repository.
>
> I agree that such a configuration would be useful. I could imagine
> this being per-path, for example, one could say that /var or /tmp
> folders (as in Unix file systems) are served by in-memory persistence
> managers.
BTW, I just noted https://issues.apache.org/jira/browse/JCR-2682 (a
feature request) that goes into the same direction, I think.
Regards,
Alex
--
Alexander Klimetschek
alexander.klimetschek@day.com
Re: Is there a way to store JackRabbit documents in two different
datastores for one repository and yet index them with Lucene
Posted by Alexander Klimetschek <ak...@day.com>.
2010/7/19 Bénigot Yves <yv...@ginerativ.fr>:
> I have two categories of documents in one repository :
>
> - some files are stored by the users, they would go in a file datastore
>
> - other files are generated, and already stored in an SQL table in a BLOB column
>
>
>
> I would like to create a specific DataStore or PersistenceManager to be able to leave the files stored in the SQL table where they are,
>
> yet define them as JackRabbit documents, and let Lucene index them
>
>
>
> Is it possible to have such a hybrid setup with two different storage seen as one repository in JackRabbit ?
The only way to do that would be by using two different workspaces for
the two use cases, and have each workspace configured differently. But
that would only apply to the persistence manager, not for the
DataStore, which is a single one for the entire repository.
I agree that such a configuration would be useful. I could imagine
this being per-path, for example, one could say that /var or /tmp
folders (as in Unix file systems) are served by in-memory persistence
managers. However, I think this is quite a big effort, something that
could be considered for Jackrabbit 3. See the dev list:
http://jackrabbit.markmail.org/search/?q=jr3
Regards,
Alex
--
Alexander Klimetschek
alexander.klimetschek@day.com
RE: Is there a way to store JackRabbit documents in two different datastores for one repository and yet index them with Lucene
Posted by Bénigot Yves <yv...@ginerativ.fr>.
I will try do what you propose.
Please allow some time for this to be done, because this is a big project already, then I will mail a report on this.
I believe it should work fine.
Thank you for your help
Yves
-----Message d'origine-----
De : Thomas Müller [mailto:thomas.mueller@day.com]
Envoyé : mardi 20 juillet 2010 10:22
À : users@jackrabbit.apache.org
Objet : Re: Is there a way to store JackRabbit documents in two different datastores for one repository and yet index them with Lucene
Hi,
To store binaries, Jackrabbit uses the datastore API which is
(relatively) stable and here to stay. The plan is to support it in
Jackrabbit 3 as well. It's a relatively simple mechanism: you add a
binary and get a unique identifier. There is even a Jackrabbit API to
get the identifier: JackrabbitValue.getContentIdentity().
What you could do is:
- Implement a new "multiplex-datastore" that supports a number of
sub-datastores. It will read objects from any datastore that contains
the entry. When storing it needs to check if it's already stored, and
only if not it will store it in the first sub-datastore.
- Configure Jackrabbit to use this multiplex-datastore.
- Store the document within you application in the second datastore
(see below). The datastore supports concurrent writes, so that can be
in a different process if you want.
- You probably need to store (stream) the object in Jackrabbit as
well, but this will not create a new entry in the data store (the
existing one is re-used). In theory, to avoid streaming the object
again, you could create a JackrabbitValue yourself, but I didn't test
that. It would be a nice feature in Jackrabbit.
The solution would look like this:
MultiplexDataStore (<= used by Jackrabbit) links to
* FileDataStore @ /data/jackabbit/ (<= used by Jackrabbit to store new objects)
* FileDataStore @ /data/app/ (<= used by your application)
Regards,
Thomas
Re: Is there a way to store JackRabbit documents in two different
datastores for one repository and yet index them with Lucene
Posted by Thomas Müller <th...@day.com>.
Hi,
To store binaries, Jackrabbit uses the datastore API which is
(relatively) stable and here to stay. The plan is to support it in
Jackrabbit 3 as well. It's a relatively simple mechanism: you add a
binary and get a unique identifier. There is even a Jackrabbit API to
get the identifier: JackrabbitValue.getContentIdentity().
What you could do is:
- Implement a new "multiplex-datastore" that supports a number of
sub-datastores. It will read objects from any datastore that contains
the entry. When storing it needs to check if it's already stored, and
only if not it will store it in the first sub-datastore.
- Configure Jackrabbit to use this multiplex-datastore.
- Store the document within you application in the second datastore
(see below). The datastore supports concurrent writes, so that can be
in a different process if you want.
- You probably need to store (stream) the object in Jackrabbit as
well, but this will not create a new entry in the data store (the
existing one is re-used). In theory, to avoid streaming the object
again, you could create a JackrabbitValue yourself, but I didn't test
that. It would be a nice feature in Jackrabbit.
The solution would look like this:
MultiplexDataStore (<= used by Jackrabbit) links to
* FileDataStore @ /data/jackabbit/ (<= used by Jackrabbit to store new objects)
* FileDataStore @ /data/app/ (<= used by your application)
Regards,
Thomas
RE: Is there a way to store JackRabbit documents in two different datastores for one repository and yet index them with Lucene
Posted by Bénigot Yves <yv...@ginerativ.fr>.
The options are :
- storing the document twice : a waste of disk space (gigabytes...)
- storing the document's URL (or reference) in JR : but I doubt that Lucene will be able to index it, otherwise it would be a great idea.
Certainly which piece of software is in charge of the document must be defined clearly. The SQL BLOB is never updated, but the inserts and deletes should be managed.
You have a point : The document would be created and stored in a table outside of JR, and I would need to call the JR API to make this BLOB a JR document, but I suspect that there is no way to insert a new document in JR without creating a copy.
Yves
-----Message d'origine-----
De : Thomas Müller [mailto:thomas.mueller@day.com]
Envoyé : mardi 20 juillet 2010 09:41
À : users@jackrabbit.apache.org
Objet : Re: Is there a way to store JackRabbit documents in two different datastores for one repository and yet index them with Lucene
Hi,
> Well I have other applications accessing (create,read) the documents in the SQL table, and I don't want to change them at once, since it is a lot of work to do.
What about storing such documents twice? Or instead of storing the
document in Jackrabbit, just store the location (URL / SQL statement
to retrieve it / ...)?
Either Jackrabbit is in charge of storing the document, or your
application is in charge (unless if you store the document twice). If
Jackrabbit is in charge, and you want to access the document (without
using Jackrabbit) directly in the table, then how do you know which
document it is?
Regards,
Thomas
Re: Is there a way to store JackRabbit documents in two different
datastores for one repository and yet index them with Lucene
Posted by Thomas Müller <th...@day.com>.
Hi,
> Well I have other applications accessing (create,read) the documents in the SQL table, and I don't want to change them at once, since it is a lot of work to do.
What about storing such documents twice? Or instead of storing the
document in Jackrabbit, just store the location (URL / SQL statement
to retrieve it / ...)?
Either Jackrabbit is in charge of storing the document, or your
application is in charge (unless if you store the document twice). If
Jackrabbit is in charge, and you want to access the document (without
using Jackrabbit) directly in the table, then how do you know which
document it is?
Regards,
Thomas
RE: Is there a way to store JackRabbit documents in two different datastores for one repository and yet index them with Lucene
Posted by Bénigot Yves <yv...@ginerativ.fr>.
Well I have other applications accessing (create,read) the documents in the SQL table, and I don't want to change them at once, since it is a lot of work to do.
Of course I could duplicate the document in both type of storage, but that would be wasteful.
I will try to think about using different workspaces, but I am afraid that this would add some constraints, like not being able to search on several workspaces in the same query. Is this correct ?
Yves
-----Message d'origine-----
De : Thomas Müller [mailto:thomas.mueller@day.com]
Envoyé : mardi 20 juillet 2010 09:04
À : users@jackrabbit.apache.org
Objet : Re: Is there a way to store JackRabbit documents in two different datastores for one repository and yet index them with Lucene
Hi,
> I have two categories of documents in one repository :
> - some files are stored by the users, they would go in a file datastore
> - other files are generated, and already stored in an SQL table in a BLOB column
Why exactly do you want them stored in two different places?
Regards,
Thomas
Re: Is there a way to store JackRabbit documents in two different
datastores for one repository and yet index them with Lucene
Posted by Thomas Müller <th...@day.com>.
Hi,
> I have two categories of documents in one repository :
> - some files are stored by the users, they would go in a file datastore
> - other files are generated, and already stored in an SQL table in a BLOB column
Why exactly do you want them stored in two different places?
Regards,
Thomas