Workspace.copy() Question ...

[KÖLL Claus <C....@TIROL.GV.AT>]

hi there ...

 

i have a nice usecase .. i have a filenode in my workspace and i should create

about 70 copies of this node. 

its a not so small pdf file (10Mb) and i am using the datastore so its no problem

the binary exists only one time but the problem is the textextractor. it will be called 70 times :-)

is it possible to reuse the fulltext index on a copy operation without new reindexing the file ?

 

thanks

 

claus

 

Re: Workspace.copy() Question ...

[Thomas Müller <th...@day.com>]

Hi,

> extended the Value interface instead of InputStream.

That would work as well: JackrabbitValue.

> if we make it like you wrote ...
> every module must handle this internally.

Yes. The modules can use standard JCR event listeners, so they are
backward compatible and implementation independent.

> it would be much finer to handle this more generic with a proxy in front of 'modules'.
> such proxy could store the dataIdentifier/result and a module key and if we try
> to process a binary a second time this proxy could check this and return without calling
> the module a second time.

If you do that, you need to define a new API. Could you describe your
solution in more detail? Not all modules have the exact same
requirements, we should make sure the solution is flexible enough.

Regards,
Thomas

AW: Workspace.copy() Question ...

[KÖLL Claus <C....@TIROL.GV.AT>]

some more thoughts ...

thomas, if we make it like you wrote ...

class VirusScanner {
    public void scan(InputStream in) throws VirusFoundException {
        if(in instanceof DataStoreInputStream) {
            DataIdentifier di = ((DataStoreInputStream) in).getDataIdentifier();
            if (hasScanned(di)) {
                // already exists, no need to copy
                return;
            } else {
                doScan(in);
                addScanned(di);
            }
        } else {
            doScan(in);
        }

every module must handle this internally.

it would be much finer to handle this more generic with a proxy in front of 'modules'.
such proxy could store the dataIdentifier/result and a module key and if we try
to process a binary a second time this proxy could check this and return without calling
the module a second time.

greets
claus

AW: Workspace.copy() Question ...

[KÖLL Claus <C....@TIROL.GV.AT>]

hi guys,

for my understanding ...

>Probably not in the Lucene index files itself. Text extraction could be used without using the Lucene index, for example to display the text content of a >PDF file. The text extraction module could store the DataIdentifier together with the extracted text ('payload'). The advantage to store this 'payload' >near the actual binary is that the data is deleted when the binary is garbage collected. So maybe it actually is better to store the 'payload' (extracted >text, virus scanner flag, thumbnail) near the binary, so it is automatically garbage collected when the binary is garbage collected. We would need to >define an API and the behavior for this 'payload storage'. It probably doesn't need to be transactional, but it needs to be consistent (a checksum). >Some kind of binary properties file maybe, with put(String key, InputStream payload), and InputStream get(String key).

... thomas you are talking about a textextraction module but i can not follow you,
As far as i understand, you will change the architectur going to modules
such like textactraction, viruscanner or thumbnailbuilder so they can store the informations with
the dataidentifier and the result in the datastore ?
if the gc will delete a entry in the datastore based on the dataIdentifier the
"near" informations will also be deleted automatilally ?

as you wrote the main problem is that we do not know if we have already processed a binary
it would be fine if we internally create a dataIdentifier of a binary stream and give them
to such modules like textextractor or what else so they can look for a result that
was already be processed and stored.

is that also what you think ;-)

Re: Workspace.copy() Question ...

[Jukka Zitting <ju...@gmail.com>]

Hi,

On Mon, Nov 17, 2008 at 11:07 AM, Thomas Müller <th...@day.com> wrote:
> Currently we don't detect that the binary already exists when using
> the regular JCR API.

We do for things like workspace.copy(...) or
propertyA.setValue(propertyB.getValue()). The only case where we don't
do that is if people explicitly get the stream from a property and
assign that to a new property, i.e.
propertyA.setValue(propertyB.getStream()) (with proper closing of the
stream, etc.). I don't think that's a frequent (or even preferred) use
case and would rather avoid adding special stream semantics just to
cover such usage.

> With the DataStoreInputStream we could do that, and it would be backward
> compatible. The DataStoreInputStream should be part of the Jackrabbit API.

If we really do need to make such functionality visible to the users,
I would much prefer if we extended the Value interface instead of
InputStream.

BR,

Jukka Zitting

Re: Workspace.copy() Question ...

[Thomas Müller <th...@day.com>]

Hi,

>> Exactly. The DataStore should also check if the InputStream is a
>> DataStoreInputStream, so maybe it doesn't need to copy the binary:
>
> IMHO we should (and currently do) handle that on a higher level, by
> tracking the DataIdentifier in InternalValue.

Currently we don't detect that the binary already exists when using
the regular JCR API. With the DataStoreInputStream we could do that,
and it would be backward compatible. The DataStoreInputStream should
be part of the Jackrabbit API.

Of course we can internally continue to use InternalValue and
BLOBFileValue to avoid using instanceof whenever possible.

> The same goes for the proposed text extraction, virus scanning, etc.
> extensions. The DataRecord interface is not a public API so we could
> easily modify it to cover such needs. Using instanceof and overloading
> the input stream is a hack.

Just adding one extra stream to binary InternalValues would only solve
the text extraction problem, but it wouldn't be a generic solution.
There would be no way for a regular (user defined) EventListener to
make sure a unique binary is only processed once. The
DataStoreInputStream would provide such a solution.

Regards,
Thomas

Re: Workspace.copy() Question ...

[Jukka Zitting <ju...@gmail.com>]

Hi,

On Mon, Nov 17, 2008 at 10:02 AM, Thomas Müller <th...@day.com> wrote:
>> But what will you do in the case if you try to copy
>> a node internaly .. the datastore should know that he must not read the binary
>> to prevent extra read and write to the datastore.
>
> Exactly. The DataStore should also check if the InputStream is a
> DataStoreInputStream, so maybe it doesn't need to copy the binary:

IMHO we should (and currently do) handle that on a higher level, by
tracking the DataIdentifier in InternalValue.

The same goes for the proposed text extraction, virus scanning, etc.
extensions. The DataRecord interface is not a public API so we could
easily modify it to cover such needs. Using instanceof and overloading
the input stream is a hack.

BR,

Jukka Zitting

Re: Workspace.copy() Question ...

[Thomas Müller <th...@day.com>]

Hi,

> would you store the dataidentifier in the index
> and so in all modules ?

Probably not in the Lucene index files itself. Text extraction could
be used without using the Lucene index, for example to display the
text content of a PDF file. The text extraction module could store the
DataIdentifier together with the extracted text ('payload'). The
advantage to store this 'payload' near the actual binary is that the
data is deleted when the binary is garbage collected. So maybe it
actually is better to store the 'payload' (extracted text, virus
scanner flag, thumbnail) near the binary, so it is automatically
garbage collected when the binary is garbage collected. We would need
to define an API and the behavior for this 'payload storage'. It
probably doesn't need to be transactional, but it needs to be
consistent (a checksum). Some kind of binary properties file maybe,
with put(String key, InputStream payload), and InputStream get(String
key).

> But what will you do in the case if you try to copy
> a node internaly .. the datastore should know that he must not read the binary
> to prevent extra read and write to the datastore.

Exactly. The DataStore should also check if the InputStream is a
DataStoreInputStream, so maybe it doesn't need to copy the binary:

if(in instanceof DataStoreInputStream) {
    DataIdentifier di = ((DataStoreInputStream) in).getDataIdentifier();
    if (exists(di)) {
        // already exists, no need to copy
        return;
    }
}
... create a new entry as done now ...

The 'instanceof' is not nice, but like that we don't have to change
the API, and we can use DataStoreInputStream also everywhere in the
JCR API (Node.setProperty(String name, InputStream in)).

> can you explain this a little bit more .. i dont know what
> viruscan and thumbnails have to do with that problem.

So far we were talking about text extraction: Text extraction should
only process each distinct binary once. But the virus scanner should
also process each distinct binary only once. And you only need to
create a thumbnail once for each distinct image. Currently there is no
way for the virus scanner to detect that it has already processed the
binary. However if we return a DataStoreInputStream, the virus scanner
module could check if it has already scanned the binary:

class VirusScanner {
    public void scan(InputStream in) throws VirusFoundException {
        if(in instanceof DataStoreInputStream) {
            DataIdentifier di = ((DataStoreInputStream) in).getDataIdentifier();
            if (hasScanned(di)) {
                // already exists, no need to copy
                return;
            } else {
                doScan(in);
                addScanned(di);
            }
        } else {
            doScan(in);
        }

The code for creating thumbnails would look similar.

Regards,
Thomas

AW: Workspace.copy() Question ...

[KÖLL Claus <C....@TIROL.GV.AT>]

Hi Thomas,

>Instead of returning an InputStream, Jackrabbit would return a
>DataStoreInputStream with the additional method getDataIdentifier().
>Then the module can read the identifier, check if the item is already
>processed, and avoid reading the data itself if this identifier is
>already processed. 

What does this exactly mean ? would you store the dataidentifier in the index 
and so in all modules ?

But what will you do in the case if you try to copy
a node internaly .. the datastore should know that he must not read the binary
to prevent extra read and write to the datastore.

>For text extraction, a separate
>file may make sense, but probably not for 'virus scan' because that's
>only a flag (you don't need the data). Thumbnails: for better
>performance you want to keep them together, and not save them
>separately (that is, in the data store).

can you explain this a little bit more .. i dont know what 
viruscan and thumbnails have to do with that problem.

i think i can not follow your thoughts at all 

greets
claus

Re: Workspace.copy() Question ...

[Thomas Müller <th...@day.com>]

Hi,

The problem is: "process the binary only once".

With 'process' we said 'text extraction', but it could be 'virus
scan', 'index', 'create a thumbnail', 'transfer' (to the client or
from the client), or 'backup' - any expensive task. I believe a good
solution is to provide the object identity to the module (the text
extraction engine, virus scanner, and so on), so that the module can
decide itself what to do.

Instead of returning an InputStream, Jackrabbit would return a
DataStoreInputStream with the additional method getDataIdentifier().
Then the module can read the identifier, check if the item is already
processed, and avoid reading the data itself if this identifier is
already processed. I believe that would be a flexible solution. How
the module stores the data for this object (the meta data) is module
specific. I don't think the best solution is to always store it in a
file or stream close to the binary. For text extraction, a separate
file may make sense, but probably not for 'virus scan' because that's
only a flag (you don't need the data). Thumbnails: for better
performance you want to keep them together, and not save them
separately (that is, in the data store).

Regards,
Thomas

Re: Workspace.copy() Question ...

[Jukka Zitting <ju...@gmail.com>]

Hi,

On Tue, Nov 11, 2008 at 10:06 AM, Thomas Müller <th...@day.com> wrote:
> It's an interesting use case, and probably quite common. It would be
> good if the text extraction would be run only once for each binary.
> However I'm not sure how this should be implemented... One solution is
> to extract the text in the data store, but that would be in the
> 'wrong' level.

An alternative would be to add an extra stream to binary
InternalValues. That stream (if present) would contain the result of
text extraction on the binary value and could then be used for
indexing.

In fact last week at the ApacheCon I was discussing with the Lucene
people about a way to store the analyzed token stream to further
optimize the re-indexing case. Apparently that should be possible with
little effort.

The problem with this is that we'd need to move the text extraction
functionality down to the persistence or item state layer. The current
configuration mechanism we have isn't too well adjusted for this and
things like using the value of the jcr:mimeType property to guide text
extraction might become quite tricky. But I don't see any fundamental
reason why those issues could not be resolved.

BR,

Jukka Zitting

AW: Workspace.copy() Question ...

[KÖLL Claus <C....@TIROL.GV.AT>]

hi thomas,

yes you are right its quite common ..
The approach with the DataIdentifier sounds good to me.
As you mentioned i don't know exactly what happens internally
if we try to copy a node.. will the binary be read and then again written
to the datastore ?

i think i will open a jira issue to write down the problems ..

greets
claus


-----Ursprüngliche Nachricht-----
Von: Thomas Müller [mailto:thomas.mueller@day.com] 
Gesendet: Dienstag, 11. November 2008 10:07
An: dev@jackrabbit.apache.org
Betreff: Re: Workspace.copy() Question ...

Hi,

> i have a nice usecase .. i have a filenode in my workspace and i should create
> about 70 copies of this node.
> its a not so small pdf file (10Mb) and i am using the datastore so its no problem
> the binary exists only one time but the problem is the textextractor. it will be called 70 times :-)
> is it possible to reuse the fulltext index on a copy operation without new reindexing the file ?

It's an interesting use case, and probably quite common. It would be
good if the text extraction would be run only once for each binary.
However I'm not sure how this should be implemented... One solution is
to extract the text in the data store, but that would be in the
'wrong' level.

What about this: the DataStore could return a special kind of
InputStream that allows to get the DataIdentifier
(DataStoreInputStream for example). The text extractor would then use
this unique identifier to ensure text for the same binary file is only
extracted once.

The same mechanism could be used to avoid copying binary data within
the same repository, and multiple repositories that share the same
data store: if the data store detects such an input stream it would
first check if the binary object already exists.

Regards,
Thomas

Re: Workspace.copy() Question ...

[Thomas Müller <th...@day.com>]

Hi,

> i have a nice usecase .. i have a filenode in my workspace and i should create
> about 70 copies of this node.
> its a not so small pdf file (10Mb) and i am using the datastore so its no problem
> the binary exists only one time but the problem is the textextractor. it will be called 70 times :-)
> is it possible to reuse the fulltext index on a copy operation without new reindexing the file ?

It's an interesting use case, and probably quite common. It would be
good if the text extraction would be run only once for each binary.
However I'm not sure how this should be implemented... One solution is
to extract the text in the data store, but that would be in the
'wrong' level.

What about this: the DataStore could return a special kind of
InputStream that allows to get the DataIdentifier
(DataStoreInputStream for example). The text extractor would then use
this unique identifier to ensure text for the same binary file is only
extracted once.

The same mechanism could be used to avoid copying binary data within
the same repository, and multiple repositories that share the same
data store: if the data store detects such an input stream it would
first check if the binary object already exists.

Regards,
Thomas