You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tika.apache.org by Jukka Zitting <ju...@gmail.com> on 2007/08/17 22:19:47 UTC
Tika use cases
Hi,
I was thinking about ways to best model the Tika interfaces, and it
seems to me that the only sane way to do that is to start with use
cases and how a client would most naturally use a toolkit like Tika.
Here are some of my initial ideas for review, feel free to add more
cases or suggest alternatives.
1) Extract structured text content from a stream (default configuration):
InputStream stream = ...;
ContentHandler handler = ...; // SAX event handler
new SomeTikaClass().parse(stream, handler);
2) Set configuration options:
SomeTikaClass tika = new SomeTikaClass();
tika.setConfigurationOption1(...);
tika.setConfigurationOption2(...);
// also composition, etc.
3) Extract metadata from a stream:
InputStream stream = ...;
Metadata metadata = new Metadata(); // Metadata container
new SomeTikaClass.parse(stream, metadata);
4) Provide external metadata as input for parsing:
InputStream stream = ...;
ContentHandler handler = ...;
Metadata metadata = new Metadata();
metadata.setFileName(...);
new SomeTikaClass.parse(stream, handler, metadata);
BR,
Jukka Zitting
Re: Tika use cases
Posted by Rida Benjelloun <ri...@doculibre.com>.
Hi Jukka,
You can create a prototype by implementing one or two parsers. This will
allow us to test it before implementing all parsers.
If you need some help, please don't hesitate.
Best Regards.
On 8/24/07, Jukka Zitting <ju...@gmail.com> wrote:
>
> Hi,
>
> On 8/24/07, Rida Benjelloun <ri...@doculibre.com> wrote:
> > I agree with your use case.
>
> Thanks!
>
> I've been thinking about this a bit more. The main thing I'm concerned
> about the current Parser classes from Lius Lite is that they always
> parse the entire document into an in-memory data structure. This can
> easily become a scalability issue and I'd like to avoid that already
> on the design level.
>
> Also, it seems to me that the current regexp and xpath features from
> Lius would work better as a layer on top of the parser code instead of
> as an integral part of it.
>
> As for my design proposal itself, I think I have a more workable
> approach to use cases 1 (extract structured content) and 3 (extract
> metadata). It looks like this:
>
> Extract metadata:
>
> InputStream stream = ...;
> Metadata metadata = new Metadata();
> SomeTikaInterface parser = new SomeTikaClass();
> parser.extractMetadata(stream, metadata);
>
> Extract structured content (and metadata as a side-effect):
>
> InputStream stream = ...;
> ContentHandler handler = ...; // SAX event handler
> Metadata metadata = new Metadata();
> SomeTikaInterface parser = new SomeTikaClass();
> parser.extractContent(stream, handler, metadata);
>
> In both cases it would be possible to feed existing metadata hints
> (like the file name, Content-Type header, or some other similar
> information) to the parser through the metadata argument.
>
> WDYT? I'd like to start going forward with some code along these
> lines, most likely by adapting/refactoring the Lius classes we already
> have.
>
> BR,
>
> Jukka Zitting
>
--
---------------------------------------------------------
Rida Benjelloun
Doculibre inc.
ridabenjelloun@apache.org
rida.benjelloun@doculibre.com
Cel: 418-262-3222
Tel: 418-353-3390
Site Web : http://www.doculibre.com
---------------------------------------------------------
Re: Tika use cases
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On 9/10/07, kbennett <kb...@bbsinc.biz> wrote:
> Thanks for responding. What you said made perfect sense. My domain
> knowledge in this area is very limited, so I apologize in advance for that.
No need to apologize. I don't consider being much of an expert myself
either, so feel free to dispute anything I say. :-)
> So a given parser (e.g. an MS Word document parser) might be instantiated at
> its first use with "global" options, that is, options for all parses, and
> then each call to extractMetadata would use that instance and be given
> file-specific options? So it might look something like this?:
Exactly! An even more concrete example would be:
// I want to extract metadata from a file I've been given
File file = ...;
// Construct a composite parser capable of parsing multiple document types
CompositeParser composite = new CompositeParser();
// Add support for MS Word documents
WordParser word = new WordParser();
word.setExtractDocumentProperties(false); // Nobody ever fills in these
word.setExtractDeletedContent(true); // I want all the secrets!
composite.addParser(word);
// Fill in all the metadata we already know
Metadata metadata = new Metadata();
metadata.assert("filename", file.getName(), Confidence.CERTAIN);
metadata.assert("content-length", file.length(), Confidence.CERTAIN);
// Extract metadata from the given file
InputStream stream = new FileInputStream(file);
try {
composite.extractMetadata(stream, metadata);
} finally {
stream.close();
}
Note that we might well want to include some convenience code to
streamline common options, but the above reflects my understanding of
a truly generic mechanism.
Some specific notes on the above example code, especially on parts I
haven't discussed before:
1) The interfaces as currently envisioned should work seamlessly with
composition and decoration. I think "compatibility" with such patterns
is highly desirable.
2) I'd like to extend the current Metadata framework from Nutch with
support for multiple (potentially conflicting) sources of information
with various confidence levels. See the above code for an early
example. Support for things like the Shared MIME info database also
require such "fuzzy" metadata.
3) After some consideration I think it's better if the parser
components would consume but never close the given input streams. IMHO
(feel free to disagree) the responsibility of closing a stream should
always be on whoever opened the stream in the first place.
BR,
Jukka Zitting
Re: Tika use cases
Posted by kbennett <kb...@bbsinc.biz>.
Jukka -
Thanks for responding. What you said made perfect sense. My domain
knowledge in this area is very limited, so I apologize in advance for that.
So a given parser (e.g. an MS Word document parser) might be instantiated at
its first use with "global" options, that is, options for all parses, and
then each call to extractMetadata would use that instance and be given
file-specific options? So it might look something like this?:
// in some parser factory class, named, say ParserFactory:
private SomeTikaInterface msWordParser;
SomeTikaInterface getMSWordParser() {
if msWordParser == null) {
msWordParser = new MSWordParser( /* the global config options */);
}
return msWordParser;
}
// ----------------- and then, where the actual parse needs to be done:
InputStream stream = ...;
Metadata metadata = new Metadata();
myParserFactoryInstance.getMSWordParser().extractMetadata(stream, metadata);
?
- Keith
Jukka Zitting wrote:
>
>
> There are really two kinds of options that could affect the way a
> parser would work. The first kind are generic options like the maximum
> amount of memory or time to use, the location of any temporary files
> to be used, etc. that don't have any direct relation to the specific
> document being parsed. The other kind are parsing hints related to the
> parsed document, like the name (and extension) of the file that
> contains the document, any MIME headers associated with the document
> (for example from a HTTP request or an email body part), etc.
>
--
View this message in context: http://www.nabble.com/Tika-use-cases-tf4287938.html#a12599158
Sent from the Apache Tika - Development mailing list archive at Nabble.com.
Re: Tika use cases
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On 9/10/07, kbennett <kb...@bbsinc.biz> wrote:
> It seems to me that options going into the parser are logically different
> from metadata coming out of the parser, and that to maximize the code's
> cohesion (see http://en.wikipedia.org/wiki/Cohesion_%28computer_science%29),
> it would be preferable to express them as two different objects.
There are really two kinds of options that could affect the way a
parser would work. The first kind are generic options like the maximum
amount of memory or time to use, the location of any temporary files
to be used, etc. that don't have any direct relation to the specific
document being parsed. The other kind are parsing hints related to the
parsed document, like the name (and extension) of the file that
contains the document, any MIME headers associated with the document
(for example from a HTTP request or an email body part), etc.
The first kind of options I'd really handle separately as JavaBean
properties or some such of the parser instances, but the second kind
is actually more or less accurate metadata about the document in
question, so IMHO it would make perfect sense to pass that information
as a part of the metadata argument.
> Also, if the metadata is the only output of the parser (as it appears to be
> in the use case), why not have the parser create the metadata object itself,
> and return it as the return value? This would seem like a more natural
> interface.
As mentioned above, I think the metadata object could (and should) be
used to pass various parsing "hints" to the parser, and that the
parser can then extend, verify, or correct the given metadata. This
approach also allows one to have a sequence of parsers that
incrementally extract more and more information from the input
document.
BR,
Jukka Zitting
Re: Tika use cases
Posted by kbennett <kb...@bbsinc.biz>.
Hi, all. I'm new here, so if I don't know what I'm talking about, feel free
to correct me. :)
It seems to me that options going into the parser are logically different
from metadata coming out of the parser, and that to maximize the code's
cohesion (see http://en.wikipedia.org/wiki/Cohesion_%28computer_science%29),
it would be preferable to express them as two different objects.
Also, if the metadata is the only output of the parser (as it appears to be
in the use case), why not have the parser create the metadata object itself,
and return it as the return value? This would seem like a more natural
interface.
So, using this approach, the code would look something like this:
InputStream stream = ...;
ParseOptions parseOptions = ...
SomeTikaInterface parser = new SomeTikaClass();
Metadata metadata = parser.extractMetadata(stream, options);...
... or, alternatively, the ParseOptions might be used to instantiate the
parser instead of being passed to the extractMetadata() method.
- Keith
Jukka Zitting wrote:
>
> Hi,
>
> On 8/25/07, Bertrand Delacretaz <bd...@apache.org> wrote:
>> On 8/24/07, Jukka Zitting <ju...@gmail.com> wrote:
>> > ...Extract metadata:
>> >
>> > InputStream stream = ...;
>> > Metadata metadata = new Metadata();
>> > SomeTikaInterface parser = new SomeTikaClass();
>> > parser.extractMetadata(stream, metadata);...
>>
>> Maybe this (and extractContent() as well) need an additional
>> TikaParseOptions parameter that sets options just for this parsing
>> call?
>
> Good point, though we could also pass all such options as a part of
> the metadata argument. If the options affect just this one document,
> then I would argue that those options might as well be a part of the
> document-specific metadata.
>
> More generic options, like the XML parser options to use when parsing
> application/xml documents, should probably be handled as JavaBean
> properties of the instantiated parser objects.
>
> BR,
>
> Jukka Zitting
>
>
--
View this message in context: http://www.nabble.com/Tika-use-cases-tf4287938.html#a12596742
Sent from the Apache Tika - Development mailing list archive at Nabble.com.
Re: Tika use cases
Posted by Bertrand Delacretaz <bd...@apache.org>.
On 8/25/07, Jukka Zitting <ju...@gmail.com> wrote:
> On 8/25/07, Bertrand Delacretaz <bd...@apache.org> wrote:
> ....> Maybe this (and extractContent() as well) need an additional
> > TikaParseOptions parameter that sets options just for this parsing
> > call?
>
> Good point, though we could also pass all such options as a part of
> the metadata argument....
Agreed, this is workable as well.
> ...More generic options, like the XML parser options to use when parsing
> application/xml documents, should probably be handled as JavaBean
> properties of the instantiated parser objects....
Sure, we'll probably need options both at the parser and at the
extract method levels.
-Bertrand
Re: Tika use cases
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On 8/25/07, Bertrand Delacretaz <bd...@apache.org> wrote:
> On 8/24/07, Jukka Zitting <ju...@gmail.com> wrote:
> > ...Extract metadata:
> >
> > InputStream stream = ...;
> > Metadata metadata = new Metadata();
> > SomeTikaInterface parser = new SomeTikaClass();
> > parser.extractMetadata(stream, metadata);...
>
> Maybe this (and extractContent() as well) need an additional
> TikaParseOptions parameter that sets options just for this parsing
> call?
Good point, though we could also pass all such options as a part of
the metadata argument. If the options affect just this one document,
then I would argue that those options might as well be a part of the
document-specific metadata.
More generic options, like the XML parser options to use when parsing
application/xml documents, should probably be handled as JavaBean
properties of the instantiated parser objects.
BR,
Jukka Zitting
Re: Tika use cases
Posted by Bertrand Delacretaz <bd...@apache.org>.
On 8/24/07, Jukka Zitting <ju...@gmail.com> wrote:
> ...Extract metadata:
>
> InputStream stream = ...;
> Metadata metadata = new Metadata();
> SomeTikaInterface parser = new SomeTikaClass();
> parser.extractMetadata(stream, metadata);...
Maybe this (and extractContent() as well) need an additional
TikaParseOptions parameter that sets options just for this parsing
call?
This might come later though, and only if really needed.
Apart from that I like the suggested API.
-Bertrand
Re: Tika use cases
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On 8/24/07, Rida Benjelloun <ri...@doculibre.com> wrote:
> I agree with your use case.
Thanks!
I've been thinking about this a bit more. The main thing I'm concerned
about the current Parser classes from Lius Lite is that they always
parse the entire document into an in-memory data structure. This can
easily become a scalability issue and I'd like to avoid that already
on the design level.
Also, it seems to me that the current regexp and xpath features from
Lius would work better as a layer on top of the parser code instead of
as an integral part of it.
As for my design proposal itself, I think I have a more workable
approach to use cases 1 (extract structured content) and 3 (extract
metadata). It looks like this:
Extract metadata:
InputStream stream = ...;
Metadata metadata = new Metadata();
SomeTikaInterface parser = new SomeTikaClass();
parser.extractMetadata(stream, metadata);
Extract structured content (and metadata as a side-effect):
InputStream stream = ...;
ContentHandler handler = ...; // SAX event handler
Metadata metadata = new Metadata();
SomeTikaInterface parser = new SomeTikaClass();
parser.extractContent(stream, handler, metadata);
In both cases it would be possible to feed existing metadata hints
(like the file name, Content-Type header, or some other similar
information) to the parser through the metadata argument.
WDYT? I'd like to start going forward with some code along these
lines, most likely by adapting/refactoring the Lius classes we already
have.
BR,
Jukka Zitting
Re: Tika use cases
Posted by Rida Benjelloun <ri...@doculibre.com>.
Hi Jukka,
I agree with your use case.
+1.
Regards.
On 8/17/07, Jukka Zitting <ju...@gmail.com> wrote:
>
> Hi,
>
> I was thinking about ways to best model the Tika interfaces, and it
> seems to me that the only sane way to do that is to start with use
> cases and how a client would most naturally use a toolkit like Tika.
> Here are some of my initial ideas for review, feel free to add more
> cases or suggest alternatives.
>
> 1) Extract structured text content from a stream (default configuration):
>
> InputStream stream = ...;
> ContentHandler handler = ...; // SAX event handler
> new SomeTikaClass().parse(stream, handler);
>
> 2) Set configuration options:
>
> SomeTikaClass tika = new SomeTikaClass();
> tika.setConfigurationOption1(...);
> tika.setConfigurationOption2(...);
> // also composition, etc.
>
> 3) Extract metadata from a stream:
>
> InputStream stream = ...;
> Metadata metadata = new Metadata(); // Metadata container
> new SomeTikaClass.parse(stream, metadata);
>
> 4) Provide external metadata as input for parsing:
>
> InputStream stream = ...;
> ContentHandler handler = ...;
> Metadata metadata = new Metadata();
> metadata.setFileName(...);
> new SomeTikaClass.parse(stream, handler, metadata);
>
> BR,
>
> Jukka Zitting
>
--
---------------------------------------------------------
Rida Benjelloun
Doculibre inc.
ridabenjelloun@apache.org
rida.benjelloun@doculibre.com
Cel: 418-262-3222
Tel: 418-353-3390
Site Web : http://www.doculibre.com
---------------------------------------------------------