You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tika.apache.org by Jukka Zitting <ju...@gmail.com> on 2007/05/03 09:06:02 UTC
Tika discussions in Amsterdam
Hi,
Quick summary of the Tika discussions from yesterday's text analysis
BOF at the ApacheCon EU. It's the next morning now, so I'm probably
missing a lot of stuff...
* release early: The consensus seemed to be that we should do an early
release using one of the existing codebases (most likely the Nutch
parser framework or the Lius codebase) instead of trying to come up
with the "perfect design" up front before doing the first release. We
perhaps should do some initial work to avoid excessive API changes
later on, but that's a secondary consideration to releasing working
code early.
* processing phases: Tika has basically two main phases of operation:
content detection and content extraction. The content detection phase
tries to detect the content type of a document given a binary stream
and optional typing metadata. The main output from this phase is the
content type of the given document, but it is also possible to output
some easily accessible metadata (image size, etc.) already during this
phase. The content extraction phase is given the binary stream and the
detected typing information, and the expected output is the structured
text content and any available metadata from the document.
* processing pipeline: There was a quick idea on possibly organizing
the Tika framework as a pipeline of content detection and extraction
components.
* plugin framework: We should design Tika so that existing plugin
frameworks can easily be used to assemble and configure Tika
components.
* structured text: We should optimally use XHTML Basic as a SAX event
stream as the "structured text content" to be extracted from
documents. Namespaces can be used to extend the format to contain
extra metadata like PDF offsets for highlighting. Other options
expressed were using plain text, custom annotations or markup
mechanisms, full document formats like odf, or perhaps a DOM tree.
* additional processing components: It should be possible to embed
additional content extraction tools like thumbnail generators, image
index generators, etc. as plugins in the Tika framework. We might even
want to add support for tools like virus and spam detectors.
* container formats: It should be possible to use Tika recursively to
process container formats like zip files or many of the video formats.
The same mechanism could also be used to handle compressed or
encrypted files.
* integration: We should design the output format of Tika so that it
is easy to map to whatever Lucene, Solr, UIMA, and other similar
projects expect as input.
* security: Tika should contain some safeguards against
denial-of-service attacks that trick a parser library to spend excess
memory or processing power on a single document.
* parser libraries: We might want to encourage external parser
libraries to join us at the ASF, but Tika by itself should not try to
reimplement or compete with existing parsers.
BR,
Jukka Zitting
Re: Tika discussions in Amsterdam
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On 5/3/07, Thilo Goetz <tw...@gmx.de> wrote:
> One other thing that we discussed was that it would make sense for some
> input formats (such as html) if Tika could produce output that allows
> mapping back to the input. In other words, it should be possible
> (optionally) to know for each character in the output text where this
> character originated in the input. This is useful, for example, for
> result highlighting.
I think the best technical solution to this (assuming we use XHTML SAX
events) is to embed such backmapping information as namespaced
attributes in the output event stream. For example a PDF document
could result in something like this:
<html xmlns="...xhtml" xmlns:pdf="...tika-pdf-annotations">
<head>...</head>
<body>
<h1 pdf:location="...">...</h1>
<p pdf:location="...">...</p>
</body>
</html>
If more granularity is needed, the parser component could produce
extra <span/> elements for example for each line or even word in the
source document:
...<span pdf:location="...">...</span>...
> This may not be something for the early releases, but it would be good
> if we could keep this option in the back of our heads when designing the
> interfaces.
Agreed. I think a namespaced annotation mechanism like the one
suggested above would be an easy and forward-compatible way to add
such functionality.
BR,
Jukka Zitting
Re: Tika discussions in Amsterdam
Posted by Thilo Goetz <tw...@gmx.de>.
Jukka Zitting wrote:
> Hi,
>
> Quick summary of the Tika discussions from yesterday's text analysis
> BOF at the ApacheCon EU. It's the next morning now, so I'm probably
> missing a lot of stuff...
One other thing that we discussed was that it would make sense for some
input formats (such as html) if Tika could produce output that allows
mapping back to the input. In other words, it should be possible
(optionally) to know for each character in the output text where this
character originated in the input. This is useful, for example, for
result highlighting.
This may not be something for the early releases, but it would be good
if we could keep this option in the back of our heads when designing the
interfaces.
--Thilo
Re: Tika discussions in Amsterdam
Posted by Rida Benjelloun <ri...@doculibre.com>.
Hi,
My answer is in the text of the email
On 5/8/07, Jukka Zitting <ju...@gmail.com> wrote:
>
> Hi,
>
> On 5/3/07, Rida Benjelloun <ri...@doculibre.com> wrote:
> > Lius is currently under apache licence. If people are interested on it
> we
> > can use it as starting point for the development of tika.
>
> I think that would be great. We discussed in the ApacheCon that
> selecting a single existing codebase as the starting point would be
> the quickest way to bootstrap our efforts, and Lius and the Nutch
> parsers are probably the best candidates for this.
>
> The only downside in doing that is that it might cause trouble later
> on when we want to refactor things to be more general. For Lius the
> main problem is tight integration with Lucene. For example the
> lius.index.Indexer class imports the following:
>
> import org.apache.lucene.document.Document;
> import org.apache.lucene.document.Field;
> import org.apache.lucene.store.Directory;
> import org.apache.lucene.store.RAMDirectory ;
>
> Optimally the Tika toolkit should have no compile-time dependencies to
> Lucene. Do you think it would be feasible to refactor the Lius classes
> to avoid the Lucene dependencies?
This dependencies can easily be removed. The indexer class will be renamed
as a Parser class and will return document content and metadata etc.. Please
see the interface suggested at the end of this email
> Structured text: lius use JDOM, XPATH and namespaces for the extraction of
>
> > structured contents.
>
> Could you describe this in more detail. What does the XML content
> model look like? I could just look at the source, but it's more
> productive if we discuss the design on the mailing list.
Xml Parser use Namespace and XPATH to extract the content, if your document
is a dublin core (dc) document, the dc properties defined in the liusconfig
xml file will be used to extract the content. Tow type of extractions,
document extraction and xml node extraction.
XML document extraction allow you to store one XML document as a lucene
document. Node indexing allow you to store a document section as lucene
document. This can be interesting, for example if you want to index RSS
document and you want to store each news as an entry in the index.
Namespace allow you to apply the correct property for XML extraction.
Example of liusconfig :
LuceneField can be renamed field and all lucene specific properties can be
removed
<!-- Dubin core extraction properties -->
<xmlFile ns="http://purl.org/dc/elements/1.1/" setBoost="2.0">
<indexer class="lius.index.xml.XmlFileIndexer">
<mime>text/xml</mime>
</indexer>
<fields>
<luceneField name="title" xpathSelect="//dc:title"
type="Text" setBoost="0.1"/>
<luceneField name="subject" xpathSelect="//dc:subject"
type="Keyword" setBoost="2.0"/>
<luceneField name="creator" xpathSelect="//dc:creator"
type="Text"/>
</fields>
</xmlFile>
<!-- ETDMS extraction properties -->
<xmlFile ns="http://www.ndltd.org/standards/metadata/etdms/1.0/
">
<indexer class="lius.index.xml.XmlFileIndexer">
<mime>text/xml</mime>
</indexer>
<fields>
<luceneField name="title" xpathSelect="//etdms:title"
type="Text"/>
<luceneField name="subject"
xpathSelect="//etdms:subject" type="Keyword"/>
<luceneField name="creator"
xpathSelect="//etdms:creator" type="Text"/>
<luceneField name="description"
xpathSelect="//etdms:description" type="Text"/>
</fields>
</xmlFile>
<!-- no namespace in the document -->
<xmlFile ns="default" setBoost="0.5">
<indexer class="lius.index.xml.XmlFileIndexer">
<mime>text/xml</mime>
</indexer>
<fields>
<luceneField name="fullText" xpathSelect="//*"
type="Text" ocurSep=" "/>
</fields>
</xmlFile>
Lius use configuration to define the content extraction model .Each Indexer
contain a methode getPopulatedLiusFields witch return a collection of
Liusfields. Each Liusfield object contain the following information :
- Name
- Value (extracted from the xml document)
- Values (multiple values)
- XPath used to extract the information
- Some Lucene properties, like field type, boost etc. This properties can be
removed.
The config file must also be adapted to remove all lucene properties.
You can see Junit pakage in the source code to see somme code example .
Also here is a small tutorial :
http://www.doculibre.com/lius/doc-1.0_en.html
> Sax could be more powerful but does not offer XPATH for the extraction
> > of contents.
It's possible to transform a SAX stream into a DOM tree for easy XPath
> access so I don't think we lose any functionality by choosing SAX over
> a DOM model. In fact it is even possible to evaluate XPath expressions
> against a live SAX stream, you just won't get full DOM nodes as the
> results.
>
> > If you have have questions about Lius do not hesitate to communicate
> with
> > me. The source code is available: http://sourceforge.net/projects/lius/
>
> Why do you have the class files instead of the java files in Lius svn?
I don't know whats happend, this is an error, I will fix it. Thanks. You can
download the latest release in sourceforge.
BR,
>
> Jukka Zitting
I Suggest to use a parser interface that will give us all the information we
need regarding a document. The interface can look like
public interface Parser {
public String getContent(...);
public Collection getStructuredContent(); //This can be metadata or other
content that user whant to extract (using XPATH, Regex etc.)
public getLangage(...);
public getMimeType(...);
public String[] getOutLinks(...);
public int getSize(...) ;
public int getNbWords(...);
Etc ...
}
I'm sorry for my bad english, If you don't understand, please let me know.
BR,
Rida Benjelloun
Re: Tika discussions in Amsterdam
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On 5/3/07, Rida Benjelloun <ri...@doculibre.com> wrote:
> Lius is currently under apache licence. If people are interested on it we
> can use it as starting point for the development of tika.
I think that would be great. We discussed in the ApacheCon that
selecting a single existing codebase as the starting point would be
the quickest way to bootstrap our efforts, and Lius and the Nutch
parsers are probably the best candidates for this.
The only downside in doing that is that it might cause trouble later
on when we want to refactor things to be more general. For Lius the
main problem is tight integration with Lucene. For example the
lius.index.Indexer class imports the following:
import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field;
import org.apache.lucene.store.Directory;
import org.apache.lucene.store.RAMDirectory;
Optimally the Tika toolkit should have no compile-time dependencies to
Lucene. Do you think it would be feasible to refactor the Lius classes
to avoid the Lucene dependencies?
> Structured text: lius use JDOM, XPATH and namespaces for the extraction of
> structured contents.
Could you describe this in more detail. What does the XML content
model look like? I could just look at the source, but it's more
productive if we discuss the design on the mailing list.
> Sax could be more powerful but does not offer XPATH for the extraction
> of contents.
It's possible to transform a SAX stream into a DOM tree for easy XPath
access so I don't think we lose any functionality by choosing SAX over
a DOM model. In fact it is even possible to evaluate XPath expressions
against a live SAX stream, you just won't get full DOM nodes as the
results.
> If you have have questions about Lius do not hesitate to communicate with
> me. The source code is available: http://sourceforge.net/projects/lius/
Why do you have the class files instead of the java files in Lius svn?
BR,
Jukka Zitting
Re: Tika discussions in Amsterdam
Posted by Rida Benjelloun <ri...@doculibre.com>.
Hello,
Lius is currently under apache licence. If people are interested on it we
can use it as starting point for the development of tika.
I think that Lius could quickly be adapted to meet all the needs that Jakka
has mentioned in his email.
Processing phases: Lius makes it possible to detect the mimetype document
but also to extract the contents and the metadata. However we should develop
the extraction of the metadata for Word Excel and PowerPoint, we could use
POI to do this. We can also use one of the mimetype detector that Jukka has
proposed.
Structure plugin: Lius doesn't have plugin architecture, it will be
necessary to develop it . However all the parseurs can be configured using
an XML file
Structured text: lius use JDOM, XPATH and namespaces for the extraction of
structured contents. Sax could be more powerful but does not offer XPATH
for the extraction of contents. Lius support opendocument format.
Lius can process ZIP files, mp3 and MPEG.
Actually Lius is connected to Lucene but could easily adapted for Solr,
UIMA and others.
I think that Tika should also extract outlinks from documents.
If you have have questions about Lius do not hesitate to communicate with
me. The source code is available: http://sourceforge.net/projects/lius/
Best regards.
Rida Benjelloun
ridabenjelloun@apache.org
On 5/3/07, Jukka Zitting <ju...@gmail.com> wrote:
>
> Hi,
>
> Quick summary of the Tika discussions from yesterday's text analysis
> BOF at the ApacheCon EU. It's the next morning now, so I'm probably
> missing a lot of stuff...
>
> * release early: The consensus seemed to be that we should do an early
> release using one of the existing codebases (most likely the Nutch
> parser framework or the Lius codebase) instead of trying to come up
> with the "perfect design" up front before doing the first release. We
> perhaps should do some initial work to avoid excessive API changes
> later on, but that's a secondary consideration to releasing working
> code early.
>
> * processing phases: Tika has basically two main phases of operation:
> content detection and content extraction. The content detection phase
> tries to detect the content type of a document given a binary stream
> and optional typing metadata. The main output from this phase is the
> content type of the given document, but it is also possible to output
> some easily accessible metadata (image size, etc.) already during this
> phase. The content extraction phase is given the binary stream and the
> detected typing information, and the expected output is the structured
> text content and any available metadata from the document.
>
> * processing pipeline: There was a quick idea on possibly organizing
> the Tika framework as a pipeline of content detection and extraction
> components.
>
> * plugin framework: We should design Tika so that existing plugin
> frameworks can easily be used to assemble and configure Tika
> components.
>
> * structured text: We should optimally use XHTML Basic as a SAX event
> stream as the "structured text content" to be extracted from
> documents. Namespaces can be used to extend the format to contain
> extra metadata like PDF offsets for highlighting. Other options
> expressed were using plain text, custom annotations or markup
> mechanisms, full document formats like odf, or perhaps a DOM tree.
>
> * additional processing components: It should be possible to embed
> additional content extraction tools like thumbnail generators, image
> index generators, etc. as plugins in the Tika framework. We might even
> want to add support for tools like virus and spam detectors.
>
> * container formats: It should be possible to use Tika recursively to
> process container formats like zip files or many of the video formats.
> The same mechanism could also be used to handle compressed or
> encrypted files.
>
> * integration: We should design the output format of Tika so that it
> is easy to map to whatever Lucene, Solr, UIMA, and other similar
> projects expect as input.
>
> * security: Tika should contain some safeguards against
> denial-of-service attacks that trick a parser library to spend excess
> memory or processing power on a single document.
>
> * parser libraries: We might want to encourage external parser
> libraries to join us at the ASF, but Tika by itself should not try to
> reimplement or compete with existing parsers.
>
> BR,
>
> Jukka Zitting
>
Re: Tika discussions in Amsterdam
Posted by Lars Trieloff <la...@trieloff.net>.
Jukka mentioned yesterday that there is a database of MIME types
available at the freedesktop.org project. The relevant URLs are:
http://www.freedesktop.org/wiki/Software_2fshared_2dmime_2dinfo
- The Mime Type Database
http://www.freedesktop.org/wiki/
Specifications_2fshared_2dmime_2dinfo_2dspec
- Mime Type Database Specification
http://hedges.net/archives/2006/11/08/java-shared-mime-info/
- A Java implementation of this specification, alabeit GPL licensed
Lars