You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tika.apache.org by ma...@apache.org on 2016/11/11 00:21:16 UTC
svn commit: r1769239 [7/15] - in /tika/site/publish: ./ 0.10/ 0.5/ 0.6/ 0.7/
0.8/ 0.9/ 1.0/ 1.1/ 1.10/ 1.11/ 1.12/ 1.13/ 1.14/ 1.2/ 1.3/ 1.4/ 1.5/ 1.6/
1.7/ 1.8/ 1.8/miredot/font/Droid_Sans/ 1.8/miredot/font/Droid_Sans_Mono/
1.8/miredot/font/Open_Sans/...
Added: tika/site/publish/1.14/configuring.html
URL: http://svn.apache.org/viewvc/tika/site/publish/1.14/configuring.html?rev=1769239&view=auto
==============================================================================
--- tika/site/publish/1.14/configuring.html (added)
+++ tika/site/publish/1.14/configuring.html Fri Nov 11 00:21:15 2016
@@ -0,0 +1,413 @@
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+ "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+
+
+
+
+
+
+<html xmlns="http://www.w3.org/1999/xhtml">
+ <head>
+ <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
+ <title>Apache Tika – Configuring Tika</title>
+ <style type="text/css" media="all">
+ @import url("../css/site.css");
+ </style>
+ <link rel="icon" type="image/png" href="../tikaNoText16.png" />
+ <script type="text/javascript">
+ function selectProvider(form) {
+ provider = form.elements['searchProvider'].value;
+ if (provider == "any") {
+ if (Math.random() > 0.5) {
+ provider = "lucid";
+ } else {
+ provider = "sl";
+ }
+ }
+ if (provider == "lucid") {
+ form.action = "http://find.searchhub.org/p:tika";
+ } else if (provider == "sl") {
+ form.action = "http://search-lucene.com/tika";
+ }
+ days = 90;
+ date = new Date();
+ date.setTime(date.getTime() + (days * 24 * 60 * 60 * 1000));
+ expires = "; expires=" + date.toGMTString();
+ document.cookie = "searchProvider=" + provider + expires + "; path=/";
+ }
+ function initProvider() {
+ if (document.cookie.length>0) {
+ cStart=document.cookie.indexOf("searchProvider=");
+ if (cStart!=-1) {
+ cStart=cStart + "searchProvider=".length;
+ cEnd=document.cookie.indexOf(";", cStart);
+ if (cEnd==-1) {
+ cEnd=document.cookie.length;
+ }
+ provider = unescape(document.cookie.substring(cStart,cEnd));
+ document.forms['searchform'].elements['searchProvider'].value = provider;
+ }
+ }
+ document.forms['searchform'].elements['q'].focus();
+ }
+ </script>
+ </head>
+ <body onLoad="initProvider();">
+ <div id="body">
+ <div id="banner">
+ <a href="http://tika.apache.org" id="bannerLeft" title="Apache Tika"
+ ><img src="http://tika.apache.org/tika.png" alt="Apache Tika"
+ width="292" height="100"/></a>
+ <a href="http://www.apache.org/" id="bannerRight"
+ title="The Apache Software Foundation"
+ ><img src="http://tika.apache.org/asf-logo.gif" alt="The Apache Software Foundation"
+ width="387" height="100"/></a>
+ </div>
+ <div id="content">
+ <!-- Licensed to the Apache Software Foundation (ASF) under one or more --><!-- contributor license agreements. See the NOTICE file distributed with --><!-- this work for additional information regarding copyright ownership. --><!-- The ASF licenses this file to You under the Apache License, Version 2.0 --><!-- (the "License"); you may not use this file except in compliance with --><!-- the License. You may obtain a copy of the License at --><!-- --><!-- http://www.apache.org/licenses/LICENSE-2.0 --><!-- --><!-- Unless required by applicable law or agreed to in writing, software --><!-- distributed under the License is distributed on an "AS IS" BASIS, --><!-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. --><!-- See the License for the specific language governing permissions and --><!-- limitations under the License. --><div class="section">
+<h2><a name="Configuring_Tika"></a>Configuring Tika</h2>
+<p>Out of the box, Apache Tika will attempt to start with all available Detectors and Parsers, running with sensible defaults. For most users, this default configuration will work well.</p>
+<p>This page gives you information on how to configure the various components of Apache Tika, such as Parsers and Detectors, if you need fine-grained control over ordering, exclusions and the like.</p>
+<ul>
+<li><a href="#Configuring_Tika">Configuring Tika</a>
+<ul>
+<li><a href="#Configuring_Parsers">Configuring Parsers</a></li>
+<li><a href="#Configuring_Detectors">Configuring Detectors</a></li>
+<li><a href="#Configuring_Mime_Types">Configuring Mime Types</a></li>
+<li><a href="#Configuring_Language_Identifiers">Configuring Language Identifiers</a></li>
+<li><a href="#Configuring_Translators">Configuring Translators</a></li>
+<li><a href="#Configuring_the_Service_Loader">Configuring the Service Loader</a>
+<ul>
+<li><a href="#Static">Static</a></li>
+<li><a href="#Dynamic">Dynamic</a></li>
+<li><a href="#Load_Error_Handling">Load Error Handling</a></li></ul></li>
+<li><a href="#Using_a_Tika_Configuration_XML_file">Using a Tika Configuration XML file</a></li></ul></li></ul>
+<div class="section">
+<h3><a name="Configuring_Parsers">Configuring Parsers</a></h3>
+<p>Through the Tika Config xml, it is possible to have a high degree of control over which parsers are or aren't used, in what order of preferences etc. It is also possible to override just certain parts, to (for example) have "default except for PDF".</p>
+<p>Currently, it is only possible to have a single parser run against a document. There is on-going discussion around fallback parsers and combining the output of multiple parsers running on a document, but none of these are available yet.</p>
+<p>To override some parser certain default behaviours, include the <tt> DefaultParser </tt> in your configuration, with excludes, then add other parser definitions in. To prevent the <tt> DefaultParser </tt> (with its auto-discovery) being used, simply omit it from your config, and list all other parsers you want instead.</p>
+<p>To override just some default behaviour, you can use a Tika Config something like this:</p>
+<div>
+<pre><?xml version="1.0" encoding="UTF-8"?>
+<properties>
+ <parsers>
+ <!-- Default Parser for most things, except for 2 mime types, and never
+ use the Executable Parser -->
+ <parser class="org.apache.tika.parser.DefaultParser">
+ <mime-exclude>image/jpeg</mime-exclude>
+ <mime-exclude>application/pdf</mime-exclude>
+ <parser-exclude class="org.apache.tika.parser.executable.ExecutableParser"/>
+ </parser>
+ <!-- Use a different parser for PDF -->
+ <parser class="org.apache.tika.parser.EmptyParser">
+ <mime>application/pdf</mime>
+ </parser>
+ </parsers>
+</properties></pre></div>
+<p>To configure things in code, the key classes to use to build up your own custom parser heirarchy are <a href="./api/org/apache/tika/parser/DefaultParser.html">org.apache.tika.parser.DefaultParser</a>, <a href="./api/org/apache/tika/parser/CompositeParser.html">org.apache.tika.parser.CompositeParser</a> and <a href="./api/org/apache/tika/parser/ParserDecorator.html">org.apache.tika.parser.ParserDecorator</a>.</p></div>
+<div class="section">
+<h3><a name="Configuring_Detectors">Configuring Detectors</a></h3>
+<p>Through the Tika Config xml, it is possible to have a high degree of control over which detectors are or aren't used, in what order of preferences etc. It is also possible to override just certain parts, to (for example) have "default except for no POIFS Container Detction".</p>
+<p>To override some detector certain default behaviours, include the <tt> DefaultDetector </tt>, with any <tt> detector-exclude </tt> entries you need, in your configuration, then add other detectors definitions in. To prevent the <tt> DefaultParser </tt> (with its auto-discovery) being used, simply omit it from your config, and list all other detectors you want instead.</p>
+<p>To override just some default behaviour, you can use a Tika Config something like this:</p>
+<div>
+<pre><?xml version="1.0" encoding="UTF-8"?>
+<properties>
+ <detectors>
+ <!-- All detectors except built-in container ones -->
+ <detector class="org.apache.tika.detect.DefaultDetector">
+ <detector-exclude class="org.apache.tika.parser.pkg.ZipContainerDetector"/>
+ <detector-exclude class="org.apache.tika.parser.microsoft.POIFSContainerDetector"/>
+ </detector>
+ </detectors>
+</properties></pre></div>
+<p>Or to just only use certain detectors, you can use a Tika Config something like this:</p>
+<div>
+<pre><?xml version="1.0" encoding="UTF-8"?>
+<properties>
+ <detectors>
+ <!-- Only use these two detectors, and ignore all others -->
+ <detector class="org.apache.tika.parser.pkg.ZipContainerDetector"/>
+ <detector class="org.apache.tika.mime.MimeTypes"/>
+ </detectors>
+</properties></pre></div>
+<p>In code, the key classes to use to build up your own custom detector heirarchy are <a href="./api/org/apache/tika/detect/DefaultDetector.html">org.apache.tika.detect.DefaultDetector</a> and <a href="./api/org/apache/tika/detect/CompositeDetector.html">org.apache.tika.detect.CompositeDetector</a>.</p></div>
+<div class="section">
+<h3><a name="Configuring_Mime_Types">Configuring Mime Types</a></h3>
+<p>TODO Mention non-standard paths, and custom mime type files</p></div>
+<div class="section">
+<h3><a name="Configuring_Language_Identifiers">Configuring Language Identifiers</a></h3>
+<p>At this time, there is no unified way to configure language identifiers. While the work on that is ongoing, for now you will need to review the <a href="./api/">Tika Javadocs</a> to see how individual identifiers are configured.</p></div>
+<div class="section">
+<h3><a name="Configuring_Translators">Configuring Translators</a></h3>
+<p>At this time, there is no unified way to configure Translators. While the work on that is ongoing, for now you will need to review the <a href="./api/">Tika Javadocs</a> to see how individual Translators are configured.</p><!-- When Translators can have their parameters configured, mention here about --><!-- specifying which single one to use in the Tika Config XML --></div>
+<div class="section">
+<h3><a name="Configuring_the_Service_Loader">Configuring the Service Loader</a></h3>
+<p>Tika has a number of service provider types such as parsers, detectors, and translators. The <a href="./api/org/apache/tika/config/ServiceLoader.html">org.apache.tika.config.ServiceLoader</a> class provides a registry of each type of provider. This allows Tika to create implementations such as <a href="./api/org/apache/tika/parser/DefaultParser.html">org.apache.tika.parser.DefaultParser</a>, <a href="./api/org/apache/tika/language/translate/DefaultTranslator.html">org.apache.tika.language.translate.DefaultTranslator</a>, and <a href="./api/org/apache/tika/detect/DefaultDetector.html">org.apache.tika.detect.DefaultDetector</a> that can match the appropriate provider to an incoming piece of content.</p>
+<p>The ServiceLoader's registry can be populated either statically or dynamically.</p>
+<div class="section">
+<h4><a name="Static"></a>Static</h4>
+<p>Static loading is the default which requires no configuration. This configuration options is used in Tika deployments where the Tika JAR files reside together in the same classloader hierarchy. The services provides are loaded from provider configuration files located within the tika-parsers JAR file at META-INF/services.</p></div>
+<div class="section">
+<h4><a name="Dynamic"></a>Dynamic</h4>
+<p>Dynamic loading may be required if the tika service providers will reside in different classloaders such as in OSGi. To allow a provider created in tika-config.xml to utilize dynamically loaded services you need to configure the ServiceLoader to be dynamic with the following configuration:</p>
+<div>
+<pre><properties>
+ <service-loader dynamic="true"/>
+ ....
+</properties></pre></div></div>
+<div class="section">
+<h4><a name="Load_Error_Handling"></a>Load Error Handling</h4>
+<p>The ServiceLoader can contains a handler to deal with errors that occur during provider initialization. For example if a class fails to initialize LoadErrorHandler deals with the exception that is thrown. This handler can be configured to:</p>
+<ul>
+<li><tt> IGNORE </tt> - (Default) Do nothing when providers fail to initialize.</li>
+<li><tt> WARN </tt> - Log a warning when providers fail to initialize.</li>
+<li><tt> THROW </tt> - Throw an exception when providers fail to initialize.</li></ul>
+<p>For example to set the LoadErrorHandler to WARN then use the following configuration:</p>
+<div>
+<pre><properties>
+ <service-loader loadErrorHandler="WARN"/>
+ ....
+</properties></pre></div></div></div>
+<div class="section">
+<h3><a name="Using_a_Tika_Configuration_XML_file">Using a Tika Configuration XML file</a></h3>
+<p>However you call Tika, the System Property of <tt> tika.config </tt> is checked first, and the Environment Variable of <tt> TIKA_CONFIG </tt> is tried next. Setting one of those will cause Tika to use your given Tika Config XML file.</p>
+<p>If you are calling Tika from your own code, then you can pass in the location of your Tika Config XML file when you construct your <tt>TikaConfig</tt> instance. From that, you can fetch your configured parser, detectors etc.</p>
+<div>
+<pre>TikaConfig config = new TikaConfig("/path/to/tika-config.xml");
+Detector detector = config.getDetector();
+Parser autoDetectParser = new AutoDetectParser(config);</pre></div>
+<p>For users of the Tika App, in addition to the sytem property and the environement variable, you can also use the <tt> --config=[tika-config.xml] </tt> option to select a different Tika Config XML file to use</p>
+<p>For users of the Tika Server, in addition to the sytem property and the environement variable, you can also use <tt> -c [tika-config.xml] </tt> or <tt> --config [tika-config.xml] </tt> options to select a different Tika Config XML file to use</p></div></div>
+ </div>
+ <div id="sidebar">
+ <div id="navigation">
+ <h5>Apache Tika</h5>
+ <ul>
+
+ <li class="none">
+ <a href="../index.html">Introduction</a>
+ </li>
+
+ <li class="none">
+ <a href="../download.html">Download</a>
+ </li>
+
+ <li class="none">
+ <a href="../contribute.html">Contribute</a>
+ </li>
+
+ <li class="none">
+ <a href="../mail-lists.html">Mailing Lists</a>
+ </li>
+
+ <li class="none">
+ <a href="http://wiki.apache.org/tika/" class="externalLink">Tika Wiki</a>
+ </li>
+
+ <li class="none">
+ <a href="https://issues.apache.org/jira/browse/TIKA" class="externalLink">Issue Tracker</a>
+ </li>
+ </ul>
+ <h5>Documentation</h5>
+ <ul>
+
+
+
+
+
+
+
+
+
+
+
+
+ <li class="expanded">
+ <a href="../1.14/index.html">Apache Tika 1.14</a>
+ <ul>
+
+ <li class="none">
+ <a href="../1.14/gettingstarted.html">Getting Started</a>
+ </li>
+
+ <li class="none">
+ <a href="../1.14/formats.html">Supported Formats</a>
+ </li>
+
+ <li class="none">
+ <a href="../1.14/parser.html">Parser API</a>
+ </li>
+
+ <li class="none">
+ <a href="../1.14/parser_guide.html">Parser 5min Quick Start Guide</a>
+ </li>
+
+ <li class="none">
+ <a href="../1.14/detection.html">Content and Language Detection</a>
+ </li>
+
+ <li class="none">
+ <strong>Configuring Tika</strong>
+ </li>
+
+ <li class="none">
+ <a href="../1.14/examples.html">Usage Examples</a>
+ </li>
+
+ <li class="none">
+ <a href="../1.14/api/">API Documentation</a>
+ </li>
+
+ <li class="none">
+ <a href="../1.14/miredot/">REST API Documentation (Miredot)</a>
+ </li>
+ </ul>
+ </li>
+
+
+
+
+
+
+
+
+
+
+
+
+ <li class="collapsed">
+ <a href="../1.13/index.html">Apache Tika 1.13</a>
+ </li>
+
+
+
+
+
+
+
+
+
+
+
+
+ <li class="collapsed">
+ <a href="../1.12/index.html">Apache Tika 1.12</a>
+ </li>
+
+
+
+
+
+
+
+
+
+
+
+
+ <li class="collapsed">
+ <a href="../1.11/index.html">Apache Tika 1.11</a>
+ </li>
+
+
+
+
+
+
+
+
+
+
+
+
+ <li class="collapsed">
+ <a href="../1.10/index.html">Apache Tika 1.10</a>
+ </li>
+ </ul>
+ <h5>The Apache Software Foundation</h5>
+ <ul>
+
+ <li class="none">
+ <a href="http://www.apache.org/foundation/" class="externalLink">About</a>
+ </li>
+
+ <li class="none">
+ <a href="http://www.apache.org/licenses/" class="externalLink">License</a>
+ </li>
+
+ <li class="none">
+ <a href="http://www.apache.org/security/" class="externalLink">Security</a>
+ </li>
+
+ <li class="none">
+ <a href="http://www.apache.org/foundation/sponsorship.html" class="externalLink">Sponsorship</a>
+ </li>
+
+ <li class="none">
+ <a href="http://www.apache.org/foundation/thanks.html" class="externalLink">Thanks</a>
+ </li>
+ </ul>
+
+ <div id="search">
+ <h5>Search with Apache Solr</h5>
+ <form action="http://search.lucidimagination.com/p:tika"
+ method="get" id="searchform">
+ <input type="text" id="query" name="q"/>
+ <select name="searchProvider" id="searchProvider">
+ <option value="any">provider</option>
+ <option value="lucid">Lucid Find</option>
+ <option value="sl">Search-Lucene</option>
+ </select>
+ <input type="submit" id="submit" value="Search" name="Search"
+ onclick="selectProvider(this.form)"/>
+ </form>
+ </div>
+
+ <div id="bookpromo">
+ <h5>Books about Tika</h5>
+ <p>
+ <a href="http://manning.com/mattmann/" title="Tika in Action"
+ ><img src="../mattmann_cover150.jpg"
+ width="150" height="186"/></a>
+ </p>
+ </div>
+ </div>
+ </div>
+ <div id="footer">
+ <p>
+ Copyright © 2016
+ <a href="http://www.apache.org/">The Apache Software Foundation</a>.
+ Site powered by <a href="http://maven.apache.org/">Apache Maven</a>.
+ Search powered by
+ <a href="http://www.lucidimagination.com">Lucid Imagination</a>
+ and <a href="http://sematext.com">Sematext</a>.
+ <br/>
+ Apache Tika, Tika, Apache, the Apache feather logo, and the Apache
+ Tika project logo are trademarks of The Apache Software Foundation.
+ </p>
+ </div>
+ </div>
+ </body>
+</html>
Added: tika/site/publish/1.14/detection.html
URL: http://svn.apache.org/viewvc/tika/site/publish/1.14/detection.html?rev=1769239&view=auto
==============================================================================
--- tika/site/publish/1.14/detection.html (added)
+++ tika/site/publish/1.14/detection.html Fri Nov 11 00:21:15 2016
@@ -0,0 +1,376 @@
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+ "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+
+
+
+
+
+
+<html xmlns="http://www.w3.org/1999/xhtml">
+ <head>
+ <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
+ <title>Apache Tika – Content Detection</title>
+ <style type="text/css" media="all">
+ @import url("../css/site.css");
+ </style>
+ <link rel="icon" type="image/png" href="../tikaNoText16.png" />
+ <script type="text/javascript">
+ function selectProvider(form) {
+ provider = form.elements['searchProvider'].value;
+ if (provider == "any") {
+ if (Math.random() > 0.5) {
+ provider = "lucid";
+ } else {
+ provider = "sl";
+ }
+ }
+ if (provider == "lucid") {
+ form.action = "http://find.searchhub.org/p:tika";
+ } else if (provider == "sl") {
+ form.action = "http://search-lucene.com/tika";
+ }
+ days = 90;
+ date = new Date();
+ date.setTime(date.getTime() + (days * 24 * 60 * 60 * 1000));
+ expires = "; expires=" + date.toGMTString();
+ document.cookie = "searchProvider=" + provider + expires + "; path=/";
+ }
+ function initProvider() {
+ if (document.cookie.length>0) {
+ cStart=document.cookie.indexOf("searchProvider=");
+ if (cStart!=-1) {
+ cStart=cStart + "searchProvider=".length;
+ cEnd=document.cookie.indexOf(";", cStart);
+ if (cEnd==-1) {
+ cEnd=document.cookie.length;
+ }
+ provider = unescape(document.cookie.substring(cStart,cEnd));
+ document.forms['searchform'].elements['searchProvider'].value = provider;
+ }
+ }
+ document.forms['searchform'].elements['q'].focus();
+ }
+ </script>
+ </head>
+ <body onLoad="initProvider();">
+ <div id="body">
+ <div id="banner">
+ <a href="http://tika.apache.org" id="bannerLeft" title="Apache Tika"
+ ><img src="http://tika.apache.org/tika.png" alt="Apache Tika"
+ width="292" height="100"/></a>
+ <a href="http://www.apache.org/" id="bannerRight"
+ title="The Apache Software Foundation"
+ ><img src="http://tika.apache.org/asf-logo.gif" alt="The Apache Software Foundation"
+ width="387" height="100"/></a>
+ </div>
+ <div id="content">
+ <!-- Licensed to the Apache Software Foundation (ASF) under one or more --><!-- contributor license agreements. See the NOTICE file distributed with --><!-- this work for additional information regarding copyright ownership. --><!-- The ASF licenses this file to You under the Apache License, Version 2.0 --><!-- (the "License"); you may not use this file except in compliance with --><!-- the License. You may obtain a copy of the License at --><!-- --><!-- http://www.apache.org/licenses/LICENSE-2.0 --><!-- --><!-- Unless required by applicable law or agreed to in writing, software --><!-- distributed under the License is distributed on an "AS IS" BASIS, --><!-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. --><!-- See the License for the specific language governing permissions and --><!-- limitations under the License. --><div class="section">
+<h2><a name="Content_Detection"></a>Content Detection</h2>
+<p>This page gives you information on how content and language detection works with Apache Tika, and how to tune the behaviour of Tika.</p>
+<ul>
+<li><a href="#Content_Detection">Content Detection</a>
+<ul>
+<li><a href="#The_Detector_Interface">The Detector Interface</a></li>
+<li><a href="#Mime_Magic_Detection">Mime Magic Detection</a></li>
+<li><a href="#Resource_Name_Based_Detection">Resource Name Based Detection</a></li>
+<li><a href="#Known_Content_Type_Detection">Known Content Type "Detection</a></li>
+<li><a href="#The_default_Mime_Types_Detector">The default Mime Types Detector</a></li>
+<li><a href="#Container_Aware_Detection">Container Aware Detection</a></li>
+<li><a href="#The_default_Tika_Detector">The default Tika Detector</a></li>
+<li><a href="#Ways_of_triggering_Detection">Ways of triggering Detection</a></li>
+<li><a href="#Language_Detection">Language Detection</a></li>
+<li><a href="#More_Examples">More Examples</a></li></ul></li></ul>
+<div class="section">
+<h3><a name="The_Detector_Interface">The Detector Interface</a></h3>
+<p>The <a href="./api/org/apache/tika/detect/Detector.html">org.apache.tika.detect.Detector</a> interface is the basis for most of the content type detection in Apache Tika. All the different ways of detecting content all implement the same common method:</p>
+<div>
+<pre>MediaType detect(java.io.InputStream input,
+ Metadata metadata) throws java.io.IOException</pre></div>
+<p>The <tt>detect</tt> method takes the stream to inspect, and a <tt>Metadata</tt> object that holds any additional information on the content. The detector will return a <a href="./api/org/apache/tika/mime/MediaType.html">MediaType</a> object describing its best guess as to the type of the file.</p>
+<p>In general, only two keys on the Metadata object are used by Detectors. These are <tt>Metadata.RESOURCE_NAME_KEY</tt> which should hold the name of the file (where known), and <tt>Metadata.CONTENT_TYPE</tt> which should hold the advertised content type of the file (eg from a webserver or a content repository).</p></div>
+<div class="section">
+<h3><a name="Mime_Magic_Detection">Mime Magic Detection</a></h3>
+<p>By looking for special ("magic") patterns of bytes near the start of the file, it is often possible to detect the type of the file. For some file types, this is a simple process. For others, typically container based formats, the magic detection may not be enough. (More detail on detecting container formats below)</p>
+<p>Tika is able to make use of a a mime magic info file, in the <a class="externalLink" href="http://www.freedesktop.org/standards/shared-mime-info">Freedesktop MIME-info</a> format to peform mime magic detection. (Note that Tika supports a few more match types than Freedesktop does)</p>
+<p>This is provided within Tika by <a href="./api/org/apache/tika/detect/MagicDetector.html">org.apache.tika.detect.MagicDetector</a>. It is most commonly access via <a href="./api/org/apache/tika/mime/MimeTypes.html">org.apache.tika.mime.MimeTypes</a>, normally sourced from the <tt>tika-mimetypes.xml</tt> and <tt>custom-mimetypes.xml</tt> files. For more information on defining your own custom mimetypes, see <a href="./parser_guide.html#Add_your_MIME-Type">the new parser guide</a>.</p></div>
+<div class="section">
+<h3><a name="Resource_Name_Based_Detection">Resource Name Based Detection</a></h3>
+<p>Where the name of the file is known, it is sometimes possible to guess the file type from the name or extension. Within the <tt>tika-mimetypes.xml</tt> file is a list of patterns which are used to identify the type from the filename.</p>
+<p>However, because files may be renamed, this method of detection is quick but not always as accurate.</p>
+<p>This is provided within Tika by <a href="./api/org/apache/tika/detect/NameDetector.html">org.apache.tika.detect.NameDetector</a>.</p></div>
+<div class="section">
+<h3><a name="Known_Content_Type_Detection">Known Content Type "Detection</a></h3>
+<p>Sometimes, the mime type for a file is already known, such as when downloading from a webserver, or when retrieving from a content store. This information can be used by detectors, such as <a href="./api/org/apache/tika/mime/MimeTypes.html">org.apache.tika.mime.MimeTypes</a>,</p></div>
+<div class="section">
+<h3><a name="The_default_Mime_Types_Detector">The default Mime Types Detector</a></h3>
+<p>By default, the mime type detection in Tika is provided by <a href="./api/org/apache/tika/mime/MimeTypes.html">org.apache.tika.mime.MimeTypes</a>. This detector makes use of <tt>tika-mimetypes.xml</tt> to power magic based and filename based detection.</p>
+<p>Firstly, magic based detection is used on the start of the file. If the file is an XML file, then the start of the XML is processed to look for root elements. Next, if available, the filename (from <tt>Metadata.RESOURCE_NAME_KEY</tt>) is then used to improve the detail of the detection, such as when magic detects a text file, and the filename hints it's really a CSV. Finally, if available, the supplied content type (from <tt>Metadata.CONTENT_TYPE</tt>) is used to further refine the type.</p></div>
+<div class="section">
+<h3><a name="Container_Aware_Detection">Container Aware Detection</a></h3>
+<p>Several common file formats are actually held within a common container format. One example is the PowerPoint .ppt and Word .doc formats, which are both held within an OLE2 container. Another is Apple iWork formats, which are actually a series of XML files within a Zip file.</p>
+<p>Using magic detection, it is easy to spot that a given file is an OLE2 document, or a Zip file. Using magic detection alone, it is very difficult (and often impossible) to tell what kind of file lives inside the container.</p>
+<p>For some use cases, speed is important, so having a quick way to know the container type is sufficient. For other cases however, you don't mind spending a bit of time (and memory!) processing the container to get a more accurate answer on its contents. For these cases, the additional container aware detectors contained in the <tt>Tika Parsers</tt> jar should be used.</p>
+<p>Tika provides a wrapping detector in the form of <a href="./api/org/apache/tika/detect/DefaultDetector.html">org.apache.tika.detect.DefaultDetector</a>. This uses the service loader to discover all available detectors, including any available container aware ones, and tries them in turn. For container aware detection, include the <tt>Tika Parsers</tt> jar and its dependencies in your project, then use DefaultDetector along with a <tt>TikaInputStream</tt>.</p>
+<p>Because these container detectors needs to read the whole file to open and inspect the container, they must be used with a <a href="./api/org/apache/tika/io/TikaInputStream.html">org.apache.tika.io.TikaInputStream</a>. If called with a regular <tt>InputStream</tt>, then all work will be done by the default Mime Magic detection only.</p>
+<p>For more information on container formats and Tika, see <a class="externalLink" href="http://wiki.apache.org/tika/MetadataDiscussion"></a></p></div>
+<div class="section">
+<h3><a name="The_default_Tika_Detector">The default Tika Detector</a></h3>
+<p>Just as with Parsers, Tika provides a special detector <a href="./api/org/apache/tika/detect/DefaultDetector.html">org.apache.tika.detect.DefaultDetector</a> which auto-detects (based on service files) the available detectors at runtime, and tries these in turn to identify the file type.</p>
+<p>If only <tt>Tika Core</tt> is available, the Default Detector will work only with Mime Magic and Resource Name detection. However, if <tt>Tika Parsers</tt> (and its dependencies!) are available, additional detectors which known about containers (such as zip and ole2) will be used as appropriate, provided that detection is being performed with a <a href="./api/org/apache/tika/io/TikaInputStream.html">org.apache.tika.io.TikaInputStream</a>. Custom detectors can also be used as desired, they simply need to be listed in a service file much as is done for <a href="./parser_guide.html#List_the_new_parser">custom parsers</a>.</p></div>
+<div class="section">
+<h3><a name="Ways_of_triggering_Detection">Ways of triggering Detection</a></h3>
+<p>The simplest way to detect is through the <a href="./api/org/apache/tika/Tika.html">Tika Facade class</a>, which provides methods to detect based on <a href="./api/org/apache/tika/Tika.html#detect(java.io.File)">File</a>, <a href="./api/org/apache/tika/Tika.html#detect(java.io.InputStream)">InputStream</a>, <a href="./api/org/apache/tika/Tika.html#detect(java.io.InputStream, java.lang.String)">InputStream and Filename</a>, <a href="./api/org/apache/tika/Tika.html#detect(java.lang.String)">Filename</a> or a few others. It works best with a File or <a href="./api/org/apache/tika/io/TikaInputStream.html">TikaInputStream</a>.</p>
+<p>Alternately, detection can be performed on a specific Detector, or using <tt>DefaultDetector</tt> to have all available Detectors used. A typical pattern would be something like:</p>
+<div>
+<pre>TikaConfig tika = new TikaConfig();
+
+for (File f : myListOfFiles) {
+ Metadata metadata = new Metadata();
+ metadata.set(Metadata.RESOURCE_NAME_KEY, f.toString());
+ String mimetype = tika.getDetector().detect(
+ TikaInputStream.get(f), metadata);
+ System.out.println("File " + f + " is " + mimetype);
+}
+for (InputStream is : myListOfStreams) {
+ String mimetype = tika.getDetector().detect(
+ TikaInputStream.get(is), new Metadata());
+ System.out.println("Stream " + is + " is " + mimetype);
+}</pre></div></div>
+<div class="section">
+<h3><a name="Language_Detection">Language Detection</a></h3>
+<p>Tika is able to help identify the language of a piece of text, which is useful when extracting text from document formats which do not include language information in their metadata.</p>
+<p>The language detection is provided by <a href="./api/org/apache/tika/language/LanguageIdentifier.html">org.apache.tika.language.LanguageIdentifier</a></p></div>
+<div class="section">
+<h3><a name="More_Examples">More Examples</a></h3>
+<p>For more examples of Detection using Apache Tika, please take a look at the <a href="./examples.html">Tika Examples page</a>.</p></div></div>
+ </div>
+ <div id="sidebar">
+ <div id="navigation">
+ <h5>Apache Tika</h5>
+ <ul>
+
+ <li class="none">
+ <a href="../index.html">Introduction</a>
+ </li>
+
+ <li class="none">
+ <a href="../download.html">Download</a>
+ </li>
+
+ <li class="none">
+ <a href="../contribute.html">Contribute</a>
+ </li>
+
+ <li class="none">
+ <a href="../mail-lists.html">Mailing Lists</a>
+ </li>
+
+ <li class="none">
+ <a href="http://wiki.apache.org/tika/" class="externalLink">Tika Wiki</a>
+ </li>
+
+ <li class="none">
+ <a href="https://issues.apache.org/jira/browse/TIKA" class="externalLink">Issue Tracker</a>
+ </li>
+ </ul>
+ <h5>Documentation</h5>
+ <ul>
+
+
+
+
+
+
+
+
+
+
+
+
+ <li class="expanded">
+ <a href="../1.14/index.html">Apache Tika 1.14</a>
+ <ul>
+
+ <li class="none">
+ <a href="../1.14/gettingstarted.html">Getting Started</a>
+ </li>
+
+ <li class="none">
+ <a href="../1.14/formats.html">Supported Formats</a>
+ </li>
+
+ <li class="none">
+ <a href="../1.14/parser.html">Parser API</a>
+ </li>
+
+ <li class="none">
+ <a href="../1.14/parser_guide.html">Parser 5min Quick Start Guide</a>
+ </li>
+
+ <li class="none">
+ <strong>Content and Language Detection</strong>
+ </li>
+
+ <li class="none">
+ <a href="../1.14/configuring.html">Configuring Tika</a>
+ </li>
+
+ <li class="none">
+ <a href="../1.14/examples.html">Usage Examples</a>
+ </li>
+
+ <li class="none">
+ <a href="../1.14/api/">API Documentation</a>
+ </li>
+
+ <li class="none">
+ <a href="../1.14/miredot/">REST API Documentation (Miredot)</a>
+ </li>
+ </ul>
+ </li>
+
+
+
+
+
+
+
+
+
+
+
+
+ <li class="collapsed">
+ <a href="../1.13/index.html">Apache Tika 1.13</a>
+ </li>
+
+
+
+
+
+
+
+
+
+
+
+
+ <li class="collapsed">
+ <a href="../1.12/index.html">Apache Tika 1.12</a>
+ </li>
+
+
+
+
+
+
+
+
+
+
+
+
+ <li class="collapsed">
+ <a href="../1.11/index.html">Apache Tika 1.11</a>
+ </li>
+
+
+
+
+
+
+
+
+
+
+
+
+ <li class="collapsed">
+ <a href="../1.10/index.html">Apache Tika 1.10</a>
+ </li>
+ </ul>
+ <h5>The Apache Software Foundation</h5>
+ <ul>
+
+ <li class="none">
+ <a href="http://www.apache.org/foundation/" class="externalLink">About</a>
+ </li>
+
+ <li class="none">
+ <a href="http://www.apache.org/licenses/" class="externalLink">License</a>
+ </li>
+
+ <li class="none">
+ <a href="http://www.apache.org/security/" class="externalLink">Security</a>
+ </li>
+
+ <li class="none">
+ <a href="http://www.apache.org/foundation/sponsorship.html" class="externalLink">Sponsorship</a>
+ </li>
+
+ <li class="none">
+ <a href="http://www.apache.org/foundation/thanks.html" class="externalLink">Thanks</a>
+ </li>
+ </ul>
+
+ <div id="search">
+ <h5>Search with Apache Solr</h5>
+ <form action="http://search.lucidimagination.com/p:tika"
+ method="get" id="searchform">
+ <input type="text" id="query" name="q"/>
+ <select name="searchProvider" id="searchProvider">
+ <option value="any">provider</option>
+ <option value="lucid">Lucid Find</option>
+ <option value="sl">Search-Lucene</option>
+ </select>
+ <input type="submit" id="submit" value="Search" name="Search"
+ onclick="selectProvider(this.form)"/>
+ </form>
+ </div>
+
+ <div id="bookpromo">
+ <h5>Books about Tika</h5>
+ <p>
+ <a href="http://manning.com/mattmann/" title="Tika in Action"
+ ><img src="../mattmann_cover150.jpg"
+ width="150" height="186"/></a>
+ </p>
+ </div>
+ </div>
+ </div>
+ <div id="footer">
+ <p>
+ Copyright © 2016
+ <a href="http://www.apache.org/">The Apache Software Foundation</a>.
+ Site powered by <a href="http://maven.apache.org/">Apache Maven</a>.
+ Search powered by
+ <a href="http://www.lucidimagination.com">Lucid Imagination</a>
+ and <a href="http://sematext.com">Sematext</a>.
+ <br/>
+ Apache Tika, Tika, Apache, the Apache feather logo, and the Apache
+ Tika project logo are trademarks of The Apache Software Foundation.
+ </p>
+ </div>
+ </div>
+ </body>
+</html>