You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tika.apache.org by "Jukka Zitting (JIRA)" <ji...@apache.org> on 2007/10/15 00:01:50 UTC
[jira] Created: (TIKA-67) Add an auto-detecting Parser
implementation
Add an auto-detecting Parser implementation
-------------------------------------------
Key: TIKA-67
URL: https://issues.apache.org/jira/browse/TIKA-67
Project: Tika
Issue Type: New Feature
Components: general
Reporter: Jukka Zitting
Assignee: Jukka Zitting
Fix For: 0.1-incubator
We should have an AutoDetectParser class that uses the MIME framework to automatically detect the type of the document being parsed, and that dispatches the parsing task to the parser class configured for the detected MIME type.
The class would work like this:
InputStream stream = ...;
ContentHandler handler = ...;
Metadata metadata = new Metadata();
metadata.set(Metadata.CONTENT_TYPE, ...); // optional content type hint
metadata.set("filename", ...); // optional file name hint
AutoDetectParser parser = new AutoDetectParser();
parser.setConfig(...); // optional TikaConfig configuration
parser.parse(stream, handler, metadata);
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Resolved: (TIKA-67) Add an auto-detecting Parser implementation
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On 10/16/07, Keith R. Bennett <kb...@bbsinc.biz> wrote:
> 1) Now that we have this AutoDetectParser, do you want (me) to remove the
> ParseUtils method that do similar/same things?
That would probably be good, as we shouldn't have too many ways to do
the same thing. In general I would prefer to avoid static methods in
order to keep the codebase modular.
We could even do something like this:
public class ConvenienceParser extends ParserDecorator {
...
public String parse(InputStream stream) throws ... {
return parse(stream, new Metadata());
}
public String parse(InputStream stream, Metadata metadata) throws ... {
StringWriter writer = new StringWriter();
parse(stream, new WriteOutContentHandler(writer), metadata);
return writer.toString();
}
public String parse(File file) throws ... {
Metadata metadata = new Metadata();
metadata.set("filename", file.getName());
InputStream stream = new FileInputStream(file);
try {
return parse(stream, metadata);
} finally {
stream.close();
}
}
... // etc.
}
> 2) Also (by the way), we no longer use RereadableInputStream. Did you want
> to remove that?
We may still have a use for RereadableInputStream later on. See also
http://www.nabble.com/RereadableInputStream-tf4609782.html for related
discussion on getting similar functionality included in commons-io.
> 3) Could we use a constant for "filename"? And should we rename it to
> something general enough to cover URL's in addition to file names (such as
> "resourceName")?
Definitely. Probably.
> 4) It looks like it unconditionally reads and inspects the header to
> determine the MIME type, even if the CONTENT_TYPE is provided. Did you want
> to allow overriding that behavior? (other than by setting a different
> config object)?
The Metadata.CONTENT_TYPE hint is just a hint. For example the HTTP
ContentType header is quite often incorrect, so a MIME magic match (if
available) is probably more accurate.
> 5) The parse() method sometimes wraps the passed stream in a
> BufferedInputStream. The caller (presumably) closes its stream after
> parse() returns, but the BufferedInputStream never gets closed, right?
As you noticed, that won't be a problem as the only resource in
BufferedInputStream that needs explicit closing is the underlying
stream.
BR,
Jukka Zitting
Re: [jira] Resolved: (TIKA-67) Add an auto-detecting Parser
implementation
Posted by "Keith R. Bennett" <kb...@bbsinc.biz>.
Jukka -
The AutoDetectParser looks good. I expect it will simplify things quite a
bit. Some questions for you:
1) Now that we have this AutoDetectParser, do you want (me) to remove the
ParseUtils method that do similar/same things?
2) Also (by the way), we no longer use RereadableInputStream. Did you want
to remove that?
3) Could we use a constant for "filename"? And should we rename it to
something general enough to cover URL's in addition to file names (such as
"resourceName")?
4) It looks like it unconditionally reads and inspects the header to
determine the MIME type, even if the CONTENT_TYPE is provided. Did you want
to allow overriding that behavior? (other than by setting a different
config object)?
5) The parse() method sometimes wraps the passed stream in a
BufferedInputStream. The caller (presumably) closes its stream after
parse() returns, but the BufferedInputStream never gets closed, right? It
looks like that's not a problem, though, because BufferedInputStream.close()
seems to do nothing other than to close its wrapped stream. From
BufferedInputStream.java:
public void close() throws IOException {
byte[] buffer;
while ( (buffer = buf) != null) {
if (bufUpdater.compareAndSet(this, buffer, null)) {
InputStream input = in;
in = null;
if (input != null)
input.close();
return;
}
// Else retry in case a new buf was CASed in fill()
}
}
- Keith
JIRA jira@apache.org wrote:
>
>
> [
> https://issues.apache.org/jira/browse/TIKA-67?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
> ]
>
> Jukka Zitting resolved TIKA-67.
> -------------------------------
>
> Resolution: Fixed
>
> Patch committed in revision 584921.
>
>> Add an auto-detecting Parser implementation
>> -------------------------------------------
>>
>> Key: TIKA-67
>> URL: https://issues.apache.org/jira/browse/TIKA-67
>> Project: Tika
>> Issue Type: New Feature
>> Components: general
>> Reporter: Jukka Zitting
>> Assignee: Jukka Zitting
>> Fix For: 0.1-incubator
>>
>> Attachments: TIKA-67.patch
>>
>>
>> We should have an AutoDetectParser class that uses the MIME framework to
>> automatically detect the type of the document being parsed, and that
>> dispatches the parsing task to the parser class configured for the
>> detected MIME type.
>> The class would work like this:
>> InputStream stream = ...;
>> ContentHandler handler = ...;
>> Metadata metadata = new Metadata();
>> metadata.set(Metadata.CONTENT_TYPE, ...); // optional content type
>> hint
>> metadata.set("filename", ...); // optional file name hint
>> AutoDetectParser parser = new AutoDetectParser();
>> parser.setConfig(...); // optional TikaConfig configuration
>> parser.parse(stream, handler, metadata);
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
>
--
View this message in context: http://www.nabble.com/-jira--Created%3A-%28TIKA-67%29-Add-an-auto-detecting-Parser-implementation-tf4623559.html#a13223061
Sent from the Apache Tika - Development mailing list archive at Nabble.com.
[jira] Updated: (TIKA-67) Add an auto-detecting Parser
implementation
Posted by "Jukka Zitting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TIKA-67?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jukka Zitting updated TIKA-67:
------------------------------
Attachment: TIKA-67.patch
The attached patch contains an implementation of the proposed AutoDetectParser class.
> Add an auto-detecting Parser implementation
> -------------------------------------------
>
> Key: TIKA-67
> URL: https://issues.apache.org/jira/browse/TIKA-67
> Project: Tika
> Issue Type: New Feature
> Components: general
> Reporter: Jukka Zitting
> Assignee: Jukka Zitting
> Fix For: 0.1-incubator
>
> Attachments: TIKA-67.patch
>
>
> We should have an AutoDetectParser class that uses the MIME framework to automatically detect the type of the document being parsed, and that dispatches the parsing task to the parser class configured for the detected MIME type.
> The class would work like this:
> InputStream stream = ...;
> ContentHandler handler = ...;
> Metadata metadata = new Metadata();
> metadata.set(Metadata.CONTENT_TYPE, ...); // optional content type hint
> metadata.set("filename", ...); // optional file name hint
> AutoDetectParser parser = new AutoDetectParser();
> parser.setConfig(...); // optional TikaConfig configuration
> parser.parse(stream, handler, metadata);
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (TIKA-67) Add an auto-detecting Parser
implementation
Posted by "Jukka Zitting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TIKA-67?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jukka Zitting resolved TIKA-67.
-------------------------------
Resolution: Fixed
Patch committed in revision 584921.
> Add an auto-detecting Parser implementation
> -------------------------------------------
>
> Key: TIKA-67
> URL: https://issues.apache.org/jira/browse/TIKA-67
> Project: Tika
> Issue Type: New Feature
> Components: general
> Reporter: Jukka Zitting
> Assignee: Jukka Zitting
> Fix For: 0.1-incubator
>
> Attachments: TIKA-67.patch
>
>
> We should have an AutoDetectParser class that uses the MIME framework to automatically detect the type of the document being parsed, and that dispatches the parsing task to the parser class configured for the detected MIME type.
> The class would work like this:
> InputStream stream = ...;
> ContentHandler handler = ...;
> Metadata metadata = new Metadata();
> metadata.set(Metadata.CONTENT_TYPE, ...); // optional content type hint
> metadata.set("filename", ...); // optional file name hint
> AutoDetectParser parser = new AutoDetectParser();
> parser.setConfig(...); // optional TikaConfig configuration
> parser.parse(stream, handler, metadata);
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (TIKA-67) Add an auto-detecting Parser
implementation
Posted by "Bertrand Delacretaz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/TIKA-67?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12534750 ]
Bertrand Delacretaz commented on TIKA-67:
-----------------------------------------
I haven't looked at the implementation details but I like the idea
> Add an auto-detecting Parser implementation
> -------------------------------------------
>
> Key: TIKA-67
> URL: https://issues.apache.org/jira/browse/TIKA-67
> Project: Tika
> Issue Type: New Feature
> Components: general
> Reporter: Jukka Zitting
> Assignee: Jukka Zitting
> Fix For: 0.1-incubator
>
> Attachments: TIKA-67.patch
>
>
> We should have an AutoDetectParser class that uses the MIME framework to automatically detect the type of the document being parsed, and that dispatches the parsing task to the parser class configured for the detected MIME type.
> The class would work like this:
> InputStream stream = ...;
> ContentHandler handler = ...;
> Metadata metadata = new Metadata();
> metadata.set(Metadata.CONTENT_TYPE, ...); // optional content type hint
> metadata.set("filename", ...); // optional file name hint
> AutoDetectParser parser = new AutoDetectParser();
> parser.setConfig(...); // optional TikaConfig configuration
> parser.parse(stream, handler, metadata);
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.