Content-Type inconsistency?

[Jérôme Charron <je...@gmail.com>]

It seems there is an inconsistency with content-type handling in Nutch:

1. The protocol level content-type header is added in content's metadata.
2. The content-type is then checked/guessed while instanciating the Content
object and stored in a private field
(at this step, the Content object can have 2 different content-types).
3. The Content's private field for content-type is used to find the good
parser.
4. Once the Parse object is constructed, the Content is no more used (=> the
guessed content-type is lost)
5. Then the index-more plugin index the raw content-type and not the guessed
one
6. As a consequence the content-type displayed in more.jsp is the raw one,
and the one used to query on type is the raw one too.

Wouldn't it be better to always use the guessed content-type all along the
process?
(except in cache.jsp, where the raw one should be used)

Jérôme

--
http://motrech.free.fr/
http://www.frutch.org/

Re: Content-Type inconsistency?

[Jérôme Charron <je...@gmail.com>]

I would like to come back on this issue:
The Content object holds two content-types:
1. The raw content-type from the protocol layer (http header in case of
http) in the Content's metadata
2. The guessed content-type in a private field content-type.

When a ParseData object is created, it takes only the Content's metadata.
So, the ParseData can only access the raw content type and not the one
guessed.

What I suggest is :
1. add a content-type parameter in the ParseData constructors (so that
Parsers  can pass the guessed content-type to ParseData).
2. The Content object stores the guessed content-type in it's metadata in a
special attribute named for instance GUESSED_CONTENT_TYPE, so that the
ParseData can access it

I think 1. is really cleanest way to implement this, but there is a lot of
code impacted => all the parsers.
Solution 2. have no impact on APIs, so the code changes are very small.

Suggestions? Comments?

Jérôme

--
http://motrech.free.fr/
http://www.frutch.org/