You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Markus Jelsma (Created) (JIRA)" <ji...@apache.org> on 2012/01/25 14:36:40 UTC
[jira] [Created] (NUTCH-1259) TikaParser should not add
Content-Type from HTTP Headers to Nutch Metadata
TikaParser should not add Content-Type from HTTP Headers to Nutch Metadata
--------------------------------------------------------------------------
Key: NUTCH-1259
URL: https://issues.apache.org/jira/browse/NUTCH-1259
Project: Nutch
Issue Type: Bug
Components: parser
Affects Versions: 1.4
Reporter: Markus Jelsma
Assignee: Markus Jelsma
Fix For: 1.5
The MIME-type detected by Tika's Detect() API is never added to a Parse's ContentMetaData or ParseMetaData. Because of this bad Content-Types will end up in the documents.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1259) TikaParser should not add
Content-Type from HTTP Headers to Nutch Metadata
Posted by "Markus Jelsma (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202483#comment-13202483 ]
Markus Jelsma commented on NUTCH-1259:
--------------------------------------
you're right. but since you're most of the time the only person reviewing and the fact this issue has your attention now, what is your opinion on this problem? ;)
> TikaParser should not add Content-Type from HTTP Headers to Nutch Metadata
> --------------------------------------------------------------------------
>
> Key: NUTCH-1259
> URL: https://issues.apache.org/jira/browse/NUTCH-1259
> Project: Nutch
> Issue Type: Bug
> Components: parser
> Affects Versions: 1.4
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.5
>
> Attachments: NUTCH-1259-1.5-1.patch
>
>
> The MIME-type detected by Tika's Detect() API is never added to a Parse's ContentMetaData or ParseMetaData. Because of this bad Content-Types will end up in the documents.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1259) TikaParser should not add
Content-Type from HTTP Headers to Nutch Metadata
Posted by "Markus Jelsma (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205548#comment-13205548 ]
Markus Jelsma commented on NUTCH-1259:
--------------------------------------
NUTCH-1024 relies on the Content-Type to be added crawldatum metadata via db.parsemeta.to.crawldb.
Anyway, i agree. Will you open another issue?
have a nice weekend :)
> TikaParser should not add Content-Type from HTTP Headers to Nutch Metadata
> --------------------------------------------------------------------------
>
> Key: NUTCH-1259
> URL: https://issues.apache.org/jira/browse/NUTCH-1259
> Project: Nutch
> Issue Type: Bug
> Components: parser
> Affects Versions: 1.4
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.5
>
> Attachments: NUTCH-1259-1.5-1.patch
>
>
> The MIME-type detected by Tika's Detect() API is never added to a Parse's ContentMetaData or ParseMetaData. Because of this bad Content-Types will end up in the documents.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (NUTCH-1259) Store detected content type in
crawldatum metadata
Posted by "Julien Nioche (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Julien Nioche updated NUTCH-1259:
---------------------------------
Summary: Store detected content type in crawldatum metadata (was: TikaParser should not add Content-Type from HTTP Headers to Nutch Metadata)
> Store detected content type in crawldatum metadata
> --------------------------------------------------
>
> Key: NUTCH-1259
> URL: https://issues.apache.org/jira/browse/NUTCH-1259
> Project: Nutch
> Issue Type: Bug
> Components: parser
> Affects Versions: 1.4
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.5
>
> Attachments: NUTCH-1259-1.5-1.patch
>
>
> The MIME-type detected by Tika's Detect() API is never added to a Parse's ContentMetaData or ParseMetaData. Because of this bad Content-Types will end up in the documents.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1259) TikaParser should not add
Content-Type from HTTP Headers to Nutch Metadata
Posted by "Julien Nioche (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205490#comment-13205490 ]
Julien Nioche commented on NUTCH-1259:
--------------------------------------
I haven't looked at NUTCH-1024. Does it take the detected value from Content or the one from the parse md?
As for storing it in the CrawlDatum that would require changing the object, its version, making sure it remains compatible etc... so I'd rather store it in the crawldatum md for now. It means that it can be overriden indeed but this is quite unlikely to happen unless you write a custom resource etc... Let's keep this option in mind for later maybe
> TikaParser should not add Content-Type from HTTP Headers to Nutch Metadata
> --------------------------------------------------------------------------
>
> Key: NUTCH-1259
> URL: https://issues.apache.org/jira/browse/NUTCH-1259
> Project: Nutch
> Issue Type: Bug
> Components: parser
> Affects Versions: 1.4
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.5
>
> Attachments: NUTCH-1259-1.5-1.patch
>
>
> The MIME-type detected by Tika's Detect() API is never added to a Parse's ContentMetaData or ParseMetaData. Because of this bad Content-Types will end up in the documents.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1259) Store detected content type in
crawldatum metadata
Posted by "Julien Nioche (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13207753#comment-13207753 ]
Julien Nioche commented on NUTCH-1259:
--------------------------------------
bq. But what about segments fetched with and without this new feature and db.parsemeta.to.crawldb=Content-Type property?
bq. I assume i'd have to update the segments before this change with the property enabled and update the segments fetched with this feature without the db.parsemeta.to.crawldb property.
yep
> Store detected content type in crawldatum metadata
> --------------------------------------------------
>
> Key: NUTCH-1259
> URL: https://issues.apache.org/jira/browse/NUTCH-1259
> Project: Nutch
> Issue Type: Bug
> Components: parser
> Affects Versions: 1.4
> Reporter: Markus Jelsma
> Assignee: Julien Nioche
> Fix For: 1.5
>
> Attachments: NUTCH-1259-1.5-1.patch
>
>
> The MIME-type detected by Tika's Detect() API is never added to a Parse's ContentMetaData or ParseMetaData. Because of this bad Content-Types will end up in the documents.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1259) TikaParser should not add
Content-Type from HTTP Headers to Nutch Metadata
Posted by "Markus Jelsma (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193030#comment-13193030 ]
Markus Jelsma commented on NUTCH-1259:
--------------------------------------
A solution would be to prevent the type to be added just like what is already being done with the title field. Now, a reliable Content-Type value is added to the ParseMetaData.
{code}
// populate Nutch metadata with Tika metadata
String[] TikaMDNames = tikamd.names();
for (String tikaMDName : TikaMDNames) {
if (tikaMDName.equalsIgnoreCase(Metadata.TITLE))
continue;
// DO NOT ADD Content-Type FROM HTTP_HEADERS, ONLY ADD THE DETECTED TYPE SEE https://issues.apache.org/jira/browse/NUTCH-1259
if (tikaMDName.equalsIgnoreCase(Metadata.CONTENT_TYPE))
continue;
// TODO what if multivalued?
nutchMetadata.add(tikaMDName, tikamd.get(tikaMDName));
}
// Only add the detected TYPE
nutchMetadata.add("Content-Type", mimeType);
{code}
> TikaParser should not add Content-Type from HTTP Headers to Nutch Metadata
> --------------------------------------------------------------------------
>
> Key: NUTCH-1259
> URL: https://issues.apache.org/jira/browse/NUTCH-1259
> Project: Nutch
> Issue Type: Bug
> Components: parser
> Affects Versions: 1.4
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.5
>
>
> The MIME-type detected by Tika's Detect() API is never added to a Parse's ContentMetaData or ParseMetaData. Because of this bad Content-Types will end up in the documents.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (NUTCH-1259) Store detected content type in
crawldatum metadata
Posted by "Julien Nioche (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Julien Nioche resolved NUTCH-1259.
----------------------------------
Resolution: Fixed
> Store detected content type in crawldatum metadata
> --------------------------------------------------
>
> Key: NUTCH-1259
> URL: https://issues.apache.org/jira/browse/NUTCH-1259
> Project: Nutch
> Issue Type: Bug
> Components: parser
> Affects Versions: 1.4
> Reporter: Markus Jelsma
> Assignee: Julien Nioche
> Fix For: 1.5
>
> Attachments: NUTCH-1259-1.5-1.patch
>
>
> The MIME-type detected by Tika's Detect() API is never added to a Parse's ContentMetaData or ParseMetaData. Because of this bad Content-Types will end up in the documents.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1259) TikaParser should not add
Content-Type from HTTP Headers to Nutch Metadata
Posted by "Markus Jelsma (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204556#comment-13204556 ]
Markus Jelsma commented on NUTCH-1259:
--------------------------------------
Hi,
Consider the following URL that produces bad output. This URL is not the only producing bad output. We've seen countless of examples that produce funky values in both content meta and parse meta, or no value at all.
http://kam.mff.cuni.cz/conferences/GraDR/
The current Nutch trunk shows us the following meta data for this URL obtained via parsechecker with only parse-tika enabled:
{code}
Content Metadata: Vary=negotiate,accept,Accept-Encoding Date=Thu, 09 Feb 2012 14:37:47 GMT Content-Length=4911 TCN=choice Content-Encoding=gzip Content-Location=index.html.bak Content-Type=application/x-trash Connection=close Accept-Ranges=bytes Server=Apache/2.2.9 (Debian) mod_auth_kerb/5.3 PHP/5.2.6-1+lenny14 with Suhosin-Patch mod_ssl/2.2.9 OpenSSL/0.9.8g
Parse Metadata: Content-Encoding=ISO-8859-1
{code}
It's an application/x-trash according to content meta and no data is available in parse meta. But, it's just an ordinary HTML page. This cannot be true, from an index point of view we will never know that this is an HTML page. With this patch enabled we will get the following output:
{code}
Content Metadata: Vary=negotiate,accept,Accept-Encoding Date=Thu, 09 Feb 2012 14:40:15 GMT Content-Length=4911 TCN=choice Content-Encoding=gzip Content-Location=index.html.bak Content-Type=application/x-trash Connection=close Accept-Ranges=bytes Server=Apache/2.2.9 (Debian) mod_auth_kerb/5.3 PHP/5.2.6-1+lenny14 with Suhosin-Patch mod_ssl/2.2.9 OpenSSL/0.9.8g
Parse Metadata: Content-Encoding=ISO-8859-1 Content-Type=text/html
{code}
For us, this solves all problems as we now only rely on Tika's MIME-detector and store it in parse meta. The value of content meta cannot be trusted. It's the same as with languages, when we do not use Tika to detect the language we get all sorts of crap.
Since the upgrade to Tika 1.0 and with NUTCH-1230 we obtain the detected MIME-type but it's not added to the parse meta. Now it is.
Do you have another suggestion?
> TikaParser should not add Content-Type from HTTP Headers to Nutch Metadata
> --------------------------------------------------------------------------
>
> Key: NUTCH-1259
> URL: https://issues.apache.org/jira/browse/NUTCH-1259
> Project: Nutch
> Issue Type: Bug
> Components: parser
> Affects Versions: 1.4
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.5
>
> Attachments: NUTCH-1259-1.5-1.patch
>
>
> The MIME-type detected by Tika's Detect() API is never added to a Parse's ContentMetaData or ParseMetaData. Because of this bad Content-Types will end up in the documents.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (NUTCH-1259) Store detected content type in
crawldatum metadata
Posted by "Julien Nioche (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Julien Nioche resolved NUTCH-1259.
----------------------------------
Resolution: Fixed
trunk => Committed revision 1243482.
> Store detected content type in crawldatum metadata
> --------------------------------------------------
>
> Key: NUTCH-1259
> URL: https://issues.apache.org/jira/browse/NUTCH-1259
> Project: Nutch
> Issue Type: Bug
> Components: parser
> Affects Versions: 1.4
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.5
>
> Attachments: NUTCH-1259-1.5-1.patch
>
>
> The MIME-type detected by Tika's Detect() API is never added to a Parse's ContentMetaData or ParseMetaData. Because of this bad Content-Types will end up in the documents.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1259) TikaParser should not add
Content-Type from HTTP Headers to Nutch Metadata
Posted by "Lewis John McGibbney (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202603#comment-13202603 ]
Lewis John McGibbney commented on NUTCH-1259:
---------------------------------------------
Hey Markus. I'm literally up to my eye balls with stuff the now so sorry for not having the time to look through your work. The best I can do is have a look tomorrow, I'll give it my all then. Thanks
> TikaParser should not add Content-Type from HTTP Headers to Nutch Metadata
> --------------------------------------------------------------------------
>
> Key: NUTCH-1259
> URL: https://issues.apache.org/jira/browse/NUTCH-1259
> Project: Nutch
> Issue Type: Bug
> Components: parser
> Affects Versions: 1.4
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.5
>
> Attachments: NUTCH-1259-1.5-1.patch
>
>
> The MIME-type detected by Tika's Detect() API is never added to a Parse's ContentMetaData or ParseMetaData. Because of this bad Content-Types will end up in the documents.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1259) TikaParser should not add
Content-Type from HTTP Headers to Nutch Metadata
Posted by "Markus Jelsma (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196034#comment-13196034 ]
Markus Jelsma commented on NUTCH-1259:
--------------------------------------
comments please.
> TikaParser should not add Content-Type from HTTP Headers to Nutch Metadata
> --------------------------------------------------------------------------
>
> Key: NUTCH-1259
> URL: https://issues.apache.org/jira/browse/NUTCH-1259
> Project: Nutch
> Issue Type: Bug
> Components: parser
> Affects Versions: 1.4
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.5
>
>
> The MIME-type detected by Tika's Detect() API is never added to a Parse's ContentMetaData or ParseMetaData. Because of this bad Content-Types will end up in the documents.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1259) TikaParser should not add
Content-Type from HTTP Headers to Nutch Metadata
Posted by "Julien Nioche (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204463#comment-13204463 ]
Julien Nioche commented on NUTCH-1259:
--------------------------------------
bq. // DO NOT ADD Content-Type FROM HTTP_HEADERS, ONLY ADD THE DETECTED TYPE SEE https://issues.apache.org/jira/browse/NUTCH-1259
hmmm, isn't that the content-type from the HTML headers instead?
Anyway, probably a good idea NOT to add it to the parse-metadata as it has already been detected from the content and stored in the content metadata, however I can't think of a reason why we'd want to duplicate that to the parse metadata as well. The value in the content metadata is the one set by the detector and should be the correct one. Or am I missing something?
> TikaParser should not add Content-Type from HTTP Headers to Nutch Metadata
> --------------------------------------------------------------------------
>
> Key: NUTCH-1259
> URL: https://issues.apache.org/jira/browse/NUTCH-1259
> Project: Nutch
> Issue Type: Bug
> Components: parser
> Affects Versions: 1.4
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.5
>
> Attachments: NUTCH-1259-1.5-1.patch
>
>
> The MIME-type detected by Tika's Detect() API is never added to a Parse's ContentMetaData or ParseMetaData. Because of this bad Content-Types will end up in the documents.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1259) TikaParser should not add
Content-Type from HTTP Headers to Nutch Metadata
Posted by "Julien Nioche (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205670#comment-13205670 ]
Julien Nioche commented on NUTCH-1259:
--------------------------------------
Nah, might as well do it in this one. Will rename it, that's all :-)
Have a nice week end too
> TikaParser should not add Content-Type from HTTP Headers to Nutch Metadata
> --------------------------------------------------------------------------
>
> Key: NUTCH-1259
> URL: https://issues.apache.org/jira/browse/NUTCH-1259
> Project: Nutch
> Issue Type: Bug
> Components: parser
> Affects Versions: 1.4
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.5
>
> Attachments: NUTCH-1259-1.5-1.patch
>
>
> The MIME-type detected by Tika's Detect() API is never added to a Parse's ContentMetaData or ParseMetaData. Because of this bad Content-Types will end up in the documents.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1259) TikaParser should not add
Content-Type from HTTP Headers to Nutch Metadata
Posted by "Markus Jelsma (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202460#comment-13202460 ]
Markus Jelsma commented on NUTCH-1259:
--------------------------------------
I'll comment on it myself then: the code above fixes the issue and adds a proper content-type to parsemeta. Consider the following URL with a very bad content-type:
http://kam.mff.cuni.cz/conferences/GraDR/
I'll upload a patch in a minute that sets the detected content type in the metadata instead
> TikaParser should not add Content-Type from HTTP Headers to Nutch Metadata
> --------------------------------------------------------------------------
>
> Key: NUTCH-1259
> URL: https://issues.apache.org/jira/browse/NUTCH-1259
> Project: Nutch
> Issue Type: Bug
> Components: parser
> Affects Versions: 1.4
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.5
>
>
> The MIME-type detected by Tika's Detect() API is never added to a Parse's ContentMetaData or ParseMetaData. Because of this bad Content-Types will end up in the documents.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1259) TikaParser should not add
Content-Type from HTTP Headers to Nutch Metadata
Posted by "Markus Jelsma (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205413#comment-13205413 ]
Markus Jelsma commented on NUTCH-1259:
--------------------------------------
Sounds good! We already store the Content-Type in de CrawlDatum's metadata for NUTCH-1024 via db.parsemeta.to.crawldb. Wouldn't it be better to store it in the CrawlDatum object itself just like the signature? Then someone cannot override it by accident.
> TikaParser should not add Content-Type from HTTP Headers to Nutch Metadata
> --------------------------------------------------------------------------
>
> Key: NUTCH-1259
> URL: https://issues.apache.org/jira/browse/NUTCH-1259
> Project: Nutch
> Issue Type: Bug
> Components: parser
> Affects Versions: 1.4
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.5
>
> Attachments: NUTCH-1259-1.5-1.patch
>
>
> The MIME-type detected by Tika's Detect() API is never added to a Parse's ContentMetaData or ParseMetaData. Because of this bad Content-Types will end up in the documents.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1259) TikaParser should not add
Content-Type from HTTP Headers to Nutch Metadata
Posted by "Julien Nioche (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202474#comment-13202474 ]
Julien Nioche commented on NUTCH-1259:
--------------------------------------
bq. I'll commit this one tomorrow unless there are objections.
Markus, I understand that you could be frustrated with having your issues not reviewed as quickly as you'd wish but it would be nice to have a bit more notice. There aren't many active committers in the project and I can't follow the pace at which you submit patches :-)
> TikaParser should not add Content-Type from HTTP Headers to Nutch Metadata
> --------------------------------------------------------------------------
>
> Key: NUTCH-1259
> URL: https://issues.apache.org/jira/browse/NUTCH-1259
> Project: Nutch
> Issue Type: Bug
> Components: parser
> Affects Versions: 1.4
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.5
>
> Attachments: NUTCH-1259-1.5-1.patch
>
>
> The MIME-type detected by Tika's Detect() API is never added to a Parse's ContentMetaData or ParseMetaData. Because of this bad Content-Types will end up in the documents.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (NUTCH-1259) TikaParser should not add
Content-Type from HTTP Headers to Nutch Metadata
Posted by "Markus Jelsma (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma updated NUTCH-1259:
---------------------------------
Attachment: NUTCH-1259-1.5-1.patch
Here's a patch for 1.5. Comments? We have this running in production and it does works very good. It completely solves the big problem of ending up with many thousands of crap content-types.
I'll commit this one tomorrow unless there are objections.
> TikaParser should not add Content-Type from HTTP Headers to Nutch Metadata
> --------------------------------------------------------------------------
>
> Key: NUTCH-1259
> URL: https://issues.apache.org/jira/browse/NUTCH-1259
> Project: Nutch
> Issue Type: Bug
> Components: parser
> Affects Versions: 1.4
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.5
>
> Attachments: NUTCH-1259-1.5-1.patch
>
>
> The MIME-type detected by Tika's Detect() API is never added to a Parse's ContentMetaData or ParseMetaData. Because of this bad Content-Types will end up in the documents.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1259) Store detected content type in
crawldatum metadata
Posted by "Julien Nioche (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13207724#comment-13207724 ]
Julien Nioche commented on NUTCH-1259:
--------------------------------------
good catch. Had overlooked the fact that the content object can be null. Could you svn up to revision 1243928 and give it a try?
Thanks
> Store detected content type in crawldatum metadata
> --------------------------------------------------
>
> Key: NUTCH-1259
> URL: https://issues.apache.org/jira/browse/NUTCH-1259
> Project: Nutch
> Issue Type: Bug
> Components: parser
> Affects Versions: 1.4
> Reporter: Markus Jelsma
> Assignee: Julien Nioche
> Fix For: 1.5
>
> Attachments: NUTCH-1259-1.5-1.patch
>
>
> The MIME-type detected by Tika's Detect() API is never added to a Parse's ContentMetaData or ParseMetaData. Because of this bad Content-Types will end up in the documents.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (NUTCH-1259) TikaParser should not add
Content-Type from HTTP Headers to Nutch Metadata
Posted by "Markus Jelsma (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma updated NUTCH-1259:
---------------------------------
Patch Info: Patch Available
> TikaParser should not add Content-Type from HTTP Headers to Nutch Metadata
> --------------------------------------------------------------------------
>
> Key: NUTCH-1259
> URL: https://issues.apache.org/jira/browse/NUTCH-1259
> Project: Nutch
> Issue Type: Bug
> Components: parser
> Affects Versions: 1.4
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.5
>
> Attachments: NUTCH-1259-1.5-1.patch
>
>
> The MIME-type detected by Tika's Detect() API is never added to a Parse's ContentMetaData or ParseMetaData. Because of this bad Content-Types will end up in the documents.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1259) TikaParser should not add
Content-Type from HTTP Headers to Nutch Metadata
Posted by "Markus Jelsma (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13206142#comment-13206142 ]
Markus Jelsma commented on NUTCH-1259:
--------------------------------------
Great. Thanks!
> TikaParser should not add Content-Type from HTTP Headers to Nutch Metadata
> --------------------------------------------------------------------------
>
> Key: NUTCH-1259
> URL: https://issues.apache.org/jira/browse/NUTCH-1259
> Project: Nutch
> Issue Type: Bug
> Components: parser
> Affects Versions: 1.4
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.5
>
> Attachments: NUTCH-1259-1.5-1.patch
>
>
> The MIME-type detected by Tika's Detect() API is never added to a Parse's ContentMetaData or ParseMetaData. Because of this bad Content-Types will end up in the documents.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1259) Store detected content type in
crawldatum metadata
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13206828#comment-13206828 ]
Hudson commented on NUTCH-1259:
-------------------------------
Integrated in nutch-trunk-maven #146 (See [https://builds.apache.org/job/nutch-trunk-maven/146/])
NUTCH-1259 Store detected content type in crawldatum metadata (Revision 1243482)
Result = SUCCESS
jnioche :
Files :
* /nutch/trunk/CHANGES.txt
* /nutch/trunk/src/java/org/apache/nutch/fetcher/Fetcher.java
> Store detected content type in crawldatum metadata
> --------------------------------------------------
>
> Key: NUTCH-1259
> URL: https://issues.apache.org/jira/browse/NUTCH-1259
> Project: Nutch
> Issue Type: Bug
> Components: parser
> Affects Versions: 1.4
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.5
>
> Attachments: NUTCH-1259-1.5-1.patch
>
>
> The MIME-type detected by Tika's Detect() API is never added to a Parse's ContentMetaData or ParseMetaData. Because of this bad Content-Types will end up in the documents.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1259) TikaParser should not add
Content-Type from HTTP Headers to Nutch Metadata
Posted by "Julien Nioche (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204619#comment-13204619 ]
Julien Nioche commented on NUTCH-1259:
--------------------------------------
Thanks for the example. Here is a summary of what is happening.
The correct Mime-type guessed by Tika is stored in the Content object. This is what is then used during the parsing step to determine which implementation of the parser should be used. This value is what you can see displayed by the parser checker e.g.
{noformat}
fetching: http://kam.mff.cuni.cz/conferences/GraDR/
parsing: http://kam.mff.cuni.cz/conferences/GraDR/
contentType: text/html
signature: 575aecee981b1aa03a145e3dc5b4de72
{noformat}
This is different from the value displayed in the content metadata which corresponds to what is returned in the protocol headers. It is also different from the value found in parse metadata which what can be found in the content. Note that there is no guarantee that these two values can be found.
Now the problem with [https://issues.apache.org/jira/browse/NUTCH-1258] is that while the ParserFilters have access to the Content object, this is not the case of the IndexingFilters. One option would be to have a bespoke Parser implementation to store a custom metadata to store the CT in the Content object (i.e. the one Tika guessed) then use that in the indexing filter. That's unnecessarily messy.
I think a cleaner approach would be to store the guessed content-type in the crawldatum metadata. This way we :
* can access it from the indexing filters (the parsing filter would still get it from Content if necessary)
* do not override the value stored in parse metadata
* can access it regardless of whether a document has been parsed or not
* have a mechanism which is independent from the actual parser used (html / tika / other)
* have the possibility of taking a different decision as to which value should be used (guessed vs protocol vs content)
* keep a trace of why such or such parser was used on a given document
This would be done in the output method of the class Fetcher.
What do you think?
> TikaParser should not add Content-Type from HTTP Headers to Nutch Metadata
> --------------------------------------------------------------------------
>
> Key: NUTCH-1259
> URL: https://issues.apache.org/jira/browse/NUTCH-1259
> Project: Nutch
> Issue Type: Bug
> Components: parser
> Affects Versions: 1.4
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.5
>
> Attachments: NUTCH-1259-1.5-1.patch
>
>
> The MIME-type detected by Tika's Detect() API is never added to a Parse's ContentMetaData or ParseMetaData. Because of this bad Content-Types will end up in the documents.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1259) Store detected content type in
crawldatum metadata
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13207764#comment-13207764 ]
Hudson commented on NUTCH-1259:
-------------------------------
Integrated in nutch-trunk-maven #149 (See [https://builds.apache.org/job/nutch-trunk-maven/149/])
BugFix : NUTCH-1259 Store detected content type in crawldatum metadata (Revision 1243928)
Result = SUCCESS
jnioche :
Files :
* /nutch/trunk/src/java/org/apache/nutch/fetcher/Fetcher.java
> Store detected content type in crawldatum metadata
> --------------------------------------------------
>
> Key: NUTCH-1259
> URL: https://issues.apache.org/jira/browse/NUTCH-1259
> Project: Nutch
> Issue Type: Bug
> Components: parser
> Affects Versions: 1.4
> Reporter: Markus Jelsma
> Assignee: Julien Nioche
> Fix For: 1.5
>
> Attachments: NUTCH-1259-1.5-1.patch
>
>
> The MIME-type detected by Tika's Detect() API is never added to a Parse's ContentMetaData or ParseMetaData. Because of this bad Content-Types will end up in the documents.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (NUTCH-1259) Store detected content type in
crawldatum metadata
Posted by "Markus Jelsma (Reopened) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma reopened NUTCH-1259:
----------------------------------
Assignee: Julien Nioche (was: Markus Jelsma)
Hey Julien, there's something wrong with this commit. We're seeing NPE's in the Fetcher without stack trace now. The fetcher doesn't die but the generated seed list is quickly terminated and few records get processed instead of millions. It looks like it's triggered when a fetch error occurs. You can reproduce this error by injecting a unknown host but it's likely to happen as well when socket time outs and related errors are thrown.
{code}
fetch of http://idonotexist.openindex.io/ failed with: java.net.UnknownHostException: idonotexist.openindex.io
fetch of http://idonotexist.openindex.io/ failed with: java.lang.NullPointerException
fetcher caught:java.lang.NullPointerException
{code}
Can you look at it?
> Store detected content type in crawldatum metadata
> --------------------------------------------------
>
> Key: NUTCH-1259
> URL: https://issues.apache.org/jira/browse/NUTCH-1259
> Project: Nutch
> Issue Type: Bug
> Components: parser
> Affects Versions: 1.4
> Reporter: Markus Jelsma
> Assignee: Julien Nioche
> Fix For: 1.5
>
> Attachments: NUTCH-1259-1.5-1.patch
>
>
> The MIME-type detected by Tika's Detect() API is never added to a Parse's ContentMetaData or ParseMetaData. Because of this bad Content-Types will end up in the documents.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1259) Store detected content type in
crawldatum metadata
Posted by "Markus Jelsma (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13207746#comment-13207746 ]
Markus Jelsma commented on NUTCH-1259:
--------------------------------------
Splendid work my friend! The fetcher runs smoothly again! I'll check out your patch for NUTCH-1258 this week.
But what about segments fetched with and without this new feature and db.parsemeta.to.crawldb=Content-Type property?
I assume i'd have to update the segments before this change with the property enabled and update the segments fetched with this feature without the db.parsemeta.to.crawldb property.
> Store detected content type in crawldatum metadata
> --------------------------------------------------
>
> Key: NUTCH-1259
> URL: https://issues.apache.org/jira/browse/NUTCH-1259
> Project: Nutch
> Issue Type: Bug
> Components: parser
> Affects Versions: 1.4
> Reporter: Markus Jelsma
> Assignee: Julien Nioche
> Fix For: 1.5
>
> Attachments: NUTCH-1259-1.5-1.patch
>
>
> The MIME-type detected by Tika's Detect() API is never added to a Parse's ContentMetaData or ParseMetaData. Because of this bad Content-Types will end up in the documents.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira