You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Stefan Groschupf (JIRA)" <ji...@apache.org> on 2005/12/09 21:51:08 UTC
[jira] Created: (NUTCH-135) http header meta data are case insensitive in the real world (e.g. Content-Type or content-type)
http header meta data are case insensitive in the real world (e.g. Content-Type or content-type)
------------------------------------------------------------------------------------------------
Key: NUTCH-135
URL: http://issues.apache.org/jira/browse/NUTCH-135
Project: Nutch
Type: Bug
Components: fetcher
Versions: 0.7.1, 0.7
Reporter: Stefan Groschupf
Priority: Critical
Fix For: 0.8-dev, 0.7.2-dev
As described in issue nutch-133, some webservers return http header meta data not standard conform case insensitive.
This provides many negative side effects, for example query thet content type from the meta data return null also in case the webserver returns a content type, but the key is not standard conform e.g. lower case. Also this has effects to the pdf parser that queries the content length etc.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
Re: [jira] Commented: (NUTCH-135) http header meta data are case insensitive in the real world (e.g. Content-Type or content-type)
Posted by Stefan Groschupf <sg...@media-style.com>.
Jack,
sorry there are now 3kb more in the patch :), please give it another
try.
Stefan
Am 10.12.2005 um 15:30 schrieb Jack Tang:
> Stefan
>
> It seemed your patch missing
> org.apache.nutch.protocol.ContentProperties class, right?
>
> /Jack
>
> On 12/10/05, Stefan Groschupf (JIRA) <ji...@apache.org> wrote:
>> [ http://issues.apache.org/jira/browse/NUTCH-135?
>> page=comments#action_12360025 ]
>>
>> Stefan Groschupf commented on NUTCH-135:
>> ----------------------------------------
>>
>> Andrzej, that is easy to add to the ContentProperties object and
>> sure I can do that. However first I would love to get a OK for
>> this patch, before I invest more time in it, since I spend to many
>> time writing stuff just for the issue archive.
>> As soon this patch is in the sources I will write a small new
>> patch (as Doug suggested, do it in small steps) to solve NUTCH-3
>>
>>> http header meta data are case insensitive in the real world
>>> (e.g. Content-Type or content-type)
>>> --------------------------------------------------------------------
>>> ----------------------------
>>>
>>> Key: NUTCH-135
>>> URL: http://issues.apache.org/jira/browse/NUTCH-135
>>> Project: Nutch
>>> Type: Bug
>>> Components: fetcher
>>> Versions: 0.7, 0.7.1
>>> Reporter: Stefan Groschupf
>>> Priority: Critical
>>> Fix For: 0.8-dev, 0.7.2-dev
>>> Attachments: contentProperties_patch.txt
>>>
>>> As described in issue nutch-133, some webservers return http
>>> header meta data not standard conform case insensitive.
>>> This provides many negative side effects, for example query thet
>>> content type from the meta data return null also in case the
>>> webserver returns a content type, but the key is not standard
>>> conform e.g. lower case. Also this has effects to the pdf parser
>>> that queries the content length etc.
>>
>> --
>> This message is automatically generated by JIRA.
>> -
>> If you think it was sent incorrectly contact one of the
>> administrators:
>> http://issues.apache.org/jira/secure/Administrators.jspa
>> -
>> For more information on JIRA, see:
>> http://www.atlassian.com/software/jira
>>
>>
>
>
> --
> Keep Discovering ... ...
> http://www.jroller.com/page/jmars
>
---------------------------------------------------------------
company: http://www.media-style.com
forum: http://www.text-mining.org
blog: http://www.find23.net
Re: [jira] Commented: (NUTCH-135) http header meta data are case insensitive in the real world (e.g. Content-Type or content-type)
Posted by Jack Tang <hi...@gmail.com>.
Stefan
It seemed your patch missing
org.apache.nutch.protocol.ContentProperties class, right?
/Jack
On 12/10/05, Stefan Groschupf (JIRA) <ji...@apache.org> wrote:
> [ http://issues.apache.org/jira/browse/NUTCH-135?page=comments#action_12360025 ]
>
> Stefan Groschupf commented on NUTCH-135:
> ----------------------------------------
>
> Andrzej, that is easy to add to the ContentProperties object and sure I can do that. However first I would love to get a OK for this patch, before I invest more time in it, since I spend to many time writing stuff just for the issue archive.
> As soon this patch is in the sources I will write a small new patch (as Doug suggested, do it in small steps) to solve NUTCH-3
>
> > http header meta data are case insensitive in the real world (e.g. Content-Type or content-type)
> > ------------------------------------------------------------------------------------------------
> >
> > Key: NUTCH-135
> > URL: http://issues.apache.org/jira/browse/NUTCH-135
> > Project: Nutch
> > Type: Bug
> > Components: fetcher
> > Versions: 0.7, 0.7.1
> > Reporter: Stefan Groschupf
> > Priority: Critical
> > Fix For: 0.8-dev, 0.7.2-dev
> > Attachments: contentProperties_patch.txt
> >
> > As described in issue nutch-133, some webservers return http header meta data not standard conform case insensitive.
> > This provides many negative side effects, for example query thet content type from the meta data return null also in case the webserver returns a content type, but the key is not standard conform e.g. lower case. Also this has effects to the pdf parser that queries the content length etc.
>
> --
> This message is automatically generated by JIRA.
> -
> If you think it was sent incorrectly contact one of the administrators:
> http://issues.apache.org/jira/secure/Administrators.jspa
> -
> For more information on JIRA, see:
> http://www.atlassian.com/software/jira
>
>
--
Keep Discovering ... ...
http://www.jroller.com/page/jmars
[jira] Commented: (NUTCH-135) http header meta data are case insensitive in the real world (e.g. Content-Type or content-type)
Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-135?page=comments#action_12359961 ]
Andrzej Bialecki commented on NUTCH-135:
-----------------------------------------
Since you already are working on this issue, I'd like to ask you to take a look at NUTCH-3, and see if you can solve this too. The problem described there is that if there are several headers with the same name, only the last value is preserved, but in some cases multiple headers make sense (see any of the existing Java models for handling HTTP or RFC822 mail messages - all of them handle multiple values per single key).
> http header meta data are case insensitive in the real world (e.g. Content-Type or content-type)
> ------------------------------------------------------------------------------------------------
>
> Key: NUTCH-135
> URL: http://issues.apache.org/jira/browse/NUTCH-135
> Project: Nutch
> Type: Bug
> Components: fetcher
> Versions: 0.7.1, 0.7
> Reporter: Stefan Groschupf
> Priority: Critical
> Fix For: 0.8-dev, 0.7.2-dev
> Attachments: contentProperties_patch.txt
>
> As described in issue nutch-133, some webservers return http header meta data not standard conform case insensitive.
> This provides many negative side effects, for example query thet content type from the meta data return null also in case the webserver returns a content type, but the key is not standard conform e.g. lower case. Also this has effects to the pdf parser that queries the content length etc.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (NUTCH-135) http header meta data are case insensitive in the real world (e.g. Content-Type or content-type)
Posted by "Jerome Charron (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-135?page=all ]
Jerome Charron updated NUTCH-135:
---------------------------------
Attachment: cached.jsp.patch
cached.jsp must be patched too.
> http header meta data are case insensitive in the real world (e.g. Content-Type or content-type)
> ------------------------------------------------------------------------------------------------
>
> Key: NUTCH-135
> URL: http://issues.apache.org/jira/browse/NUTCH-135
> Project: Nutch
> Type: Bug
> Components: fetcher
> Versions: 0.7, 0.7.1
> Reporter: Stefan Groschupf
> Priority: Critical
> Fix For: 0.8-dev, 0.7.2-dev
> Attachments: cached.jsp.patch, contentProperties_patch.txt, contentProperties_patch_WithContentProperties.txt
>
> As described in issue nutch-133, some webservers return http header meta data not standard conform case insensitive.
> This provides many negative side effects, for example query thet content type from the meta data return null also in case the webserver returns a content type, but the key is not standard conform e.g. lower case. Also this has effects to the pdf parser that queries the content length etc.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (NUTCH-135) http header meta data are case insensitive in the real world (e.g. Content-Type or content-type)
Posted by "Stefan Groschupf (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-135?page=comments#action_12360025 ]
Stefan Groschupf commented on NUTCH-135:
----------------------------------------
Andrzej, that is easy to add to the ContentProperties object and sure I can do that. However first I would love to get a OK for this patch, before I invest more time in it, since I spend to many time writing stuff just for the issue archive.
As soon this patch is in the sources I will write a small new patch (as Doug suggested, do it in small steps) to solve NUTCH-3
> http header meta data are case insensitive in the real world (e.g. Content-Type or content-type)
> ------------------------------------------------------------------------------------------------
>
> Key: NUTCH-135
> URL: http://issues.apache.org/jira/browse/NUTCH-135
> Project: Nutch
> Type: Bug
> Components: fetcher
> Versions: 0.7, 0.7.1
> Reporter: Stefan Groschupf
> Priority: Critical
> Fix For: 0.8-dev, 0.7.2-dev
> Attachments: contentProperties_patch.txt
>
> As described in issue nutch-133, some webservers return http header meta data not standard conform case insensitive.
> This provides many negative side effects, for example query thet content type from the meta data return null also in case the webserver returns a content type, but the key is not standard conform e.g. lower case. Also this has effects to the pdf parser that queries the content length etc.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (NUTCH-135) http header meta data are case insensitive in the real world (e.g. Content-Type or content-type)
Posted by "Stefan Groschupf (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-135?page=all ]
Stefan Groschupf updated NUTCH-135:
-----------------------------------
Attachment: contentProperties_patch_WithContentProperties.txt
missed to add the contentproperties itself to the version control... thanks Jack!
> http header meta data are case insensitive in the real world (e.g. Content-Type or content-type)
> ------------------------------------------------------------------------------------------------
>
> Key: NUTCH-135
> URL: http://issues.apache.org/jira/browse/NUTCH-135
> Project: Nutch
> Type: Bug
> Components: fetcher
> Versions: 0.7, 0.7.1
> Reporter: Stefan Groschupf
> Priority: Critical
> Fix For: 0.8-dev, 0.7.2-dev
> Attachments: contentProperties_patch.txt, contentProperties_patch_WithContentProperties.txt
>
> As described in issue nutch-133, some webservers return http header meta data not standard conform case insensitive.
> This provides many negative side effects, for example query thet content type from the meta data return null also in case the webserver returns a content type, but the key is not standard conform e.g. lower case. Also this has effects to the pdf parser that queries the content length etc.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Resolved: (NUTCH-135) http header meta data are case insensitive in the real world (e.g. Content-Type or content-type)
Posted by "Jerome Charron (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-135?page=all ]
Jerome Charron resolved NUTCH-135:
----------------------------------
Fix Version: (was: 0.7.2-dev)
Resolution: Fixed
Committed to trunk (to be merged into branche 0.7?)
Thanks Stefan.
I have performed unit and functional tests, but I don't have resources for a wide and intensive test.
If someone can perform such test, it would be greatly apreciated.
Note: During my tests, I notice some strange content-types returned by de.yahoo.com and all de.yahoo related files. The content-type returned by the protocol layer to the Content constructor is always text/plain, but when performing some wget on these sites the content-type in headers is text/html ... sorry, I don't have time for more investigations..
> http header meta data are case insensitive in the real world (e.g. Content-Type or content-type)
> ------------------------------------------------------------------------------------------------
>
> Key: NUTCH-135
> URL: http://issues.apache.org/jira/browse/NUTCH-135
> Project: Nutch
> Type: Bug
> Components: fetcher
> Versions: 0.7, 0.7.1
> Reporter: Stefan Groschupf
> Priority: Critical
> Fix For: 0.8-dev
> Attachments: cached.jsp.patch, contentProperties_patch.txt, contentProperties_patch_WithContentProperties.txt
>
> As described in issue nutch-133, some webservers return http header meta data not standard conform case insensitive.
> This provides many negative side effects, for example query thet content type from the meta data return null also in case the webserver returns a content type, but the key is not standard conform e.g. lower case. Also this has effects to the pdf parser that queries the content length etc.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (NUTCH-135) http header meta data are case insensitive in the real world (e.g. Content-Type or content-type)
Posted by "Stefan Groschupf (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/NUTCH-135?page=all ]
Stefan Groschupf updated NUTCH-135:
-----------------------------------
Attachment: contentProperties_patch.txt
As Doug suggested a patch using TreeMap String.CASE_INSENSITIVE_ORDER that solve the problem of case insensitive http header or general case insensitve content meta data.
In general I see two different ways to solve the problem. First leave the API as it is and extend a Properties object to overwriting its methods by using behind the sence a TreeMap. This solution would also require to copy some data between the properties object and treemap back and for several times, since the nutch code uses a Properties object in the content constructor. The other choice would be to change the API of the content object to cleanly document that a other object, that has a different behavior than the properties object is used. The negative thing on this solution is that there are many small changes in the nutch code base.
However I decide for a clean way, the last way, since I don't like code that does some things behind the sence that developers would not expect. So I introduced a tiny ContentProperties object and changed the Content construtor to use the ContentProperties object instead of the java.util.Properties object. The new ContentProperties has a similar API as the Properties class but use case insensitve keys. I changed all classes that use the content object to use the new ContentProperties until object instantiation and I also extend the Content test case to test if case insensitive keys are now supported.
Feel free to give constructive improvement suggestions, but also please let get us this done as soon as possible since from my point of view this is a critical issue. All testcases pass on my box, but please double check before commiting.
> http header meta data are case insensitive in the real world (e.g. Content-Type or content-type)
> ------------------------------------------------------------------------------------------------
>
> Key: NUTCH-135
> URL: http://issues.apache.org/jira/browse/NUTCH-135
> Project: Nutch
> Type: Bug
> Components: fetcher
> Versions: 0.7.1, 0.7
> Reporter: Stefan Groschupf
> Priority: Critical
> Fix For: 0.8-dev, 0.7.2-dev
> Attachments: contentProperties_patch.txt
>
> As described in issue nutch-133, some webservers return http header meta data not standard conform case insensitive.
> This provides many negative side effects, for example query thet content type from the meta data return null also in case the webserver returns a content type, but the key is not standard conform e.g. lower case. Also this has effects to the pdf parser that queries the content length etc.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira