You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Marcin Okraszewski (JIRA)" <ji...@apache.org> on 2009/05/28 23:13:45 UTC

[jira] Created: (NUTCH-740) Configuration option to override default language for fetched pages.

Configuration option to override default language for fetched pages.
--------------------------------------------------------------------

                 Key: NUTCH-740
                 URL: https://issues.apache.org/jira/browse/NUTCH-740
             Project: Nutch
          Issue Type: Improvement
          Components: fetcher
    Affects Versions: 1.0.0, 0.9.0
            Reporter: Marcin Okraszewski
             Fix For: 1.0.0


By default "Accept-Language" HTTP request header is set to English. Unfortunately this value is hard coded and seems there is no way to override it. As a result you may index English version of pages even though you would prefer it in different language. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (NUTCH-740) Configuration option to override default language for fetched pages.

Posted by "Otis Gospodnetic (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/NUTCH-740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Otis Gospodnetic updated NUTCH-740:
-----------------------------------

             Priority: Minor  (was: Major)
    Affects Version/s:     (was: 0.9.0)
        Fix Version/s:     (was: 1.0.0)
                       1.1
             Assignee: Otis Gospodnetic

> Configuration option to override default language for fetched pages.
> --------------------------------------------------------------------
>
>                 Key: NUTCH-740
>                 URL: https://issues.apache.org/jira/browse/NUTCH-740
>             Project: Nutch
>          Issue Type: Improvement
>          Components: fetcher
>    Affects Versions: 1.0.0
>            Reporter: Marcin Okraszewski
>            Assignee: Otis Gospodnetic
>            Priority: Minor
>             Fix For: 1.1
>
>         Attachments: AcceptLanguage.patch
>
>
> By default "Accept-Language" HTTP request header is set to English. Unfortunately this value is hard coded and seems there is no way to override it. As a result you may index English version of pages even though you would prefer it in different language. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (NUTCH-740) Configuration option to override default language for fetched pages.

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/NUTCH-740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12848537#action_12848537 ] 

Hudson commented on NUTCH-740:
------------------------------

Integrated in Nutch-trunk #1104 (See [http://hudson.zones.apache.org/hudson/job/Nutch-trunk/1104/])
     Configuration option to override default language for fetched pages


> Configuration option to override default language for fetched pages.
> --------------------------------------------------------------------
>
>                 Key: NUTCH-740
>                 URL: https://issues.apache.org/jira/browse/NUTCH-740
>             Project: Nutch
>          Issue Type: Improvement
>          Components: fetcher
>    Affects Versions: 1.0.0
>            Reporter: Marcin Okraszewski
>            Assignee: Julien Nioche
>            Priority: Minor
>             Fix For: 1.1
>
>         Attachments: AcceptLanguage.patch, AcceptLanguage_trunk_2009-06-09.patch, NUTCH-740.patch
>
>
> By default "Accept-Language" HTTP request header is set to English. Unfortunately this value is hard coded and seems there is no way to override it. As a result you may index English version of pages even though you would prefer it in different language. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Closed: (NUTCH-740) Configuration option to override default language for fetched pages.

Posted by "Julien Nioche (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/NUTCH-740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Julien Nioche closed NUTCH-740.
-------------------------------

    Resolution: Fixed
      Assignee: Julien Nioche

Committed in rev 926003
Thanks Marcin for contributing this patch

> Configuration option to override default language for fetched pages.
> --------------------------------------------------------------------
>
>                 Key: NUTCH-740
>                 URL: https://issues.apache.org/jira/browse/NUTCH-740
>             Project: Nutch
>          Issue Type: Improvement
>          Components: fetcher
>    Affects Versions: 1.0.0
>            Reporter: Marcin Okraszewski
>            Assignee: Julien Nioche
>            Priority: Minor
>             Fix For: 1.1
>
>         Attachments: AcceptLanguage.patch, AcceptLanguage_trunk_2009-06-09.patch, NUTCH-740.patch
>
>
> By default "Accept-Language" HTTP request header is set to English. Unfortunately this value is hard coded and seems there is no way to override it. As a result you may index English version of pages even though you would prefer it in different language. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (NUTCH-740) Configuration option to override default language for fetched pages.

Posted by "Julien Nioche (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/NUTCH-740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12845886#action_12845886 ] 

Julien Nioche commented on NUTCH-740:
-------------------------------------

A nice contribution but should not this be applied to the *protocol-http* plugin as well e.g. in HttpResponse?

> Configuration option to override default language for fetched pages.
> --------------------------------------------------------------------
>
>                 Key: NUTCH-740
>                 URL: https://issues.apache.org/jira/browse/NUTCH-740
>             Project: Nutch
>          Issue Type: Improvement
>          Components: fetcher
>    Affects Versions: 1.0.0
>            Reporter: Marcin Okraszewski
>            Assignee: Otis Gospodnetic
>            Priority: Minor
>             Fix For: 1.1
>
>         Attachments: AcceptLanguage.patch, AcceptLanguage_trunk_2009-06-09.patch
>
>
> By default "Accept-Language" HTTP request header is set to English. Unfortunately this value is hard coded and seems there is no way to override it. As a result you may index English version of pages even though you would prefer it in different language. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (NUTCH-740) Configuration option to override default language for fetched pages.

Posted by "Julien Nioche (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/NUTCH-740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Julien Nioche updated NUTCH-740:
--------------------------------

    Attachment: NUTCH-740.patch

Slightly modified version of the patch with modifs for protocol-http.
will commit shortly

> Configuration option to override default language for fetched pages.
> --------------------------------------------------------------------
>
>                 Key: NUTCH-740
>                 URL: https://issues.apache.org/jira/browse/NUTCH-740
>             Project: Nutch
>          Issue Type: Improvement
>          Components: fetcher
>    Affects Versions: 1.0.0
>            Reporter: Marcin Okraszewski
>            Priority: Minor
>             Fix For: 1.1
>
>         Attachments: AcceptLanguage.patch, AcceptLanguage_trunk_2009-06-09.patch, NUTCH-740.patch
>
>
> By default "Accept-Language" HTTP request header is set to English. Unfortunately this value is hard coded and seems there is no way to override it. As a result you may index English version of pages even though you would prefer it in different language. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (NUTCH-740) Configuration option to override default language for fetched pages.

Posted by "Marcin Okraszewski (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/NUTCH-740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Marcin Okraszewski updated NUTCH-740:
-------------------------------------

    Attachment: AcceptLanguage_trunk_2009-06-09.patch

It does apply, but with Fuzz factor set to 2. Here is the ported patch.

> Configuration option to override default language for fetched pages.
> --------------------------------------------------------------------
>
>                 Key: NUTCH-740
>                 URL: https://issues.apache.org/jira/browse/NUTCH-740
>             Project: Nutch
>          Issue Type: Improvement
>          Components: fetcher
>    Affects Versions: 1.0.0
>            Reporter: Marcin Okraszewski
>            Assignee: Otis Gospodnetic
>            Priority: Minor
>             Fix For: 1.1
>
>         Attachments: AcceptLanguage.patch, AcceptLanguage_trunk_2009-06-09.patch
>
>
> By default "Accept-Language" HTTP request header is set to English. Unfortunately this value is hard coded and seems there is no way to override it. As a result you may index English version of pages even though you would prefer it in different language. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (NUTCH-740) Configuration option to override default language for fetched pages.

Posted by "Marcin Okraszewski (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/NUTCH-740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Marcin Okraszewski updated NUTCH-740:
-------------------------------------

    Attachment: AcceptLanguage.patch

The patch which allows overriding of "Accept-Language" header. The patch is done on 1.0 code. 

> Configuration option to override default language for fetched pages.
> --------------------------------------------------------------------
>
>                 Key: NUTCH-740
>                 URL: https://issues.apache.org/jira/browse/NUTCH-740
>             Project: Nutch
>          Issue Type: Improvement
>          Components: fetcher
>    Affects Versions: 0.9.0, 1.0.0
>            Reporter: Marcin Okraszewski
>             Fix For: 1.0.0
>
>         Attachments: AcceptLanguage.patch
>
>
> By default "Accept-Language" HTTP request header is set to English. Unfortunately this value is hard coded and seems there is no way to override it. As a result you may index English version of pages even though you would prefer it in different language. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (NUTCH-740) Configuration option to override default language for fetched pages.

Posted by "Doğacan Güney (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/NUTCH-740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717061#action_12717061 ] 

Doğacan Güney commented on NUTCH-740:
-------------------------------------

+1 for the idea but your patch does not apply to trunk.

> Configuration option to override default language for fetched pages.
> --------------------------------------------------------------------
>
>                 Key: NUTCH-740
>                 URL: https://issues.apache.org/jira/browse/NUTCH-740
>             Project: Nutch
>          Issue Type: Improvement
>          Components: fetcher
>    Affects Versions: 1.0.0
>            Reporter: Marcin Okraszewski
>            Assignee: Otis Gospodnetic
>            Priority: Minor
>             Fix For: 1.1
>
>         Attachments: AcceptLanguage.patch
>
>
> By default "Accept-Language" HTTP request header is set to English. Unfortunately this value is hard coded and seems there is no way to override it. As a result you may index English version of pages even though you would prefer it in different language. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (NUTCH-740) Configuration option to override default language for fetched pages.

Posted by "Otis Gospodnetic (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/NUTCH-740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Otis Gospodnetic updated NUTCH-740:
-----------------------------------

    Assignee:     (was: Otis Gospodnetic)

> Configuration option to override default language for fetched pages.
> --------------------------------------------------------------------
>
>                 Key: NUTCH-740
>                 URL: https://issues.apache.org/jira/browse/NUTCH-740
>             Project: Nutch
>          Issue Type: Improvement
>          Components: fetcher
>    Affects Versions: 1.0.0
>            Reporter: Marcin Okraszewski
>            Priority: Minor
>             Fix For: 1.1
>
>         Attachments: AcceptLanguage.patch, AcceptLanguage_trunk_2009-06-09.patch
>
>
> By default "Accept-Language" HTTP request header is set to English. Unfortunately this value is hard coded and seems there is no way to override it. As a result you may index English version of pages even though you would prefer it in different language. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.