You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Markus Jelsma (JIRA)" <ji...@apache.org> on 2011/06/07 22:05:58 UTC
[jira] [Created] (NUTCH-1006) meta equiv with single quotes not
accepted
meta equiv with single quotes not accepted
------------------------------------------
Key: NUTCH-1006
URL: https://issues.apache.org/jira/browse/NUTCH-1006
Project: Nutch
Issue Type: Bug
Components: parser
Affects Versions: 1.2, 1.3, 2.0
Reporter: Markus Jelsma
Fix For: 2.0
As posted by Alex F:
the regex metaPattern inside org.apache.nutch.parse.html.HtmlParser is not
suitable for sites using single quotes for <meta http-equiv....>
Example: <meta http-equiv='Content-Type' content='text/html;
charset=iso-8859-1'>
We experienced a couple of pages with that kind of quotes and Nutch-1.2
was not able to handle it.
Is there any fallback or would it be good to use the following
regex: "<meta\\s+([^>]*http-equiv=(\"|')?content-type(\"|')?[^>]*)>" (single
or regular quotes are accepted)?
See this thread:
http://lucene.472066.n3.nabble.com/Character-encoding-on-Html-Pages-td3034850.html
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (NUTCH-1006) meta equiv with single quotes not
accepted
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma updated NUTCH-1006:
---------------------------------
Affects Version/s: 1.4
Fix Version/s: 1.4
> meta equiv with single quotes not accepted
> ------------------------------------------
>
> Key: NUTCH-1006
> URL: https://issues.apache.org/jira/browse/NUTCH-1006
> Project: Nutch
> Issue Type: Bug
> Components: parser
> Affects Versions: 1.2, 1.3, 1.4, 2.0
> Reporter: Markus Jelsma
> Fix For: 1.4, 2.0
>
>
> As posted by Alex F:
> the regex metaPattern inside org.apache.nutch.parse.html.HtmlParser is not
> suitable for sites using single quotes for <meta http-equiv....>
> Example: <meta http-equiv='Content-Type' content='text/html;
> charset=iso-8859-1'>
> We experienced a couple of pages with that kind of quotes and Nutch-1.2
> was not able to handle it.
> Is there any fallback or would it be good to use the following
> regex: "<meta\\s+([^>]*http-equiv=(\"|')?content-type(\"|')?[^>]*)>" (single
> or regular quotes are accepted)?
> See this thread:
> http://lucene.472066.n3.nabble.com/Character-encoding-on-Html-Pages-td3034850.html
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (NUTCH-1006) meta equiv with single quotes not
accepted
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma updated NUTCH-1006:
---------------------------------
Attachment: NUTCH-1006-104.patch
Patch for 1.4. This changes the regex to allow double and single quotes.
> meta equiv with single quotes not accepted
> ------------------------------------------
>
> Key: NUTCH-1006
> URL: https://issues.apache.org/jira/browse/NUTCH-1006
> Project: Nutch
> Issue Type: Bug
> Components: parser
> Affects Versions: 1.2, 1.3, 1.4, 2.0
> Reporter: Markus Jelsma
> Fix For: 1.4, 2.0
>
> Attachments: NUTCH-1006-104.patch
>
>
> As posted by Alex F:
> the regex metaPattern inside org.apache.nutch.parse.html.HtmlParser is not
> suitable for sites using single quotes for <meta http-equiv....>
> Example: <meta http-equiv='Content-Type' content='text/html;
> charset=iso-8859-1'>
> We experienced a couple of pages with that kind of quotes and Nutch-1.2
> was not able to handle it.
> Is there any fallback or would it be good to use the following
> regex: "<meta\\s+([^>]*http-equiv=(\"|')?content-type(\"|')?[^>]*)>" (single
> or regular quotes are accepted)?
> See this thread:
> http://lucene.472066.n3.nabble.com/Character-encoding-on-Html-Pages-td3034850.html
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (NUTCH-1006) meta equiv with single quotes not
accepted
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma reassigned NUTCH-1006:
------------------------------------
Assignee: Markus Jelsma
> meta equiv with single quotes not accepted
> ------------------------------------------
>
> Key: NUTCH-1006
> URL: https://issues.apache.org/jira/browse/NUTCH-1006
> Project: Nutch
> Issue Type: Bug
> Components: parser
> Affects Versions: 1.2, 1.3, 1.4, 2.0
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4, 2.0
>
> Attachments: NUTCH-1006-104.patch, NUTCH-1006-2.0.patch
>
>
> As posted by Alex F:
> the regex metaPattern inside org.apache.nutch.parse.html.HtmlParser is not
> suitable for sites using single quotes for <meta http-equiv....>
> Example: <meta http-equiv='Content-Type' content='text/html;
> charset=iso-8859-1'>
> We experienced a couple of pages with that kind of quotes and Nutch-1.2
> was not able to handle it.
> Is there any fallback or would it be good to use the following
> regex: "<meta\\s+([^>]*http-equiv=(\"|')?content-type(\"|')?[^>]*)>" (single
> or regular quotes are accepted)?
> See this thread:
> http://lucene.472066.n3.nabble.com/Character-encoding-on-Html-Pages-td3034850.html
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (NUTCH-1006) meta equiv with single
quotes not accepted
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052529#comment-13052529 ]
Markus Jelsma edited comment on NUTCH-1006 at 6/21/11 12:54 PM:
----------------------------------------------------------------
Patch for 1.4. This changes the regex to allow double and single quotes.
edit: tested and it works.
was (Author: markus17):
Patch for 1.4. This changes the regex to allow double and single quotes.
> meta equiv with single quotes not accepted
> ------------------------------------------
>
> Key: NUTCH-1006
> URL: https://issues.apache.org/jira/browse/NUTCH-1006
> Project: Nutch
> Issue Type: Bug
> Components: parser
> Affects Versions: 1.2, 1.3, 1.4, 2.0
> Reporter: Markus Jelsma
> Fix For: 1.4, 2.0
>
> Attachments: NUTCH-1006-104.patch
>
>
> As posted by Alex F:
> the regex metaPattern inside org.apache.nutch.parse.html.HtmlParser is not
> suitable for sites using single quotes for <meta http-equiv....>
> Example: <meta http-equiv='Content-Type' content='text/html;
> charset=iso-8859-1'>
> We experienced a couple of pages with that kind of quotes and Nutch-1.2
> was not able to handle it.
> Is there any fallback or would it be good to use the following
> regex: "<meta\\s+([^>]*http-equiv=(\"|')?content-type(\"|')?[^>]*)>" (single
> or regular quotes are accepted)?
> See this thread:
> http://lucene.472066.n3.nabble.com/Character-encoding-on-Html-Pages-td3034850.html
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1006) meta equiv with single quotes not
accepted
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13053238#comment-13053238 ]
Markus Jelsma commented on NUTCH-1006:
--------------------------------------
Objections? Commit?
> meta equiv with single quotes not accepted
> ------------------------------------------
>
> Key: NUTCH-1006
> URL: https://issues.apache.org/jira/browse/NUTCH-1006
> Project: Nutch
> Issue Type: Bug
> Components: parser
> Affects Versions: 1.2, 1.3, 1.4, 2.0
> Reporter: Markus Jelsma
> Fix For: 1.4, 2.0
>
> Attachments: NUTCH-1006-104.patch, NUTCH-1006-2.0.patch
>
>
> As posted by Alex F:
> the regex metaPattern inside org.apache.nutch.parse.html.HtmlParser is not
> suitable for sites using single quotes for <meta http-equiv....>
> Example: <meta http-equiv='Content-Type' content='text/html;
> charset=iso-8859-1'>
> We experienced a couple of pages with that kind of quotes and Nutch-1.2
> was not able to handle it.
> Is there any fallback or would it be good to use the following
> regex: "<meta\\s+([^>]*http-equiv=(\"|')?content-type(\"|')?[^>]*)>" (single
> or regular quotes are accepted)?
> See this thread:
> http://lucene.472066.n3.nabble.com/Character-encoding-on-Html-Pages-td3034850.html
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (NUTCH-1006) meta equiv with single quotes not
accepted
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma resolved NUTCH-1006.
----------------------------------
Resolution: Fixed
Committed for 1.4 in rev. 1139329 and for trunk in rev. 1139331. Thanks Alex for pointing out this issue.
> meta equiv with single quotes not accepted
> ------------------------------------------
>
> Key: NUTCH-1006
> URL: https://issues.apache.org/jira/browse/NUTCH-1006
> Project: Nutch
> Issue Type: Bug
> Components: parser
> Affects Versions: 1.2, 1.3, 1.4, 2.0
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4, 2.0
>
> Attachments: NUTCH-1006-104.patch, NUTCH-1006-2.0.patch
>
>
> As posted by Alex F:
> the regex metaPattern inside org.apache.nutch.parse.html.HtmlParser is not
> suitable for sites using single quotes for <meta http-equiv....>
> Example: <meta http-equiv='Content-Type' content='text/html;
> charset=iso-8859-1'>
> We experienced a couple of pages with that kind of quotes and Nutch-1.2
> was not able to handle it.
> Is there any fallback or would it be good to use the following
> regex: "<meta\\s+([^>]*http-equiv=(\"|')?content-type(\"|')?[^>]*)>" (single
> or regular quotes are accepted)?
> See this thread:
> http://lucene.472066.n3.nabble.com/Character-encoding-on-Html-Pages-td3034850.html
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (NUTCH-1006) meta equiv with single quotes not
accepted
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma updated NUTCH-1006:
---------------------------------
Attachment: NUTCH-1006-2.0.patch
Patch for 2.0.
> meta equiv with single quotes not accepted
> ------------------------------------------
>
> Key: NUTCH-1006
> URL: https://issues.apache.org/jira/browse/NUTCH-1006
> Project: Nutch
> Issue Type: Bug
> Components: parser
> Affects Versions: 1.2, 1.3, 1.4, 2.0
> Reporter: Markus Jelsma
> Fix For: 1.4, 2.0
>
> Attachments: NUTCH-1006-104.patch, NUTCH-1006-2.0.patch
>
>
> As posted by Alex F:
> the regex metaPattern inside org.apache.nutch.parse.html.HtmlParser is not
> suitable for sites using single quotes for <meta http-equiv....>
> Example: <meta http-equiv='Content-Type' content='text/html;
> charset=iso-8859-1'>
> We experienced a couple of pages with that kind of quotes and Nutch-1.2
> was not able to handle it.
> Is there any fallback or would it be good to use the following
> regex: "<meta\\s+([^>]*http-equiv=(\"|')?content-type(\"|')?[^>]*)>" (single
> or regular quotes are accepted)?
> See this thread:
> http://lucene.472066.n3.nabble.com/Character-encoding-on-Html-Pages-td3034850.html
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1006) meta equiv with single quotes not
accepted
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13056990#comment-13056990 ]
Hudson commented on NUTCH-1006:
-------------------------------
Integrated in Nutch-trunk #1530 (See [https://builds.apache.org/job/Nutch-trunk/1530/])
> meta equiv with single quotes not accepted
> ------------------------------------------
>
> Key: NUTCH-1006
> URL: https://issues.apache.org/jira/browse/NUTCH-1006
> Project: Nutch
> Issue Type: Bug
> Components: parser
> Affects Versions: 1.2, 1.3, 1.4, 2.0
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4, 2.0
>
> Attachments: NUTCH-1006-104.patch, NUTCH-1006-2.0.patch
>
>
> As posted by Alex F:
> the regex metaPattern inside org.apache.nutch.parse.html.HtmlParser is not
> suitable for sites using single quotes for <meta http-equiv....>
> Example: <meta http-equiv='Content-Type' content='text/html;
> charset=iso-8859-1'>
> We experienced a couple of pages with that kind of quotes and Nutch-1.2
> was not able to handle it.
> Is there any fallback or would it be good to use the following
> regex: "<meta\\s+([^>]*http-equiv=(\"|')?content-type(\"|')?[^>]*)>" (single
> or regular quotes are accepted)?
> See this thread:
> http://lucene.472066.n3.nabble.com/Character-encoding-on-Html-Pages-td3034850.html
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (NUTCH-1006) meta equiv with single quotes not
accepted
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma closed NUTCH-1006.
--------------------------------
> meta equiv with single quotes not accepted
> ------------------------------------------
>
> Key: NUTCH-1006
> URL: https://issues.apache.org/jira/browse/NUTCH-1006
> Project: Nutch
> Issue Type: Bug
> Components: parser
> Affects Versions: 1.2, 1.3, 1.4, 2.0
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.4, 2.0
>
> Attachments: NUTCH-1006-104.patch, NUTCH-1006-2.0.patch
>
>
> As posted by Alex F:
> the regex metaPattern inside org.apache.nutch.parse.html.HtmlParser is not
> suitable for sites using single quotes for <meta http-equiv....>
> Example: <meta http-equiv='Content-Type' content='text/html;
> charset=iso-8859-1'>
> We experienced a couple of pages with that kind of quotes and Nutch-1.2
> was not able to handle it.
> Is there any fallback or would it be good to use the following
> regex: "<meta\\s+([^>]*http-equiv=(\"|')?content-type(\"|')?[^>]*)>" (single
> or regular quotes are accepted)?
> See this thread:
> http://lucene.472066.n3.nabble.com/Character-encoding-on-Html-Pages-td3034850.html
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (NUTCH-1006) meta equiv with single quotes not
accepted
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma updated NUTCH-1006:
---------------------------------
Patch Info: [Patch Available]
> meta equiv with single quotes not accepted
> ------------------------------------------
>
> Key: NUTCH-1006
> URL: https://issues.apache.org/jira/browse/NUTCH-1006
> Project: Nutch
> Issue Type: Bug
> Components: parser
> Affects Versions: 1.2, 1.3, 1.4, 2.0
> Reporter: Markus Jelsma
> Fix For: 1.4, 2.0
>
> Attachments: NUTCH-1006-104.patch
>
>
> As posted by Alex F:
> the regex metaPattern inside org.apache.nutch.parse.html.HtmlParser is not
> suitable for sites using single quotes for <meta http-equiv....>
> Example: <meta http-equiv='Content-Type' content='text/html;
> charset=iso-8859-1'>
> We experienced a couple of pages with that kind of quotes and Nutch-1.2
> was not able to handle it.
> Is there any fallback or would it be good to use the following
> regex: "<meta\\s+([^>]*http-equiv=(\"|')?content-type(\"|')?[^>]*)>" (single
> or regular quotes are accepted)?
> See this thread:
> http://lucene.472066.n3.nabble.com/Character-encoding-on-Html-Pages-td3034850.html
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira