You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Arni Sumarlidason (JIRA)" <ji...@apache.org> on 2012/10/26 03:31:12 UTC
[jira] [Created] (NUTCH-1481) When using MySQL as storage unicode
characters within URLS cause nutch to fail
Arni Sumarlidason created NUTCH-1481:
----------------------------------------
Summary: When using MySQL as storage unicode characters within URLS cause nutch to fail
Key: NUTCH-1481
URL: https://issues.apache.org/jira/browse/NUTCH-1481
Project: Nutch
Issue Type: Bug
Components: crawldb
Affects Versions: 2.1
Environment: mysql 5.5.28 on centos
Reporter: Arni Sumarlidason
MySQL's (innodb) primary key / unique key is restricted to 767 bytes.. currently the url of a web page is used as a primary key in nutch storage.
when using latin1 character set on the 'id' column @ length 767 bytes/characters; unicode characters in urls cause jdbc to throw an exception,
java.io.IOException: java.sql.BatchUpdateException: Incorrect string value: '\xE2\x80\x8' for column 'id' at row 1
when using utf8mb4 character set on the 'id' column @ length 190 characters / 760 bytes to fully support unicode characters; the field length becomes insufficient
It may be better to use a hash of the url as the primary key instead of the url itself. This would allow urls of any length and full utf8 support.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1481) When using MySQL as storage unicode
characters within URLS cause nutch to fail
Posted by "Arni Sumarlidason (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484927#comment-13484927 ]
Arni Sumarlidason commented on NUTCH-1481:
------------------------------------------
I was going to post a work around, but haven't decided where. I actually modified gora-sql-0.1.1-incubating.jar::org.apache.gora.sql.store.SqlStore.flush() to be "less sensitive" to failures. I logged all the URLS that were skipped, around 0.0003% of 150,000 so far, some what insignificant. However, I think if users are going to use less than 4b/character for their 'id' field I don't think the job should fail.
I haven't been able to spend a ton of time familiarizing myself with the nutch source, but it may be possible to add a parameter to webpage that is a hash of the key and setting it to unique in the database. Then we can remove the primary key constraint from ID and use a utf8mb4 character set with a good length.
I am operating under the assumption that the primary key's sole purpose is to reject duplicates in the database.
please advise.
> When using MySQL as storage unicode characters within URLS cause nutch to fail
> ------------------------------------------------------------------------------
>
> Key: NUTCH-1481
> URL: https://issues.apache.org/jira/browse/NUTCH-1481
> Project: Nutch
> Issue Type: Bug
> Components: crawldb
> Affects Versions: 2.1
> Environment: mysql 5.5.28 on centos
> Reporter: Arni Sumarlidason
> Labels: database, sql, unicode, utf8
>
> MySQL's (innodb) primary key / unique key is restricted to 767 bytes.. currently the url of a web page is used as a primary key in nutch storage.
> when using latin1 character set on the 'id' column @ length 767 bytes/characters; unicode characters in urls cause jdbc to throw an exception,
> java.io.IOException: java.sql.BatchUpdateException: Incorrect string value: '\xE2\x80\x8' for column 'id' at row 1
> when using utf8mb4 character set on the 'id' column @ length 190 characters / 760 bytes to fully support unicode characters; the field length becomes insufficient
> It may be better to use a hash of the url as the primary key instead of the url itself. This would allow urls of any length and full utf8 support.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1481) When using MySQL as storage unicode
characters within URLS cause nutch to fail
Posted by "James Sullivan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13493768#comment-13493768 ]
James Sullivan commented on NUTCH-1481:
---------------------------------------
There is a way around the 190 character restriction up to 767 or 768 characters which should be good enough for most URLs. Use the following options for a recent version of MySQL.
innodb_file_format=barracuda
innodb_file_per_table=true
innodb_large_prefix=true
ROW_FORMAT=COMPRESSED
For step by step instructions I've updated http://nlp.solutions.asia/?p=180.
The hash is probably a better long-term solution (given URL is stored in other fields as well) but probably involves mores work.
> When using MySQL as storage unicode characters within URLS cause nutch to fail
> ------------------------------------------------------------------------------
>
> Key: NUTCH-1481
> URL: https://issues.apache.org/jira/browse/NUTCH-1481
> Project: Nutch
> Issue Type: Bug
> Components: crawldb
> Affects Versions: 2.1
> Environment: mysql 5.5.28 on centos
> Reporter: Arni Sumarlidason
> Labels: database, sql, unicode, utf8
>
> MySQL's (innodb) primary key / unique key is restricted to 767 bytes.. currently the url of a web page is used as a primary key in nutch storage.
> when using latin1 character set on the 'id' column @ length 767 bytes/characters; unicode characters in urls cause jdbc to throw an exception,
> java.io.IOException: java.sql.BatchUpdateException: Incorrect string value: '\xE2\x80\x8' for column 'id' at row 1
> when using utf8mb4 character set on the 'id' column @ length 190 characters / 760 bytes to fully support unicode characters; the field length becomes insufficient
> It may be better to use a hash of the url as the primary key instead of the url itself. This would allow urls of any length and full utf8 support.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1481) When using MySQL as storage unicode
characters within URLS cause nutch to fail
Posted by "Lewis John McGibbney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484909#comment-13484909 ]
Lewis John McGibbney commented on NUTCH-1481:
---------------------------------------------
Do you have a suggestion to fix?
> When using MySQL as storage unicode characters within URLS cause nutch to fail
> ------------------------------------------------------------------------------
>
> Key: NUTCH-1481
> URL: https://issues.apache.org/jira/browse/NUTCH-1481
> Project: Nutch
> Issue Type: Bug
> Components: crawldb
> Affects Versions: 2.1
> Environment: mysql 5.5.28 on centos
> Reporter: Arni Sumarlidason
> Labels: database, sql, unicode, utf8
>
> MySQL's (innodb) primary key / unique key is restricted to 767 bytes.. currently the url of a web page is used as a primary key in nutch storage.
> when using latin1 character set on the 'id' column @ length 767 bytes/characters; unicode characters in urls cause jdbc to throw an exception,
> java.io.IOException: java.sql.BatchUpdateException: Incorrect string value: '\xE2\x80\x8' for column 'id' at row 1
> when using utf8mb4 character set on the 'id' column @ length 190 characters / 760 bytes to fully support unicode characters; the field length becomes insufficient
> It may be better to use a hash of the url as the primary key instead of the url itself. This would allow urls of any length and full utf8 support.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1481) When using MySQL as storage unicode
characters within URLS cause nutch to fail
Posted by "Yang Xiao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504326#comment-13504326 ]
Yang Xiao commented on NUTCH-1481:
----------------------------------
It's not right to have unicode in URLs. It's more convenient to use url as id. Adding a url normalizer plug-in would be a better solution.
> When using MySQL as storage unicode characters within URLS cause nutch to fail
> ------------------------------------------------------------------------------
>
> Key: NUTCH-1481
> URL: https://issues.apache.org/jira/browse/NUTCH-1481
> Project: Nutch
> Issue Type: Bug
> Components: crawldb
> Affects Versions: 2.1
> Environment: mysql 5.5.28 on centos
> Reporter: Arni Sumarlidason
> Labels: database, sql, unicode, utf8
>
> MySQL's (innodb) primary key / unique key is restricted to 767 bytes.. currently the url of a web page is used as a primary key in nutch storage.
> when using latin1 character set on the 'id' column @ length 767 bytes/characters; unicode characters in urls cause jdbc to throw an exception,
> java.io.IOException: java.sql.BatchUpdateException: Incorrect string value: '\xE2\x80\x8' for column 'id' at row 1
> when using utf8mb4 character set on the 'id' column @ length 190 characters / 760 bytes to fully support unicode characters; the field length becomes insufficient
> It may be better to use a hash of the url as the primary key instead of the url itself. This would allow urls of any length and full utf8 support.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1481) When using MySQL as storage unicode
characters within URLS cause nutch to fail
Posted by "Arni Sumarlidason (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485143#comment-13485143 ]
Arni Sumarlidason commented on NUTCH-1481:
------------------------------------------
built a patch, testing now
> When using MySQL as storage unicode characters within URLS cause nutch to fail
> ------------------------------------------------------------------------------
>
> Key: NUTCH-1481
> URL: https://issues.apache.org/jira/browse/NUTCH-1481
> Project: Nutch
> Issue Type: Bug
> Components: crawldb
> Affects Versions: 2.1
> Environment: mysql 5.5.28 on centos
> Reporter: Arni Sumarlidason
> Labels: database, sql, unicode, utf8
>
> MySQL's (innodb) primary key / unique key is restricted to 767 bytes.. currently the url of a web page is used as a primary key in nutch storage.
> when using latin1 character set on the 'id' column @ length 767 bytes/characters; unicode characters in urls cause jdbc to throw an exception,
> java.io.IOException: java.sql.BatchUpdateException: Incorrect string value: '\xE2\x80\x8' for column 'id' at row 1
> when using utf8mb4 character set on the 'id' column @ length 190 characters / 760 bytes to fully support unicode characters; the field length becomes insufficient
> It may be better to use a hash of the url as the primary key instead of the url itself. This would allow urls of any length and full utf8 support.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira