[jira] [Commented] (NUTCH-1481) When using MySQL as storage unicode characters within URLS cause nutch to fail

["James Sullivan (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NUTCH-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13493768#comment-13493768 ] 

James Sullivan commented on NUTCH-1481:
---------------------------------------

There is a way around the 190 character restriction up to 767 or 768 characters which should be good enough for most URLs. Use the following options for a recent version of MySQL.

innodb_file_format=barracuda
innodb_file_per_table=true
innodb_large_prefix=true
ROW_FORMAT=COMPRESSED

For step by step instructions I've updated http://nlp.solutions.asia/?p=180.

The hash is probably a better long-term solution (given URL is stored in other fields as well) but probably involves mores work.

                
> When using MySQL as storage unicode characters within URLS cause nutch to fail
> ------------------------------------------------------------------------------
>
>                 Key: NUTCH-1481
>                 URL: https://issues.apache.org/jira/browse/NUTCH-1481
>             Project: Nutch
>          Issue Type: Bug
>          Components: crawldb
>    Affects Versions: 2.1
>         Environment: mysql 5.5.28 on centos
>            Reporter: Arni Sumarlidason
>              Labels: database, sql, unicode, utf8
>
> MySQL's (innodb) primary key / unique key is restricted to 767 bytes.. currently the url of a web page is used as a primary key in nutch storage.
> when using latin1 character set on the 'id' column @ length 767 bytes/characters; unicode characters in urls cause jdbc to throw an exception,
> java.io.IOException: java.sql.BatchUpdateException: Incorrect string value: '\xE2\x80\x8' for column 'id' at row 1
> when using utf8mb4 character set on the 'id' column @ length 190 characters / 760 bytes to fully support unicode characters; the field length becomes insufficient
> It may be better to use a hash of the url as the primary key instead of the url itself. This would allow urls of any length and full utf8 support.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira