You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Lewis John McGibbney (JIRA)" <ji...@apache.org> on 2014/09/28 00:02:34 UTC
[jira] [Resolved] (NUTCH-1481) When using MySQL as storage unicode
characters within URLS cause nutch to fail
[ https://issues.apache.org/jira/browse/NUTCH-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lewis John McGibbney resolved NUTCH-1481.
-----------------------------------------
Resolution: Won't Fix
Fix Version/s: (was: 2.4)
2.3
We will not be fixing this, MySQL is no loner supported in Gora.
> When using MySQL as storage unicode characters within URLS cause nutch to fail
> ------------------------------------------------------------------------------
>
> Key: NUTCH-1481
> URL: https://issues.apache.org/jira/browse/NUTCH-1481
> Project: Nutch
> Issue Type: Bug
> Components: crawldb
> Affects Versions: 2.1
> Environment: mysql 5.5.28 on centos
> Reporter: Arni Sumarlidason
> Labels: database, sql, unicode, utf8
> Fix For: 2.3
>
>
> MySQL's (innodb) primary key / unique key is restricted to 767 bytes.. currently the url of a web page is used as a primary key in nutch storage.
> when using latin1 character set on the 'id' column @ length 767 bytes/characters; unicode characters in urls cause jdbc to throw an exception,
> java.io.IOException: java.sql.BatchUpdateException: Incorrect string value: '\xE2\x80\x8' for column 'id' at row 1
> when using utf8mb4 character set on the 'id' column @ length 190 characters / 760 bytes to fully support unicode characters; the field length becomes insufficient
> It may be better to use a hash of the url as the primary key instead of the url itself. This would allow urls of any length and full utf8 support.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)