You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Carl Steinbach (JIRA)" <ji...@apache.org> on 2010/05/24 18:34:24 UTC
[jira] Created: (HIVE-1364) Increase the maximum length of
SERDEPROPERTIES values (currently 767 characters)
Increase the maximum length of SERDEPROPERTIES values (currently 767 characters)
--------------------------------------------------------------------------------
Key: HIVE-1364
URL: https://issues.apache.org/jira/browse/HIVE-1364
Project: Hadoop Hive
Issue Type: Bug
Components: Metastore
Reporter: Carl Steinbach
The value component of a SERDEPROPERTIES key/value pair is currently limited
to a maximum length of 767 characters. I believe that the motivation for limiting the length to
767 characters is that this value is the maximum allowed length of an index in
a MySQL database running on the InnoDB engine: http://bugs.mysql.com/bug.php?id=13315
* The Metastore OR mapping currently limits many fields (including SERDEPROPERTIES.PARAM_VALUE) to a maximum length of 767 characters despite the fact that these fields are not indexed.
* The maximum length of a VARCHAR value in MySQL 5.0.3 and later is 65,535.
* We can expect many users to hit the 767 character limit on SERDEPROPERTIES.PARAM_VALUE when using the hbase.columns.mapping serdeproperty to map a table that has many columns.
I propose increasing the maximum allowed length of SERDEPROPERTIES.PARAM_VALUE to 8192.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1364) Increase the maximum length of
SERDEPROPERTIES values (currently 767 characters)
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891753#action_12891753 ]
John Sichi commented on HIVE-1364:
----------------------------------
I think we should go ahead with changing everything. Even ftype could get large for complex nested types, right Ashish?
But I really think if we're going to do this, we should go to LOB unless I'm wrong about it being possible to do in-place.
Side note: I recently found out that in InnoDB (at least the version in use at Facebook), the first 512 bytes of each LOB is stored inline in the row, and there's an overall inline row limit of 8K, or something like that. So a table with a lot of LOB columns can still hit a limit, which is annoying.
> Increase the maximum length of SERDEPROPERTIES values (currently 767 characters)
> --------------------------------------------------------------------------------
>
> Key: HIVE-1364
> URL: https://issues.apache.org/jira/browse/HIVE-1364
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.5.0
> Reporter: Carl Steinbach
> Assignee: Carl Steinbach
> Fix For: 0.6.0, 0.7.0
>
> Attachments: HIVE-1364.2.patch.txt, HIVE-1364.patch
>
>
> The value component of a SERDEPROPERTIES key/value pair is currently limited
> to a maximum length of 767 characters. I believe that the motivation for limiting the length to
> 767 characters is that this value is the maximum allowed length of an index in
> a MySQL database running on the InnoDB engine: http://bugs.mysql.com/bug.php?id=13315
> * The Metastore OR mapping currently limits many fields (including SERDEPROPERTIES.PARAM_VALUE) to a maximum length of 767 characters despite the fact that these fields are not indexed.
> * The maximum length of a VARCHAR value in MySQL 5.0.3 and later is 65,535.
> * We can expect many users to hit the 767 character limit on SERDEPROPERTIES.PARAM_VALUE when using the hbase.columns.mapping serdeproperty to map a table that has many columns.
> I propose increasing the maximum allowed length of SERDEPROPERTIES.PARAM_VALUE to 8192.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1364) Increase the maximum length of
SERDEPROPERTIES values (currently 767 characters)
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879635#action_12879635 ]
John Sichi commented on HIVE-1364:
----------------------------------
Also, PART_NAME in table PARTITIONS needs to remain as is, since it is covered by an index.
> Increase the maximum length of SERDEPROPERTIES values (currently 767 characters)
> --------------------------------------------------------------------------------
>
> Key: HIVE-1364
> URL: https://issues.apache.org/jira/browse/HIVE-1364
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.5.0
> Reporter: Carl Steinbach
> Assignee: Carl Steinbach
> Fix For: 0.6.0
>
> Attachments: HIVE-1364.patch
>
>
> The value component of a SERDEPROPERTIES key/value pair is currently limited
> to a maximum length of 767 characters. I believe that the motivation for limiting the length to
> 767 characters is that this value is the maximum allowed length of an index in
> a MySQL database running on the InnoDB engine: http://bugs.mysql.com/bug.php?id=13315
> * The Metastore OR mapping currently limits many fields (including SERDEPROPERTIES.PARAM_VALUE) to a maximum length of 767 characters despite the fact that these fields are not indexed.
> * The maximum length of a VARCHAR value in MySQL 5.0.3 and later is 65,535.
> * We can expect many users to hit the 767 character limit on SERDEPROPERTIES.PARAM_VALUE when using the hbase.columns.mapping serdeproperty to map a table that has many columns.
> I propose increasing the maximum allowed length of SERDEPROPERTIES.PARAM_VALUE to 8192.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1364) Increase the maximum length of
SERDEPROPERTIES values (currently 767 characters)
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879648#action_12879648 ]
John Sichi commented on HIVE-1364:
----------------------------------
I'm not sure we need to use the deprecation approach. In Java land, it's all just String regardless of the underlying character precision in the DB.
For existing metastores, people are already going to need to run upgrade SQL commands against their metastore DB's when upgrading to 0.6 because of the new support for views. We can just add on to those scripts.
> Increase the maximum length of SERDEPROPERTIES values (currently 767 characters)
> --------------------------------------------------------------------------------
>
> Key: HIVE-1364
> URL: https://issues.apache.org/jira/browse/HIVE-1364
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.5.0
> Reporter: Carl Steinbach
> Assignee: Carl Steinbach
> Fix For: 0.6.0
>
> Attachments: HIVE-1364.2.patch.txt, HIVE-1364.patch
>
>
> The value component of a SERDEPROPERTIES key/value pair is currently limited
> to a maximum length of 767 characters. I believe that the motivation for limiting the length to
> 767 characters is that this value is the maximum allowed length of an index in
> a MySQL database running on the InnoDB engine: http://bugs.mysql.com/bug.php?id=13315
> * The Metastore OR mapping currently limits many fields (including SERDEPROPERTIES.PARAM_VALUE) to a maximum length of 767 characters despite the fact that these fields are not indexed.
> * The maximum length of a VARCHAR value in MySQL 5.0.3 and later is 65,535.
> * We can expect many users to hit the 767 character limit on SERDEPROPERTIES.PARAM_VALUE when using the hbase.columns.mapping serdeproperty to map a table that has many columns.
> I propose increasing the maximum allowed length of SERDEPROPERTIES.PARAM_VALUE to 8192.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1364) Increase the maximum length of
SERDEPROPERTIES values (currently 767 characters)
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Steinbach updated HIVE-1364:
---------------------------------
Fix Version/s: 0.7.0
> Increase the maximum length of SERDEPROPERTIES values (currently 767 characters)
> --------------------------------------------------------------------------------
>
> Key: HIVE-1364
> URL: https://issues.apache.org/jira/browse/HIVE-1364
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.5.0
> Reporter: Carl Steinbach
> Assignee: Carl Steinbach
> Fix For: 0.6.0, 0.7.0
>
> Attachments: HIVE-1364.2.patch.txt, HIVE-1364.patch
>
>
> The value component of a SERDEPROPERTIES key/value pair is currently limited
> to a maximum length of 767 characters. I believe that the motivation for limiting the length to
> 767 characters is that this value is the maximum allowed length of an index in
> a MySQL database running on the InnoDB engine: http://bugs.mysql.com/bug.php?id=13315
> * The Metastore OR mapping currently limits many fields (including SERDEPROPERTIES.PARAM_VALUE) to a maximum length of 767 characters despite the fact that these fields are not indexed.
> * The maximum length of a VARCHAR value in MySQL 5.0.3 and later is 65,535.
> * We can expect many users to hit the 767 character limit on SERDEPROPERTIES.PARAM_VALUE when using the hbase.columns.mapping serdeproperty to map a table that has many columns.
> I propose increasing the maximum allowed length of SERDEPROPERTIES.PARAM_VALUE to 8192.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1364) Increase the maximum length of
SERDEPROPERTIES values (currently 767 characters)
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12914311#action_12914311 ]
John Sichi commented on HIVE-1364:
----------------------------------
+1 from me pending metastore testing by Paul.
> Increase the maximum length of SERDEPROPERTIES values (currently 767 characters)
> --------------------------------------------------------------------------------
>
> Key: HIVE-1364
> URL: https://issues.apache.org/jira/browse/HIVE-1364
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.5.0
> Reporter: Carl Steinbach
> Assignee: Carl Steinbach
> Fix For: 0.6.0, 0.7.0
>
> Attachments: HIVE-1364.2.patch.txt, HIVE-1364.3.backport-060.patch.txt, HIVE-1364.3.patch.txt, HIVE-1364.4.backport-060.patch.txt, HIVE-1364.4.patch.txt, HIVE-1364.patch
>
>
> The value component of a SERDEPROPERTIES key/value pair is currently limited
> to a maximum length of 767 characters. I believe that the motivation for limiting the length to
> 767 characters is that this value is the maximum allowed length of an index in
> a MySQL database running on the InnoDB engine: http://bugs.mysql.com/bug.php?id=13315
> * The Metastore OR mapping currently limits many fields (including SERDEPROPERTIES.PARAM_VALUE) to a maximum length of 767 characters despite the fact that these fields are not indexed.
> * The maximum length of a VARCHAR value in MySQL 5.0.3 and later is 65,535.
> * We can expect many users to hit the 767 character limit on SERDEPROPERTIES.PARAM_VALUE when using the hbase.columns.mapping serdeproperty to map a table that has many columns.
> I propose increasing the maximum allowed length of SERDEPROPERTIES.PARAM_VALUE to 8192.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1364) Increase the maximum length of
SERDEPROPERTIES values (currently 767 characters)
Posted by "Ashish Thusoo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881324#action_12881324 ]
Ashish Thusoo commented on HIVE-1364:
-------------------------------------
Also why do we make everything 4000 bytes - I presume things like ftype will never hit that limit.
Also changes to upgrade SQL should also be a part of the patch, no? Where are the scripts for the view change located?
> Increase the maximum length of SERDEPROPERTIES values (currently 767 characters)
> --------------------------------------------------------------------------------
>
> Key: HIVE-1364
> URL: https://issues.apache.org/jira/browse/HIVE-1364
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.5.0
> Reporter: Carl Steinbach
> Assignee: Carl Steinbach
> Fix For: 0.6.0
>
> Attachments: HIVE-1364.2.patch.txt, HIVE-1364.patch
>
>
> The value component of a SERDEPROPERTIES key/value pair is currently limited
> to a maximum length of 767 characters. I believe that the motivation for limiting the length to
> 767 characters is that this value is the maximum allowed length of an index in
> a MySQL database running on the InnoDB engine: http://bugs.mysql.com/bug.php?id=13315
> * The Metastore OR mapping currently limits many fields (including SERDEPROPERTIES.PARAM_VALUE) to a maximum length of 767 characters despite the fact that these fields are not indexed.
> * The maximum length of a VARCHAR value in MySQL 5.0.3 and later is 65,535.
> * We can expect many users to hit the 767 character limit on SERDEPROPERTIES.PARAM_VALUE when using the hbase.columns.mapping serdeproperty to map a table that has many columns.
> I propose increasing the maximum allowed length of SERDEPROPERTIES.PARAM_VALUE to 8192.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1364) Increase the maximum length of various
metastore fields, and remove TYPE_NAME from COLUMNS primary key
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Steinbach updated HIVE-1364:
---------------------------------
Fix Version/s: (was: 0.7.0)
> Increase the maximum length of various metastore fields, and remove TYPE_NAME from COLUMNS primary key
> ------------------------------------------------------------------------------------------------------
>
> Key: HIVE-1364
> URL: https://issues.apache.org/jira/browse/HIVE-1364
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.5.0
> Reporter: Carl Steinbach
> Assignee: Carl Steinbach
> Fix For: 0.6.0
>
> Attachments: HIVE-1364.2.patch.txt, HIVE-1364.3.backport-060.patch.txt, HIVE-1364.3.patch.txt, HIVE-1364.4.backport-060.patch.txt, HIVE-1364.4.patch.txt, HIVE-1364.patch
>
>
> The value component of a SERDEPROPERTIES key/value pair is currently limited
> to a maximum length of 767 characters. I believe that the motivation for limiting the length to
> 767 characters is that this value is the maximum allowed length of an index in
> a MySQL database running on the InnoDB engine: http://bugs.mysql.com/bug.php?id=13315
> * The Metastore OR mapping currently limits many fields (including SERDEPROPERTIES.PARAM_VALUE) to a maximum length of 767 characters despite the fact that these fields are not indexed.
> * The maximum length of a VARCHAR value in MySQL 5.0.3 and later is 65,535.
> * We can expect many users to hit the 767 character limit on SERDEPROPERTIES.PARAM_VALUE when using the hbase.columns.mapping serdeproperty to map a table that has many columns.
> I propose increasing the maximum allowed length of SERDEPROPERTIES.PARAM_VALUE to 8192.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1364) Increase the maximum length of
SERDEPROPERTIES values (currently 767 characters)
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12910361#action_12910361 ]
John Sichi commented on HIVE-1364:
----------------------------------
(Actually TYPE_NAME is what was needed for HIVE-1632, not FTYPE, but I think we should address both.)
> Increase the maximum length of SERDEPROPERTIES values (currently 767 characters)
> --------------------------------------------------------------------------------
>
> Key: HIVE-1364
> URL: https://issues.apache.org/jira/browse/HIVE-1364
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.5.0
> Reporter: Carl Steinbach
> Assignee: Carl Steinbach
> Fix For: 0.6.0, 0.7.0
>
> Attachments: HIVE-1364.2.patch.txt, HIVE-1364.patch
>
>
> The value component of a SERDEPROPERTIES key/value pair is currently limited
> to a maximum length of 767 characters. I believe that the motivation for limiting the length to
> 767 characters is that this value is the maximum allowed length of an index in
> a MySQL database running on the InnoDB engine: http://bugs.mysql.com/bug.php?id=13315
> * The Metastore OR mapping currently limits many fields (including SERDEPROPERTIES.PARAM_VALUE) to a maximum length of 767 characters despite the fact that these fields are not indexed.
> * The maximum length of a VARCHAR value in MySQL 5.0.3 and later is 65,535.
> * We can expect many users to hit the 767 character limit on SERDEPROPERTIES.PARAM_VALUE when using the hbase.columns.mapping serdeproperty to map a table that has many columns.
> I propose increasing the maximum allowed length of SERDEPROPERTIES.PARAM_VALUE to 8192.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1364) Increase the maximum length of
SERDEPROPERTIES values (currently 767 characters)
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12870752#action_12870752 ]
Carl Steinbach commented on HIVE-1364:
--------------------------------------
The max length of a VARCHAR in Oracle (9i/10g/11g) is 4000 bytes, so
I suppose it makes sense to use 4000 as the default size for string properties
that we don't need to index.
> Increase the maximum length of SERDEPROPERTIES values (currently 767 characters)
> --------------------------------------------------------------------------------
>
> Key: HIVE-1364
> URL: https://issues.apache.org/jira/browse/HIVE-1364
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.5.0
> Reporter: Carl Steinbach
>
> The value component of a SERDEPROPERTIES key/value pair is currently limited
> to a maximum length of 767 characters. I believe that the motivation for limiting the length to
> 767 characters is that this value is the maximum allowed length of an index in
> a MySQL database running on the InnoDB engine: http://bugs.mysql.com/bug.php?id=13315
> * The Metastore OR mapping currently limits many fields (including SERDEPROPERTIES.PARAM_VALUE) to a maximum length of 767 characters despite the fact that these fields are not indexed.
> * The maximum length of a VARCHAR value in MySQL 5.0.3 and later is 65,535.
> * We can expect many users to hit the 767 character limit on SERDEPROPERTIES.PARAM_VALUE when using the hbase.columns.mapping serdeproperty to map a table that has many columns.
> I propose increasing the maximum allowed length of SERDEPROPERTIES.PARAM_VALUE to 8192.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1364) Increase the maximum length of
SERDEPROPERTIES values (currently 767 characters)
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Steinbach updated HIVE-1364:
---------------------------------
Fix Version/s: 0.6.0
> Increase the maximum length of SERDEPROPERTIES values (currently 767 characters)
> --------------------------------------------------------------------------------
>
> Key: HIVE-1364
> URL: https://issues.apache.org/jira/browse/HIVE-1364
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.5.0
> Reporter: Carl Steinbach
> Assignee: Carl Steinbach
> Fix For: 0.6.0
>
> Attachments: HIVE-1364.patch
>
>
> The value component of a SERDEPROPERTIES key/value pair is currently limited
> to a maximum length of 767 characters. I believe that the motivation for limiting the length to
> 767 characters is that this value is the maximum allowed length of an index in
> a MySQL database running on the InnoDB engine: http://bugs.mysql.com/bug.php?id=13315
> * The Metastore OR mapping currently limits many fields (including SERDEPROPERTIES.PARAM_VALUE) to a maximum length of 767 characters despite the fact that these fields are not indexed.
> * The maximum length of a VARCHAR value in MySQL 5.0.3 and later is 65,535.
> * We can expect many users to hit the 767 character limit on SERDEPROPERTIES.PARAM_VALUE when using the hbase.columns.mapping serdeproperty to map a table that has many columns.
> I propose increasing the maximum allowed length of SERDEPROPERTIES.PARAM_VALUE to 8192.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1364) Increase the maximum length of
SERDEPROPERTIES values (currently 767 characters)
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879093#action_12879093 ]
John Sichi commented on HIVE-1364:
----------------------------------
Couldn't we just use a LOB? So far so good with views (except for the Oracle mapping which we still need to address to get it to use CLOB instead of LONG VARCHAR).
> Increase the maximum length of SERDEPROPERTIES values (currently 767 characters)
> --------------------------------------------------------------------------------
>
> Key: HIVE-1364
> URL: https://issues.apache.org/jira/browse/HIVE-1364
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.5.0
> Reporter: Carl Steinbach
> Assignee: Carl Steinbach
> Attachments: HIVE-1364.patch
>
>
> The value component of a SERDEPROPERTIES key/value pair is currently limited
> to a maximum length of 767 characters. I believe that the motivation for limiting the length to
> 767 characters is that this value is the maximum allowed length of an index in
> a MySQL database running on the InnoDB engine: http://bugs.mysql.com/bug.php?id=13315
> * The Metastore OR mapping currently limits many fields (including SERDEPROPERTIES.PARAM_VALUE) to a maximum length of 767 characters despite the fact that these fields are not indexed.
> * The maximum length of a VARCHAR value in MySQL 5.0.3 and later is 65,535.
> * We can expect many users to hit the 767 character limit on SERDEPROPERTIES.PARAM_VALUE when using the hbase.columns.mapping serdeproperty to map a table that has many columns.
> I propose increasing the maximum allowed length of SERDEPROPERTIES.PARAM_VALUE to 8192.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1364) Increase the maximum length of
SERDEPROPERTIES values (currently 767 characters)
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881361#action_12881361 ]
John Sichi commented on HIVE-1364:
----------------------------------
Currently the view scripts are only in the wiki:
http://wiki.apache.org/hadoop/Hive/ViewDev#Metastore_Upgrades
Per discussion with Ashish, we should open a separate JIRA issue for (at a minimum) packaging up example MySQL migration scripts (cumulative across all schema changes from 0.5 to 0.6) and explaining what to do with them in the release notes. Carl, do you want to take that on as part of release mgmt?
> Increase the maximum length of SERDEPROPERTIES values (currently 767 characters)
> --------------------------------------------------------------------------------
>
> Key: HIVE-1364
> URL: https://issues.apache.org/jira/browse/HIVE-1364
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.5.0
> Reporter: Carl Steinbach
> Assignee: Carl Steinbach
> Fix For: 0.6.0
>
> Attachments: HIVE-1364.2.patch.txt, HIVE-1364.patch
>
>
> The value component of a SERDEPROPERTIES key/value pair is currently limited
> to a maximum length of 767 characters. I believe that the motivation for limiting the length to
> 767 characters is that this value is the maximum allowed length of an index in
> a MySQL database running on the InnoDB engine: http://bugs.mysql.com/bug.php?id=13315
> * The Metastore OR mapping currently limits many fields (including SERDEPROPERTIES.PARAM_VALUE) to a maximum length of 767 characters despite the fact that these fields are not indexed.
> * The maximum length of a VARCHAR value in MySQL 5.0.3 and later is 65,535.
> * We can expect many users to hit the 767 character limit on SERDEPROPERTIES.PARAM_VALUE when using the hbase.columns.mapping serdeproperty to map a table that has many columns.
> I propose increasing the maximum allowed length of SERDEPROPERTIES.PARAM_VALUE to 8192.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1364) Increase the maximum length of
various metastore fields, and remove TYPE_NAME from COLUMNS primary key
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12918356#action_12918356 ]
Namit Jain commented on HIVE-1364:
----------------------------------
We tested this on our production metastore db.
The downtime should be acceptable - I will start the tests
+1
> Increase the maximum length of various metastore fields, and remove TYPE_NAME from COLUMNS primary key
> ------------------------------------------------------------------------------------------------------
>
> Key: HIVE-1364
> URL: https://issues.apache.org/jira/browse/HIVE-1364
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.5.0
> Reporter: Carl Steinbach
> Assignee: Carl Steinbach
> Fix For: 0.6.0, 0.7.0
>
> Attachments: HIVE-1364.2.patch.txt, HIVE-1364.3.backport-060.patch.txt, HIVE-1364.3.patch.txt, HIVE-1364.4.backport-060.patch.txt, HIVE-1364.4.patch.txt, HIVE-1364.patch
>
>
> The value component of a SERDEPROPERTIES key/value pair is currently limited
> to a maximum length of 767 characters. I believe that the motivation for limiting the length to
> 767 characters is that this value is the maximum allowed length of an index in
> a MySQL database running on the InnoDB engine: http://bugs.mysql.com/bug.php?id=13315
> * The Metastore OR mapping currently limits many fields (including SERDEPROPERTIES.PARAM_VALUE) to a maximum length of 767 characters despite the fact that these fields are not indexed.
> * The maximum length of a VARCHAR value in MySQL 5.0.3 and later is 65,535.
> * We can expect many users to hit the 767 character limit on SERDEPROPERTIES.PARAM_VALUE when using the hbase.columns.mapping serdeproperty to map a table that has many columns.
> I propose increasing the maximum allowed length of SERDEPROPERTIES.PARAM_VALUE to 8192.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1364) Increase the maximum length of
SERDEPROPERTIES values (currently 767 characters)
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12908592#action_12908592 ]
John Sichi commented on HIVE-1364:
----------------------------------
See HIVE-1632 for an ftype use case.
> Increase the maximum length of SERDEPROPERTIES values (currently 767 characters)
> --------------------------------------------------------------------------------
>
> Key: HIVE-1364
> URL: https://issues.apache.org/jira/browse/HIVE-1364
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.5.0
> Reporter: Carl Steinbach
> Assignee: Carl Steinbach
> Fix For: 0.6.0, 0.7.0
>
> Attachments: HIVE-1364.2.patch.txt, HIVE-1364.patch
>
>
> The value component of a SERDEPROPERTIES key/value pair is currently limited
> to a maximum length of 767 characters. I believe that the motivation for limiting the length to
> 767 characters is that this value is the maximum allowed length of an index in
> a MySQL database running on the InnoDB engine: http://bugs.mysql.com/bug.php?id=13315
> * The Metastore OR mapping currently limits many fields (including SERDEPROPERTIES.PARAM_VALUE) to a maximum length of 767 characters despite the fact that these fields are not indexed.
> * The maximum length of a VARCHAR value in MySQL 5.0.3 and later is 65,535.
> * We can expect many users to hit the 767 character limit on SERDEPROPERTIES.PARAM_VALUE when using the hbase.columns.mapping serdeproperty to map a table that has many columns.
> I propose increasing the maximum allowed length of SERDEPROPERTIES.PARAM_VALUE to 8192.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1364) Increase the maximum length of
SERDEPROPERTIES values (currently 767 characters)
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12909395#action_12909395 ]
John Sichi commented on HIVE-1364:
----------------------------------
Just hit a case where TYPE_NAME in the COLUMNS table (limit 128 characters) was waaay too short. Is there a good reason the type name is part of the primary key for the COLUMNS table?
> Increase the maximum length of SERDEPROPERTIES values (currently 767 characters)
> --------------------------------------------------------------------------------
>
> Key: HIVE-1364
> URL: https://issues.apache.org/jira/browse/HIVE-1364
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.5.0
> Reporter: Carl Steinbach
> Assignee: Carl Steinbach
> Fix For: 0.6.0, 0.7.0
>
> Attachments: HIVE-1364.2.patch.txt, HIVE-1364.patch
>
>
> The value component of a SERDEPROPERTIES key/value pair is currently limited
> to a maximum length of 767 characters. I believe that the motivation for limiting the length to
> 767 characters is that this value is the maximum allowed length of an index in
> a MySQL database running on the InnoDB engine: http://bugs.mysql.com/bug.php?id=13315
> * The Metastore OR mapping currently limits many fields (including SERDEPROPERTIES.PARAM_VALUE) to a maximum length of 767 characters despite the fact that these fields are not indexed.
> * The maximum length of a VARCHAR value in MySQL 5.0.3 and later is 65,535.
> * We can expect many users to hit the 767 character limit on SERDEPROPERTIES.PARAM_VALUE when using the hbase.columns.mapping serdeproperty to map a table that has many columns.
> I propose increasing the maximum allowed length of SERDEPROPERTIES.PARAM_VALUE to 8192.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1364) Increase the maximum length of
SERDEPROPERTIES values (currently 767 characters)
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Steinbach updated HIVE-1364:
---------------------------------
Attachment: HIVE-1364.4.patch.txt
HIVE-1364.4.backport-060.patch.txt
Updated version of the patch with changes requested by John.
> Increase the maximum length of SERDEPROPERTIES values (currently 767 characters)
> --------------------------------------------------------------------------------
>
> Key: HIVE-1364
> URL: https://issues.apache.org/jira/browse/HIVE-1364
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.5.0
> Reporter: Carl Steinbach
> Assignee: Carl Steinbach
> Fix For: 0.6.0, 0.7.0
>
> Attachments: HIVE-1364.2.patch.txt, HIVE-1364.3.backport-060.patch.txt, HIVE-1364.3.patch.txt, HIVE-1364.4.backport-060.patch.txt, HIVE-1364.4.patch.txt, HIVE-1364.patch
>
>
> The value component of a SERDEPROPERTIES key/value pair is currently limited
> to a maximum length of 767 characters. I believe that the motivation for limiting the length to
> 767 characters is that this value is the maximum allowed length of an index in
> a MySQL database running on the InnoDB engine: http://bugs.mysql.com/bug.php?id=13315
> * The Metastore OR mapping currently limits many fields (including SERDEPROPERTIES.PARAM_VALUE) to a maximum length of 767 characters despite the fact that these fields are not indexed.
> * The maximum length of a VARCHAR value in MySQL 5.0.3 and later is 65,535.
> * We can expect many users to hit the 767 character limit on SERDEPROPERTIES.PARAM_VALUE when using the hbase.columns.mapping serdeproperty to map a table that has many columns.
> I propose increasing the maximum allowed length of SERDEPROPERTIES.PARAM_VALUE to 8192.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1364) Increase the maximum length of various
metastore fields, and remove TYPE_NAME from COLUMNS primary key
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-1364:
-----------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Committed to both 0.6 and trunk. Thanks Carl
> Increase the maximum length of various metastore fields, and remove TYPE_NAME from COLUMNS primary key
> ------------------------------------------------------------------------------------------------------
>
> Key: HIVE-1364
> URL: https://issues.apache.org/jira/browse/HIVE-1364
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.5.0
> Reporter: Carl Steinbach
> Assignee: Carl Steinbach
> Fix For: 0.6.0, 0.7.0
>
> Attachments: HIVE-1364.2.patch.txt, HIVE-1364.3.backport-060.patch.txt, HIVE-1364.3.patch.txt, HIVE-1364.4.backport-060.patch.txt, HIVE-1364.4.patch.txt, HIVE-1364.patch
>
>
> The value component of a SERDEPROPERTIES key/value pair is currently limited
> to a maximum length of 767 characters. I believe that the motivation for limiting the length to
> 767 characters is that this value is the maximum allowed length of an index in
> a MySQL database running on the InnoDB engine: http://bugs.mysql.com/bug.php?id=13315
> * The Metastore OR mapping currently limits many fields (including SERDEPROPERTIES.PARAM_VALUE) to a maximum length of 767 characters despite the fact that these fields are not indexed.
> * The maximum length of a VARCHAR value in MySQL 5.0.3 and later is 65,535.
> * We can expect many users to hit the 767 character limit on SERDEPROPERTIES.PARAM_VALUE when using the hbase.columns.mapping serdeproperty to map a table that has many columns.
> I propose increasing the maximum allowed length of SERDEPROPERTIES.PARAM_VALUE to 8192.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1364) Increase the maximum length of
SERDEPROPERTIES values (currently 767 characters)
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Steinbach updated HIVE-1364:
---------------------------------
Status: Patch Available (was: Open)
> Increase the maximum length of SERDEPROPERTIES values (currently 767 characters)
> --------------------------------------------------------------------------------
>
> Key: HIVE-1364
> URL: https://issues.apache.org/jira/browse/HIVE-1364
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.5.0
> Reporter: Carl Steinbach
> Assignee: Carl Steinbach
> Fix For: 0.6.0
>
> Attachments: HIVE-1364.2.patch.txt, HIVE-1364.patch
>
>
> The value component of a SERDEPROPERTIES key/value pair is currently limited
> to a maximum length of 767 characters. I believe that the motivation for limiting the length to
> 767 characters is that this value is the maximum allowed length of an index in
> a MySQL database running on the InnoDB engine: http://bugs.mysql.com/bug.php?id=13315
> * The Metastore OR mapping currently limits many fields (including SERDEPROPERTIES.PARAM_VALUE) to a maximum length of 767 characters despite the fact that these fields are not indexed.
> * The maximum length of a VARCHAR value in MySQL 5.0.3 and later is 65,535.
> * We can expect many users to hit the 767 character limit on SERDEPROPERTIES.PARAM_VALUE when using the hbase.columns.mapping serdeproperty to map a table that has many columns.
> I propose increasing the maximum allowed length of SERDEPROPERTIES.PARAM_VALUE to 8192.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1364) Increase the maximum length of
SERDEPROPERTIES values (currently 767 characters)
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Steinbach updated HIVE-1364:
---------------------------------
Affects Version/s: 0.5.0
> Increase the maximum length of SERDEPROPERTIES values (currently 767 characters)
> --------------------------------------------------------------------------------
>
> Key: HIVE-1364
> URL: https://issues.apache.org/jira/browse/HIVE-1364
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.5.0
> Reporter: Carl Steinbach
>
> The value component of a SERDEPROPERTIES key/value pair is currently limited
> to a maximum length of 767 characters. I believe that the motivation for limiting the length to
> 767 characters is that this value is the maximum allowed length of an index in
> a MySQL database running on the InnoDB engine: http://bugs.mysql.com/bug.php?id=13315
> * The Metastore OR mapping currently limits many fields (including SERDEPROPERTIES.PARAM_VALUE) to a maximum length of 767 characters despite the fact that these fields are not indexed.
> * The maximum length of a VARCHAR value in MySQL 5.0.3 and later is 65,535.
> * We can expect many users to hit the 767 character limit on SERDEPROPERTIES.PARAM_VALUE when using the hbase.columns.mapping serdeproperty to map a table that has many columns.
> I propose increasing the maximum allowed length of SERDEPROPERTIES.PARAM_VALUE to 8192.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1364) Increase the maximum length of
SERDEPROPERTIES values (currently 767 characters)
Posted by "Prasad Chakka (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879092#action_12879092 ]
Prasad Chakka commented on HIVE-1364:
-------------------------------------
it used to be much higher in the beginning but quite a few users reported problems on some mysql dbs. 767 seemed to work most dbs. before committing this can someone test this on some different dbs (with and without UTF encoding)?
> Increase the maximum length of SERDEPROPERTIES values (currently 767 characters)
> --------------------------------------------------------------------------------
>
> Key: HIVE-1364
> URL: https://issues.apache.org/jira/browse/HIVE-1364
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.5.0
> Reporter: Carl Steinbach
> Assignee: Carl Steinbach
> Attachments: HIVE-1364.patch
>
>
> The value component of a SERDEPROPERTIES key/value pair is currently limited
> to a maximum length of 767 characters. I believe that the motivation for limiting the length to
> 767 characters is that this value is the maximum allowed length of an index in
> a MySQL database running on the InnoDB engine: http://bugs.mysql.com/bug.php?id=13315
> * The Metastore OR mapping currently limits many fields (including SERDEPROPERTIES.PARAM_VALUE) to a maximum length of 767 characters despite the fact that these fields are not indexed.
> * The maximum length of a VARCHAR value in MySQL 5.0.3 and later is 65,535.
> * We can expect many users to hit the 767 character limit on SERDEPROPERTIES.PARAM_VALUE when using the hbase.columns.mapping serdeproperty to map a table that has many columns.
> I propose increasing the maximum allowed length of SERDEPROPERTIES.PARAM_VALUE to 8192.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1364) Increase the maximum length of
SERDEPROPERTIES values (currently 767 characters)
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12914196#action_12914196 ]
John Sichi commented on HIVE-1364:
----------------------------------
Per discussion in IRC, we should not change the precision for identifiers. Here's the revert list (plus one change for MStorageDescriptor's TYPE_NAME).
MFieldSchema
FNAME
MType
TYPE_NAME
FIELD_NAME
MTable
TBL_NAME
PKEY_NAME
PARAM_KEY
TBL_TYPE
MSerDeInfo
NAME
PARAM_KEY
MOrder
COL_NAME
MStorageDescriptor
COLUMN_NAME (all instances)
PARAM_KEY
TYPE_NAME should actually be 4000 since it's really a type signature, not a type name, and we're getting rid of the indexing for it
MPartition
PARAM_KEY
MIndex
INDEX_NAME
PARAM_KEY
> Increase the maximum length of SERDEPROPERTIES values (currently 767 characters)
> --------------------------------------------------------------------------------
>
> Key: HIVE-1364
> URL: https://issues.apache.org/jira/browse/HIVE-1364
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.5.0
> Reporter: Carl Steinbach
> Assignee: Carl Steinbach
> Fix For: 0.6.0, 0.7.0
>
> Attachments: HIVE-1364.2.patch.txt, HIVE-1364.3.backport-060.patch.txt, HIVE-1364.3.patch.txt, HIVE-1364.patch
>
>
> The value component of a SERDEPROPERTIES key/value pair is currently limited
> to a maximum length of 767 characters. I believe that the motivation for limiting the length to
> 767 characters is that this value is the maximum allowed length of an index in
> a MySQL database running on the InnoDB engine: http://bugs.mysql.com/bug.php?id=13315
> * The Metastore OR mapping currently limits many fields (including SERDEPROPERTIES.PARAM_VALUE) to a maximum length of 767 characters despite the fact that these fields are not indexed.
> * The maximum length of a VARCHAR value in MySQL 5.0.3 and later is 65,535.
> * We can expect many users to hit the 767 character limit on SERDEPROPERTIES.PARAM_VALUE when using the hbase.columns.mapping serdeproperty to map a table that has many columns.
> I propose increasing the maximum allowed length of SERDEPROPERTIES.PARAM_VALUE to 8192.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1364) Increase the maximum length of
SERDEPROPERTIES values (currently 767 characters)
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12914162#action_12914162 ]
HBase Review Board commented on HIVE-1364:
------------------------------------------
Message from: "Carl Steinbach" <ca...@cloudera.com>
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/895/
-----------------------------------------------------------
Review request for Hive Developers and John Sichi.
Summary
-------
The patch increases the length of various properties in the Metastore OR mapping. Properties which are currently indexed, or which we may want to index in the future were increased to a length of 767 bytes. Properties which are not indexed and which we are unlikely to ever want to index were increased to a max length of 4000 bytes. I also removed the PK constraint on the COLUMNS.TYPE_NAME field.
This addresses bug HIVE-1364.
http://issues.apache.org/jira/browse/HIVE-1364
Diffs
-----
metastore/src/model/package.jdo 527f4b2
Diff: http://review.cloudera.org/r/895/diff
Testing
-------
Thanks,
Carl
> Increase the maximum length of SERDEPROPERTIES values (currently 767 characters)
> --------------------------------------------------------------------------------
>
> Key: HIVE-1364
> URL: https://issues.apache.org/jira/browse/HIVE-1364
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.5.0
> Reporter: Carl Steinbach
> Assignee: Carl Steinbach
> Fix For: 0.6.0, 0.7.0
>
> Attachments: HIVE-1364.2.patch.txt, HIVE-1364.3.backport-060.patch.txt, HIVE-1364.3.patch.txt, HIVE-1364.patch
>
>
> The value component of a SERDEPROPERTIES key/value pair is currently limited
> to a maximum length of 767 characters. I believe that the motivation for limiting the length to
> 767 characters is that this value is the maximum allowed length of an index in
> a MySQL database running on the InnoDB engine: http://bugs.mysql.com/bug.php?id=13315
> * The Metastore OR mapping currently limits many fields (including SERDEPROPERTIES.PARAM_VALUE) to a maximum length of 767 characters despite the fact that these fields are not indexed.
> * The maximum length of a VARCHAR value in MySQL 5.0.3 and later is 65,535.
> * We can expect many users to hit the 767 character limit on SERDEPROPERTIES.PARAM_VALUE when using the hbase.columns.mapping serdeproperty to map a table that has many columns.
> I propose increasing the maximum allowed length of SERDEPROPERTIES.PARAM_VALUE to 8192.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1364) Increase the maximum length of
SERDEPROPERTIES values (currently 767 characters)
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891447#action_12891447 ]
Carl Steinbach commented on HIVE-1364:
--------------------------------------
MetaStore upgrade scripts are covered in HIVE-1427.
@ John & Ashish: Please tell me how you want me to change this patch. I'm willing to revert these limits to their original values. Are there any properties besides ftype which you want me to revert? Should I revert everything except SERDEPROPERTIES.PARAM_VALUE?
> Increase the maximum length of SERDEPROPERTIES values (currently 767 characters)
> --------------------------------------------------------------------------------
>
> Key: HIVE-1364
> URL: https://issues.apache.org/jira/browse/HIVE-1364
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.5.0
> Reporter: Carl Steinbach
> Assignee: Carl Steinbach
> Fix For: 0.6.0, 0.7.0
>
> Attachments: HIVE-1364.2.patch.txt, HIVE-1364.patch
>
>
> The value component of a SERDEPROPERTIES key/value pair is currently limited
> to a maximum length of 767 characters. I believe that the motivation for limiting the length to
> 767 characters is that this value is the maximum allowed length of an index in
> a MySQL database running on the InnoDB engine: http://bugs.mysql.com/bug.php?id=13315
> * The Metastore OR mapping currently limits many fields (including SERDEPROPERTIES.PARAM_VALUE) to a maximum length of 767 characters despite the fact that these fields are not indexed.
> * The maximum length of a VARCHAR value in MySQL 5.0.3 and later is 65,535.
> * We can expect many users to hit the 767 character limit on SERDEPROPERTIES.PARAM_VALUE when using the hbase.columns.mapping serdeproperty to map a table that has many columns.
> I propose increasing the maximum allowed length of SERDEPROPERTIES.PARAM_VALUE to 8192.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1364) Increase the maximum length of
SERDEPROPERTIES values (currently 767 characters)
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879643#action_12879643 ]
Carl Steinbach commented on HIVE-1364:
--------------------------------------
@Prasad: It's possible that people who ran into problems were before were using a version of MySQL older than 5.0.3. These versions supported a 255 byte max length for VARCHARs. It's also possible that older versions of the package.jdo mapping contained more indexes, in which case the 767 byte limit holds. Also, UTF encoding should not make a difference since these are byte lengths, not character lengths.
@John: I think using LOBs is the right approach, but perhaps we should handle that problem in a different ticket? I don't think we can just change the mapping to use LOB instead of VARCHAR, and will instead have to add a new LOB column, deprecate the old VARCHAR column, and create an accessor that is capable of using either column.
> Increase the maximum length of SERDEPROPERTIES values (currently 767 characters)
> --------------------------------------------------------------------------------
>
> Key: HIVE-1364
> URL: https://issues.apache.org/jira/browse/HIVE-1364
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.5.0
> Reporter: Carl Steinbach
> Assignee: Carl Steinbach
> Fix For: 0.6.0
>
> Attachments: HIVE-1364.2.patch.txt, HIVE-1364.patch
>
>
> The value component of a SERDEPROPERTIES key/value pair is currently limited
> to a maximum length of 767 characters. I believe that the motivation for limiting the length to
> 767 characters is that this value is the maximum allowed length of an index in
> a MySQL database running on the InnoDB engine: http://bugs.mysql.com/bug.php?id=13315
> * The Metastore OR mapping currently limits many fields (including SERDEPROPERTIES.PARAM_VALUE) to a maximum length of 767 characters despite the fact that these fields are not indexed.
> * The maximum length of a VARCHAR value in MySQL 5.0.3 and later is 65,535.
> * We can expect many users to hit the 767 character limit on SERDEPROPERTIES.PARAM_VALUE when using the hbase.columns.mapping serdeproperty to map a table that has many columns.
> I propose increasing the maximum allowed length of SERDEPROPERTIES.PARAM_VALUE to 8192.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1364) Increase the maximum length of
SERDEPROPERTIES values (currently 767 characters)
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12914251#action_12914251 ]
HBase Review Board commented on HIVE-1364:
------------------------------------------
Message from: "Carl Steinbach" <ca...@cloudera.com>
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/895/
-----------------------------------------------------------
(Updated 2010-09-23 14:45:46.673298)
Review request for Hive Developers and John Sichi.
Summary
-------
The patch increases the length of various properties in the Metastore OR mapping. Properties which are currently indexed, or which we may want to index in the future were increased to a length of 767 bytes. Properties which are not indexed and which we are unlikely to ever want to index were increased to a max length of 4000 bytes. I also removed the PK constraint on the COLUMNS.TYPE_NAME field.
This addresses bug HIVE-1364.
http://issues.apache.org/jira/browse/HIVE-1364
Diffs (updated)
-----
metastore/src/model/package.jdo 527f4b2
Diff: http://review.cloudera.org/r/895/diff
Testing
-------
Thanks,
Carl
> Increase the maximum length of SERDEPROPERTIES values (currently 767 characters)
> --------------------------------------------------------------------------------
>
> Key: HIVE-1364
> URL: https://issues.apache.org/jira/browse/HIVE-1364
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.5.0
> Reporter: Carl Steinbach
> Assignee: Carl Steinbach
> Fix For: 0.6.0, 0.7.0
>
> Attachments: HIVE-1364.2.patch.txt, HIVE-1364.3.backport-060.patch.txt, HIVE-1364.3.patch.txt, HIVE-1364.4.backport-060.patch.txt, HIVE-1364.4.patch.txt, HIVE-1364.patch
>
>
> The value component of a SERDEPROPERTIES key/value pair is currently limited
> to a maximum length of 767 characters. I believe that the motivation for limiting the length to
> 767 characters is that this value is the maximum allowed length of an index in
> a MySQL database running on the InnoDB engine: http://bugs.mysql.com/bug.php?id=13315
> * The Metastore OR mapping currently limits many fields (including SERDEPROPERTIES.PARAM_VALUE) to a maximum length of 767 characters despite the fact that these fields are not indexed.
> * The maximum length of a VARCHAR value in MySQL 5.0.3 and later is 65,535.
> * We can expect many users to hit the 767 character limit on SERDEPROPERTIES.PARAM_VALUE when using the hbase.columns.mapping serdeproperty to map a table that has many columns.
> I propose increasing the maximum allowed length of SERDEPROPERTIES.PARAM_VALUE to 8192.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1364) Increase the maximum length of
SERDEPROPERTIES values (currently 767 characters)
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Steinbach updated HIVE-1364:
---------------------------------
Attachment: HIVE-1364.3.patch.txt
HIVE-1364.3.backport-060.patch.txt
> Increase the maximum length of SERDEPROPERTIES values (currently 767 characters)
> --------------------------------------------------------------------------------
>
> Key: HIVE-1364
> URL: https://issues.apache.org/jira/browse/HIVE-1364
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.5.0
> Reporter: Carl Steinbach
> Assignee: Carl Steinbach
> Fix For: 0.6.0, 0.7.0
>
> Attachments: HIVE-1364.2.patch.txt, HIVE-1364.3.backport-060.patch.txt, HIVE-1364.3.patch.txt, HIVE-1364.patch
>
>
> The value component of a SERDEPROPERTIES key/value pair is currently limited
> to a maximum length of 767 characters. I believe that the motivation for limiting the length to
> 767 characters is that this value is the maximum allowed length of an index in
> a MySQL database running on the InnoDB engine: http://bugs.mysql.com/bug.php?id=13315
> * The Metastore OR mapping currently limits many fields (including SERDEPROPERTIES.PARAM_VALUE) to a maximum length of 767 characters despite the fact that these fields are not indexed.
> * The maximum length of a VARCHAR value in MySQL 5.0.3 and later is 65,535.
> * We can expect many users to hit the 767 character limit on SERDEPROPERTIES.PARAM_VALUE when using the hbase.columns.mapping serdeproperty to map a table that has many columns.
> I propose increasing the maximum allowed length of SERDEPROPERTIES.PARAM_VALUE to 8192.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HIVE-1364) Increase the maximum length of
SERDEPROPERTIES values (currently 767 characters)
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Steinbach reassigned HIVE-1364:
------------------------------------
Assignee: Carl Steinbach
> Increase the maximum length of SERDEPROPERTIES values (currently 767 characters)
> --------------------------------------------------------------------------------
>
> Key: HIVE-1364
> URL: https://issues.apache.org/jira/browse/HIVE-1364
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.5.0
> Reporter: Carl Steinbach
> Assignee: Carl Steinbach
> Attachments: HIVE-1364.patch
>
>
> The value component of a SERDEPROPERTIES key/value pair is currently limited
> to a maximum length of 767 characters. I believe that the motivation for limiting the length to
> 767 characters is that this value is the maximum allowed length of an index in
> a MySQL database running on the InnoDB engine: http://bugs.mysql.com/bug.php?id=13315
> * The Metastore OR mapping currently limits many fields (including SERDEPROPERTIES.PARAM_VALUE) to a maximum length of 767 characters despite the fact that these fields are not indexed.
> * The maximum length of a VARCHAR value in MySQL 5.0.3 and later is 65,535.
> * We can expect many users to hit the 767 character limit on SERDEPROPERTIES.PARAM_VALUE when using the hbase.columns.mapping serdeproperty to map a table that has many columns.
> I propose increasing the maximum allowed length of SERDEPROPERTIES.PARAM_VALUE to 8192.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1364) Increase the maximum length of
SERDEPROPERTIES values (currently 767 characters)
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881416#action_12881416 ]
Carl Steinbach commented on HIVE-1364:
--------------------------------------
bq. Also why do we make everything 4000 bytes - I presume things like ftype will never hit that limit.
Currently the ORM is the de facto enforcement mechanism for string length limitations. I think this is a bad approach since 1) users can work around it by manually altering the underlying tables, and 2) the limits are stated in terms of bytes so the actual length restriction in terms of number of characters will depend on the character set of the underlying DB. In light of this I bumped every size limit to 4000 bytes, and also because I did not want to try to predict which property length limit someone would next bump into. I'm willing to revert these limits to their original values. Are there any properties besides ftype which you want me to revert? Should I revert everything except SERDEPROPERTIES.PARAM_VALUE?
bq. Also changes to upgrade SQL should also be a part of the patch, no? Where are the scripts for the view change located?
I'll update the patch with the necessary scripts. Should these go in bin/ or somewhere under metastore/ ?
@John: Yes, I think this falls under the responsibility of the release manager. I will take care of it.
I think the current approach of using the ORM as the de facto enforcement mechanism for checking
> Increase the maximum length of SERDEPROPERTIES values (currently 767 characters)
> --------------------------------------------------------------------------------
>
> Key: HIVE-1364
> URL: https://issues.apache.org/jira/browse/HIVE-1364
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.5.0
> Reporter: Carl Steinbach
> Assignee: Carl Steinbach
> Fix For: 0.6.0
>
> Attachments: HIVE-1364.2.patch.txt, HIVE-1364.patch
>
>
> The value component of a SERDEPROPERTIES key/value pair is currently limited
> to a maximum length of 767 characters. I believe that the motivation for limiting the length to
> 767 characters is that this value is the maximum allowed length of an index in
> a MySQL database running on the InnoDB engine: http://bugs.mysql.com/bug.php?id=13315
> * The Metastore OR mapping currently limits many fields (including SERDEPROPERTIES.PARAM_VALUE) to a maximum length of 767 characters despite the fact that these fields are not indexed.
> * The maximum length of a VARCHAR value in MySQL 5.0.3 and later is 65,535.
> * We can expect many users to hit the 767 character limit on SERDEPROPERTIES.PARAM_VALUE when using the hbase.columns.mapping serdeproperty to map a table that has many columns.
> I propose increasing the maximum allowed length of SERDEPROPERTIES.PARAM_VALUE to 8192.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1364) Increase the maximum length of
SERDEPROPERTIES values (currently 767 characters)
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Sichi updated HIVE-1364:
-----------------------------
Status: Open (was: Patch Available)
Canceling patch since I think we should widen TYPE_NAME and also drop it from the PRIMARY KEY on the COLUMNS table.
> Increase the maximum length of SERDEPROPERTIES values (currently 767 characters)
> --------------------------------------------------------------------------------
>
> Key: HIVE-1364
> URL: https://issues.apache.org/jira/browse/HIVE-1364
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.5.0
> Reporter: Carl Steinbach
> Assignee: Carl Steinbach
> Fix For: 0.6.0, 0.7.0
>
> Attachments: HIVE-1364.2.patch.txt, HIVE-1364.patch
>
>
> The value component of a SERDEPROPERTIES key/value pair is currently limited
> to a maximum length of 767 characters. I believe that the motivation for limiting the length to
> 767 characters is that this value is the maximum allowed length of an index in
> a MySQL database running on the InnoDB engine: http://bugs.mysql.com/bug.php?id=13315
> * The Metastore OR mapping currently limits many fields (including SERDEPROPERTIES.PARAM_VALUE) to a maximum length of 767 characters despite the fact that these fields are not indexed.
> * The maximum length of a VARCHAR value in MySQL 5.0.3 and later is 65,535.
> * We can expect many users to hit the 767 character limit on SERDEPROPERTIES.PARAM_VALUE when using the hbase.columns.mapping serdeproperty to map a table that has many columns.
> I propose increasing the maximum allowed length of SERDEPROPERTIES.PARAM_VALUE to 8192.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1364) Increase the maximum length of
SERDEPROPERTIES values (currently 767 characters)
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Steinbach updated HIVE-1364:
---------------------------------
Status: Patch Available (was: Open)
> Increase the maximum length of SERDEPROPERTIES values (currently 767 characters)
> --------------------------------------------------------------------------------
>
> Key: HIVE-1364
> URL: https://issues.apache.org/jira/browse/HIVE-1364
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.5.0
> Reporter: Carl Steinbach
> Assignee: Carl Steinbach
> Fix For: 0.6.0, 0.7.0
>
> Attachments: HIVE-1364.2.patch.txt, HIVE-1364.3.backport-060.patch.txt, HIVE-1364.3.patch.txt, HIVE-1364.patch
>
>
> The value component of a SERDEPROPERTIES key/value pair is currently limited
> to a maximum length of 767 characters. I believe that the motivation for limiting the length to
> 767 characters is that this value is the maximum allowed length of an index in
> a MySQL database running on the InnoDB engine: http://bugs.mysql.com/bug.php?id=13315
> * The Metastore OR mapping currently limits many fields (including SERDEPROPERTIES.PARAM_VALUE) to a maximum length of 767 characters despite the fact that these fields are not indexed.
> * The maximum length of a VARCHAR value in MySQL 5.0.3 and later is 65,535.
> * We can expect many users to hit the 767 character limit on SERDEPROPERTIES.PARAM_VALUE when using the hbase.columns.mapping serdeproperty to map a table that has many columns.
> I propose increasing the maximum allowed length of SERDEPROPERTIES.PARAM_VALUE to 8192.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1364) Increase the maximum length of
SERDEPROPERTIES values (currently 767 characters)
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Steinbach updated HIVE-1364:
---------------------------------
Attachment: HIVE-1364.patch
> Increase the maximum length of SERDEPROPERTIES values (currently 767 characters)
> --------------------------------------------------------------------------------
>
> Key: HIVE-1364
> URL: https://issues.apache.org/jira/browse/HIVE-1364
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.5.0
> Reporter: Carl Steinbach
> Attachments: HIVE-1364.patch
>
>
> The value component of a SERDEPROPERTIES key/value pair is currently limited
> to a maximum length of 767 characters. I believe that the motivation for limiting the length to
> 767 characters is that this value is the maximum allowed length of an index in
> a MySQL database running on the InnoDB engine: http://bugs.mysql.com/bug.php?id=13315
> * The Metastore OR mapping currently limits many fields (including SERDEPROPERTIES.PARAM_VALUE) to a maximum length of 767 characters despite the fact that these fields are not indexed.
> * The maximum length of a VARCHAR value in MySQL 5.0.3 and later is 65,535.
> * We can expect many users to hit the 767 character limit on SERDEPROPERTIES.PARAM_VALUE when using the hbase.columns.mapping serdeproperty to map a table that has many columns.
> I propose increasing the maximum allowed length of SERDEPROPERTIES.PARAM_VALUE to 8192.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1364) Increase the maximum length of various
metastore fields, and remove TYPE_NAME from COLUMNS primary key
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Sichi updated HIVE-1364:
-----------------------------
Summary: Increase the maximum length of various metastore fields, and remove TYPE_NAME from COLUMNS primary key (was: Increase the maximum length of SERDEPROPERTIES values (currently 767 characters))
> Increase the maximum length of various metastore fields, and remove TYPE_NAME from COLUMNS primary key
> ------------------------------------------------------------------------------------------------------
>
> Key: HIVE-1364
> URL: https://issues.apache.org/jira/browse/HIVE-1364
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.5.0
> Reporter: Carl Steinbach
> Assignee: Carl Steinbach
> Fix For: 0.6.0, 0.7.0
>
> Attachments: HIVE-1364.2.patch.txt, HIVE-1364.3.backport-060.patch.txt, HIVE-1364.3.patch.txt, HIVE-1364.4.backport-060.patch.txt, HIVE-1364.4.patch.txt, HIVE-1364.patch
>
>
> The value component of a SERDEPROPERTIES key/value pair is currently limited
> to a maximum length of 767 characters. I believe that the motivation for limiting the length to
> 767 characters is that this value is the maximum allowed length of an index in
> a MySQL database running on the InnoDB engine: http://bugs.mysql.com/bug.php?id=13315
> * The Metastore OR mapping currently limits many fields (including SERDEPROPERTIES.PARAM_VALUE) to a maximum length of 767 characters despite the fact that these fields are not indexed.
> * The maximum length of a VARCHAR value in MySQL 5.0.3 and later is 65,535.
> * We can expect many users to hit the 767 character limit on SERDEPROPERTIES.PARAM_VALUE when using the hbase.columns.mapping serdeproperty to map a table that has many columns.
> I propose increasing the maximum allowed length of SERDEPROPERTIES.PARAM_VALUE to 8192.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1364) Increase the maximum length of
SERDEPROPERTIES values (currently 767 characters)
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Steinbach updated HIVE-1364:
---------------------------------
Attachment: HIVE-1364.2.patch.txt
HIVE-1364.2.patch.txt:
* Change PARTITIONS.PART_NAME max length back to 767
> Increase the maximum length of SERDEPROPERTIES values (currently 767 characters)
> --------------------------------------------------------------------------------
>
> Key: HIVE-1364
> URL: https://issues.apache.org/jira/browse/HIVE-1364
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.5.0
> Reporter: Carl Steinbach
> Assignee: Carl Steinbach
> Fix For: 0.6.0
>
> Attachments: HIVE-1364.2.patch.txt, HIVE-1364.patch
>
>
> The value component of a SERDEPROPERTIES key/value pair is currently limited
> to a maximum length of 767 characters. I believe that the motivation for limiting the length to
> 767 characters is that this value is the maximum allowed length of an index in
> a MySQL database running on the InnoDB engine: http://bugs.mysql.com/bug.php?id=13315
> * The Metastore OR mapping currently limits many fields (including SERDEPROPERTIES.PARAM_VALUE) to a maximum length of 767 characters despite the fact that these fields are not indexed.
> * The maximum length of a VARCHAR value in MySQL 5.0.3 and later is 65,535.
> * We can expect many users to hit the 767 character limit on SERDEPROPERTIES.PARAM_VALUE when using the hbase.columns.mapping serdeproperty to map a table that has many columns.
> I propose increasing the maximum allowed length of SERDEPROPERTIES.PARAM_VALUE to 8192.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.