You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Zheng Shao (JIRA)" <ji...@apache.org> on 2009/02/23 02:21:01 UTC
[jira] Created: (HIVE-298) Make branch 0.2 forward compatible with
tables created with LazySimpleSerDe
Make branch 0.2 forward compatible with tables created with LazySimpleSerDe
---------------------------------------------------------------------------
Key: HIVE-298
URL: https://issues.apache.org/jira/browse/HIVE-298
Project: Hadoop Hive
Issue Type: Improvement
Reporter: Zheng Shao
We can easily make branch 0.2 forward-compatible with tables created with LazySimpleSerDe (starting branch 0.3), since LazySimpleSerDe has exactly the same serialized format as TCTLSeparatedProtocol and MetadataTypedColumnsetSerDe.
The plan is to directly map it to TCTLSeparatedProtocal.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-298) Make branch 0.2 forward compatible
with tables created with LazySimpleSerDe
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12675828#action_12675828 ]
Zheng Shao commented on HIVE-298:
---------------------------------
To clarify, I did find some shortName related code, but they are all for backward compatibility. We don't generate short names for serdes any more.
> Make branch 0.2 forward compatible with tables created with LazySimpleSerDe
> ---------------------------------------------------------------------------
>
> Key: HIVE-298
> URL: https://issues.apache.org/jira/browse/HIVE-298
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Zheng Shao
> Attachments: HIVE-298.1.patch
>
>
> We can easily make branch 0.2 forward-compatible with tables created with LazySimpleSerDe (starting branch 0.3), since LazySimpleSerDe has exactly the same serialized format as TCTLSeparatedProtocol and MetadataTypedColumnsetSerDe.
> The plan is to directly map it to TCTLSeparatedProtocal.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-298) Make branch 0.2 forward compatible
with tables created with LazySimpleSerDe
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12676443#action_12676443 ]
dhruba borthakur commented on HIVE-298:
---------------------------------------
I like this approach versus making branches forward-compatible. There is still some inconsistency, because the change to the 0.20 branch is more like a new feature rather than a bug fix.
> Make branch 0.2 forward compatible with tables created with LazySimpleSerDe
> ---------------------------------------------------------------------------
>
> Key: HIVE-298
> URL: https://issues.apache.org/jira/browse/HIVE-298
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Attachments: HIVE-298.1.patch, HIVE-298.2.patch
>
>
> We can easily make branch 0.2 forward-compatible with tables created with LazySimpleSerDe (starting branch 0.3), since LazySimpleSerDe has exactly the same serialized format as TCTLSeparatedProtocol and MetadataTypedColumnsetSerDe.
> The plan is to directly map it to TCTLSeparatedProtocal.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-298) Make branch 0.2 forward compatible
with tables created with LazySimpleSerDe
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12676439#action_12676439 ]
Joydeep Sen Sarma commented on HIVE-298:
----------------------------------------
+1
> Make branch 0.2 forward compatible with tables created with LazySimpleSerDe
> ---------------------------------------------------------------------------
>
> Key: HIVE-298
> URL: https://issues.apache.org/jira/browse/HIVE-298
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Attachments: HIVE-298.1.patch, HIVE-298.2.patch
>
>
> We can easily make branch 0.2 forward-compatible with tables created with LazySimpleSerDe (starting branch 0.3), since LazySimpleSerDe has exactly the same serialized format as TCTLSeparatedProtocol and MetadataTypedColumnsetSerDe.
> The plan is to directly map it to TCTLSeparatedProtocal.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HIVE-298) Make branch 0.2 forward compatible with
tables created with LazySimpleSerDe
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao resolved HIVE-298.
-----------------------------
Resolution: Fixed
Fix Version/s: 0.2.0
Release Note: HIVE-298. Add LazySimpleSerDe. (zshao)
Hadoop Flags: [Reviewed]
Committed revision 747644.
> Make branch 0.2 forward compatible with tables created with LazySimpleSerDe
> ---------------------------------------------------------------------------
>
> Key: HIVE-298
> URL: https://issues.apache.org/jira/browse/HIVE-298
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Fix For: 0.2.0
>
> Attachments: HIVE-298.1.patch, HIVE-298.2.patch
>
>
> We can easily make branch 0.2 forward-compatible with tables created with LazySimpleSerDe (starting branch 0.3), since LazySimpleSerDe has exactly the same serialized format as TCTLSeparatedProtocol and MetadataTypedColumnsetSerDe.
> The plan is to directly map it to TCTLSeparatedProtocal.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-298) Make branch 0.2 forward compatible
with tables created with LazySimpleSerDe
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12675819#action_12675819 ]
Joydeep Sen Sarma commented on HIVE-298:
----------------------------------------
This doesn't make sense .. (can't keep putting forward compatibility changes in older branches).
Perhaps we need to have some layer of indirection (I think for a while hive1 - we had aliases for serdes) - so that we can default to different implementations (for delimited text deserialization) in different versions without breaking compatibility. thoughts?
> Make branch 0.2 forward compatible with tables created with LazySimpleSerDe
> ---------------------------------------------------------------------------
>
> Key: HIVE-298
> URL: https://issues.apache.org/jira/browse/HIVE-298
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Zheng Shao
>
> We can easily make branch 0.2 forward-compatible with tables created with LazySimpleSerDe (starting branch 0.3), since LazySimpleSerDe has exactly the same serialized format as TCTLSeparatedProtocol and MetadataTypedColumnsetSerDe.
> The plan is to directly map it to TCTLSeparatedProtocal.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-298) Make branch 0.2 forward compatible
with tables created with LazySimpleSerDe
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12676119#action_12676119 ]
Joydeep Sen Sarma commented on HIVE-298:
----------------------------------------
why don't we just checkin LazySerDe into the 0.2 branch (but not all the code changes in Hive that default to using it).
that way - there's no added risk to 0.2 - and it's available for testing.
> Make branch 0.2 forward compatible with tables created with LazySimpleSerDe
> ---------------------------------------------------------------------------
>
> Key: HIVE-298
> URL: https://issues.apache.org/jira/browse/HIVE-298
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Attachments: HIVE-298.1.patch
>
>
> We can easily make branch 0.2 forward-compatible with tables created with LazySimpleSerDe (starting branch 0.3), since LazySimpleSerDe has exactly the same serialized format as TCTLSeparatedProtocol and MetadataTypedColumnsetSerDe.
> The plan is to directly map it to TCTLSeparatedProtocal.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-298) Make branch 0.2 forward compatible
with tables created with LazySimpleSerDe
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12675821#action_12675821 ]
Zheng Shao commented on HIVE-298:
---------------------------------
The alias for serde (shortNames) turned out to be a pain while we were migrating hive1. Java does not automatically load classes, so in order to support serde aliases we would have to load all classes which takes more than 10 seconds in usual case.
As I remember that is why we have moved away from shortNames (there are no shortNames in the code or in the MetaStore right now)
We actually still have serde name mappings in both serde 1.5 and serde 2 code, to make sure they are compatible with each other.
I agree we can't keep putting forward compatibility patches. This is just meant for safer migration from 0.2 to 0.3. There won't be such a need for 0.2 to 0.4.
Without this patch it is harder to push people to test LazySimpleSerDe (since table created by them won't be usable by other people unless they are all on branch 0.3).
But I am OK to go without it, as long as we can get enough people to test out LazySimpleSerDe.
> Make branch 0.2 forward compatible with tables created with LazySimpleSerDe
> ---------------------------------------------------------------------------
>
> Key: HIVE-298
> URL: https://issues.apache.org/jira/browse/HIVE-298
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Zheng Shao
>
> We can easily make branch 0.2 forward-compatible with tables created with LazySimpleSerDe (starting branch 0.3), since LazySimpleSerDe has exactly the same serialized format as TCTLSeparatedProtocol and MetadataTypedColumnsetSerDe.
> The plan is to directly map it to TCTLSeparatedProtocal.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-298) Make branch 0.2 forward compatible
with tables created with LazySimpleSerDe
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12676218#action_12676218 ]
Zheng Shao commented on HIVE-298:
---------------------------------
@Joydeep, this is the new patch of adding LazySimpleSerDe to branch 0.2 (without changing the default).
> Make branch 0.2 forward compatible with tables created with LazySimpleSerDe
> ---------------------------------------------------------------------------
>
> Key: HIVE-298
> URL: https://issues.apache.org/jira/browse/HIVE-298
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Attachments: HIVE-298.1.patch, HIVE-298.2.patch
>
>
> We can easily make branch 0.2 forward-compatible with tables created with LazySimpleSerDe (starting branch 0.3), since LazySimpleSerDe has exactly the same serialized format as TCTLSeparatedProtocol and MetadataTypedColumnsetSerDe.
> The plan is to directly map it to TCTLSeparatedProtocal.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-298) Make branch 0.2 forward compatible with
tables created with LazySimpleSerDe
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Steinbach updated HIVE-298:
--------------------------------
Fix Version/s: (was: 0.6.0)
0.3.0
> Make branch 0.2 forward compatible with tables created with LazySimpleSerDe
> ---------------------------------------------------------------------------
>
> Key: HIVE-298
> URL: https://issues.apache.org/jira/browse/HIVE-298
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Fix For: 0.3.0
>
> Attachments: HIVE-298.1.patch, HIVE-298.2.patch
>
>
> We can easily make branch 0.2 forward-compatible with tables created with LazySimpleSerDe (starting branch 0.3), since LazySimpleSerDe has exactly the same serialized format as TCTLSeparatedProtocol and MetadataTypedColumnsetSerDe.
> The plan is to directly map it to TCTLSeparatedProtocal.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-298) Make branch 0.2 forward compatible with
tables created with LazySimpleSerDe
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated HIVE-298:
----------------------------
Attachment: HIVE-298.1.patch
Map LazySimpleSerDe with TCTLSeparatedProtocol.
> Make branch 0.2 forward compatible with tables created with LazySimpleSerDe
> ---------------------------------------------------------------------------
>
> Key: HIVE-298
> URL: https://issues.apache.org/jira/browse/HIVE-298
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Zheng Shao
> Attachments: HIVE-298.1.patch
>
>
> We can easily make branch 0.2 forward-compatible with tables created with LazySimpleSerDe (starting branch 0.3), since LazySimpleSerDe has exactly the same serialized format as TCTLSeparatedProtocol and MetadataTypedColumnsetSerDe.
> The plan is to directly map it to TCTLSeparatedProtocal.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HIVE-298) Make branch 0.2 forward compatible with
tables created with LazySimpleSerDe
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao reassigned HIVE-298:
-------------------------------
Assignee: Zheng Shao
> Make branch 0.2 forward compatible with tables created with LazySimpleSerDe
> ---------------------------------------------------------------------------
>
> Key: HIVE-298
> URL: https://issues.apache.org/jira/browse/HIVE-298
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Attachments: HIVE-298.1.patch
>
>
> We can easily make branch 0.2 forward-compatible with tables created with LazySimpleSerDe (starting branch 0.3), since LazySimpleSerDe has exactly the same serialized format as TCTLSeparatedProtocol and MetadataTypedColumnsetSerDe.
> The plan is to directly map it to TCTLSeparatedProtocal.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-298) Make branch 0.2 forward compatible with
tables created with LazySimpleSerDe
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated HIVE-298:
----------------------------
Attachment: HIVE-298.2.patch
Adding LazySimpleSerDe to branch 0.2. We don't change the default.
Tested with a table created by trunk. It works.
> Make branch 0.2 forward compatible with tables created with LazySimpleSerDe
> ---------------------------------------------------------------------------
>
> Key: HIVE-298
> URL: https://issues.apache.org/jira/browse/HIVE-298
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Attachments: HIVE-298.1.patch, HIVE-298.2.patch
>
>
> We can easily make branch 0.2 forward-compatible with tables created with LazySimpleSerDe (starting branch 0.3), since LazySimpleSerDe has exactly the same serialized format as TCTLSeparatedProtocol and MetadataTypedColumnsetSerDe.
> The plan is to directly map it to TCTLSeparatedProtocal.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.