You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Travis Crawford (Created) (JIRA)" <ji...@apache.org> on 2012/01/31 19:00:11 UTC
[jira] [Created] (HIVE-2767) Optionally use framed transport with
metastore
Optionally use framed transport with metastore
----------------------------------------------
Key: HIVE-2767
URL: https://issues.apache.org/jira/browse/HIVE-2767
Project: Hive
Issue Type: New Feature
Components: Metastore
Reporter: Travis Crawford
Users may want/need to use thrift's framed transport when communicating with the Hive MetaStore. This patch adds a new property {{hive.metastore.thrift.framed.transport.enabled}} that enables the framed transport (defaults to off, aka no change from before the patch). This property must be set for both clients and the HMS server.
It wasn't immediately clear how to use the framed transport with SASL, so as written an exception is thrown if you try starting the server with both options. If SASL and the framed transport will indeed work together I can update the patch (although I don't have a secured environment to test in).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2767) Optionally use framed transport with
metastore
Posted by "Travis Crawford (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250718#comment-13250718 ]
Travis Crawford commented on HIVE-2767:
---------------------------------------
Cool - thanks for the pointer! I'll watch that issue and afterwards rebase if necessary and update.
> Optionally use framed transport with metastore
> ----------------------------------------------
>
> Key: HIVE-2767
> URL: https://issues.apache.org/jira/browse/HIVE-2767
> Project: Hive
> Issue Type: New Feature
> Components: Metastore
> Reporter: Travis Crawford
> Assignee: Travis Crawford
> Attachments: HIVE-2767.D2661.1.patch, HIVE-2767.D2661.2.patch, HIVE-2767.D2661.3.patch, HIVE-2767.patch.txt, HIVE-2767_a.patch.txt
>
>
> Users may want/need to use thrift's framed transport when communicating with the Hive MetaStore. This patch adds a new property {{hive.metastore.thrift.framed.transport.enabled}} that enables the framed transport (defaults to off, aka no change from before the patch). This property must be set for both clients and the HMS server.
> It wasn't immediately clear how to use the framed transport with SASL, so as written an exception is thrown if you try starting the server with both options. If SASL and the framed transport will indeed work together I can update the patch (although I don't have a secured environment to test in).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HIVE-2767) Optionally use framed transport with
metastore
Posted by "Ashutosh Chauhan (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Chauhan resolved HIVE-2767.
------------------------------------
Resolution: Fixed
Fix Version/s: 0.10
Committed. Thanks, Travis!
> Optionally use framed transport with metastore
> ----------------------------------------------
>
> Key: HIVE-2767
> URL: https://issues.apache.org/jira/browse/HIVE-2767
> Project: Hive
> Issue Type: New Feature
> Components: Metastore
> Reporter: Travis Crawford
> Assignee: Travis Crawford
> Fix For: 0.10
>
> Attachments: HIVE-2767.D2661.1.patch, HIVE-2767.D2661.2.patch, HIVE-2767.D2661.3.patch, HIVE-2767.patch.txt, HIVE-2767_a.patch.txt
>
>
> Users may want/need to use thrift's framed transport when communicating with the Hive MetaStore. This patch adds a new property {{hive.metastore.thrift.framed.transport.enabled}} that enables the framed transport (defaults to off, aka no change from before the patch). This property must be set for both clients and the HMS server.
> It wasn't immediately clear how to use the framed transport with SASL, so as written an exception is thrown if you try starting the server with both options. If SASL and the framed transport will indeed work together I can update the patch (although I don't have a secured environment to test in).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2767) Optionally use framed transport with
metastore
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HIVE-2767:
------------------------------
Attachment: HIVE-2767.D2661.3.patch
travis updated the revision "HIVE-2767 [jira] Optionally use framed transport with metastore".
Reviewers: JIRA, ashutoshc
Update per comments from review. Add option to hive-default.xml, and disallow framed transport when using SASL as that has not been tested.
REVISION DETAIL
https://reviews.facebook.net/D2661
AFFECTED FILES
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
conf/hive-default.xml.template
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
> Optionally use framed transport with metastore
> ----------------------------------------------
>
> Key: HIVE-2767
> URL: https://issues.apache.org/jira/browse/HIVE-2767
> Project: Hive
> Issue Type: New Feature
> Components: Metastore
> Reporter: Travis Crawford
> Assignee: Travis Crawford
> Attachments: HIVE-2767.D2661.1.patch, HIVE-2767.D2661.2.patch, HIVE-2767.D2661.3.patch, HIVE-2767.patch.txt, HIVE-2767_a.patch.txt
>
>
> Users may want/need to use thrift's framed transport when communicating with the Hive MetaStore. This patch adds a new property {{hive.metastore.thrift.framed.transport.enabled}} that enables the framed transport (defaults to off, aka no change from before the patch). This property must be set for both clients and the HMS server.
> It wasn't immediately clear how to use the framed transport with SASL, so as written an exception is thrown if you try starting the server with both options. If SASL and the framed transport will indeed work together I can update the patch (although I don't have a secured environment to test in).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2767) Optionally use framed transport with
metastore
Posted by "Travis Crawford (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250707#comment-13250707 ]
Travis Crawford commented on HIVE-2767:
---------------------------------------
@ashutosh - Yeah this test does seem problematic. It works fine in IntelliJ but I can't get it to pass with the command you gave, even when doing a clean trunk build. Looking at Jenkins the test works fine, so perhaps its something to do with my machine (osx 10.7.3).
I'm looking into why the test doesn't work on trunk, then will see if this change affects it.
> Optionally use framed transport with metastore
> ----------------------------------------------
>
> Key: HIVE-2767
> URL: https://issues.apache.org/jira/browse/HIVE-2767
> Project: Hive
> Issue Type: New Feature
> Components: Metastore
> Reporter: Travis Crawford
> Assignee: Travis Crawford
> Attachments: HIVE-2767.D2661.1.patch, HIVE-2767.D2661.2.patch, HIVE-2767.D2661.3.patch, HIVE-2767.patch.txt, HIVE-2767_a.patch.txt
>
>
> Users may want/need to use thrift's framed transport when communicating with the Hive MetaStore. This patch adds a new property {{hive.metastore.thrift.framed.transport.enabled}} that enables the framed transport (defaults to off, aka no change from before the patch). This property must be set for both clients and the HMS server.
> It wasn't immediately clear how to use the framed transport with SASL, so as written an exception is thrown if you try starting the server with both options. If SASL and the framed transport will indeed work together I can update the patch (although I don't have a secured environment to test in).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-2767) Optionally use framed transport with
metastore
Posted by "Ashutosh Chauhan (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Chauhan reassigned HIVE-2767:
--------------------------------------
Assignee: Julien Le Dem (was: Travis Crawford)
Not to the latest trunk : ) Patch fails to apply. Also, can you use arc to submit the patch. Instructions are here: https://cwiki.apache.org/confluence/display/Hive/PhabricatorCodeReview
> Optionally use framed transport with metastore
> ----------------------------------------------
>
> Key: HIVE-2767
> URL: https://issues.apache.org/jira/browse/HIVE-2767
> Project: Hive
> Issue Type: New Feature
> Components: Metastore
> Reporter: Travis Crawford
> Assignee: Julien Le Dem
> Attachments: HIVE-2767.patch.txt, HIVE-2767_a.patch.txt
>
>
> Users may want/need to use thrift's framed transport when communicating with the Hive MetaStore. This patch adds a new property {{hive.metastore.thrift.framed.transport.enabled}} that enables the framed transport (defaults to off, aka no change from before the patch). This property must be set for both clients and the HMS server.
> It wasn't immediately clear how to use the framed transport with SASL, so as written an exception is thrown if you try starting the server with both options. If SASL and the framed transport will indeed work together I can update the patch (although I don't have a secured environment to test in).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2767) Optionally use framed transport with
metastore
Posted by "Ashutosh Chauhan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13217743#comment-13217743 ]
Ashutosh Chauhan commented on HIVE-2767:
----------------------------------------
Thanks Travis for explaining the usecase. Makes sense. Without that I was shooting in dark about why you need Framed Transport. : ) Yeah, update the patch. I will test and commit.
> Optionally use framed transport with metastore
> ----------------------------------------------
>
> Key: HIVE-2767
> URL: https://issues.apache.org/jira/browse/HIVE-2767
> Project: Hive
> Issue Type: New Feature
> Components: Metastore
> Reporter: Travis Crawford
> Assignee: Travis Crawford
> Attachments: HIVE-2767.patch.txt
>
>
> Users may want/need to use thrift's framed transport when communicating with the Hive MetaStore. This patch adds a new property {{hive.metastore.thrift.framed.transport.enabled}} that enables the framed transport (defaults to off, aka no change from before the patch). This property must be set for both clients and the HMS server.
> It wasn't immediately clear how to use the framed transport with SASL, so as written an exception is thrown if you try starting the server with both options. If SASL and the framed transport will indeed work together I can update the patch (although I don't have a secured environment to test in).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2767) Optionally use framed transport with
metastore
Posted by "Travis Crawford (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13217694#comment-13217694 ]
Travis Crawford commented on HIVE-2767:
---------------------------------------
Thanks for the feedback Ashutosh - if this is something you'd consider adding I can update based on current trunk if things have changed.
While there may be perf gains, this is needed to integrate with our compute grid. Basically a wrapper registers the metastore host:port in zookeeper, and a thrift framed-transport-only proxy proxies metastore requests based on its ZK registration. This lets us launch the metastore on our mesos cluster and still give clients the hard-coded host:port they expect.
> Optionally use framed transport with metastore
> ----------------------------------------------
>
> Key: HIVE-2767
> URL: https://issues.apache.org/jira/browse/HIVE-2767
> Project: Hive
> Issue Type: New Feature
> Components: Metastore
> Reporter: Travis Crawford
> Assignee: Travis Crawford
> Attachments: HIVE-2767.patch.txt
>
>
> Users may want/need to use thrift's framed transport when communicating with the Hive MetaStore. This patch adds a new property {{hive.metastore.thrift.framed.transport.enabled}} that enables the framed transport (defaults to off, aka no change from before the patch). This property must be set for both clients and the HMS server.
> It wasn't immediately clear how to use the framed transport with SASL, so as written an exception is thrown if you try starting the server with both options. If SASL and the framed transport will indeed work together I can update the patch (although I don't have a secured environment to test in).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2767) Optionally use framed transport with
metastore
Posted by "Ashutosh Chauhan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13249582#comment-13249582 ]
Ashutosh Chauhan commented on HIVE-2767:
----------------------------------------
After your patch test {{ant test -Dtestcase=TestHiveServerSessions}} is timing out. Can you take a look?
> Optionally use framed transport with metastore
> ----------------------------------------------
>
> Key: HIVE-2767
> URL: https://issues.apache.org/jira/browse/HIVE-2767
> Project: Hive
> Issue Type: New Feature
> Components: Metastore
> Reporter: Travis Crawford
> Assignee: Travis Crawford
> Attachments: HIVE-2767.D2661.1.patch, HIVE-2767.D2661.2.patch, HIVE-2767.D2661.3.patch, HIVE-2767.patch.txt, HIVE-2767_a.patch.txt
>
>
> Users may want/need to use thrift's framed transport when communicating with the Hive MetaStore. This patch adds a new property {{hive.metastore.thrift.framed.transport.enabled}} that enables the framed transport (defaults to off, aka no change from before the patch). This property must be set for both clients and the HMS server.
> It wasn't immediately clear how to use the framed transport with SASL, so as written an exception is thrown if you try starting the server with both options. If SASL and the framed transport will indeed work together I can update the patch (although I don't have a secured environment to test in).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2767) Optionally use framed transport with
metastore
Posted by "Julien Le Dem (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13217794#comment-13217794 ]
Julien Le Dem commented on HIVE-2767:
-------------------------------------
I have an updated patch, as Hive changed since this patch. Will post it soon.
> Optionally use framed transport with metastore
> ----------------------------------------------
>
> Key: HIVE-2767
> URL: https://issues.apache.org/jira/browse/HIVE-2767
> Project: Hive
> Issue Type: New Feature
> Components: Metastore
> Reporter: Travis Crawford
> Assignee: Travis Crawford
> Attachments: HIVE-2767.patch.txt
>
>
> Users may want/need to use thrift's framed transport when communicating with the Hive MetaStore. This patch adds a new property {{hive.metastore.thrift.framed.transport.enabled}} that enables the framed transport (defaults to off, aka no change from before the patch). This property must be set for both clients and the HMS server.
> It wasn't immediately clear how to use the framed transport with SASL, so as written an exception is thrown if you try starting the server with both options. If SASL and the framed transport will indeed work together I can update the patch (although I don't have a secured environment to test in).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-2767) Optionally use framed transport with
metastore
Posted by "Travis Crawford (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Travis Crawford reassigned HIVE-2767:
-------------------------------------
Assignee: Travis Crawford (was: Julien Le Dem)
> Optionally use framed transport with metastore
> ----------------------------------------------
>
> Key: HIVE-2767
> URL: https://issues.apache.org/jira/browse/HIVE-2767
> Project: Hive
> Issue Type: New Feature
> Components: Metastore
> Reporter: Travis Crawford
> Assignee: Travis Crawford
> Attachments: HIVE-2767.D2661.1.patch, HIVE-2767.patch.txt, HIVE-2767_a.patch.txt
>
>
> Users may want/need to use thrift's framed transport when communicating with the Hive MetaStore. This patch adds a new property {{hive.metastore.thrift.framed.transport.enabled}} that enables the framed transport (defaults to off, aka no change from before the patch). This property must be set for both clients and the HMS server.
> It wasn't immediately clear how to use the framed transport with SASL, so as written an exception is thrown if you try starting the server with both options. If SASL and the framed transport will indeed work together I can update the patch (although I don't have a secured environment to test in).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2767) Optionally use framed transport with
metastore
Posted by "Ashutosh Chauhan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Chauhan updated HIVE-2767:
-----------------------------------
Status: Open (was: Patch Available)
Patch needs to be rebased.
> Optionally use framed transport with metastore
> ----------------------------------------------
>
> Key: HIVE-2767
> URL: https://issues.apache.org/jira/browse/HIVE-2767
> Project: Hive
> Issue Type: New Feature
> Components: Metastore
> Reporter: Travis Crawford
> Assignee: Julien Le Dem
> Attachments: HIVE-2767.patch.txt, HIVE-2767_a.patch.txt
>
>
> Users may want/need to use thrift's framed transport when communicating with the Hive MetaStore. This patch adds a new property {{hive.metastore.thrift.framed.transport.enabled}} that enables the framed transport (defaults to off, aka no change from before the patch). This property must be set for both clients and the HMS server.
> It wasn't immediately clear how to use the framed transport with SASL, so as written an exception is thrown if you try starting the server with both options. If SASL and the framed transport will indeed work together I can update the patch (although I don't have a secured environment to test in).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2767) Optionally use framed transport with
metastore
Posted by "Ashutosh Chauhan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250711#comment-13250711 ]
Ashutosh Chauhan commented on HIVE-2767:
----------------------------------------
@Travis,
It has nothing to do with your patch. Problem exists on trunk. See, HIVE-2937 And ya, its a race condition, so it doesn't show up always.
> Optionally use framed transport with metastore
> ----------------------------------------------
>
> Key: HIVE-2767
> URL: https://issues.apache.org/jira/browse/HIVE-2767
> Project: Hive
> Issue Type: New Feature
> Components: Metastore
> Reporter: Travis Crawford
> Assignee: Travis Crawford
> Attachments: HIVE-2767.D2661.1.patch, HIVE-2767.D2661.2.patch, HIVE-2767.D2661.3.patch, HIVE-2767.patch.txt, HIVE-2767_a.patch.txt
>
>
> Users may want/need to use thrift's framed transport when communicating with the Hive MetaStore. This patch adds a new property {{hive.metastore.thrift.framed.transport.enabled}} that enables the framed transport (defaults to off, aka no change from before the patch). This property must be set for both clients and the HMS server.
> It wasn't immediately clear how to use the framed transport with SASL, so as written an exception is thrown if you try starting the server with both options. If SASL and the framed transport will indeed work together I can update the patch (although I don't have a secured environment to test in).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2767) Optionally use framed transport with
metastore
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252739#comment-13252739 ]
Phabricator commented on HIVE-2767:
-----------------------------------
ashutoshc has accepted the revision "HIVE-2767 [jira] Optionally use framed transport with metastore".
+1 will commit soon.
REVISION DETAIL
https://reviews.facebook.net/D2661
BRANCH
HIVE-2767_optional_framed_transport
> Optionally use framed transport with metastore
> ----------------------------------------------
>
> Key: HIVE-2767
> URL: https://issues.apache.org/jira/browse/HIVE-2767
> Project: Hive
> Issue Type: New Feature
> Components: Metastore
> Reporter: Travis Crawford
> Assignee: Travis Crawford
> Fix For: 0.10
>
> Attachments: HIVE-2767.D2661.1.patch, HIVE-2767.D2661.2.patch, HIVE-2767.D2661.3.patch, HIVE-2767.patch.txt, HIVE-2767_a.patch.txt
>
>
> Users may want/need to use thrift's framed transport when communicating with the Hive MetaStore. This patch adds a new property {{hive.metastore.thrift.framed.transport.enabled}} that enables the framed transport (defaults to off, aka no change from before the patch). This property must be set for both clients and the HMS server.
> It wasn't immediately clear how to use the framed transport with SASL, so as written an exception is thrown if you try starting the server with both options. If SASL and the framed transport will indeed work together I can update the patch (although I don't have a secured environment to test in).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2767) Optionally use framed transport with
metastore
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HIVE-2767:
------------------------------
Attachment: HIVE-2767.D2661.1.patch
travis requested code review of "HIVE-2767 [jira] Optionally use framed transport with metastore".
Reviewers: JIRA
Add support for optionally using the thrift framed transport, enabling integration with environments where that is necessary.
Users may want/need to use thrift's framed transport when communicating with the Hive MetaStore. This patch adds a new property hive.metastore.thrift.framed.transport.enabled that enables the framed transport (defaults to off, aka no change from before the patch). This property must be set for both clients and the HMS server.
It wasn't immediately clear how to use the framed transport with SASL, so as written an exception is thrown if you try starting the server with both options. If SASL and the framed transport will indeed work together I can update the patch (although I don't have a secured environment to test in).
TEST PLAN
Tested locally that client and server can connect, both with and without the flag. Tests pass.
REVISION DETAIL
https://reviews.facebook.net/D2661
AFFECTED FILES
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
MANAGE HERALD DIFFERENTIAL RULES
https://reviews.facebook.net/herald/view/differential/
WHY DID I GET THIS EMAIL?
https://reviews.facebook.net/herald/transcript/6111/
Tip: use the X-Herald-Rules header to filter Herald messages in your client.
> Optionally use framed transport with metastore
> ----------------------------------------------
>
> Key: HIVE-2767
> URL: https://issues.apache.org/jira/browse/HIVE-2767
> Project: Hive
> Issue Type: New Feature
> Components: Metastore
> Reporter: Travis Crawford
> Assignee: Julien Le Dem
> Attachments: HIVE-2767.D2661.1.patch, HIVE-2767.patch.txt, HIVE-2767_a.patch.txt
>
>
> Users may want/need to use thrift's framed transport when communicating with the Hive MetaStore. This patch adds a new property {{hive.metastore.thrift.framed.transport.enabled}} that enables the framed transport (defaults to off, aka no change from before the patch). This property must be set for both clients and the HMS server.
> It wasn't immediately clear how to use the framed transport with SASL, so as written an exception is thrown if you try starting the server with both options. If SASL and the framed transport will indeed work together I can update the patch (although I don't have a secured environment to test in).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2767) Optionally use framed transport with
metastore
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252746#comment-13252746 ]
Phabricator commented on HIVE-2767:
-----------------------------------
travis has committed the revision "HIVE-2767 [jira] Optionally use framed transport with metastore".
Change committed by hashutosh.
REVISION DETAIL
https://reviews.facebook.net/D2661
COMMIT
https://reviews.facebook.net/rHIVE1325446
> Optionally use framed transport with metastore
> ----------------------------------------------
>
> Key: HIVE-2767
> URL: https://issues.apache.org/jira/browse/HIVE-2767
> Project: Hive
> Issue Type: New Feature
> Components: Metastore
> Reporter: Travis Crawford
> Assignee: Travis Crawford
> Fix For: 0.10
>
> Attachments: HIVE-2767.D2661.1.patch, HIVE-2767.D2661.2.patch, HIVE-2767.D2661.3.patch, HIVE-2767.patch.txt, HIVE-2767_a.patch.txt
>
>
> Users may want/need to use thrift's framed transport when communicating with the Hive MetaStore. This patch adds a new property {{hive.metastore.thrift.framed.transport.enabled}} that enables the framed transport (defaults to off, aka no change from before the patch). This property must be set for both clients and the HMS server.
> It wasn't immediately clear how to use the framed transport with SASL, so as written an exception is thrown if you try starting the server with both options. If SASL and the framed transport will indeed work together I can update the patch (although I don't have a secured environment to test in).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2767) Optionally use framed transport with
metastore
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13249300#comment-13249300 ]
Phabricator commented on HIVE-2767:
-----------------------------------
ashutoshc has requested changes to the revision "HIVE-2767 [jira] Optionally use framed transport with metastore".
What will happen if the config variable is set to true in client, but is false in server and vice-versa?
INLINE COMMENTS
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:269 This also needs to be added in conf/hive-default.xml.template which serve as a documentation.
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:3020 In your earlier version of patch you were throwing exception in case both secure mode as well as framed transport is set to true. If you have tested with both true and it works then its fine, otherwise we should throw an exception here.
REVISION DETAIL
https://reviews.facebook.net/D2661
BRANCH
HIVE-2767_optional_framed_transport
> Optionally use framed transport with metastore
> ----------------------------------------------
>
> Key: HIVE-2767
> URL: https://issues.apache.org/jira/browse/HIVE-2767
> Project: Hive
> Issue Type: New Feature
> Components: Metastore
> Reporter: Travis Crawford
> Assignee: Travis Crawford
> Attachments: HIVE-2767.D2661.1.patch, HIVE-2767.patch.txt, HIVE-2767_a.patch.txt
>
>
> Users may want/need to use thrift's framed transport when communicating with the Hive MetaStore. This patch adds a new property {{hive.metastore.thrift.framed.transport.enabled}} that enables the framed transport (defaults to off, aka no change from before the patch). This property must be set for both clients and the HMS server.
> It wasn't immediately clear how to use the framed transport with SASL, so as written an exception is thrown if you try starting the server with both options. If SASL and the framed transport will indeed work together I can update the patch (although I don't have a secured environment to test in).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2767) Optionally use framed transport with
metastore
Posted by "Travis Crawford (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Travis Crawford updated HIVE-2767:
----------------------------------
Attachment: HIVE-2767.patch.txt
> Optionally use framed transport with metastore
> ----------------------------------------------
>
> Key: HIVE-2767
> URL: https://issues.apache.org/jira/browse/HIVE-2767
> Project: Hive
> Issue Type: New Feature
> Components: Metastore
> Reporter: Travis Crawford
> Attachments: HIVE-2767.patch.txt
>
>
> Users may want/need to use thrift's framed transport when communicating with the Hive MetaStore. This patch adds a new property {{hive.metastore.thrift.framed.transport.enabled}} that enables the framed transport (defaults to off, aka no change from before the patch). This property must be set for both clients and the HMS server.
> It wasn't immediately clear how to use the framed transport with SASL, so as written an exception is thrown if you try starting the server with both options. If SASL and the framed transport will indeed work together I can update the patch (although I don't have a secured environment to test in).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2767) Optionally use framed transport with
metastore
Posted by "Julien Le Dem (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Julien Le Dem updated HIVE-2767:
--------------------------------
Attachment: HIVE-2767_a.patch.txt
patch updated to latest Hive
> Optionally use framed transport with metastore
> ----------------------------------------------
>
> Key: HIVE-2767
> URL: https://issues.apache.org/jira/browse/HIVE-2767
> Project: Hive
> Issue Type: New Feature
> Components: Metastore
> Reporter: Travis Crawford
> Assignee: Travis Crawford
> Attachments: HIVE-2767.patch.txt, HIVE-2767_a.patch.txt
>
>
> Users may want/need to use thrift's framed transport when communicating with the Hive MetaStore. This patch adds a new property {{hive.metastore.thrift.framed.transport.enabled}} that enables the framed transport (defaults to off, aka no change from before the patch). This property must be set for both clients and the HMS server.
> It wasn't immediately clear how to use the framed transport with SASL, so as written an exception is thrown if you try starting the server with both options. If SASL and the framed transport will indeed work together I can update the patch (although I don't have a secured environment to test in).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2767) Optionally use framed transport with
metastore
Posted by "Travis Crawford (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Travis Crawford updated HIVE-2767:
----------------------------------
Status: Patch Available (was: Open)
> Optionally use framed transport with metastore
> ----------------------------------------------
>
> Key: HIVE-2767
> URL: https://issues.apache.org/jira/browse/HIVE-2767
> Project: Hive
> Issue Type: New Feature
> Components: Metastore
> Reporter: Travis Crawford
> Attachments: HIVE-2767.patch.txt
>
>
> Users may want/need to use thrift's framed transport when communicating with the Hive MetaStore. This patch adds a new property {{hive.metastore.thrift.framed.transport.enabled}} that enables the framed transport (defaults to off, aka no change from before the patch). This property must be set for both clients and the HMS server.
> It wasn't immediately clear how to use the framed transport with SASL, so as written an exception is thrown if you try starting the server with both options. If SASL and the framed transport will indeed work together I can update the patch (although I don't have a secured environment to test in).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2767) Optionally use framed transport with
metastore
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13249411#comment-13249411 ]
Phabricator commented on HIVE-2767:
-----------------------------------
travis has commented on the revision "HIVE-2767 [jira] Optionally use framed transport with metastore".
If just one side of the connection is using the framed transport the connection will fail, likely after the timeout.
As this is something that is setup just once per site its not likely to change often, and it defaults off, so the risk of casual misconfiguration is low.
Any suggestions on how to better clue the user in that the transport may be the issue?
INLINE COMMENTS
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:269 Good suggestion, done.
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:3020 Good suggestion. I don't have an environment to test SASL in, so I'll add the check back.
REVISION DETAIL
https://reviews.facebook.net/D2661
BRANCH
HIVE-2767_optional_framed_transport
> Optionally use framed transport with metastore
> ----------------------------------------------
>
> Key: HIVE-2767
> URL: https://issues.apache.org/jira/browse/HIVE-2767
> Project: Hive
> Issue Type: New Feature
> Components: Metastore
> Reporter: Travis Crawford
> Assignee: Travis Crawford
> Attachments: HIVE-2767.D2661.1.patch, HIVE-2767.patch.txt, HIVE-2767_a.patch.txt
>
>
> Users may want/need to use thrift's framed transport when communicating with the Hive MetaStore. This patch adds a new property {{hive.metastore.thrift.framed.transport.enabled}} that enables the framed transport (defaults to off, aka no change from before the patch). This property must be set for both clients and the HMS server.
> It wasn't immediately clear how to use the framed transport with SASL, so as written an exception is thrown if you try starting the server with both options. If SASL and the framed transport will indeed work together I can update the patch (although I don't have a secured environment to test in).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2767) Optionally use framed transport with
metastore
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HIVE-2767:
------------------------------
Attachment: HIVE-2767.D2661.2.patch
travis updated the revision "HIVE-2767 [jira] Optionally use framed transport with metastore".
Reviewers: JIRA, ashutoshc
-
REVISION DETAIL
https://reviews.facebook.net/D2661
AFFECTED FILES
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
conf/hive-default.xml.template
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
> Optionally use framed transport with metastore
> ----------------------------------------------
>
> Key: HIVE-2767
> URL: https://issues.apache.org/jira/browse/HIVE-2767
> Project: Hive
> Issue Type: New Feature
> Components: Metastore
> Reporter: Travis Crawford
> Assignee: Travis Crawford
> Attachments: HIVE-2767.D2661.1.patch, HIVE-2767.D2661.2.patch, HIVE-2767.patch.txt, HIVE-2767_a.patch.txt
>
>
> Users may want/need to use thrift's framed transport when communicating with the Hive MetaStore. This patch adds a new property {{hive.metastore.thrift.framed.transport.enabled}} that enables the framed transport (defaults to off, aka no change from before the patch). This property must be set for both clients and the HMS server.
> It wasn't immediately clear how to use the framed transport with SASL, so as written an exception is thrown if you try starting the server with both options. If SASL and the framed transport will indeed work together I can update the patch (although I don't have a secured environment to test in).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2767) Optionally use framed transport with
metastore
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253072#comment-13253072 ]
Hudson commented on HIVE-2767:
------------------------------
Integrated in Hive-trunk-h0.21 #1370 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1370/])
HIVE-2767 [jira] Optionally use framed transport with metastore
(Travis Crawford via Ashutosh Chauhan)
Summary:
Add support for optionally using the thrift framed transport, enabling
integration with environments where that is necessary.
Users may want/need to use thrift's framed transport when communicating with the
Hive MetaStore. This patch adds a new property
hive.metastore.thrift.framed.transport.enabled that enables the framed transport
(defaults to off, aka no change from before the patch). This property must be
set for both clients and the HMS server.
It wasn't immediately clear how to use the framed transport with SASL, so as
written an exception is thrown if you try starting the server with both options.
If SASL and the framed transport will indeed work together I can update the
patch (although I don't have a secured environment to test in).
Test Plan:
Tested locally that client and server can connect, both with and
without the flag. Tests pass.
Reviewers: JIRA, ashutoshc
Reviewed By: ashutoshc
Differential Revision: https://reviews.facebook.net/D2661 (Revision 1325446)
Result = SUCCESS
hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1325446
Files :
* /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
* /hive/trunk/conf/hive-default.xml.template
* /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
* /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
> Optionally use framed transport with metastore
> ----------------------------------------------
>
> Key: HIVE-2767
> URL: https://issues.apache.org/jira/browse/HIVE-2767
> Project: Hive
> Issue Type: New Feature
> Components: Metastore
> Reporter: Travis Crawford
> Assignee: Travis Crawford
> Fix For: 0.10
>
> Attachments: HIVE-2767.D2661.1.patch, HIVE-2767.D2661.2.patch, HIVE-2767.D2661.3.patch, HIVE-2767.patch.txt, HIVE-2767_a.patch.txt
>
>
> Users may want/need to use thrift's framed transport when communicating with the Hive MetaStore. This patch adds a new property {{hive.metastore.thrift.framed.transport.enabled}} that enables the framed transport (defaults to off, aka no change from before the patch). This property must be set for both clients and the HMS server.
> It wasn't immediately clear how to use the framed transport with SASL, so as written an exception is thrown if you try starting the server with both options. If SASL and the framed transport will indeed work together I can update the patch (although I don't have a secured environment to test in).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-2767) Optionally use framed transport with
metastore
Posted by "Ashutosh Chauhan (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Chauhan reassigned HIVE-2767:
--------------------------------------
Assignee: Travis Crawford
Patch looks ok to me. I presume this is for performance. Just curious, have you seen substantial performance gains because of this. Since, most of the rpc call in MetaStore are of fairly small payload. To me, it appears it won't bring in much perf gains.
> Optionally use framed transport with metastore
> ----------------------------------------------
>
> Key: HIVE-2767
> URL: https://issues.apache.org/jira/browse/HIVE-2767
> Project: Hive
> Issue Type: New Feature
> Components: Metastore
> Reporter: Travis Crawford
> Assignee: Travis Crawford
> Attachments: HIVE-2767.patch.txt
>
>
> Users may want/need to use thrift's framed transport when communicating with the Hive MetaStore. This patch adds a new property {{hive.metastore.thrift.framed.transport.enabled}} that enables the framed transport (defaults to off, aka no change from before the patch). This property must be set for both clients and the HMS server.
> It wasn't immediately clear how to use the framed transport with SASL, so as written an exception is thrown if you try starting the server with both options. If SASL and the framed transport will indeed work together I can update the patch (although I don't have a secured environment to test in).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira