You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Enis Soztutar (Created) (JIRA)" <ji...@apache.org> on 2012/01/31 02:20:10 UTC
[jira] [Created] (HIVE-2764) Obtain delegation tokens for MR jobs
in secure hbase setup
Obtain delegation tokens for MR jobs in secure hbase setup
------------------------------------------------------------
Key: HIVE-2764
URL: https://issues.apache.org/jira/browse/HIVE-2764
Project: Hive
Issue Type: Improvement
Components: HBase Handler, Security
Reporter: Enis Soztutar
Assignee: Enis Soztutar
As discussed in HCATALOG-244, in a secure hbase setup with 0.92, we need to obtain delegation tokens for hbase and save it in jobconf, so that tasks can access region servers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2764) Obtain delegation tokens for MR jobs
in secure hbase setup
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HIVE-2764:
------------------------------
Attachment: HIVE-2764.D2205.4.patch
enis updated the revision "HIVE-2764 [jira] Obtain delegation tokens for MR jobs in secure hbase setup".
Reviewers: JIRA, ashutoshc
Fixed failing tests
REVISION DETAIL
https://reviews.facebook.net/D2205
AFFECTED FILES
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java
ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java
ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
ql/src/java/org/apache/hadoop/hive/ql/io/HiveNullValueSequenceFileOutputFormat.java
ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormat.java
ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormatImpl.java
ql/src/java/org/apache/hadoop/hive/ql/io/HiveSequenceFileOutputFormat.java
ql/src/java/org/apache/hadoop/hive/ql/io/RCFileOutputFormat.java
ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java
ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java
shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java
shims/src/common-secure/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java
shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java
> Obtain delegation tokens for MR jobs in secure hbase setup
> ------------------------------------------------------------
>
> Key: HIVE-2764
> URL: https://issues.apache.org/jira/browse/HIVE-2764
> Project: Hive
> Issue Type: Improvement
> Components: HBase Handler, Security
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
> Attachments: HIVE-2764.D2205.1.patch, HIVE-2764.D2205.2.patch, HIVE-2764.D2205.3.patch, HIVE-2764.D2205.4.patch, HIVE-2764_v0.patch
>
>
> As discussed in HCATALOG-244, in a secure hbase setup with 0.92, we need to obtain delegation tokens for hbase and save it in jobconf, so that tasks can access region servers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2764) Obtain delegation tokens for MR jobs
in secure hbase setup
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HIVE-2764:
------------------------------
Attachment: HIVE-2764.D2205.2.patch
enis updated the revision "HIVE-2764 [jira] Obtain delegation tokens for MR jobs in secure hbase setup".
Reviewers: JIRA
Realized that HiveHFileOutputFormat does not need delegation tokens. Duh!
REVISION DETAIL
https://reviews.facebook.net/D2205
AFFECTED FILES
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java
ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java
ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
ql/src/java/org/apache/hadoop/hive/ql/io/HiveNullValueSequenceFileOutputFormat.java
ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormat.java
ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormatImpl.java
ql/src/java/org/apache/hadoop/hive/ql/io/HiveSequenceFileOutputFormat.java
ql/src/java/org/apache/hadoop/hive/ql/io/RCFileOutputFormat.java
ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java
shims/src/test/org/apache/hadoop/hive/thrift/TestZooKeeperTokenStore.java
> Obtain delegation tokens for MR jobs in secure hbase setup
> ------------------------------------------------------------
>
> Key: HIVE-2764
> URL: https://issues.apache.org/jira/browse/HIVE-2764
> Project: Hive
> Issue Type: Improvement
> Components: HBase Handler, Security
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
> Attachments: HIVE-2764.D2205.1.patch, HIVE-2764.D2205.2.patch, HIVE-2764_v0.patch
>
>
> As discussed in HCATALOG-244, in a secure hbase setup with 0.92, we need to obtain delegation tokens for hbase and save it in jobconf, so that tasks can access region servers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2764) Obtain delegation tokens for MR jobs
in secure hbase setup
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13246820#comment-13246820 ]
Phabricator commented on HIVE-2764:
-----------------------------------
enis has commented on the revision "HIVE-2764 [jira] Obtain delegation tokens for MR jobs in secure hbase setup".
Strange, all of the tests passed for me except for one. I rerun that and it passed as well. Can you rerun the tests or post the logs ?
REVISION DETAIL
https://reviews.facebook.net/D2205
> Obtain delegation tokens for MR jobs in secure hbase setup
> ------------------------------------------------------------
>
> Key: HIVE-2764
> URL: https://issues.apache.org/jira/browse/HIVE-2764
> Project: Hive
> Issue Type: Improvement
> Components: HBase Handler, Security
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
> Attachments: HIVE-2764.D2205.1.patch, HIVE-2764.D2205.2.patch, HIVE-2764.D2205.3.patch, HIVE-2764_v0.patch
>
>
> As discussed in HCATALOG-244, in a secure hbase setup with 0.92, we need to obtain delegation tokens for hbase and save it in jobconf, so that tasks can access region servers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2764) Obtain delegation tokens for MR jobs
in secure hbase setup
Posted by "Ashutosh Chauhan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13246830#comment-13246830 ]
Ashutosh Chauhan commented on HIVE-2764:
----------------------------------------
In TestCliDriver, following tests failed:
* alter_concatenate_indexed_table.q
* alter_merge.q
* alter_merge_2.q
* alter_merge_stats.q
* concatenate_inherit_table_location.q
* create_merge_compressed.q
* escape2.q
I re-ran them again individually, and they again failed.
> Obtain delegation tokens for MR jobs in secure hbase setup
> ------------------------------------------------------------
>
> Key: HIVE-2764
> URL: https://issues.apache.org/jira/browse/HIVE-2764
> Project: Hive
> Issue Type: Improvement
> Components: HBase Handler, Security
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
> Attachments: HIVE-2764.D2205.1.patch, HIVE-2764.D2205.2.patch, HIVE-2764.D2205.3.patch, HIVE-2764_v0.patch
>
>
> As discussed in HCATALOG-244, in a secure hbase setup with 0.92, we need to obtain delegation tokens for hbase and save it in jobconf, so that tasks can access region servers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2764) Obtain delegation tokens for MR jobs
in secure hbase setup
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13245883#comment-13245883 ]
Phabricator commented on HIVE-2764:
-----------------------------------
enis has commented on the revision "HIVE-2764 [jira] Obtain delegation tokens for MR jobs in secure hbase setup".
INLINE COMMENTS
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java:104 Yes, that is correct. It was never invoked.
ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java:268 HadoopShims.setNullOutputFormat() not only sets the OF, but sets the null output committer, etc. So we still need that call.
ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java:769 checkOS() is called at the job submission time from the client, but FileSinkOperator.initializeOp() is called from the task context when the job is running. So we need to initialize the output format.
ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java:781 I though HOF was hive-private. You are saying that, we need this for maintaining binary BC? We are also changing the HOF to extent OF, so all HOF implementations have to be fixed (to include the OF methods) and recompiled.
If for binary BC, I can use reflection to check whether the method is there.
ql/src/java/org/apache/hadoop/hive/ql/io/HiveNullValueSequenceFileOutputFormat.java:27 It is a rearrange for the import statement.
REVISION DETAIL
https://reviews.facebook.net/D2205
BRANCH
hbase-dt-of
> Obtain delegation tokens for MR jobs in secure hbase setup
> ------------------------------------------------------------
>
> Key: HIVE-2764
> URL: https://issues.apache.org/jira/browse/HIVE-2764
> Project: Hive
> Issue Type: Improvement
> Components: HBase Handler, Security
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
> Attachments: HIVE-2764.D2205.1.patch, HIVE-2764.D2205.2.patch, HIVE-2764_v0.patch
>
>
> As discussed in HCATALOG-244, in a secure hbase setup with 0.92, we need to obtain delegation tokens for hbase and save it in jobconf, so that tasks can access region servers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HIVE-2764) Obtain delegation tokens for MR jobs
in secure hbase setup
Posted by "Ashutosh Chauhan (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Chauhan resolved HIVE-2764.
------------------------------------
Resolution: Fixed
Fix Version/s: 0.9.0
Checked-in trunk. Thanks, Enis!
> Obtain delegation tokens for MR jobs in secure hbase setup
> ------------------------------------------------------------
>
> Key: HIVE-2764
> URL: https://issues.apache.org/jira/browse/HIVE-2764
> Project: Hive
> Issue Type: Improvement
> Components: HBase Handler, Security
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
> Fix For: 0.9.0
>
> Attachments: HIVE-2764.D2205.1.patch, HIVE-2764.D2205.2.patch, HIVE-2764.D2205.3.patch, HIVE-2764.D2205.4.patch, HIVE-2764.D2205.5.patch, HIVE-2764_v0.patch
>
>
> As discussed in HCATALOG-244, in a secure hbase setup with 0.92, we need to obtain delegation tokens for hbase and save it in jobconf, so that tasks can access region servers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2764) Obtain delegation tokens for MR jobs
in secure hbase setup
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250410#comment-13250410 ]
Hudson commented on HIVE-2764:
------------------------------
Integrated in Hive-trunk-h0.21 #1364 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1364/])
HIVE-2764: Obtain delegation tokens for MR jobs in secure hbase setup (Enis Soztutar via Ashutosh Chauhan) (Revision 1311418)
Result = SUCCESS
hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311418
Files :
* /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
* /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java
* /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveNullValueSequenceFileOutputFormat.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormat.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormatImpl.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveSequenceFileOutputFormat.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/RCFileOutputFormat.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java
* /hive/trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java
* /hive/trunk/shims/src/common-secure/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java
* /hive/trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java
> Obtain delegation tokens for MR jobs in secure hbase setup
> ------------------------------------------------------------
>
> Key: HIVE-2764
> URL: https://issues.apache.org/jira/browse/HIVE-2764
> Project: Hive
> Issue Type: Improvement
> Components: HBase Handler, Security
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
> Fix For: 0.9.0
>
> Attachments: HIVE-2764.D2205.1.patch, HIVE-2764.D2205.2.patch, HIVE-2764.D2205.3.patch, HIVE-2764.D2205.4.patch, HIVE-2764.D2205.5.patch, HIVE-2764_v0.patch
>
>
> As discussed in HCATALOG-244, in a secure hbase setup with 0.92, we need to obtain delegation tokens for hbase and save it in jobconf, so that tasks can access region servers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2764) Obtain delegation tokens for MR jobs
in secure hbase setup
Posted by "Ashutosh Chauhan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247075#comment-13247075 ]
Ashutosh Chauhan commented on HIVE-2764:
----------------------------------------
Those tests passed. So, thats a progress. But following new ones failed.
* rcfile_merge3.q
* rcfile_createas1.q
> Obtain delegation tokens for MR jobs in secure hbase setup
> ------------------------------------------------------------
>
> Key: HIVE-2764
> URL: https://issues.apache.org/jira/browse/HIVE-2764
> Project: Hive
> Issue Type: Improvement
> Components: HBase Handler, Security
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
> Attachments: HIVE-2764.D2205.1.patch, HIVE-2764.D2205.2.patch, HIVE-2764.D2205.3.patch, HIVE-2764.D2205.4.patch, HIVE-2764_v0.patch
>
>
> As discussed in HCATALOG-244, in a secure hbase setup with 0.92, we need to obtain delegation tokens for hbase and save it in jobconf, so that tasks can access region servers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2764) Obtain delegation tokens for MR jobs
in secure hbase setup
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225683#comment-13225683 ]
Phabricator commented on HIVE-2764:
-----------------------------------
enis has commented on the revision "HIVE-2764 [jira] Obtain delegation tokens for MR jobs in secure hbase setup".
Some comments about the patch:
Changing the StorageHandler.configureJobProperties() to be called with the actual job is kind of intrusive, so I have given up on that approach, and go with the plan of obtaining the delegation tokens from the IF.getSplits() and OF.checkOutputSpecs() time. However, one problem with this approach is that although on the input side it works fine, on the output side, Hive does not use OF's properly.
So, this patch adds a HiveOutputFormatImpl class, and changes the HOF to extend OF. The actual usage of OFs are not changed as a part of this patch, since it is out of the scope of this issue. However, not the HOFImpl class delgates the checkOutputSpecs() call to the underlying OutputFormat, so that if the OF wants to do any initialization, it can.
REVISION DETAIL
https://reviews.facebook.net/D2205
> Obtain delegation tokens for MR jobs in secure hbase setup
> ------------------------------------------------------------
>
> Key: HIVE-2764
> URL: https://issues.apache.org/jira/browse/HIVE-2764
> Project: Hive
> Issue Type: Improvement
> Components: HBase Handler, Security
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
> Attachments: HIVE-2764.D2205.1.patch, HIVE-2764_v0.patch
>
>
> As discussed in HCATALOG-244, in a secure hbase setup with 0.92, we need to obtain delegation tokens for hbase and save it in jobconf, so that tasks can access region servers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2764) Obtain delegation tokens for MR jobs
in secure hbase setup
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13235106#comment-13235106 ]
Phabricator commented on HIVE-2764:
-----------------------------------
ashutoshc has requested changes to the revision "HIVE-2764 [jira] Obtain delegation tokens for MR jobs in secure hbase setup".
Added comments. Also, patch needs to be rebased.
INLINE COMMENTS
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java:104 I understand from this patch that checkOutputSpecs() is never called. But, see an existing impl here. So, then before this patch, this code was never getting invoked ?
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java:214 I don't think you want to override this function and leave a no-op here. I think you still want super-class checkOS to be called. Isn't it ?
ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormatImpl.java:53 This is not supposed to get called, right ? In that case this should throw exception, instead of creating object and returning it.
ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormatImpl.java:45-49 I dont' see any use of this, you can as well keep it in local variable in checkOS()
ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormatImpl.java:75 Instead of throwing Runtime, you should throw IOException which method signature allows you to do.
ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java:268 This results in setting of NullOF in all the shims. We should no longer do that, since we are resetting it again in following line.
ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java:769 Is this null check necessary? At the time, checkOS is called, hiveOF would already have been initialized, no?
ql/src/java/org/apache/hadoop/hive/ql/io/HiveNullValueSequenceFileOutputFormat.java:27 Looks like this import statement inadvertently got added.
ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java:781 To maintain backward-compat you should first check whether hiveOF has the method checkOS, and call it only if it has.
REVISION DETAIL
https://reviews.facebook.net/D2205
BRANCH
hbase-dt-of
> Obtain delegation tokens for MR jobs in secure hbase setup
> ------------------------------------------------------------
>
> Key: HIVE-2764
> URL: https://issues.apache.org/jira/browse/HIVE-2764
> Project: Hive
> Issue Type: Improvement
> Components: HBase Handler, Security
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
> Attachments: HIVE-2764.D2205.1.patch, HIVE-2764.D2205.2.patch, HIVE-2764_v0.patch
>
>
> As discussed in HCATALOG-244, in a secure hbase setup with 0.92, we need to obtain delegation tokens for hbase and save it in jobconf, so that tasks can access region servers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2764) Obtain delegation tokens for MR jobs
in secure hbase setup
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HIVE-2764:
------------------------------
Attachment: HIVE-2764.D2205.1.patch
enis requested code review of "HIVE-2764 [jira] Obtain delegation tokens for MR jobs in secure hbase setup".
Reviewers: JIRA
HIVE-2764. Obtain delegation tokens for MR jobs in secure hbase setup.Title:
HIVE-2764. Obtain delegation tokens for MR jobs in secure hbase setup.
As discussed in HCATALOG-244, in a secure hbase setup with 0.92, we need to obtain delegation tokens for hbase and save it in jobconf, so that tasks can access region servers.
TEST PLAN
Tested manually by setting up a kerberos cluster with hadoop
+ hbase security enabled.
REVISION DETAIL
https://reviews.facebook.net/D2205
AFFECTED FILES
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java
ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java
ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
ql/src/java/org/apache/hadoop/hive/ql/io/HiveNullValueSequenceFileOutputFormat.java
ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormat.java
ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormatImpl.java
ql/src/java/org/apache/hadoop/hive/ql/io/HiveSequenceFileOutputFormat.java
ql/src/java/org/apache/hadoop/hive/ql/io/RCFileOutputFormat.java
ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java
MANAGE HERALD DIFFERENTIAL RULES
https://reviews.facebook.net/herald/view/differential/
WHY DID I GET THIS EMAIL?
https://reviews.facebook.net/herald/transcript/4839/
Tip: use the X-Herald-Rules header to filter Herald messages in your client.
> Obtain delegation tokens for MR jobs in secure hbase setup
> ------------------------------------------------------------
>
> Key: HIVE-2764
> URL: https://issues.apache.org/jira/browse/HIVE-2764
> Project: Hive
> Issue Type: Improvement
> Components: HBase Handler, Security
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
> Attachments: HIVE-2764.D2205.1.patch, HIVE-2764_v0.patch
>
>
> As discussed in HCATALOG-244, in a secure hbase setup with 0.92, we need to obtain delegation tokens for hbase and save it in jobconf, so that tasks can access region servers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2764) Obtain delegation tokens for MR jobs
in secure hbase setup
Posted by "Enis Soztutar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Enis Soztutar updated HIVE-2764:
--------------------------------
Attachment: HIVE-2764_v0.patch
Attaching a patch for demonstration of yet another approach of adding the Credentials to TableDesc. This patch is still WIP.
I have tested this with fully secure hadoop +hbase. Compiles with -Dhadoop.version=1.0.0
> Obtain delegation tokens for MR jobs in secure hbase setup
> ------------------------------------------------------------
>
> Key: HIVE-2764
> URL: https://issues.apache.org/jira/browse/HIVE-2764
> Project: Hive
> Issue Type: Improvement
> Components: HBase Handler, Security
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
> Attachments: HIVE-2764_v0.patch
>
>
> As discussed in HCATALOG-244, in a secure hbase setup with 0.92, we need to obtain delegation tokens for hbase and save it in jobconf, so that tasks can access region servers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2764) Obtain delegation tokens for MR jobs
in secure hbase setup
Posted by "Francis Liu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222994#comment-13222994 ]
Francis Liu commented on HIVE-2764:
-----------------------------------
Having making getCredentials() exposed as a TableDesc member sounds ok to me as it really is a special property of JobConf.
Though it breaks the expectation of tableDesc being readonly and jobProperties being the only one mutable for configureTableProperties(). I wonder if we can remove jobProperties altogether and use tableDesc.jobProperties to set properties. As tableDesc.jobProperties is always a new HashMap when configureTableProperties() is called?
> Obtain delegation tokens for MR jobs in secure hbase setup
> ------------------------------------------------------------
>
> Key: HIVE-2764
> URL: https://issues.apache.org/jira/browse/HIVE-2764
> Project: Hive
> Issue Type: Improvement
> Components: HBase Handler, Security
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
> Attachments: HIVE-2764_v0.patch
>
>
> As discussed in HCATALOG-244, in a secure hbase setup with 0.92, we need to obtain delegation tokens for hbase and save it in jobconf, so that tasks can access region servers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2764) Obtain delegation tokens for MR jobs
in secure hbase setup
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13245964#comment-13245964 ]
Phabricator commented on HIVE-2764:
-----------------------------------
enis has commented on the revision "HIVE-2764 [jira] Obtain delegation tokens for MR jobs in secure hbase setup".
INLINE COMMENTS
ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormatImpl.java:53 Hadoop framework calls the getRR() although we do not emit any records. So we need some kind of an empty implementation.
REVISION DETAIL
https://reviews.facebook.net/D2205
BRANCH
hbase-dt-of
> Obtain delegation tokens for MR jobs in secure hbase setup
> ------------------------------------------------------------
>
> Key: HIVE-2764
> URL: https://issues.apache.org/jira/browse/HIVE-2764
> Project: Hive
> Issue Type: Improvement
> Components: HBase Handler, Security
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
> Attachments: HIVE-2764.D2205.1.patch, HIVE-2764.D2205.2.patch, HIVE-2764_v0.patch
>
>
> As discussed in HCATALOG-244, in a secure hbase setup with 0.92, we need to obtain delegation tokens for hbase and save it in jobconf, so that tasks can access region servers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2764) Obtain delegation tokens for MR jobs
in secure hbase setup
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13246816#comment-13246816 ]
Phabricator commented on HIVE-2764:
-----------------------------------
ashutoshc has commented on the revision "HIVE-2764 [jira] Obtain delegation tokens for MR jobs in secure hbase setup".
Changes look good. But, I see plenty of failures while running tests.
REVISION DETAIL
https://reviews.facebook.net/D2205
> Obtain delegation tokens for MR jobs in secure hbase setup
> ------------------------------------------------------------
>
> Key: HIVE-2764
> URL: https://issues.apache.org/jira/browse/HIVE-2764
> Project: Hive
> Issue Type: Improvement
> Components: HBase Handler, Security
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
> Attachments: HIVE-2764.D2205.1.patch, HIVE-2764.D2205.2.patch, HIVE-2764.D2205.3.patch, HIVE-2764_v0.patch
>
>
> As discussed in HCATALOG-244, in a secure hbase setup with 0.92, we need to obtain delegation tokens for hbase and save it in jobconf, so that tasks can access region servers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2764) Obtain delegation tokens for MR jobs
in secure hbase setup
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HIVE-2764:
------------------------------
Attachment: HIVE-2764.D2205.3.patch
enis updated the revision "HIVE-2764 [jira] Obtain delegation tokens for MR jobs in secure hbase setup".
Reviewers: JIRA, ashutoshc
incorporated review suggestions:
- change HadoopShims.setNullOutputFormat() to .prepareJobOutput()
- BC check for HiveOutputFormat.checkOutputSpecs() call.
- HiveHFileOutputFormat.checkOS() now delegates to underlying OF.
- minor changes per review.
- running tests right now.
REVISION DETAIL
https://reviews.facebook.net/D2205
AFFECTED FILES
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java
ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java
ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
ql/src/java/org/apache/hadoop/hive/ql/io/HiveNullValueSequenceFileOutputFormat.java
ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormat.java
ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormatImpl.java
ql/src/java/org/apache/hadoop/hive/ql/io/HiveSequenceFileOutputFormat.java
ql/src/java/org/apache/hadoop/hive/ql/io/RCFileOutputFormat.java
ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java
ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java
shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java
shims/src/common-secure/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java
shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java
> Obtain delegation tokens for MR jobs in secure hbase setup
> ------------------------------------------------------------
>
> Key: HIVE-2764
> URL: https://issues.apache.org/jira/browse/HIVE-2764
> Project: Hive
> Issue Type: Improvement
> Components: HBase Handler, Security
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
> Attachments: HIVE-2764.D2205.1.patch, HIVE-2764.D2205.2.patch, HIVE-2764.D2205.3.patch, HIVE-2764_v0.patch
>
>
> As discussed in HCATALOG-244, in a secure hbase setup with 0.92, we need to obtain delegation tokens for hbase and save it in jobconf, so that tasks can access region servers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2764) Obtain delegation tokens for MR jobs
in secure hbase setup
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HIVE-2764:
------------------------------
Attachment: HIVE-2764.D2205.5.patch
enis updated the revision "HIVE-2764 [jira] Obtain delegation tokens for MR jobs in secure hbase setup".
Reviewers: JIRA, ashutoshc
Fixed one more issue with the tests. All tests pass now.
REVISION DETAIL
https://reviews.facebook.net/D2205
AFFECTED FILES
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java
ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java
ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
ql/src/java/org/apache/hadoop/hive/ql/io/HiveNullValueSequenceFileOutputFormat.java
ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormat.java
ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormatImpl.java
ql/src/java/org/apache/hadoop/hive/ql/io/HiveSequenceFileOutputFormat.java
ql/src/java/org/apache/hadoop/hive/ql/io/RCFileOutputFormat.java
ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java
ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java
shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java
shims/src/common-secure/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java
shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java
> Obtain delegation tokens for MR jobs in secure hbase setup
> ------------------------------------------------------------
>
> Key: HIVE-2764
> URL: https://issues.apache.org/jira/browse/HIVE-2764
> Project: Hive
> Issue Type: Improvement
> Components: HBase Handler, Security
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
> Attachments: HIVE-2764.D2205.1.patch, HIVE-2764.D2205.2.patch, HIVE-2764.D2205.3.patch, HIVE-2764.D2205.4.patch, HIVE-2764.D2205.5.patch, HIVE-2764_v0.patch
>
>
> As discussed in HCATALOG-244, in a secure hbase setup with 0.92, we need to obtain delegation tokens for hbase and save it in jobconf, so that tasks can access region servers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2764) Obtain delegation tokens for MR jobs
in secure hbase setup
Posted by "Enis Soztutar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222862#comment-13222862 ]
Enis Soztutar commented on HIVE-2764:
-------------------------------------
We had some discussion with Ashutosh about how to best approach this. As a part of this issue, we have to obtain delegation tokens from the storage handler (hbase in this case), and then pass this to the submitted job. However, HiveStorageHandler.configureTableJobProperties() does not expose the JobConf object, or an interface to configure the job's credentials. There seems a couple of options we can go for:
1. At HiveStorageHandler.configureTableJobProperties(), create and save the job credentials somewhere, and at the time of job submit, merge all the credentials before submitting the job.
2. Change the HiveStorageHandler.configureTableJobProperties() to get the actual Job to be submitted, so that Storage handlers have a real shot at configuring the job.
3. Add a method like HiveStorageHandler.configureJobCredentials(), and call it from HIF.getSplits() and HiveHBaseTableOutputFormat.checkOutputSpecs().
Number 1 and 3 seems hackish, and we will want to give the storagehandlers ability to configure the actual job. So we are leaning towards that. But this will also be merged with HIVE-2773. I am still trying to make this work, so any comments / suggestions are more than welcome.
> Obtain delegation tokens for MR jobs in secure hbase setup
> ------------------------------------------------------------
>
> Key: HIVE-2764
> URL: https://issues.apache.org/jira/browse/HIVE-2764
> Project: Hive
> Issue Type: Improvement
> Components: HBase Handler, Security
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
>
> As discussed in HCATALOG-244, in a secure hbase setup with 0.92, we need to obtain delegation tokens for hbase and save it in jobconf, so that tasks can access region servers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2764) Obtain delegation tokens for MR jobs
in secure hbase setup
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13245921#comment-13245921 ]
Phabricator commented on HIVE-2764:
-----------------------------------
ashutoshc has commented on the revision "HIVE-2764 [jira] Obtain delegation tokens for MR jobs in secure hbase setup".
INLINE COMMENTS
ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java:268 Sorry, I wasn't clear. What I meant was we should remove setting of OF class inside setNullOF() method, because its redundant as we set it again later in this method.
ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java:781 Yes, please use reflection to check if method is there and then only call.
REVISION DETAIL
https://reviews.facebook.net/D2205
BRANCH
hbase-dt-of
> Obtain delegation tokens for MR jobs in secure hbase setup
> ------------------------------------------------------------
>
> Key: HIVE-2764
> URL: https://issues.apache.org/jira/browse/HIVE-2764
> Project: Hive
> Issue Type: Improvement
> Components: HBase Handler, Security
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
> Attachments: HIVE-2764.D2205.1.patch, HIVE-2764.D2205.2.patch, HIVE-2764_v0.patch
>
>
> As discussed in HCATALOG-244, in a secure hbase setup with 0.92, we need to obtain delegation tokens for hbase and save it in jobconf, so that tasks can access region servers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira