You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Wei-Chiu Chuang (Jira)" <ji...@apache.org> on 2025/04/14 22:44:00 UTC
[jira] [Assigned] (HADOOP-19249) Getting NullPointerException when the unauthorised user tries to perform the key operation
[ https://issues.apache.org/jira/browse/HADOOP-19249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wei-Chiu Chuang reassigned HADOOP-19249:
----------------------------------------
Assignee: Dhaval Shah
> Getting NullPointerException when the unauthorised user tries to perform the key operation
> ------------------------------------------------------------------------------------------
>
> Key: HADOOP-19249
> URL: https://issues.apache.org/jira/browse/HADOOP-19249
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, security
> Reporter: Dhaval Shah
> Assignee: Dhaval Shah
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.5.0, 3.4.1
>
>
> While validating the tomcat 9.x in apache Ranger when user doesn't have appropriate permission in Ranger policies we faced the NPE for key operation using hadoop cmd.
> *Problem :*
> _Functionally -_ We are facing the NPE while performing key operations from hadoop cmd with the user not having permission in policy on cluster with tomcat v9.x. However with curl to Ranger KSM Server is working as expected.
> _Technically -_ Getting response message as null on client side in hadoop-common at [KMSClientProvider.java|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java#L565]
> *E.G.*
> _with Ranger KMS tomcat v9.x_
> {code:java}
> hadoop key list
> The list subcommand displays the keynames contained within
> a particular provider as configured in core-site.xml or
> specified with the -provider argument. -metadata displays
> the metadata. If -strict is supplied, fail immediately if
> the provider requires a password and none is given.
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hadoop.crypto.key.KeyShell.prettifyException(KeyShell.java:541)
> at org.apache.hadoop.crypto.key.KeyShell.printException(KeyShell.java:536)
> at org.apache.hadoop.tools.CommandShell.run(CommandShell.java:79)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:81)
> at org.apache.hadoop.crypto.key.KeyShell.main(KeyShell.java:553) {code}
> _on_ _Ranger KMS_ _tomcat v8.5.x_
> {code:java}
> hadoop key list
> The list subcommand displays the keynames contained within
> a particular provider as configured in core-site.xml or
> specified with the -provider argument. -metadata displays
> the metadata. If -strict is supplied, fail immediately ifthe provider requires a password and none is given.
> Executing command failed with the following exception: AuthorizationException: User:xyzuser not allowed to do 'GET_KEYS'{code}
> *Debug logs on Ranger KMS Server side*
> 1.) Added logs in [KMSExceptionsProvider.java|https://github.com/apache/ranger/blob/master/kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSExceptionsProvider.java] in method _createResponse()_ and _toResponse()_ where we are generating response to send it to client i.e. _hadoop-common_
> Logs are exactly same on both the tomcat scenario. Refer below the added logs, detailed logs will be available in ranger kms log file on cluster.
> {code:java}
> 2024-07-25 11:35:51,452 INFO org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider: [https-jsse-nio-9494-exec-2]: ==== Entered into toResponse =========
> 2024-07-25 11:35:51,452 INFO org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider: [https-jsse-nio-9494-exec-2]: ==== exception =========org.apache.hadoop.security.authorize.AuthorizationException: User:systest not allowed to do 'GET_KEYS'
> 2024-07-25 11:35:51,452 INFO org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider: [https-jsse-nio-9494-exec-2]: ==== exception.getClass() =========class org.apache.hadoop.security.authorize.AuthorizationException
> 2024-07-25 11:35:51,452 INFO org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider: [https-jsse-nio-9494-exec-2]: ==== AuthorizationException =========
> 2024-07-25 11:35:51,452 WARN org.apache.hadoop.crypto.key.kms.server.KMS: [https-jsse-nio-9494-exec-2]: User systest@ROOT.COMOPS.SITE (auth:KERBEROS) request GET https://ccycloud-1.ss-tomcat-test1.root.comops.site:9494/kms/v1/keys/names caused exception.
> org.apache.hadoop.security.authorize.AuthorizationException: User:systest not allowed to do 'GET_KEYS'
> 2024-07-25 11:35:51,452 INFO org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider: [https-jsse-nio-9494-exec-2]: ===== Entered into createResponse ======
> 2024-07-25 11:35:51,452 INFO org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider: [https-jsse-nio-9494-exec-2]: ==== status ======= Forbidden
> 2024-07-25 11:35:51,452 INFO org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider: [https-jsse-nio-9494-exec-2]: ======= ex ======= org.apache.hadoop.security.authorize.AuthorizationException: User:systest not allowed to do 'GET_KEYS'
> 2024-07-25 11:35:51,452 INFO org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider: [https-jsse-nio-9494-exec-2]: ======= ex.getStackTrace() ======= [Ljava.lang.StackTraceElement;@3e75ae9d
> 2024-07-25 11:35:51,452 INFO org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider: [https-jsse-nio-9494-exec-2]: ======= ex.getMessage() ======= User:systest not allowed to do 'GET_KEYS'
> 2024-07-25 11:35:51,452 INFO org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider: [https-jsse-nio-9494-exec-2]: ======= ex.toString() ======= org.apache.hadoop.security.authorize.AuthorizationException: User:systest not allowed to do 'GET_KEYS' {code}
> 2.) Also added logs in [KMSExceptionsProvider.java|https://github.com/apache/ranger/blob/master/kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSExceptionsProvider.java]
> Adding logs in code base
> {code:java}
> public void setStatus(int sc, String sm) {
> LOG.info("========= setStatus with message============ ");
> statusCode = sc;
> msg = sm;
> LOG.info("========= sc ============ " +sc);
> LOG.info("========= msg ============ " +msg);
> if(sc == 403) {
> LOG.info("===== its 403 ====");
> super.setStatus(sc, sm);
> } else{
> super.setStatus(sc, sm);
> }
> } {code}
> LOGS:
> {code:java}
> 2024-07-25 11:35:51,460 INFO org.apache.hadoop.crypto.key.kms.server.KMSAuthenticationFilter: [https-jsse-nio-9494-exec-2]: ========= setStatus with message============
> 2024-07-25 11:35:51,460 INFO org.apache.hadoop.crypto.key.kms.server.KMSAuthenticationFilter: [https-jsse-nio-9494-exec-2]: ========= sc ============ 403
> 2024-07-25 11:35:51,460 INFO org.apache.hadoop.crypto.key.kms.server.KMSAuthenticationFilter: [https-jsse-nio-9494-exec-2]: ========= msg ============ Forbidden
> 2024-07-25 11:35:51,460 INFO org.apache.hadoop.crypto.key.kms.server.KMSAuthenticationFilter: [https-jsse-nio-9494-exec-2]: ===== its 403 ==== {code}
> This explains that the KMS server is sending the code and message appropriately.
> *Debug logs on Hadoop Common Client side*
> 1.) Added logs in [HttpExceptionUtils.java|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HttpExceptionUtils.java] to make sure whether appropriate response is received.
> Logs will be available in ranger kms log file.
> {code:java}
> 2024-07-25 11:35:51,453 INFO org.apache.hadoop.util.HttpExceptionUtils: [https-jsse-nio-9494-exec-2]: ====== Entered into createJerseyExceptionResponse ====
> 2024-07-25 11:35:51,453 INFO org.apache.hadoop.util.HttpExceptionUtils: [https-jsse-nio-9494-exec-2]: ========== ex ======== org.apache.hadoop.security.authorize.AuthorizationException: User:systest not allowed to do 'GET_KEYS'
> 2024-07-25 11:35:51,454 INFO org.apache.hadoop.util.HttpExceptionUtils: [https-jsse-nio-9494-exec-2]: ========== ex.getMessage ======== User:systest not allowed to do 'GET_KEYS'
> 2024-07-25 11:35:51,454 INFO org.apache.hadoop.util.HttpExceptionUtils: [https-jsse-nio-9494-exec-2]: ========== status ======== Forbidden
> 2024-07-25 11:35:51,454 INFO org.apache.hadoop.util.HttpExceptionUtils: [https-jsse-nio-9494-exec-2]: ========== status.getStatusCode ======== 403
> 2024-07-25 11:35:51,454 INFO org.apache.hadoop.util.HttpExceptionUtils: [https-jsse-nio-9494-exec-2]: ========== status.getReasonPhrase ======== Forbidden
> 2024-07-25 11:35:51,454 INFO org.apache.hadoop.util.HttpExceptionUtils: [https-jsse-nio-9494-exec-2]: ======= response ======== com.sun.jersey.core.spi.factory.ResponseImpl@5bd8a59b {code}
> 2.) Added logs exactly before NPE occurs i.e. [KMSClientProvider.java|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java#L564]
> Adding logs in code base
> LOG.info(" =========== conn ======== " + conn);
> Map<String, List<String>> map = conn.getHeaderFields();
> LOG.info("======= map ======== " + map);for (Map.Entry<String, List<String>> entry : map.entrySet()) {
> LOG.info("=============== " + "Key : " + entry.getKey() + " ,Value : " + entry.getValue());
> }
> LOG.info(" =========== conn.getResponseMessage ======== " + conn.getResponseMessage());
> LOG.info(" =========== conn.getResponseCode ======== " + conn.getResponseCode());if ((conn.getResponseCode() == HttpURLConnection.HTTP_FORBIDDEN
> && (conn.getResponseMessage().equals(ANONYMOUS_REQUESTS_DISALLOWED) ||
> conn.getResponseMessage().contains(INVALID_SIGNATURE)))
> || conn.getResponseCode() == HttpURLConnection.HTTP_UNAUTHORIZED) {
> LOGS: This logs gets printed on terminal where we execute hadoop cmd .
> _with Ranger KMS tomcat v9.x_
> {code:java}
> hadoop key list
> 24/07/25 11:38:15 INFO kms.KMSClientProvider: ======== Entered into call ========
> 24/07/25 11:38:15 INFO kms.KMSClientProvider: =========== conn ======== sun.net.www.protocol.https.DelegateHttpsURLConnection:https://ccycloud-1.ss-tomcat-test1.root.comops.site:9494/kms/v1/keys/names
> 24/07/25 11:38:15 INFO kms.KMSClientProvider: ======= map ======== {Keep-Alive=[timeout=60], null=[HTTP/1.1 403], Strict-Transport-Security=[max-age=31536000; includeSubDomains; preload], Server=[Apache Ranger], Connection=[keep-alive], Content-Length=[220], Date=[Thu, 25 Jul 2024 11:38:15 GMT], Content-Type=[application/json]}
> 24/07/25 11:38:15 INFO kms.KMSClientProvider: =============== Key : Keep-Alive ,Value : [timeout=60]
> 24/07/25 11:38:15 INFO kms.KMSClientProvider: =============== Key : null ,Value : [HTTP/1.1 403]
> 24/07/25 11:38:15 INFO kms.KMSClientProvider: =============== Key : Strict-Transport-Security ,Value : [max-age=31536000; includeSubDomains; preload]
> 24/07/25 11:38:15 INFO kms.KMSClientProvider: =============== Key : Server ,Value : [Apache Ranger]
> 24/07/25 11:38:15 INFO kms.KMSClientProvider: =============== Key : Connection ,Value : [keep-alive]
> 24/07/25 11:38:15 INFO kms.KMSClientProvider: =============== Key : Content-Length ,Value : [220]
> 24/07/25 11:38:15 INFO kms.KMSClientProvider: =============== Key : Date ,Value : [Thu, 25 Jul 2024 11:38:15 GMT]
> 24/07/25 11:38:15 INFO kms.KMSClientProvider: =============== Key : Content-Type ,Value : [application/json]
> 24/07/25 11:38:15 INFO kms.KMSClientProvider: =========== conn.getResponseMessage ======== null
> 24/07/25 11:38:15 INFO kms.KMSClientProvider: =========== conn.getResponseCode ======== 403
> list [-provider <provider>] [-strict] [-metadata] [-help]:
> The list subcommand displays the keynames contained within
> a particular provider as configured in core-site.xml or
> specified with the -provider argument. -metadata displays
> the metadata. If -strict is supplied, fail immediately if
> the provider requires a password and none is given.
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hadoop.crypto.key.KeyShell.prettifyException(KeyShell.java:541)
> at org.apache.hadoop.crypto.key.KeyShell.printException(KeyShell.java:536)
> at org.apache.hadoop.tools.CommandShell.run(CommandShell.java:79)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:81)
> at org.apache.hadoop.crypto.key.KeyShell.main(KeyShell.java:553) {code}
> _with Ranger KMS tomcat v8.5.x_
> hadoop key list
> 24/07/25 11:02:25 INFO kms.KMSClientProvider: ======== Entered into call ========
> 24/07/25 11:02:25 INFO kms.KMSClientProvider: =========== conn ======== sun.net.www.protocol.https.DelegateHttpsURLConnection:https://ccycloud-1.ds-tomcat-test1.root.comops.site:9494/kms/v1/keys/names24/07/25 11:02:25 INFO kms.KMSClientProvider: ======= map ======== {Keep-Alive=[timeout=60], null=[HTTP/1.1 403 Forbidden], Strict-Transport-Security=[max-age=31536000; includeSubDomains; preload], Server=[Apache Ranger], Connection=[keep-alive], Content-Length=[220], Date=[Thu, 25 Jul 2024 11:02:25 GMT], Content-Type=[application/json]}
> 24/07/25 11:02:25 INFO kms.KMSClientProvider: =============== Key : Keep-Alive ,Value : [timeout=60]
> 24/07/25 11:02:25 INFO kms.KMSClientProvider: =============== Key : null ,Value : [HTTP/1.1 403 Forbidden]
> 24/07/25 11:02:25 INFO kms.KMSClientProvider: =============== Key : Strict-Transport-Security ,Value : [max-age=31536000; includeSubDomains; preload]
> 24/07/25 11:02:25 INFO kms.KMSClientProvider: =============== Key : Server ,Value : [Apache Ranger]
> 24/07/25 11:02:25 INFO kms.KMSClientProvider: =============== Key : Connection ,Value : [keep-alive]
> 24/07/25 11:02:25 INFO kms.KMSClientProvider: =============== Key : Content-Length ,Value : [220]
> 24/07/25 11:02:25 INFO kms.KMSClientProvider: =============== Key : Date ,Value : [Thu, 25 Jul 2024 11:02:25 GMT]
> 24/07/25 11:02:25 INFO kms.KMSClientProvider: =============== Key : Content-Type ,Value : [application/json]
> 24/07/25 11:02:25 INFO kms.KMSClientProvider: =========== conn.getResponseMessage ======== Forbidden
> 24/07/25 11:02:25 INFO kms.KMSClientProvider: =========== conn.getResponseCode ======== 403
> Cannot list keys for KeyProvider: org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider@209da20d
> list [-provider <provider>] [-strict] [-metadata] [-help]:
> The list subcommand displays the keynames contained within
> a particular provider as configured in core-site.xml or
> specified with the -provider argument. -metadata displays
> the metadata. If -strict is supplied, fail immediately ifthe provider requires a password and none is given.
> Executing command failed with the following exception: AuthorizationException: User:xyzuser not allowed to do 'GET_KEYS'
> Please notice
> _with tomcat v9.x : *Key : null ,Value : [HTTP/1.1 403]*_
> _with. tomcat v8.5.x : *Key : null ,Value : [HTTP/1.1 403 Forbidden]*_
> **
> Message "Forbidden" is not present with tomcat v9.x.
> It seems that tomcat v9.x is not setting the message and hadoop-common is trying to get where we are facing NPE.
> Also checked for _*org.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER*_ but its not available in tomcat 9.x
> Ref:
> Tomcat Doc for 8.5.x [https://tomcat.apache.org/tomcat-8.5-doc/api/org/apache/coyote/Constants.html#USE_CUSTOM_STATUS_MSG_IN_HEADER]
> Tomcat Doc for 9.x [https://tomcat.apache.org/tomcat-9.0-doc/api/org/apache/coyote/Constants.html]
> Thanks
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org