You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Pakapoj Tulsuk (Jira)" <ji...@apache.org> on 2021/05/07 05:22:00 UTC
[jira] [Comment Edited] (HDDS-5193) Permission Deny when using
auth:TOKEN
[ https://issues.apache.org/jira/browse/HDDS-5193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17340569#comment-17340569 ]
Pakapoj Tulsuk edited comment on HDDS-5193 at 5/7/21, 5:21 AM:
---------------------------------------------------------------
{{PERMISSION_DENIED}} issue throwing this error only when we auth to ozone using {{auth:token}} - {{auth:token}} was used on spark executors so it suppose happen on both *client* and *cluster* mode (however we found it when we deploy on *cluster* mode)
{code:java}
2021-05-06 12:12:11,945 [OM StateMachine ApplyTransaction Thread - 0] DEBUG org.apache.hadoop.ozone.om.KeyManagerImpl: user:pakapoj_tul@DEV.TAP (auth:SIMPLE) has access rights for key:mykey1/106186534408552448 :false
2021-05-06 12:12:11,945 [OM StateMachine ApplyTransaction Thread - 0] DEBUG org.apache.hadoop.ozone.om.lock.OzoneManagerLock: Release read BUCKET_LOCK, lock on resource /vol1/bucket1
2021-05-06 12:12:11,945 [OM StateMachine ApplyTransaction Thread - 0] WARN org.apache.hadoop.ozone.om.OzoneManager: User pakapoj_tul@DEV.TAP doesn't have WRITE permission to access key /vol1/bucket1/mykey1/106186534408552448
2021-05-06 12:12:11,947 [OMDoubleBufferFlushThread] DEBUG org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer: Sync Iteration 3 flushed transactions in this iteration1
2021-05-06 12:12:11,947 [OM StateMachine ApplyTransaction Thread - 0] ERROR org.apache.hadoop.ozone.om.request.key.OMKeyCommitRequest: Key commit failed. Volume:vol1, Bucket:bucket1, Key:mykey1.
PERMISSION_DENIED org.apache.hadoop.ozone.om.exceptions.OMException: User pakapoj_tul@DEV.TAP doesn't have WRITE permission to access key vol1 bucket1 mykey1/106186534408552448
at org.apache.hadoop.ozone.om.OzoneManager.checkAcls(OzoneManager.java:1803)
at org.apache.hadoop.ozone.om.request.OMClientRequest.checkAcls(OMClientRequest.java:207)
at org.apache.hadoop.ozone.om.request.OMClientRequest.checkAcls(OMClientRequest.java:185)
at org.apache.hadoop.ozone.om.request.key.OMKeyRequest.checkKeyAcls(OMKeyRequest.java:437)
at org.apache.hadoop.ozone.om.request.key.OMKeyRequest.checkKeyAclsInOpenKeyTable(OMKeyRequest.java:485)
at org.apache.hadoop.ozone.om.request.key.OMKeyCommitRequest.validateAndUpdateCache(OMKeyCommitRequest.java:139)
at org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleWriteRequest(OzoneManagerRequestHandler.java:227)
at org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.runCommand(OzoneManagerStateMachine.java:415)
at org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.lambda$applyTransaction$1(OzoneManagerStateMachine.java:240)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2021-05-06 12:12:11,947 [OMDoubleBufferFlushThread] DEBUG org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine: ComputeAndUpdateLastAppliedIndex due to SM is (t:82, i:3665)
{code}
last week i have confirm the reproduction by using {{auth:token}} and also check on the client side, RPC call, which seem to be normal. where the error will happed when {{commitKey}} is requested to OM. from that point I understand that it’s the server side who response the *denial*.
so I went to OM log, see above, during the failure it says it failed on {{[org.apache.hadoop.ozone.om|http://org.apache.hadoop.ozone.om/]}}{{.OzoneManager.checkAcls(OzoneManager.java:1803)}} I went there and It looks fine then I move on to {{org.apache.hadoop.ozone.om.KeyManagerImpl}} on {{checkAcls}} since it the one who pass the value to the function and print out some log… and I got
{code:java}
[TOKEN]
2021-05-06 12:12:11,945 [OM StateMachine ApplyTransaction Thread - 0] DEBUG org.apache.hadoop.ozone.om.helpers.OzoneAclUtil: Type: user, A Name: pakapoj_tul, username: pakapoj_tul@DEV.TAP, aclToCheck: WRITE, rights: {7}
[KERBEROS]
2021-05-06 12:13:35,305 [OM StateMachine ApplyTransaction Thread - 0] DEBUG org.apache.hadoop.ozone.om.helpers.OzoneAclUtil: Type: user, A Name: pakapoj_tul@DEV.TAP, username: pakapoj_tul@DEV.TAP, aclToCheck: WRITE, rights: {7}{code}
It looks like when we auth with TOKEN the username resolve into {{pakapoj_tul}} but KERBEROS is {{pakapoj_tul@DEV.TAP}}
In my understanding *pakapoj_tul* is username and *pakapoj_tul@DEV.TAP* is principle which both is the same identity on DFS but the code {{OzoneAclUtil.java#L105}} says
{code:java}
if (a.getName().equals(username))
return checkIfAclBitIsSet(aclToCheck, rights);{code}
so it’s not equal, so I workaround the issue by add function {{normalizedUsername}} to remove realm and apply it before the comparison
was (Author: pakapoj):
*Explanation* {{PERMISSION_DENIED}} issue throwing this error only when we auth to ozone using {{auth:token}} - {{auth:token}} was used on spark executors so it suppose happen on both *client* and *cluster* mode (however we found it when we deploy on *cluster* mode)
2021-05-06 12:12:11,945 [OM StateMachine ApplyTransaction Thread - 0] DEBUG org.apache.hadoop.ozone.om.KeyManagerImpl: [user:pakapoj_tul@DEV.TAP]
(auth:SIMPLE) has access rights for key:mykey1/106186534408552448 :false
2021-05-06 12:12:11,945 [OM StateMachine ApplyTransaction Thread - 0] DEBUG org.apache.hadoop.ozone.om.lock.OzoneManagerLock: Release read BUCKET_LOCK, lock on resource /vol1/bucket1
2021-05-06 12:12:11,945 [OM StateMachine ApplyTransaction Thread - 0] WARN org.apache.hadoop.ozone.om.OzoneManager: User pakapoj_tul@DEV.TAP doesn't have WRITE permission to access key /vol1/bucket1/mykey1/106186534408552448
2021-05-06 12:12:11,947 [OMDoubleBufferFlushThread] DEBUG org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer: Sync Iteration 3 flushed transactions in this iteration1
2021-05-06 12:12:11,947 [OM StateMachine ApplyTransaction Thread - 0] ERROR org.apache.hadoop.ozone.om.request.key.OMKeyCommitRequest: Key commit failed. Volume:vol1, Bucket:bucket1, Key:mykey1.
PERMISSION_DENIED org.apache.hadoop.ozone.om.exceptions.OMException: User pakapoj_tul@DEV.TAP doesn't have WRITE permission to access key vol1 bucket1 mykey1/106186534408552448
at org.apache.hadoop.ozone.om.OzoneManager.checkAcls(OzoneManager.java:1803)
at org.apache.hadoop.ozone.om.request.OMClientRequest.checkAcls(OMClientRequest.java:207)
at org.apache.hadoop.ozone.om.request.OMClientRequest.checkAcls(OMClientRequest.java:185)
at org.apache.hadoop.ozone.om.request.key.OMKeyRequest.checkKeyAcls(OMKeyRequest.java:437)
at org.apache.hadoop.ozone.om.request.key.OMKeyRequest.checkKeyAclsInOpenKeyTable(OMKeyRequest.java:485)
at org.apache.hadoop.ozone.om.request.key.OMKeyCommitRequest.validateAndUpdateCache(OMKeyCommitRequest.java:139)
at org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleWriteRequest(OzoneManagerRequestHandler.java:227)
at org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.runCommand(OzoneManagerStateMachine.java:415)
at org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.lambda$applyTransaction$1(OzoneManagerStateMachine.java:240)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2021-05-06 12:12:11,947 [OMDoubleBufferFlushThread] DEBUG org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine: ComputeAndUpdateLastAppliedIndex due to SM is (t:82, i:3665)
last week it have confirm the reproduction by using {{auth:token}} and also check on the client side, RPC call, which seem to be normal. where the error will happed when {{commitKey}} is requested to OM. from that point I understand that it’s the server side who response the *denial*.
so I went to OM log, see above, during the failure it says it failed on {{[org.apache.hadoop.ozone.om|http://org.apache.hadoop.ozone.om/]}}{{.OzoneManager.checkAcls(OzoneManager.java:1803)}} I went there and It looks fine then I move on to {{org.apache.hadoop.ozone.om.KeyManagerImpl}} on {{checkAcls}} since it the one who pass the value to the function and print out some log… and I got
[TOKEN]
2021-05-06 12:12:11,945 [OM StateMachine ApplyTransaction Thread - 0] DEBUG org.apache.hadoop.ozone.om.helpers.OzoneAclUtil: Type: user, A Name: pakapoj_tul, username: pakapoj_tul@DEV.TAP, aclToCheck: WRITE, rights: \{7}[KERBEROS]
2021-05-06 12:13:35,305 [OM StateMachine ApplyTransaction Thread - 0] DEBUG org.apache.hadoop.ozone.om.helpers.OzoneAclUtil: Type: user, A Name: pakapoj_tul@DEV.TAP, username: pakapoj_tul@DEV.TAP, aclToCheck: WRITE, rights: \{7}
It looks like when we auth with TOKEN the username resolve into {{pakapoj_tul}} but KERBEROS is {{pakapoj_tul@DEV.TAP}}
In my understanding *pakapoj_tul* is username and *pakapoj_tul@DEV.TAP* is principle which both is the same identity on DFS but the code {{OzoneAclUtil.java#L105}} says
if (a.getName().equals(username))
return checkIfAclBitIsSet(aclToCheck, rights);
so it’s not equal, so I workaround the issue by add function {{normalizedUsername}} to remove realm and apply it before the comparison
> Permission Deny when using auth:TOKEN
> -------------------------------------
>
> Key: HDDS-5193
> URL: https://issues.apache.org/jira/browse/HDDS-5193
> Project: Apache Ozone
> Issue Type: Bug
> Components: OM
> Affects Versions: 1.1.0
> Environment: A Ozone (version 1.1 build from source) cluster with 3 master 3 datanode deploy on baremetal(VMs) running CentOS 7
>
> Reporter: Pakapoj Tulsuk
> Priority: Major
> Attachments: ozone-client-kerberos.log, ozone-client-token.log, ozone-java-client.java
>
>
> Hi I’m got stuck on the permission issue where I gonna write the data, a text file to a ozone path {{/vol1/bucket1/mykey}} * with {{auth:KERBEROS}} It be able to complete the task
> {code:java}
> 2021-04-29 11:49:01,145 Socket Reader #1 for port 9862 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for pakapoj_tul@DEV.TAP (auth:KERBEROS) from ip.ip.ip.ip:40294 * with auth:TOKEN It got stuck on this error despite the given permission to /vol1 /bucket1 see below
> 2021-04-29 11:49:08,327 Socket Reader #1 for port 9862 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for pakapoj_tul@DEV.TAP (auth:TOKEN) from ip.ip.ip.ip:40412
> 2021-04-29 11:49:12,228 Socket Reader #1 for port 9862 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for pakapoj_tul@DEV.TAP (auth:TOKEN) from ip.ip.ip.ip:35266
> 2021-04-29 11:49:14,671 [OM StateMachine ApplyTransaction Thread - 0] WARN org.apache.hadoop.ozone.om.OzoneManager: User pakapoj_tul@DEV.TAP doesn't have WRITE permission to access key /vol1/bucket1/mykey/_temporary/0/_temporary/attempt_202104290449105826106778232640855_0000_m_000000_0/part-00000-9f9c4fcc-5e96-43ee-b53e-913a06729109-c000.txt/106146807974133768
> 2021-04-29 11:49:14,672 [OM StateMachine ApplyTransaction Thread - 0] ERROR org.apache.hadoop.ozone.om.request.key.OMKeyCommitRequest: Key commit failed. Volume:vol1, Bucket:bucket1, Key:mykey/_temporary/0/_temporary/attempt_202104290449105826106778232640855_0000_m_000000_0/part-00000-9f9c4fcc-5e96-43ee-b53e-913a06729109-c000.txt.
> PERMISSION_DENIED org.apache.hadoop.ozone.om.exceptions.OMException: User pakapoj_tul@DEV.TAP doesn't have WRITE permission to access key vol1 bucket1 mykey/_temporary/0/_temporary/attempt_202104290449105826106778232640855_0000_m_000000_0/part-00000-9f9c4fcc-5e96-43ee-b53e-913a06729109-c000.txt/106146807974133768
> at org.apache.hadoop.ozone.om.OzoneManager.checkAcls(OzoneManager.java:1803)
> at org.apache.hadoop.ozone.om.request.OMClientRequest.checkAcls(OMClientRequest.java:207)
> at org.apache.hadoop.ozone.om.request.OMClientRequest.checkAcls(OMClientRequest.java:185)
> at org.apache.hadoop.ozone.om.request.key.OMKeyRequest.checkKeyAcls(OMKeyRequest.java:437)
> at org.apache.hadoop.ozone.om.request.key.OMKeyRequest.checkKeyAclsInOpenKeyTable(OMKeyRequest.java:485)
> at org.apache.hadoop.ozone.om.request.key.OMKeyCommitRequest.validateAndUpdateCache(OMKeyCommitRequest.java:139)
> at org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleWriteRequest(OzoneManagerRequestHandler.java:227)
> at org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.runCommand(OzoneManagerStateMachine.java:415)
> at org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.lambda$applyTransaction$1(OzoneManagerStateMachine.java:240)
> at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> *Given Permission*
> {code:java}
> $ ozone sh vol getacl /vol1/
> [ {
> "type" : "USER",
> "name" : "pakapoj_tul",
> "aclScope" : "ACCESS",
> "aclList" : [ "WRITE", "ALL" ]
> }, {
> "type" : "USER",
> "name" : "pakapoj_tul@DEV.TAP",
> "aclScope" : "ACCESS",
> "aclList" : [ "WRITE", "ALL" ]
> }, {
> "type" : "USER",
> "name" : "ozone-admin@DEV.TAP",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> }, {
> "type" : "GROUP",
> "name" : "ozone-admin",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> }, {
> "type" : "GROUP",
> "name" : "ozone-users",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> } ]
> $ ozone sh bucket getacl /vol1/bucket1/
> [ {
> "type" : "USER",
> "name" : "ozone-admin@DEV.TAP",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> }, {
> "type" : "GROUP",
> "name" : "ozone-admin",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> }, {
> "type" : "GROUP",
> "name" : "ozone-users",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> }, {
> "type" : "USER",
> "name" : "pakapoj_tul@DEV.TAP",
> "aclScope" : "ACCESS",
> "aclList" : [ "WRITE", "ALL" ]
> }, {
> "type" : "USER",
> "name" : "pakapoj_tul",
> "aclScope" : "ACCESS",
> "aclList" : [ "WRITE", "ALL" ]
> } ]
> $ ozone sh key getacl /vol1/bucket1/mykey/
> [ {
> "type" : "USER",
> "name" : "ozone-admin@DEV.TAP",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> }, {
> "type" : "GROUP",
> "name" : "ozone-admin",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> }, {
> "type" : "GROUP",
> "name" : "ozone-users",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> }, {
> "type" : "USER",
> "name" : "pakapoj_tul@DEV.TAP",
> "aclScope" : "ACCESS",
> "aclList" : [ "WRITE", "ALL" ]
> }, {
> "type" : "USER",
> "name" : "pakapoj_tul",
> "aclScope" : "ACCESS",
> "aclList" : [ "WRITE", "ALL" ]
> } ]{code}
>
> The spark code was deployed in Kubernetes in spark cluster mode. Then, the error would happed on spark executors side when the do {{commitKey}} with {{auth:TOKEN}} , BTW the spark driver was using {{auth:KERBEROS}} .
>
> so I reproduce using ozone java client writing to ozone with {{OzoneClient}} using # Token by {{export HADOOP_TOKEN_FILE_LOCATION=credential/ozone.token}} before running the program
> # Keytab by running {{/usr/bin/kinit -kt credential/pakapoj_tul.keytab pakapoj_tul@DEV.TAP}} before running the program
> the code, output for #1 and #2 (DEBUG) is in attach
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org