You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Pakapoj Tulsuk (Jira)" <ji...@apache.org> on 2021/06/16 07:14:00 UTC
[jira] [Comment Edited] (HDDS-5193) Permission Deny when using
auth:TOKEN
[ https://issues.apache.org/jira/browse/HDDS-5193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17364111#comment-17364111 ]
Pakapoj Tulsuk edited comment on HDDS-5193 at 6/16/21, 7:13 AM:
----------------------------------------------------------------
Hi update
I think I found root causes of the event. As I mentioned in the description that I'm running spark on cluster mode against kerberized ozone with authorization enable. The application intents to read an orc file from ozone and write it to ozone.
code.
{code:java}
val data = spark.read.format("orc").load(src)
data.write.format("orc").save(des)
{code}
given
{code:java}
src = "ofs://dev-ozone/vol1/bucket1/sample_seq_data.snappy.orc"
des = "ofs://dev-ozone/vol1/bucket1/mykey"{code}
the ERROR happened on {{save}} function where it invoke {{createFile}} from {{OzoneBucket}} in this sequence
# From *spark driver* using {{auth:KERBEROS}} so it have {{pakapoj_tul@DEV.TAP}} identity and create temp file and dir such as
{{/vol1/bucket1/mykey/_temporary/0/_temporary/attempt_202106151221294802544738954213121_0000_m_000000_0/}} with acl {{USER:pakapoj_tul@DEV.TAP:ALL}} and related groups from local user {{pakapoj_tul}} from client local machine
# From *spark executor* using {{auth:TOKEN}} so it have {{pakapoj_tul}} identity and create data files in *the temp dir* since the dir have acl {{USER:pakapoj_tul@DEV.TAP:ALL}} so {{pakapoj_tul}} is not allowed. the ERROR is here
# In the end, after all executors finish writing files to temp dir the driver will move those temp dir to user specified destination, in the case is {{"ofs://dev-ozone/vol1/bucket1/mykey"}}
IMHO, in kerberized cluster identity {{pakapoj_tul@DEV.TAP and pakapoj_tul}} are equal so I propose to use getShortUsername() instead of getShortUsername() from UGI interface when assign and/or compare any ACLs. To leverage {{auth_to_local}} property to translate any identity from {{auth:KERBEROS}} to plain username which should be consistent with identity from {{auth:TOKEN}}
was (Author: pakapoj):
Hi update
I think I found root causes of the event. As I mentioned in the description that I'm running spark on cluster mode against kerberized ozone with authorization enable. The application intents to read an orc file from ozone and write it to ozone.
code.
{code:java}
val data = spark.read.format("orc").load(src)
data.write.format("orc").save(des)
{code}
given
{code:java}
src = "ofs://dev-ozone/vol1/bucket1/sample_seq_data.snappy.orc"
des = "ofs://dev-ozone/vol1/bucket1/mykey"{code}
the ERROR happened on {{save}} function where it invoke {{createFile}} from {{OzoneBucket}} in this sequence
# From *spark driver* using {{auth:KERBEROS}} so it have {{pakapoj_tul@DEV.TAP}} identity and create temp file and dir such as
{{/vol1/bucket1/mykey/_temporary/0/_temporary/attempt_202106151221294802544738954213121_0000_m_000000_0/}} with acl {{USER:pakapoj_tul@DEV.TAP:ALL}} and related groups from local user {{pakapoj_tul}} from client local machine
# From *spark executor* using {{auth:TOKEN}} so it have {{pakapoj_tul}} identity and create data files in *the temp dir* since the dir have acl {{USER:pakapoj_tul@DEV.TAP:ALL}} so {{pakapoj_tul}} is not allowed. the ERROR is here
# In the end, after all executors finish writing files to temp dir the driver will move those temp dir to user specified destination, in the case is {{"ofs://dev-ozone/vol1/bucket1/mykey"}}
IMHO, in kerberized cluster identity {{pakapoj_tul@DEV.TAP }}and {{pakapoj_tul}} are equal so I propose to use getShortUsername() instead of getShortUsername() from UGI interface when assign and/or compare any ACLs. To leverage {{auth_to_local}} property to translate any identity from {{auth:KERBEROS}} to plain username which should be consistent with identity from {{auth:TOKEN}}
> Permission Deny when using auth:TOKEN
> -------------------------------------
>
> Key: HDDS-5193
> URL: https://issues.apache.org/jira/browse/HDDS-5193
> Project: Apache Ozone
> Issue Type: Bug
> Components: OM
> Affects Versions: 1.1.0
> Environment: A Ozone (version 1.1 build from source) cluster with 3 master 3 datanode deploy on baremetal(VMs) running CentOS 7
>
> Reporter: Pakapoj Tulsuk
> Priority: Critical
> Attachments: ozone-client-kerberos.log, ozone-client-token.log, ozone-java-client.java
>
>
> Hi I’m got stuck on the permission issue where I gonna write the data, a text file to a ozone path {{/vol1/bucket1/mykey}} * with {{auth:KERBEROS}} It be able to complete the task
> {code:java}
> 2021-04-29 11:49:01,145 Socket Reader #1 for port 9862 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for pakapoj_tul@DEV.TAP (auth:KERBEROS) from ip.ip.ip.ip:40294 * with auth:TOKEN It got stuck on this error despite the given permission to /vol1 /bucket1 see below
> 2021-04-29 11:49:08,327 Socket Reader #1 for port 9862 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for pakapoj_tul@DEV.TAP (auth:TOKEN) from ip.ip.ip.ip:40412
> 2021-04-29 11:49:12,228 Socket Reader #1 for port 9862 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for pakapoj_tul@DEV.TAP (auth:TOKEN) from ip.ip.ip.ip:35266
> 2021-04-29 11:49:14,671 [OM StateMachine ApplyTransaction Thread - 0] WARN org.apache.hadoop.ozone.om.OzoneManager: User pakapoj_tul@DEV.TAP doesn't have WRITE permission to access key /vol1/bucket1/mykey/_temporary/0/_temporary/attempt_202104290449105826106778232640855_0000_m_000000_0/part-00000-9f9c4fcc-5e96-43ee-b53e-913a06729109-c000.txt/106146807974133768
> 2021-04-29 11:49:14,672 [OM StateMachine ApplyTransaction Thread - 0] ERROR org.apache.hadoop.ozone.om.request.key.OMKeyCommitRequest: Key commit failed. Volume:vol1, Bucket:bucket1, Key:mykey/_temporary/0/_temporary/attempt_202104290449105826106778232640855_0000_m_000000_0/part-00000-9f9c4fcc-5e96-43ee-b53e-913a06729109-c000.txt.
> PERMISSION_DENIED org.apache.hadoop.ozone.om.exceptions.OMException: User pakapoj_tul@DEV.TAP doesn't have WRITE permission to access key vol1 bucket1 mykey/_temporary/0/_temporary/attempt_202104290449105826106778232640855_0000_m_000000_0/part-00000-9f9c4fcc-5e96-43ee-b53e-913a06729109-c000.txt/106146807974133768
> at org.apache.hadoop.ozone.om.OzoneManager.checkAcls(OzoneManager.java:1803)
> at org.apache.hadoop.ozone.om.request.OMClientRequest.checkAcls(OMClientRequest.java:207)
> at org.apache.hadoop.ozone.om.request.OMClientRequest.checkAcls(OMClientRequest.java:185)
> at org.apache.hadoop.ozone.om.request.key.OMKeyRequest.checkKeyAcls(OMKeyRequest.java:437)
> at org.apache.hadoop.ozone.om.request.key.OMKeyRequest.checkKeyAclsInOpenKeyTable(OMKeyRequest.java:485)
> at org.apache.hadoop.ozone.om.request.key.OMKeyCommitRequest.validateAndUpdateCache(OMKeyCommitRequest.java:139)
> at org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleWriteRequest(OzoneManagerRequestHandler.java:227)
> at org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.runCommand(OzoneManagerStateMachine.java:415)
> at org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.lambda$applyTransaction$1(OzoneManagerStateMachine.java:240)
> at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> *Given Permission*
> {code:java}
> $ ozone sh vol getacl /vol1/
> [ {
> "type" : "USER",
> "name" : "pakapoj_tul",
> "aclScope" : "ACCESS",
> "aclList" : [ "WRITE", "ALL" ]
> }, {
> "type" : "USER",
> "name" : "pakapoj_tul@DEV.TAP",
> "aclScope" : "ACCESS",
> "aclList" : [ "WRITE", "ALL" ]
> }, {
> "type" : "USER",
> "name" : "ozone-admin@DEV.TAP",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> }, {
> "type" : "GROUP",
> "name" : "ozone-admin",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> }, {
> "type" : "GROUP",
> "name" : "ozone-users",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> } ]
> $ ozone sh bucket getacl /vol1/bucket1/
> [ {
> "type" : "USER",
> "name" : "ozone-admin@DEV.TAP",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> }, {
> "type" : "GROUP",
> "name" : "ozone-admin",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> }, {
> "type" : "GROUP",
> "name" : "ozone-users",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> }, {
> "type" : "USER",
> "name" : "pakapoj_tul@DEV.TAP",
> "aclScope" : "ACCESS",
> "aclList" : [ "WRITE", "ALL" ]
> }, {
> "type" : "USER",
> "name" : "pakapoj_tul",
> "aclScope" : "ACCESS",
> "aclList" : [ "WRITE", "ALL" ]
> } ]
> $ ozone sh key getacl /vol1/bucket1/mykey/
> [ {
> "type" : "USER",
> "name" : "ozone-admin@DEV.TAP",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> }, {
> "type" : "GROUP",
> "name" : "ozone-admin",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> }, {
> "type" : "GROUP",
> "name" : "ozone-users",
> "aclScope" : "ACCESS",
> "aclList" : [ "ALL" ]
> }, {
> "type" : "USER",
> "name" : "pakapoj_tul@DEV.TAP",
> "aclScope" : "ACCESS",
> "aclList" : [ "WRITE", "ALL" ]
> }, {
> "type" : "USER",
> "name" : "pakapoj_tul",
> "aclScope" : "ACCESS",
> "aclList" : [ "WRITE", "ALL" ]
> } ]{code}
>
> The spark code was deployed in Kubernetes in spark cluster mode. Then, the error would happed on spark executors side when the do {{commitKey}} with {{auth:TOKEN}} , BTW the spark driver was using {{auth:KERBEROS}} .
>
> so I reproduce using ozone java client writing to ozone with {{OzoneClient}} using # Token by {{export HADOOP_TOKEN_FILE_LOCATION=credential/ozone.token}} before running the program
> # Keytab by running {{/usr/bin/kinit -kt credential/pakapoj_tul.keytab pakapoj_tul@DEV.TAP}} before running the program
> the code, output for #1 and #2 (DEBUG) is in attach
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org