You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "lschampion (via GitHub)" <gi...@apache.org> on 2025/11/24 10:01:03 UTC
[I] [Bug] execute query very slow when ranger authorization configured in hive catalog? [doris]
lschampion opened a new issue, #58304:
URL: https://github.com/apache/doris/issues/58304
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.
### Version
3.0.10
### What's Wrong?
when i use an user with all table authorization execute query "show tables", it takes 0.05 second.
but when i user an user the minior authorization (for example: with priviliege on the database), it takes 14 seconds.
and it is not only for the query "show table", when other select query the situation occurs too
there about are 700+ table in the database of hive catalog.
i konw that the ranger cache work well, and i check the cache in local directory.
### What You Expected?
speed up the ranger check engine
### How to Reproduce?
_No response_
### Anything Else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] execute query very slow when ranger authorization configured in hive catalog? [doris]
Posted by "lschampion (via GitHub)" <gi...@apache.org>.
lschampion commented on issue #58304:
URL: https://github.com/apache/doris/issues/58304#issuecomment-3631047842
i got the privilege check duration of every table is about 20-40 ms, so that realy takes time. and i add a guava cache to class of AccessControllerManager. the cache would record the result for 5min, so after query, privilege of query faster, example: fist time execute "show tables" takes 17 seoncds, the second would take 0.01 seconds. but the privileges would update as a delay job, which might equal to the cache invalidation time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] execute query very slow when ranger authorization configured in hive catalog? [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #58304: [Bug] execute query very slow when ranger authorization configured in hive catalog?
URL: https://github.com/apache/doris/issues/58304
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] execute query very slow when ranger authorization configured in hive catalog? [doris]
Posted by "lschampion (via GitHub)" <gi...@apache.org>.
lschampion commented on issue #58304:
URL: https://github.com/apache/doris/issues/58304#issuecomment-3625948645
@morningman
this only small part of the log
i print the elapse when execute "show tables;" command. RangerDorisAccessController run the examination, and there are 719 tables in the databases and there are 2880 log records like below. that real takes time. i sum all the elapse together, it is 17 seonds, which matches the command site shows (719 rows in set (17.01 sec)).
example： 2025-12-08 13:22:13,091 INFO (mysql-nio-pool-1|128) [RangerDorisAccessController.checkPrivilege():149] checkPrivileges [priv predicate: OR, Admin_priv,Select_priv,Load_priv,Alter_priv,Create_priv,Drop_priv,Show_view_priv] for user [myuser] takes [25] mills about resource map [{catalog=hive}]
I found that the privilege check the global privilege for everytable, it is real unnecessay!
maybe we should set a cache when checking privileges,then may the checks readuce to less than 10.
2025-12-08 13:22:12,970 INFO (mysql-nio-pool-1|128) [RangerDorisAccessController.checkPrivilege():149] checkPrivileges [priv predicate: OR, Admin_priv,Select_priv,Load_priv,Alter_priv,Create_priv,Drop_priv,Show_view_priv] for user [myuser] takes [1] mills about resource map [{database=managerment, catalog=hive}]
2025-12-08 13:22:12,970 DEBUG (mysql-nio-pool-1|128) [RangerDorisAccessController.checkPrivilegeByPlugin():109] ranger request: RangerAccessRequestImpl={resource={RangerResourceImpl={ownerUser={null} elements={global=*; } }} accessType={ADMIN} user={myuser} userGroups={} userRoles={} accessTime={Mon Dec 08 13:22:12 CST 2025} clientIPAddress={10.165.24.104} forwardedAddresses={} remoteIPAddress={null} clientType={doris} action={ADMIN} requestData={null} sessionId={null} resourceMatchingScope={SELF} clusterName={null} clusterType={doris} context={} }
2025-12-08 13:22:12,970 DEBUG (mysql-nio-pool-1|128) [RangerDorisAccessController.checkPrivilegeByPlugin():109] ranger request: RangerAccessRequestImpl={resource={RangerResourceImpl={ownerUser={null} elements={global=*; } }} accessType={SELECT} user={myuser} userGroups={} userRoles={} accessTime={Mon Dec 08 13:22:12 CST 2025} clientIPAddress={10.165.24.104} forwardedAddresses={} remoteIPAddress={null} clientType={doris} action={SELECT} requestData={null} sessionId={null} resourceMatchingScope={SELF} clusterName={null} clusterType={doris} context={} }
2025-12-08 13:22:12,970 DEBUG (mysql-nio-pool-1|128) [RangerDorisAccessController.checkPrivilegeByPlugin():109] ranger request: RangerAccessRequestImpl={resource={RangerResourceImpl={ownerUser={null} elements={global=*; } }} accessType={LOAD} user={myuser} userGroups={} userRoles={} accessTime={Mon Dec 08 13:22:12 CST 2025} clientIPAddress={10.165.24.104} forwardedAddresses={} remoteIPAddress={null} clientType={doris} action={LOAD} requestData={null} sessionId={null} resourceMatchingScope={SELF} clusterName={null} clusterType={doris} context={} }
2025-12-08 13:22:12,971 DEBUG (mysql-nio-pool-1|128) [RangerDorisAccessController.checkPrivilegeByPlugin():109] ranger request: RangerAccessRequestImpl={resource={RangerResourceImpl={ownerUser={null} elements={global=*; } }} accessType={ALTER} user={myuser} userGroups={} userRoles={} accessTime={Mon Dec 08 13:22:12 CST 2025} clientIPAddress={10.165.24.104} forwardedAddresses={} remoteIPAddress={null} clientType={doris} action={ALTER} requestData={null} sessionId={null} resourceMatchingScope={SELF} clusterName={null} clusterType={doris} context={} }
2025-12-08 13:22:12,971 DEBUG (mysql-nio-pool-1|128) [RangerDorisAccessController.checkPrivilegeByPlugin():109] ranger request: RangerAccessRequestImpl={resource={RangerResourceImpl={ownerUser={null} elements={global=*; } }} accessType={CREATE} user={myuser} userGroups={} userRoles={} accessTime={Mon Dec 08 13:22:12 CST 2025} clientIPAddress={10.165.24.104} forwardedAddresses={} remoteIPAddress={null} clientType={doris} action={CREATE} requestData={null} sessionId={null} resourceMatchingScope={SELF} clusterName={null} clusterType={doris} context={} }
2025-12-08 13:22:12,971 DEBUG (mysql-nio-pool-1|128) [RangerDorisAccessController.checkPrivilegeByPlugin():109] ranger request: RangerAccessRequestImpl={resource={RangerResourceImpl={ownerUser={null} elements={global=*; } }} accessType={DROP} user={myuser} userGroups={} userRoles={} accessTime={Mon Dec 08 13:22:12 CST 2025} clientIPAddress={10.165.24.104} forwardedAddresses={} remoteIPAddress={null} clientType={doris} action={DROP} requestData={null} sessionId={null} resourceMatchingScope={SELF} clusterName={null} clusterType={doris} context={} }
2025-12-08 13:22:12,971 DEBUG (mysql-nio-pool-1|128) [RangerDorisAccessController.checkPrivilegeByPlugin():109] ranger request: RangerAccessRequestImpl={resource={RangerResourceImpl={ownerUser={null} elements={global=*; } }} accessType={SHOW_VIEW} user={myuser} userGroups={} userRoles={} accessTime={Mon Dec 08 13:22:12 CST 2025} clientIPAddress={10.165.24.104} forwardedAddresses={} remoteIPAddress={null} clientType={doris} action={SHOW_VIEW} requestData={null} sessionId={null} resourceMatchingScope={SELF} clusterName={null} clusterType={doris} context={} }
2025-12-08 13:22:12,971 INFO (mysql-nio-pool-1|128) [RangerDorisAccessController.checkPrivilege():149] checkPrivileges [priv predicate: OR, Admin_priv,Select_priv,Load_priv,Alter_priv,Create_priv,Drop_priv,Show_view_priv] for user [myuser] takes [1] mills about resource map [{global=*}]
2025-12-08 13:22:12,971 INFO (mysql-nio-pool-1|128) [RangerDorisAccessController.checkTblPriv():219] check tbl [managerment.some_table] [priv predicate: OR, Admin_priv,Select_priv,Load_priv,Alter_priv,Create_priv,Drop_priv,Show_view_priv] of privilege for user ['myuser'@'10.165.24.104'] with ctl [hive]
2025-12-08 13:22:12,971 DEBUG (mysql-nio-pool-1|128) [RangerDorisAccessController.checkPrivilegeByPlugin():109] ranger request: RangerAccessRequestImpl={resource={RangerResourceImpl={ownerUser={null} elements={global=*; } }} accessType={ADMIN} user={myuser} userGroups={} userRoles={} accessTime={Mon Dec 08 13:22:12 CST 2025} clientIPAddress={10.165.24.104} forwardedAddresses={} remoteIPAddress={null} clientType={doris} action={ADMIN} requestData={null} sessionId={null} resourceMatchingScope={SELF} clusterName={null} clusterType={doris} context={} }
2025-12-08 13:22:12,971 DEBUG (mysql-nio-pool-1|128) [RangerDorisAccessController.checkPrivilegeByPlugin():109] ranger request: RangerAccessRequestImpl={resource={RangerResourceImpl={ownerUser={null} elements={global=*; } }} accessType={SELECT} user={myuser} userGroups={} userRoles={} accessTime={Mon Dec 08 13:22:12 CST 2025} clientIPAddress={10.165.24.104} forwardedAddresses={} remoteIPAddress={null} clientType={doris} action={SELECT} requestData={null} sessionId={null} resourceMatchingScope={SELF} clusterName={null} clusterType={doris} context={} }
2025-12-08 13:22:12,972 DEBUG (mysql-nio-pool-1|128) [RangerDorisAccessController.checkPrivilegeByPlugin():109] ranger request: RangerAccessRequestImpl={resource={RangerResourceImpl={ownerUser={null} elements={global=*; } }} accessType={LOAD} user={myuser} userGroups={} userRoles={} accessTime={Mon Dec 08 13:22:12 CST 2025} clientIPAddress={10.165.24.104} forwardedAddresses={} remoteIPAddress={null} clientType={doris} action={LOAD} requestData={null} sessionId={null} resourceMatchingScope={SELF} clusterName={null} clusterType={doris} context={} }
2025-12-08 13:22:12,972 DEBUG (mysql-nio-pool-1|128) [RangerDorisAccessController.checkPrivilegeByPlugin():109] ranger request: RangerAccessRequestImpl={resource={RangerResourceImpl={ownerUser={null} elements={global=*; } }} accessType={ALTER} user={myuser} userGroups={} userRoles={} accessTime={Mon Dec 08 13:22:12 CST 2025} clientIPAddress={10.165.24.104} forwardedAddresses={} remoteIPAddress={null} clientType={doris} action={ALTER} requestData={null} sessionId={null} resourceMatchingScope={SELF} clusterName={null} clusterType={doris} context={} }
2025-12-08 13:22:12,972 DEBUG (mysql-nio-pool-1|128) [RangerDorisAccessController.checkPrivilegeByPlugin():109] ranger request: RangerAccessRequestImpl={resource={RangerResourceImpl={ownerUser={null} elements={global=*; } }} accessType={CREATE} user={myuser} userGroups={} userRoles={} accessTime={Mon Dec 08 13:22:12 CST 2025} clientIPAddress={10.165.24.104} forwardedAddresses={} remoteIPAddress={null} clientType={doris} action={CREATE} requestData={null} sessionId={null} resourceMatchingScope={SELF} clusterName={null} clusterType={doris} context={} }
2025-12-08 13:22:12,972 DEBUG (mysql-nio-pool-1|128) [RangerDorisAccessController.checkPrivilegeByPlugin():109] ranger request: RangerAccessRequestImpl={resource={RangerResourceImpl={ownerUser={null} elements={global=*; } }} accessType={DROP} user={myuser} userGroups={} userRoles={} accessTime={Mon Dec 08 13:22:12 CST 2025} clientIPAddress={10.165.24.104} forwardedAddresses={} remoteIPAddress={null} clientType={doris} action={DROP} requestData={null} sessionId={null} resourceMatchingScope={SELF} clusterName={null} clusterType={doris} context={} }
2025-12-08 13:22:12,972 DEBUG (mysql-nio-pool-1|128) [RangerDorisAccessController.checkPrivilegeByPlugin():109] ranger request: RangerAccessRequestImpl={resource={RangerResourceImpl={ownerUser={null} elements={global=*; } }} accessType={SHOW_VIEW} user={myuser} userGroups={} userRoles={} accessTime={Mon Dec 08 13:22:12 CST 2025} clientIPAddress={10.165.24.104} forwardedAddresses={} remoteIPAddress={null} clientType={doris} action={SHOW_VIEW} requestData={null} sessionId={null} resourceMatchingScope={SELF} clusterName={null} clusterType={doris} context={} }
2025-12-08 13:22:12,972 INFO (mysql-nio-pool-1|128) [RangerDorisAccessController.checkPrivilege():149] checkPrivileges [priv predicate: OR, Admin_priv,Select_priv,Load_priv,Alter_priv,Create_priv,Drop_priv,Show_view_priv] for user [myuser] takes [1] mills about resource map [{global=*}]
2025-12-08 13:22:12,972 DEBUG (mysql-nio-pool-1|128) [RangerDorisAccessController.checkPrivilegeByPlugin():109] ranger request: RangerAccessRequestImpl={resource={RangerResourceImpl={ownerUser={null} elements={catalog=hive; } }} accessType={ADMIN} user={myuser} userGroups={} userRoles={} accessTime={Mon Dec 08 13:22:12 CST 2025} clientIPAddress={10.165.24.104} forwardedAddresses={} remoteIPAddress={null} clientType={doris} action={ADMIN} requestData={null} sessionId={null} resourceMatchingScope={SELF} clusterName={null} clusterType={doris} context={} }
2025-12-08 13:22:12,977 DEBUG (mysql-nio-pool-1|128) [RangerDorisAccessController.checkPrivilegeByPlugin():109] ranger request: RangerAccessRequestImpl={resource={RangerResourceImpl={ownerUser={null} elements={catalog=hive; } }} accessType={SELECT} user={myuser} userGroups={} userRoles={} accessTime={Mon Dec 08 13:22:12 CST 2025} clientIPAddress={10.165.24.104} forwardedAddresses={} remoteIPAddress={null} clientType={doris} action={SELECT} requestData={null} sessionId={null} resourceMatchingScope={SELF} clusterName={null} clusterType={doris} context={} }
2025-12-08 13:22:12,982 DEBUG (mysql-nio-pool-1|128) [RangerDorisAccessController.checkPrivilegeByPlugin():109] ranger request: RangerAccessRequestImpl={resource={RangerResourceImpl={ownerUser={null} elements={catalog=hive; } }} accessType={LOAD} user={myuser} userGroups={} userRoles={} accessTime={Mon Dec 08 13:22:12 CST 2025} clientIPAddress={10.165.24.104} forwardedAddresses={} remoteIPAddress={null} clientType={doris} action={LOAD} requestData={null} sessionId={null} resourceMatchingScope={SELF} clusterName={null} clusterType={doris} context={} }
2025-12-08 13:22:12,987 DEBUG (mysql-nio-pool-1|128) [RangerDorisAccessController.checkPrivilegeByPlugin():109] ranger request: RangerAccessRequestImpl={resource={RangerResourceImpl={ownerUser={null} elements={catalog=hive; } }} accessType={ALTER} user={myuser} userGroups={} userRoles={} accessTime={Mon Dec 08 13:22:12 CST 2025} clientIPAddress={10.165.24.104} forwardedAddresses={} remoteIPAddress={null} clientType={doris} action={ALTER} requestData={null} sessionId={null} resourceMatchingScope={SELF} clusterName={null} clusterType={doris} context={} }
2025-12-08 13:22:12,991 DEBUG (mysql-nio-pool-1|128) [RangerDorisAccessController.checkPrivilegeByPlugin():109] ranger request: RangerAccessRequestImpl={resource={RangerResourceImpl={ownerUser={null} elements={catalog=hive; } }} accessType={CREATE} user={myuser} userGroups={} userRoles={} accessTime={Mon Dec 08 13:22:12 CST 2025} clientIPAddress={10.165.24.104} forwardedAddresses={} remoteIPAddress={null} clientType={doris} action={CREATE} requestData={null} sessionId={null} resourceMatchingScope={SELF} clusterName={null} clusterType={doris} context={} }
2025-12-08 13:22:12,996 DEBUG (mysql-nio-pool-1|128) [RangerDorisAccessController.checkPrivilegeByPlugin():109] ranger request: RangerAccessRequestImpl={resource={RangerResourceImpl={ownerUser={null} elements={catalog=hive; } }} accessType={DROP} user={myuser} userGroups={} userRoles={} accessTime={Mon Dec 08 13:22:12 CST 2025} clientIPAddress={10.165.24.104} forwardedAddresses={} remoteIPAddress={null} clientType={doris} action={DROP} requestData={null} sessionId={null} resourceMatchingScope={SELF} clusterName={null} clusterType={doris} context={} }
2025-12-08 13:22:13,000 DEBUG (mysql-nio-pool-1|128) [RangerDorisAccessController.checkPrivilegeByPlugin():109] ranger request: RangerAccessRequestImpl={resource={RangerResourceImpl={ownerUser={null} elements={catalog=hive; } }} accessType={SHOW_VIEW} user={myuser} userGroups={} userRoles={} accessTime={Mon Dec 08 13:22:13 CST 2025} clientIPAddress={10.165.24.104} forwardedAddresses={} remoteIPAddress={null} clientType={doris} action={SHOW_VIEW} requestData={null} sessionId={null} resourceMatchingScope={SELF} clusterName={null} clusterType={doris} context={} }
2025-12-08 13:22:13,005 INFO (mysql-nio-pool-1|128) [RangerDorisAccessController.checkPrivilege():149] checkPrivileges [priv predicate: OR, Admin_priv,Select_priv,Load_priv,Alter_priv,Create_priv,Drop_priv,Show_view_priv] for user [myuser] takes [32] mills about resource map [{catalog=hive}]
2025-12-08 13:22:13,005 DEBUG (mysql-nio-pool-1|128) [RangerDorisAccessController.checkPrivilegeByPlugin():109] ranger request: RangerAccessRequestImpl={resource={RangerResourceImpl={ownerUser={null} elements={database=managerment; catalog=hive; } }} accessType={ADMIN} user={myuser} userGroups={} userRoles={} accessTime={Mon Dec 08 13:22:13 CST 2025} clientIPAddress={10.165.24.104} forwardedAddresses={} remoteIPAddress={null} clientType={doris} action={ADMIN} requestData={null} sessionId={null} resourceMatchingScope={SELF} clusterName={null} clusterType={doris} context={} }
2025-12-08 13:22:13,005 DEBUG (mysql-nio-pool-1|128) [RangerDorisAccessController.checkPrivilegeByPlugin():109] ranger request: RangerAccessRequestImpl={resource={RangerResourceImpl={ownerUser={null} elements={database=managerment; catalog=hive; } }} accessType={SELECT} user={myuser} userGroups={} userRoles={} accessTime={Mon Dec 08 13:22:13 CST 2025} clientIPAddress={10.165.24.104} forwardedAddresses={} remoteIPAddress={null} clientType={doris} action={SELECT} requestData={null} sessionId={null} resourceMatchingScope={SELF} clusterName={null} clusterType={doris} context={} }
2025-12-08 13:22:13,006 INFO (mysql-nio-pool-1|128) [RangerDorisAccessController.checkPrivilege():149] checkPrivileges [priv predicate: OR, Admin_priv,Select_priv,Load_priv,Alter_priv,Create_priv,Drop_priv,Show_view_priv] for user [myuser] takes [0] mills about resource map [{database=managerment, catalog=hive}]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] execute query very slow when ranger authorization configured in hive catalog? [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on issue #58304:
URL: https://github.com/apache/doris/issues/58304#issuecomment-3588367626
You can provide a jstack result of FE when executing this slow command.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] execute query very slow when ranger authorization configured in hive catalog? [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #58304:
URL: https://github.com/apache/doris/issues/58304#issuecomment-4644624581
We're closing this issue because it hasn't been updated in a while.
If you'd like to revive this issue, please reopen it and feel free a maintainer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org