You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ranger.apache.org by Guru Thejus Arveti <th...@gmail.com> on 2024/11/13 16:51:39 UTC
Review Request 75271: RANGER-4996: Made meta attributes fields in policy rest api response configurable for retrieval
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/75271/
-----------------------------------------------------------
Review request for ranger and Pradeep Agrawal.
Bugs: RANGER-4996
https://issues.apache.org/jira/browse/RANGER-4996
Repository: ranger
Description
-------
RANGER-4996: Made meta attributes fields in policy rest api response configurable for retrieval
Diffs
-----
security-admin/src/main/java/org/apache/ranger/common/RangerServicePoliciesCache.java c8d1e5d08
security-admin/src/main/resources/conf.dist/ranger-admin-default-site.xml 1af339010
Diff: https://reviews.apache.org/r/75271/diff/1/
Testing
-------
Tested the /service/public/v2/api/policy rest api from swagger and as well as from postman and verified that createTime, updateTime, createdBy, updatedBy fields are being retrieved in the response when the ranger.admin.api.include.policy.meta.attributes flag is set
Thanks,
Guru Thejus Arveti
Re: Review Request 75271: RANGER-4996: Made meta attributes fields in policy rest api response configurable for retrieval
Posted by Guru Thejus Arveti <th...@gmail.com>.
> On Nov. 14, 2024, 4:44 p.m., Madhan Neethiraj wrote:
> > security-admin/src/main/java/org/apache/ranger/common/RangerServicePoliciesCache.java
> > Lines 551 (patched)
> > <https://reviews.apache.org/r/75271/diff/1/?file=2294568#file2294568line551>
> >
> > Policies in RangerServicePoliciesCache are provided to plugins. Pruning of unused attributes was done to reduce the payload, especially where there are a large number of policies. I would suggest to not update the cache for the reported issue.
> >
> > If a list of APIs that require pruned attributes, consider updating these APIs to directly read from the database instead of from cache.
Got it. Made the changes accordingly. Please review the new iteration. Also, made pruning unused policy attributes while storing policies in cache at all times, previously pruning was missing during deltas/policy updates. And as well, currently retrieving and setting policy meta time data from db just while returning policies if the flag is set for the api.
- Guru Thejus
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/75271/#review227055
-----------------------------------------------------------
On Nov. 19, 2024, 7:01 a.m., Guru Thejus Arveti wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/75271/
> -----------------------------------------------------------
>
> (Updated Nov. 19, 2024, 7:01 a.m.)
>
>
> Review request for ranger, Madhan Neethiraj and Pradeep Agrawal.
>
>
> Bugs: RANGER-4996
> https://issues.apache.org/jira/browse/RANGER-4996
>
>
> Repository: ranger
>
>
> Description
> -------
>
> RANGER-4996: Made meta attributes fields in policy rest api response configurable for retrieval
>
>
> Diffs
> -----
>
> agents-common/src/main/java/org/apache/ranger/plugin/store/ServiceStore.java 97072391e
> security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java 0dee87734
> security-admin/src/main/java/org/apache/ranger/common/RangerServicePoliciesCache.java c8d1e5d08
> security-admin/src/main/java/org/apache/ranger/rest/PublicAPIsv2.java b93b63f85
> security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java ce3a95766
> security-admin/src/main/java/org/apache/ranger/service/RangerBaseModelService.java 26f76578e
> security-admin/src/main/resources/conf.dist/ranger-admin-default-site.xml 1af339010
>
>
> Diff: https://reviews.apache.org/r/75271/diff/2/
>
>
> Testing
> -------
>
> Tested the /service/public/v2/api/policy rest api from swagger and as well as from postman and verified that createTime, updateTime, createdBy, updatedBy fields are being retrieved in the response when the ranger.admin.api.include.policy.meta.attributes flag is set
>
>
> Thanks,
>
> Guru Thejus Arveti
>
>
Re: Review Request 75271: RANGER-4996: Made meta attributes fields in policy rest api response configurable for retrieval
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/75271/#review227055
-----------------------------------------------------------
security-admin/src/main/java/org/apache/ranger/common/RangerServicePoliciesCache.java
Lines 551 (patched)
<https://reviews.apache.org/r/75271/#comment315346>
Policies in RangerServicePoliciesCache are provided to plugins. Pruning of unused attributes was done to reduce the payload, especially where there are a large number of policies. I would suggest to not update the cache for the reported issue.
If a list of APIs that require pruned attributes, consider updating these APIs to directly read from the database instead of from cache.
- Madhan Neethiraj
On Nov. 13, 2024, 4:51 p.m., Guru Thejus Arveti wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/75271/
> -----------------------------------------------------------
>
> (Updated Nov. 13, 2024, 4:51 p.m.)
>
>
> Review request for ranger and Pradeep Agrawal.
>
>
> Bugs: RANGER-4996
> https://issues.apache.org/jira/browse/RANGER-4996
>
>
> Repository: ranger
>
>
> Description
> -------
>
> RANGER-4996: Made meta attributes fields in policy rest api response configurable for retrieval
>
>
> Diffs
> -----
>
> security-admin/src/main/java/org/apache/ranger/common/RangerServicePoliciesCache.java c8d1e5d08
> security-admin/src/main/resources/conf.dist/ranger-admin-default-site.xml 1af339010
>
>
> Diff: https://reviews.apache.org/r/75271/diff/1/
>
>
> Testing
> -------
>
> Tested the /service/public/v2/api/policy rest api from swagger and as well as from postman and verified that createTime, updateTime, createdBy, updatedBy fields are being retrieved in the response when the ranger.admin.api.include.policy.meta.attributes flag is set
>
>
> Thanks,
>
> Guru Thejus Arveti
>
>
Re: Review Request 75271: RANGER-4996: Made meta attributes fields in policy rest api response configurable for retrieval
Posted by Pradeep Agrawal <pr...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/75271/#review227054
-----------------------------------------------------------
Ship it!
Ship It!
- Pradeep Agrawal
On Nov. 13, 2024, 4:51 p.m., Guru Thejus Arveti wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/75271/
> -----------------------------------------------------------
>
> (Updated Nov. 13, 2024, 4:51 p.m.)
>
>
> Review request for ranger and Pradeep Agrawal.
>
>
> Bugs: RANGER-4996
> https://issues.apache.org/jira/browse/RANGER-4996
>
>
> Repository: ranger
>
>
> Description
> -------
>
> RANGER-4996: Made meta attributes fields in policy rest api response configurable for retrieval
>
>
> Diffs
> -----
>
> security-admin/src/main/java/org/apache/ranger/common/RangerServicePoliciesCache.java c8d1e5d08
> security-admin/src/main/resources/conf.dist/ranger-admin-default-site.xml 1af339010
>
>
> Diff: https://reviews.apache.org/r/75271/diff/1/
>
>
> Testing
> -------
>
> Tested the /service/public/v2/api/policy rest api from swagger and as well as from postman and verified that createTime, updateTime, createdBy, updatedBy fields are being retrieved in the response when the ranger.admin.api.include.policy.meta.attributes flag is set
>
>
> Thanks,
>
> Guru Thejus Arveti
>
>
Re: Review Request 75271: RANGER-4996: Made meta attributes fields in policy rest api response configurable for retrieval
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/75271/#review227078
-----------------------------------------------------------
security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java
Lines 404 (patched)
<https://reviews.apache.org/r/75271/#comment315363>
Instead of a configuration, consider having this as a query-param in ServiceREST.getPolicies() API (or PublicAPIsV2.getPolicies()). This will eliminate the overhead for existing users of the API (who don't need meta attributes).
boolean includeMetaAttributes = Boolean.parseBoolean(request.getParameter("includeMetaAttributes"));
- Madhan Neethiraj
On Nov. 19, 2024, 7:01 a.m., Guru Thejus Arveti wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/75271/
> -----------------------------------------------------------
>
> (Updated Nov. 19, 2024, 7:01 a.m.)
>
>
> Review request for ranger, Madhan Neethiraj and Pradeep Agrawal.
>
>
> Bugs: RANGER-4996
> https://issues.apache.org/jira/browse/RANGER-4996
>
>
> Repository: ranger
>
>
> Description
> -------
>
> RANGER-4996: Made meta attributes fields in policy rest api response configurable for retrieval
>
>
> Diffs
> -----
>
> agents-common/src/main/java/org/apache/ranger/plugin/store/ServiceStore.java 97072391e
> security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java 0dee87734
> security-admin/src/main/java/org/apache/ranger/common/RangerServicePoliciesCache.java c8d1e5d08
> security-admin/src/main/java/org/apache/ranger/rest/PublicAPIsv2.java b93b63f85
> security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java ce3a95766
> security-admin/src/main/java/org/apache/ranger/service/RangerBaseModelService.java 26f76578e
> security-admin/src/main/resources/conf.dist/ranger-admin-default-site.xml 1af339010
>
>
> Diff: https://reviews.apache.org/r/75271/diff/2/
>
>
> Testing
> -------
>
> Tested the /service/public/v2/api/policy rest api from swagger and as well as from postman and verified that createTime, updateTime, createdBy, updatedBy fields are being retrieved in the response when the ranger.admin.api.include.policy.meta.attributes flag is set
>
>
> Thanks,
>
> Guru Thejus Arveti
>
>
Re: Review Request 75271: RANGER-4996: Made meta attributes fields in policy rest api response configurable for retrieval
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/75271/#review227077
-----------------------------------------------------------
security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java
Lines 376 (patched)
<https://reviews.apache.org/r/75271/#comment315360>
includeUnusedPolicyAttributes => includePolicyMetaAttributes
security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java
Lines 2683 (patched)
<https://reviews.apache.org/r/75271/#comment315362>
fillPolicyMetaAttributes() => getPoliciesWithMetaAttributes()
security-admin/src/main/java/org/apache/ranger/service/RangerBaseModelService.java
Lines 343 (patched)
<https://reviews.apache.org/r/75271/#comment315361>
- Query text should be registered in jpa_named_queries.xml, with something like:
<named-query name="XXPolicy.getMetaAttributesForPolicies">
<query>SELECT obj.id, obj.createdBy, obj.createTime, obj.updatedBy, obj.updateTime FROM XXPolicy obj WHERE obj.id IN :policyIds</query>
</named-query>
- move RangerBaseModelService.getResourcesByIds(List<Long> ids) => XXPolicyDao.getMetaAttributesForPolicies(List<Long> policyIds)
- Madhan Neethiraj
On Nov. 19, 2024, 7:01 a.m., Guru Thejus Arveti wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/75271/
> -----------------------------------------------------------
>
> (Updated Nov. 19, 2024, 7:01 a.m.)
>
>
> Review request for ranger, Madhan Neethiraj and Pradeep Agrawal.
>
>
> Bugs: RANGER-4996
> https://issues.apache.org/jira/browse/RANGER-4996
>
>
> Repository: ranger
>
>
> Description
> -------
>
> RANGER-4996: Made meta attributes fields in policy rest api response configurable for retrieval
>
>
> Diffs
> -----
>
> agents-common/src/main/java/org/apache/ranger/plugin/store/ServiceStore.java 97072391e
> security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java 0dee87734
> security-admin/src/main/java/org/apache/ranger/common/RangerServicePoliciesCache.java c8d1e5d08
> security-admin/src/main/java/org/apache/ranger/rest/PublicAPIsv2.java b93b63f85
> security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java ce3a95766
> security-admin/src/main/java/org/apache/ranger/service/RangerBaseModelService.java 26f76578e
> security-admin/src/main/resources/conf.dist/ranger-admin-default-site.xml 1af339010
>
>
> Diff: https://reviews.apache.org/r/75271/diff/2/
>
>
> Testing
> -------
>
> Tested the /service/public/v2/api/policy rest api from swagger and as well as from postman and verified that createTime, updateTime, createdBy, updatedBy fields are being retrieved in the response when the ranger.admin.api.include.policy.meta.attributes flag is set
>
>
> Thanks,
>
> Guru Thejus Arveti
>
>
Re: Review Request 75271: RANGER-4996: Made meta attributes fields in policy rest api response configurable for retrieval
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/75271/#review227111
-----------------------------------------------------------
Ship it!
Ship It!
- Madhan Neethiraj
On Dec. 5, 2024, 4:44 a.m., Guru Thejus Arveti wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/75271/
> -----------------------------------------------------------
>
> (Updated Dec. 5, 2024, 4:44 a.m.)
>
>
> Review request for ranger, Madhan Neethiraj and Pradeep Agrawal.
>
>
> Bugs: RANGER-4996
> https://issues.apache.org/jira/browse/RANGER-4996
>
>
> Repository: ranger
>
>
> Description
> -------
>
> RANGER-4996: Made meta attributes fields in policy rest api response configurable for retrieval
>
>
> Diffs
> -----
>
> agents-common/src/main/java/org/apache/ranger/plugin/store/ServiceStore.java 97072391e
> security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java 0dee87734
> security-admin/src/main/java/org/apache/ranger/common/RangerServicePoliciesCache.java c8d1e5d08
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyDao.java c65b961fc
> security-admin/src/main/java/org/apache/ranger/rest/PublicAPIsv2.java b93b63f85
> security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java ce3a95766
> security-admin/src/main/resources/META-INF/jpa_named_queries.xml 1a2acd188
>
>
> Diff: https://reviews.apache.org/r/75271/diff/4/
>
>
> Testing
> -------
>
> Tested the /service/public/v2/api/policy rest api from swagger and as well as from postman and verified that createTime, updateTime, createdBy, updatedBy fields are being retrieved in the response when the ranger.admin.api.include.policy.meta.attributes flag is set
>
>
> Thanks,
>
> Guru Thejus Arveti
>
>
Re: Review Request 75271: RANGER-4996: Made meta attributes fields in policy rest api response configurable for retrieval
Posted by Guru Thejus Arveti <th...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/75271/
-----------------------------------------------------------
(Updated Dec. 5, 2024, 4:44 a.m.)
Review request for ranger, Madhan Neethiraj and Pradeep Agrawal.
Bugs: RANGER-4996
https://issues.apache.org/jira/browse/RANGER-4996
Repository: ranger
Description
-------
RANGER-4996: Made meta attributes fields in policy rest api response configurable for retrieval
Diffs (updated)
-----
agents-common/src/main/java/org/apache/ranger/plugin/store/ServiceStore.java 97072391e
security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java 0dee87734
security-admin/src/main/java/org/apache/ranger/common/RangerServicePoliciesCache.java c8d1e5d08
security-admin/src/main/java/org/apache/ranger/db/XXPolicyDao.java c65b961fc
security-admin/src/main/java/org/apache/ranger/rest/PublicAPIsv2.java b93b63f85
security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java ce3a95766
security-admin/src/main/resources/META-INF/jpa_named_queries.xml 1a2acd188
Diff: https://reviews.apache.org/r/75271/diff/4/
Changes: https://reviews.apache.org/r/75271/diff/3-4/
Testing
-------
Tested the /service/public/v2/api/policy rest api from swagger and as well as from postman and verified that createTime, updateTime, createdBy, updatedBy fields are being retrieved in the response when the ranger.admin.api.include.policy.meta.attributes flag is set
Thanks,
Guru Thejus Arveti
Re: Review Request 75271: RANGER-4996: Made meta attributes fields in policy rest api response configurable for retrieval
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/75271/#review227110
-----------------------------------------------------------
security-admin/src/main/java/org/apache/ranger/rest/PublicAPIsv2.java
Lines 553 (patched)
<https://reviews.apache.org/r/75271/#comment315393>
Consider replacing #553 with the following, for better readability. Also, rename getPoliciesWithMetaAttributesIfNeeded() => getPoliciesWithMetaAttributes().
if (includeMetaAttributes) {
ret = seviceREST.getPoliciesWithMetaAttributes(ret);
}
...
return ret;
- Madhan Neethiraj
On Nov. 26, 2024, 7:17 a.m., Guru Thejus Arveti wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/75271/
> -----------------------------------------------------------
>
> (Updated Nov. 26, 2024, 7:17 a.m.)
>
>
> Review request for ranger, Madhan Neethiraj and Pradeep Agrawal.
>
>
> Bugs: RANGER-4996
> https://issues.apache.org/jira/browse/RANGER-4996
>
>
> Repository: ranger
>
>
> Description
> -------
>
> RANGER-4996: Made meta attributes fields in policy rest api response configurable for retrieval
>
>
> Diffs
> -----
>
> agents-common/src/main/java/org/apache/ranger/plugin/store/ServiceStore.java 97072391e
> security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java 0dee87734
> security-admin/src/main/java/org/apache/ranger/common/RangerServicePoliciesCache.java c8d1e5d08
> security-admin/src/main/java/org/apache/ranger/db/XXPolicyDao.java c65b961fc
> security-admin/src/main/java/org/apache/ranger/rest/PublicAPIsv2.java b93b63f85
> security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java ce3a95766
> security-admin/src/main/resources/META-INF/jpa_named_queries.xml 1a2acd188
>
>
> Diff: https://reviews.apache.org/r/75271/diff/3/
>
>
> Testing
> -------
>
> Tested the /service/public/v2/api/policy rest api from swagger and as well as from postman and verified that createTime, updateTime, createdBy, updatedBy fields are being retrieved in the response when the ranger.admin.api.include.policy.meta.attributes flag is set
>
>
> Thanks,
>
> Guru Thejus Arveti
>
>
Re: Review Request 75271: RANGER-4996: Made meta attributes fields in policy rest api response configurable for retrieval
Posted by Guru Thejus Arveti <th...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/75271/
-----------------------------------------------------------
(Updated Nov. 26, 2024, 7:17 a.m.)
Review request for ranger, Madhan Neethiraj and Pradeep Agrawal.
Bugs: RANGER-4996
https://issues.apache.org/jira/browse/RANGER-4996
Repository: ranger
Description
-------
RANGER-4996: Made meta attributes fields in policy rest api response configurable for retrieval
Diffs (updated)
-----
agents-common/src/main/java/org/apache/ranger/plugin/store/ServiceStore.java 97072391e
security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java 0dee87734
security-admin/src/main/java/org/apache/ranger/common/RangerServicePoliciesCache.java c8d1e5d08
security-admin/src/main/java/org/apache/ranger/db/XXPolicyDao.java c65b961fc
security-admin/src/main/java/org/apache/ranger/rest/PublicAPIsv2.java b93b63f85
security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java ce3a95766
security-admin/src/main/resources/META-INF/jpa_named_queries.xml 1a2acd188
Diff: https://reviews.apache.org/r/75271/diff/3/
Changes: https://reviews.apache.org/r/75271/diff/2-3/
Testing
-------
Tested the /service/public/v2/api/policy rest api from swagger and as well as from postman and verified that createTime, updateTime, createdBy, updatedBy fields are being retrieved in the response when the ranger.admin.api.include.policy.meta.attributes flag is set
Thanks,
Guru Thejus Arveti
Re: Review Request 75271: RANGER-4996: Made meta attributes fields in policy rest api response configurable for retrieval
Posted by Guru Thejus Arveti <th...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/75271/
-----------------------------------------------------------
(Updated Nov. 19, 2024, 7:01 a.m.)
Review request for ranger, Madhan Neethiraj and Pradeep Agrawal.
Changes
-------
Made pruning unused policy attributes while storing policies in cache at all times, previously pruning was missing during deltas/policy updates. And as well, currently retrieving and setting policy meta time data from db while returning policies if the flag is set
Bugs: RANGER-4996
https://issues.apache.org/jira/browse/RANGER-4996
Repository: ranger
Description
-------
RANGER-4996: Made meta attributes fields in policy rest api response configurable for retrieval
Diffs (updated)
-----
agents-common/src/main/java/org/apache/ranger/plugin/store/ServiceStore.java 97072391e
security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java 0dee87734
security-admin/src/main/java/org/apache/ranger/common/RangerServicePoliciesCache.java c8d1e5d08
security-admin/src/main/java/org/apache/ranger/rest/PublicAPIsv2.java b93b63f85
security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java ce3a95766
security-admin/src/main/java/org/apache/ranger/service/RangerBaseModelService.java 26f76578e
security-admin/src/main/resources/conf.dist/ranger-admin-default-site.xml 1af339010
Diff: https://reviews.apache.org/r/75271/diff/2/
Changes: https://reviews.apache.org/r/75271/diff/1-2/
Testing
-------
Tested the /service/public/v2/api/policy rest api from swagger and as well as from postman and verified that createTime, updateTime, createdBy, updatedBy fields are being retrieved in the response when the ranger.admin.api.include.policy.meta.attributes flag is set
Thanks,
Guru Thejus Arveti