You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ranger.apache.org by Abhay Kulkarni <ak...@hortonworks.com> on 2019/05/20 22:19:59 UTC
Review Request 70682: RANGER-2420: Ranger spends 36% of CPU in
ObjectMapper
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70682/
-----------------------------------------------------------
Review request for ranger, Madhan Neethiraj, Pradeep Agrawal, Ramesh Mani, and Velmurugan Periasamy.
Bugs: RANGER-2420
https://issues.apache.org/jira/browse/RANGER-2420
Repository: ranger
Description
-------
Ranger uses ObjectMapper to convert to/from JSON. In a profile of a workload (an Impala authorization test) I see that 36% of the ranger CPU is spent in these functions. 26% of total CPU is in the '_findRootDeserializer' method, which gets cached if a type is deserialized multiple times. However, the caching is only effective if the ObjectMapper is reused. JSONUtil appears to create a new ObjectMapper on every call, which defeats the caching.
Patch ensures that there is only one instance of ObjectMapper created in a process.
Diffs
-----
agents-common/src/main/java/org/apache/ranger/admin/client/datatype/GrantRevokeData.java a8d3c6af5
agents-common/src/main/java/org/apache/ranger/admin/client/datatype/RESTResponse.java 333f56504
agents-common/src/main/java/org/apache/ranger/plugin/audit/RangerDefaultAuditHandler.java 5e2c49211
agents-common/src/main/java/org/apache/ranger/plugin/util/JsonUtilsV2.java PRE-CREATION
kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStoreProvider.java bc692937b
kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java d3e0064db
kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONWriter.java 3674e7a87
knox-agent/src/main/java/org/apache/ranger/services/knox/client/KnoxClient.java 0c83ef9bb
security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java 35dc9405b
security-admin/src/main/java/org/apache/ranger/common/JSONUtil.java 5bc1baf58
security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 0e7cd8f86
security-admin/src/main/java/org/apache/ranger/service/RangerDataHistService.java a96d648a9
security-admin/src/main/java/org/apache/ranger/service/RangerPolicyService.java 02763670e
security-admin/src/main/java/org/apache/ranger/service/XAssetService.java 5525584f0
security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java ac9af5eb4
security-admin/src/test/java/org/apache/ranger/service/TestRangerDataHistService.java e1a236e5f
Diff: https://reviews.apache.org/r/70682/diff/1/
Testing
-------
Ran all unit tests successfully.
Thanks,
Abhay Kulkarni
Re: Review Request 70682: RANGER-2420: Ranger spends 36% of CPU in
ObjectMapper
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70682/#review215392
-----------------------------------------------------------
Ship it!
Ship It!
- Madhan Neethiraj
On May 20, 2019, 11:41 p.m., Abhay Kulkarni wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70682/
> -----------------------------------------------------------
>
> (Updated May 20, 2019, 11:41 p.m.)
>
>
> Review request for ranger, Madhan Neethiraj, Pradeep Agrawal, Ramesh Mani, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-2420
> https://issues.apache.org/jira/browse/RANGER-2420
>
>
> Repository: ranger
>
>
> Description
> -------
>
> Ranger uses ObjectMapper to convert to/from JSON. In a profile of a workload (an Impala authorization test) I see that 36% of the ranger CPU is spent in these functions. 26% of total CPU is in the '_findRootDeserializer' method, which gets cached if a type is deserialized multiple times. However, the caching is only effective if the ObjectMapper is reused. JSONUtil appears to create a new ObjectMapper on every call, which defeats the caching.
>
> Patch ensures that there is only one instance of ObjectMapper created in a process.
>
>
> Diffs
> -----
>
> agents-common/src/main/java/org/apache/ranger/admin/client/datatype/GrantRevokeData.java a8d3c6af5
> agents-common/src/main/java/org/apache/ranger/admin/client/datatype/RESTResponse.java 333f56504
> agents-common/src/main/java/org/apache/ranger/plugin/audit/RangerDefaultAuditHandler.java 5e2c49211
> agents-common/src/main/java/org/apache/ranger/plugin/util/JsonUtilsV2.java PRE-CREATION
> kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStoreProvider.java bc692937b
> kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java d3e0064db
> kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONWriter.java 3674e7a87
> knox-agent/src/main/java/org/apache/ranger/services/knox/client/KnoxClient.java 0c83ef9bb
> security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java 35dc9405b
> security-admin/src/main/java/org/apache/ranger/common/JSONUtil.java 5bc1baf58
> security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 0e7cd8f86
> security-admin/src/main/java/org/apache/ranger/service/RangerDataHistService.java a96d648a9
> security-admin/src/main/java/org/apache/ranger/service/RangerPolicyService.java 02763670e
> security-admin/src/main/java/org/apache/ranger/service/XAssetService.java 5525584f0
> security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java ac9af5eb4
> security-admin/src/test/java/org/apache/ranger/service/TestRangerDataHistService.java e1a236e5f
>
>
> Diff: https://reviews.apache.org/r/70682/diff/2/
>
>
> Testing
> -------
>
> Ran all unit tests successfully.
>
>
> Thanks,
>
> Abhay Kulkarni
>
>
Re: Review Request 70682: RANGER-2420: Ranger spends 36% of CPU in
ObjectMapper
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70682/#review215552
-----------------------------------------------------------
Ship it!
Ship It!
- Madhan Neethiraj
On May 28, 2019, 7:07 p.m., Abhay Kulkarni wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70682/
> -----------------------------------------------------------
>
> (Updated May 28, 2019, 7:07 p.m.)
>
>
> Review request for ranger, Madhan Neethiraj, Pradeep Agrawal, Ramesh Mani, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-2420
> https://issues.apache.org/jira/browse/RANGER-2420
>
>
> Repository: ranger
>
>
> Description
> -------
>
> Ranger uses ObjectMapper to convert to/from JSON. In a profile of a workload (an Impala authorization test) I see that 36% of the ranger CPU is spent in these functions. 26% of total CPU is in the '_findRootDeserializer' method, which gets cached if a type is deserialized multiple times. However, the caching is only effective if the ObjectMapper is reused. JSONUtil appears to create a new ObjectMapper on every call, which defeats the caching.
>
> Patch ensures that there is only one instance of ObjectMapper created in a process.
>
>
> Diffs
> -----
>
> agents-common/src/main/java/org/apache/ranger/admin/client/datatype/GrantRevokeData.java a8d3c6af5
> agents-common/src/main/java/org/apache/ranger/admin/client/datatype/RESTResponse.java 333f56504
> agents-common/src/main/java/org/apache/ranger/plugin/audit/RangerDefaultAuditHandler.java 5e2c49211
> agents-common/src/main/java/org/apache/ranger/plugin/util/JsonUtilsV2.java PRE-CREATION
> kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStoreProvider.java bc692937b
> kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java d3e0064db
> kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONWriter.java 3674e7a87
> knox-agent/src/main/java/org/apache/ranger/services/knox/client/KnoxClient.java 0c83ef9bb
> security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java 35dc9405b
> security-admin/src/main/java/org/apache/ranger/common/JSONUtil.java 5bc1baf58
> security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java b5caea8ef
> security-admin/src/main/java/org/apache/ranger/service/RangerDataHistService.java a96d648a9
> security-admin/src/main/java/org/apache/ranger/service/RangerPolicyService.java 411744237
> security-admin/src/main/java/org/apache/ranger/service/XAssetService.java 5525584f0
> security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java ac9af5eb4
> security-admin/src/test/java/org/apache/ranger/service/TestRangerDataHistService.java e1a236e5f
>
>
> Diff: https://reviews.apache.org/r/70682/diff/4/
>
>
> Testing
> -------
>
> Ran all unit tests successfully.
>
>
> Thanks,
>
> Abhay Kulkarni
>
>
Re: Review Request 70682: RANGER-2420: Ranger spends 36% of CPU in
ObjectMapper
Posted by Abhay Kulkarni <ak...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70682/
-----------------------------------------------------------
(Updated May 28, 2019, 7:07 p.m.)
Review request for ranger, Madhan Neethiraj, Pradeep Agrawal, Ramesh Mani, and Velmurugan Periasamy.
Changes
-------
Instead of using process-wide unique ObjectMapper, create a new ObjectMapper for every thread.
Bugs: RANGER-2420
https://issues.apache.org/jira/browse/RANGER-2420
Repository: ranger
Description
-------
Ranger uses ObjectMapper to convert to/from JSON. In a profile of a workload (an Impala authorization test) I see that 36% of the ranger CPU is spent in these functions. 26% of total CPU is in the '_findRootDeserializer' method, which gets cached if a type is deserialized multiple times. However, the caching is only effective if the ObjectMapper is reused. JSONUtil appears to create a new ObjectMapper on every call, which defeats the caching.
Patch ensures that there is only one instance of ObjectMapper created in a process.
Diffs (updated)
-----
agents-common/src/main/java/org/apache/ranger/admin/client/datatype/GrantRevokeData.java a8d3c6af5
agents-common/src/main/java/org/apache/ranger/admin/client/datatype/RESTResponse.java 333f56504
agents-common/src/main/java/org/apache/ranger/plugin/audit/RangerDefaultAuditHandler.java 5e2c49211
agents-common/src/main/java/org/apache/ranger/plugin/util/JsonUtilsV2.java PRE-CREATION
kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStoreProvider.java bc692937b
kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java d3e0064db
kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONWriter.java 3674e7a87
knox-agent/src/main/java/org/apache/ranger/services/knox/client/KnoxClient.java 0c83ef9bb
security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java 35dc9405b
security-admin/src/main/java/org/apache/ranger/common/JSONUtil.java 5bc1baf58
security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java b5caea8ef
security-admin/src/main/java/org/apache/ranger/service/RangerDataHistService.java a96d648a9
security-admin/src/main/java/org/apache/ranger/service/RangerPolicyService.java 411744237
security-admin/src/main/java/org/apache/ranger/service/XAssetService.java 5525584f0
security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java ac9af5eb4
security-admin/src/test/java/org/apache/ranger/service/TestRangerDataHistService.java e1a236e5f
Diff: https://reviews.apache.org/r/70682/diff/4/
Changes: https://reviews.apache.org/r/70682/diff/3-4/
Testing
-------
Ran all unit tests successfully.
Thanks,
Abhay Kulkarni
Re: Review Request 70682: RANGER-2420: Ranger spends 36% of CPU in
ObjectMapper
Posted by Abhay Kulkarni <ak...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70682/
-----------------------------------------------------------
(Updated May 21, 2019, 12:41 a.m.)
Review request for ranger, Madhan Neethiraj, Pradeep Agrawal, Ramesh Mani, and Velmurugan Periasamy.
Changes
-------
Addressed review comments
Bugs: RANGER-2420
https://issues.apache.org/jira/browse/RANGER-2420
Repository: ranger
Description
-------
Ranger uses ObjectMapper to convert to/from JSON. In a profile of a workload (an Impala authorization test) I see that 36% of the ranger CPU is spent in these functions. 26% of total CPU is in the '_findRootDeserializer' method, which gets cached if a type is deserialized multiple times. However, the caching is only effective if the ObjectMapper is reused. JSONUtil appears to create a new ObjectMapper on every call, which defeats the caching.
Patch ensures that there is only one instance of ObjectMapper created in a process.
Diffs (updated)
-----
agents-common/src/main/java/org/apache/ranger/admin/client/datatype/GrantRevokeData.java a8d3c6af5
agents-common/src/main/java/org/apache/ranger/admin/client/datatype/RESTResponse.java 333f56504
agents-common/src/main/java/org/apache/ranger/plugin/audit/RangerDefaultAuditHandler.java 5e2c49211
agents-common/src/main/java/org/apache/ranger/plugin/util/JsonUtilsV2.java PRE-CREATION
kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStoreProvider.java bc692937b
kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java d3e0064db
kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONWriter.java 3674e7a87
knox-agent/src/main/java/org/apache/ranger/services/knox/client/KnoxClient.java 0c83ef9bb
security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java 35dc9405b
security-admin/src/main/java/org/apache/ranger/common/JSONUtil.java 5bc1baf58
security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 0e7cd8f86
security-admin/src/main/java/org/apache/ranger/service/RangerDataHistService.java a96d648a9
security-admin/src/main/java/org/apache/ranger/service/RangerPolicyService.java 02763670e
security-admin/src/main/java/org/apache/ranger/service/XAssetService.java 5525584f0
security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java ac9af5eb4
security-admin/src/test/java/org/apache/ranger/service/TestRangerDataHistService.java e1a236e5f
Diff: https://reviews.apache.org/r/70682/diff/3/
Changes: https://reviews.apache.org/r/70682/diff/2-3/
Testing
-------
Ran all unit tests successfully.
Thanks,
Abhay Kulkarni
Re: Review Request 70682: RANGER-2420: Ranger spends 36% of CPU in
ObjectMapper
Posted by Abhay Kulkarni <ak...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70682/
-----------------------------------------------------------
(Updated May 20, 2019, 11:41 p.m.)
Review request for ranger, Madhan Neethiraj, Pradeep Agrawal, Ramesh Mani, and Velmurugan Periasamy.
Changes
-------
Addressed review comments
Bugs: RANGER-2420
https://issues.apache.org/jira/browse/RANGER-2420
Repository: ranger
Description
-------
Ranger uses ObjectMapper to convert to/from JSON. In a profile of a workload (an Impala authorization test) I see that 36% of the ranger CPU is spent in these functions. 26% of total CPU is in the '_findRootDeserializer' method, which gets cached if a type is deserialized multiple times. However, the caching is only effective if the ObjectMapper is reused. JSONUtil appears to create a new ObjectMapper on every call, which defeats the caching.
Patch ensures that there is only one instance of ObjectMapper created in a process.
Diffs (updated)
-----
agents-common/src/main/java/org/apache/ranger/admin/client/datatype/GrantRevokeData.java a8d3c6af5
agents-common/src/main/java/org/apache/ranger/admin/client/datatype/RESTResponse.java 333f56504
agents-common/src/main/java/org/apache/ranger/plugin/audit/RangerDefaultAuditHandler.java 5e2c49211
agents-common/src/main/java/org/apache/ranger/plugin/util/JsonUtilsV2.java PRE-CREATION
kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStoreProvider.java bc692937b
kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java d3e0064db
kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONWriter.java 3674e7a87
knox-agent/src/main/java/org/apache/ranger/services/knox/client/KnoxClient.java 0c83ef9bb
security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java 35dc9405b
security-admin/src/main/java/org/apache/ranger/common/JSONUtil.java 5bc1baf58
security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 0e7cd8f86
security-admin/src/main/java/org/apache/ranger/service/RangerDataHistService.java a96d648a9
security-admin/src/main/java/org/apache/ranger/service/RangerPolicyService.java 02763670e
security-admin/src/main/java/org/apache/ranger/service/XAssetService.java 5525584f0
security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java ac9af5eb4
security-admin/src/test/java/org/apache/ranger/service/TestRangerDataHistService.java e1a236e5f
Diff: https://reviews.apache.org/r/70682/diff/2/
Changes: https://reviews.apache.org/r/70682/diff/1-2/
Testing
-------
Ran all unit tests successfully.
Thanks,
Abhay Kulkarni
Re: Review Request 70682: RANGER-2420: Ranger spends 36% of CPU in
ObjectMapper
Posted by Abhay Kulkarni <ak...@hortonworks.com>.
> On May 20, 2019, 11:18 p.m., Madhan Neethiraj wrote:
> > kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStoreProvider.java
> > Line 220 (original), 220 (patched)
> > <https://reviews.apache.org/r/70682/diff/1/?file=2145939#file2145939line220>
> >
> > Consider avoiding call to getMapper() by replacing with:
> > JsonUtilsV2.readMapToString(attributes)
> >
> > Please review other such usage as well.
Map interface (attributes is a Map) is not serializable, so cannot use it as an argument to JsonUtilsV2.readMapToString().
- Abhay
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70682/#review215388
-----------------------------------------------------------
On May 20, 2019, 10:19 p.m., Abhay Kulkarni wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70682/
> -----------------------------------------------------------
>
> (Updated May 20, 2019, 10:19 p.m.)
>
>
> Review request for ranger, Madhan Neethiraj, Pradeep Agrawal, Ramesh Mani, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-2420
> https://issues.apache.org/jira/browse/RANGER-2420
>
>
> Repository: ranger
>
>
> Description
> -------
>
> Ranger uses ObjectMapper to convert to/from JSON. In a profile of a workload (an Impala authorization test) I see that 36% of the ranger CPU is spent in these functions. 26% of total CPU is in the '_findRootDeserializer' method, which gets cached if a type is deserialized multiple times. However, the caching is only effective if the ObjectMapper is reused. JSONUtil appears to create a new ObjectMapper on every call, which defeats the caching.
>
> Patch ensures that there is only one instance of ObjectMapper created in a process.
>
>
> Diffs
> -----
>
> agents-common/src/main/java/org/apache/ranger/admin/client/datatype/GrantRevokeData.java a8d3c6af5
> agents-common/src/main/java/org/apache/ranger/admin/client/datatype/RESTResponse.java 333f56504
> agents-common/src/main/java/org/apache/ranger/plugin/audit/RangerDefaultAuditHandler.java 5e2c49211
> agents-common/src/main/java/org/apache/ranger/plugin/util/JsonUtilsV2.java PRE-CREATION
> kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStoreProvider.java bc692937b
> kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java d3e0064db
> kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONWriter.java 3674e7a87
> knox-agent/src/main/java/org/apache/ranger/services/knox/client/KnoxClient.java 0c83ef9bb
> security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java 35dc9405b
> security-admin/src/main/java/org/apache/ranger/common/JSONUtil.java 5bc1baf58
> security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 0e7cd8f86
> security-admin/src/main/java/org/apache/ranger/service/RangerDataHistService.java a96d648a9
> security-admin/src/main/java/org/apache/ranger/service/RangerPolicyService.java 02763670e
> security-admin/src/main/java/org/apache/ranger/service/XAssetService.java 5525584f0
> security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java ac9af5eb4
> security-admin/src/test/java/org/apache/ranger/service/TestRangerDataHistService.java e1a236e5f
>
>
> Diff: https://reviews.apache.org/r/70682/diff/2/
>
>
> Testing
> -------
>
> Ran all unit tests successfully.
>
>
> Thanks,
>
> Abhay Kulkarni
>
>
Re: Review Request 70682: RANGER-2420: Ranger spends 36% of CPU in
ObjectMapper
Posted by Abhay Kulkarni <ak...@hortonworks.com>.
> On May 20, 2019, 11:18 p.m., Madhan Neethiraj wrote:
> > kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStoreProvider.java
> > Line 220 (original), 220 (patched)
> > <https://reviews.apache.org/r/70682/diff/1/?file=2145939#file2145939line220>
> >
> > Consider avoiding call to getMapper() by replacing with:
> > JsonUtilsV2.readMapToString(attributes)
> >
> > Please review other such usage as well.
>
> Abhay Kulkarni wrote:
> Map interface (attributes is a Map) is not serializable, so cannot use it as an argument to JsonUtilsV2.readMapToString().
Using JsonUtilsV2.mapToJson() API.
- Abhay
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70682/#review215388
-----------------------------------------------------------
On May 20, 2019, 11:41 p.m., Abhay Kulkarni wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70682/
> -----------------------------------------------------------
>
> (Updated May 20, 2019, 11:41 p.m.)
>
>
> Review request for ranger, Madhan Neethiraj, Pradeep Agrawal, Ramesh Mani, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-2420
> https://issues.apache.org/jira/browse/RANGER-2420
>
>
> Repository: ranger
>
>
> Description
> -------
>
> Ranger uses ObjectMapper to convert to/from JSON. In a profile of a workload (an Impala authorization test) I see that 36% of the ranger CPU is spent in these functions. 26% of total CPU is in the '_findRootDeserializer' method, which gets cached if a type is deserialized multiple times. However, the caching is only effective if the ObjectMapper is reused. JSONUtil appears to create a new ObjectMapper on every call, which defeats the caching.
>
> Patch ensures that there is only one instance of ObjectMapper created in a process.
>
>
> Diffs
> -----
>
> agents-common/src/main/java/org/apache/ranger/admin/client/datatype/GrantRevokeData.java a8d3c6af5
> agents-common/src/main/java/org/apache/ranger/admin/client/datatype/RESTResponse.java 333f56504
> agents-common/src/main/java/org/apache/ranger/plugin/audit/RangerDefaultAuditHandler.java 5e2c49211
> agents-common/src/main/java/org/apache/ranger/plugin/util/JsonUtilsV2.java PRE-CREATION
> kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStoreProvider.java bc692937b
> kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java d3e0064db
> kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONWriter.java 3674e7a87
> knox-agent/src/main/java/org/apache/ranger/services/knox/client/KnoxClient.java 0c83ef9bb
> security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java 35dc9405b
> security-admin/src/main/java/org/apache/ranger/common/JSONUtil.java 5bc1baf58
> security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 0e7cd8f86
> security-admin/src/main/java/org/apache/ranger/service/RangerDataHistService.java a96d648a9
> security-admin/src/main/java/org/apache/ranger/service/RangerPolicyService.java 02763670e
> security-admin/src/main/java/org/apache/ranger/service/XAssetService.java 5525584f0
> security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java ac9af5eb4
> security-admin/src/test/java/org/apache/ranger/service/TestRangerDataHistService.java e1a236e5f
>
>
> Diff: https://reviews.apache.org/r/70682/diff/2/
>
>
> Testing
> -------
>
> Ran all unit tests successfully.
>
>
> Thanks,
>
> Abhay Kulkarni
>
>
Re: Review Request 70682: RANGER-2420: Ranger spends 36% of CPU in
ObjectMapper
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70682/#review215388
-----------------------------------------------------------
Fix it, then Ship it!
agents-common/src/main/java/org/apache/ranger/plugin/util/JsonUtilsV2.java
Lines 50 (patched)
<https://reviews.apache.org/r/70682/#comment302076>
Consider following renames, for better readability:
readMapToString() => mapToJson()
readListToString() => listToJson()
writeObjectAsString() => objToJson()
writeJsonToJavaObject() => jsonToObj()
kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStoreProvider.java
Line 220 (original), 220 (patched)
<https://reviews.apache.org/r/70682/#comment302077>
Consider avoiding call to getMapper() by replacing with:
JsonUtilsV2.readMapToString(attributes)
Please review other such usage as well.
- Madhan Neethiraj
On May 20, 2019, 10:19 p.m., Abhay Kulkarni wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70682/
> -----------------------------------------------------------
>
> (Updated May 20, 2019, 10:19 p.m.)
>
>
> Review request for ranger, Madhan Neethiraj, Pradeep Agrawal, Ramesh Mani, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-2420
> https://issues.apache.org/jira/browse/RANGER-2420
>
>
> Repository: ranger
>
>
> Description
> -------
>
> Ranger uses ObjectMapper to convert to/from JSON. In a profile of a workload (an Impala authorization test) I see that 36% of the ranger CPU is spent in these functions. 26% of total CPU is in the '_findRootDeserializer' method, which gets cached if a type is deserialized multiple times. However, the caching is only effective if the ObjectMapper is reused. JSONUtil appears to create a new ObjectMapper on every call, which defeats the caching.
>
> Patch ensures that there is only one instance of ObjectMapper created in a process.
>
>
> Diffs
> -----
>
> agents-common/src/main/java/org/apache/ranger/admin/client/datatype/GrantRevokeData.java a8d3c6af5
> agents-common/src/main/java/org/apache/ranger/admin/client/datatype/RESTResponse.java 333f56504
> agents-common/src/main/java/org/apache/ranger/plugin/audit/RangerDefaultAuditHandler.java 5e2c49211
> agents-common/src/main/java/org/apache/ranger/plugin/util/JsonUtilsV2.java PRE-CREATION
> kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStoreProvider.java bc692937b
> kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java d3e0064db
> kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONWriter.java 3674e7a87
> knox-agent/src/main/java/org/apache/ranger/services/knox/client/KnoxClient.java 0c83ef9bb
> security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java 35dc9405b
> security-admin/src/main/java/org/apache/ranger/common/JSONUtil.java 5bc1baf58
> security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 0e7cd8f86
> security-admin/src/main/java/org/apache/ranger/service/RangerDataHistService.java a96d648a9
> security-admin/src/main/java/org/apache/ranger/service/RangerPolicyService.java 02763670e
> security-admin/src/main/java/org/apache/ranger/service/XAssetService.java 5525584f0
> security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java ac9af5eb4
> security-admin/src/test/java/org/apache/ranger/service/TestRangerDataHistService.java e1a236e5f
>
>
> Diff: https://reviews.apache.org/r/70682/diff/1/
>
>
> Testing
> -------
>
> Ran all unit tests successfully.
>
>
> Thanks,
>
> Abhay Kulkarni
>
>
Re: Review Request 70682: RANGER-2420: Ranger spends 36% of CPU in
ObjectMapper
Posted by Ramesh Mani <rm...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70682/#review215386
-----------------------------------------------------------
Ship it!
Ship It!
- Ramesh Mani
On May 20, 2019, 10:19 p.m., Abhay Kulkarni wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70682/
> -----------------------------------------------------------
>
> (Updated May 20, 2019, 10:19 p.m.)
>
>
> Review request for ranger, Madhan Neethiraj, Pradeep Agrawal, Ramesh Mani, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-2420
> https://issues.apache.org/jira/browse/RANGER-2420
>
>
> Repository: ranger
>
>
> Description
> -------
>
> Ranger uses ObjectMapper to convert to/from JSON. In a profile of a workload (an Impala authorization test) I see that 36% of the ranger CPU is spent in these functions. 26% of total CPU is in the '_findRootDeserializer' method, which gets cached if a type is deserialized multiple times. However, the caching is only effective if the ObjectMapper is reused. JSONUtil appears to create a new ObjectMapper on every call, which defeats the caching.
>
> Patch ensures that there is only one instance of ObjectMapper created in a process.
>
>
> Diffs
> -----
>
> agents-common/src/main/java/org/apache/ranger/admin/client/datatype/GrantRevokeData.java a8d3c6af5
> agents-common/src/main/java/org/apache/ranger/admin/client/datatype/RESTResponse.java 333f56504
> agents-common/src/main/java/org/apache/ranger/plugin/audit/RangerDefaultAuditHandler.java 5e2c49211
> agents-common/src/main/java/org/apache/ranger/plugin/util/JsonUtilsV2.java PRE-CREATION
> kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStoreProvider.java bc692937b
> kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java d3e0064db
> kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONWriter.java 3674e7a87
> knox-agent/src/main/java/org/apache/ranger/services/knox/client/KnoxClient.java 0c83ef9bb
> security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java 35dc9405b
> security-admin/src/main/java/org/apache/ranger/common/JSONUtil.java 5bc1baf58
> security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 0e7cd8f86
> security-admin/src/main/java/org/apache/ranger/service/RangerDataHistService.java a96d648a9
> security-admin/src/main/java/org/apache/ranger/service/RangerPolicyService.java 02763670e
> security-admin/src/main/java/org/apache/ranger/service/XAssetService.java 5525584f0
> security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java ac9af5eb4
> security-admin/src/test/java/org/apache/ranger/service/TestRangerDataHistService.java e1a236e5f
>
>
> Diff: https://reviews.apache.org/r/70682/diff/1/
>
>
> Testing
> -------
>
> Ran all unit tests successfully.
>
>
> Thanks,
>
> Abhay Kulkarni
>
>