You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ranger.apache.org by "caozhiqiang (Jira)" <ji...@apache.org> on 2021/08/31 11:57:00 UTC
[jira] [Comment Edited] (RANGER-3386) apache CollectionUtils
package reduce ranger agent performance and should be replaced
[ https://issues.apache.org/jira/browse/RANGER-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17407276#comment-17407276 ]
caozhiqiang edited comment on RANGER-3386 at 8/31/21, 11:56 AM:
----------------------------------------------------------------
OK. I will make this changer to my company's private branch. Thank you.
was (Author: caozhiqiang):
OK. I will make this changer to my company's private branch. Thank you.
> apache CollectionUtils package reduce ranger agent performance and should be replaced
> -------------------------------------------------------------------------------------
>
> Key: RANGER-3386
> URL: https://issues.apache.org/jira/browse/RANGER-3386
> Project: Ranger
> Issue Type: Improvement
> Components: plugins
> Affects Versions: 2.0.1
> Reporter: caozhiqiang
> Assignee: caozhiqiang
> Priority: Major
> Attachments: CollectionUtils.png, RANGER-3386-branch-2.0.1.001.patch, RANGER-3386-branch-2.0.1.002.patch
>
>
> org.apache.commons.collections.CollectionUtils' performance is too poor and reduce the performance of ranger plugins, particularly with hdfs. There are too many places used CollectionUtils.isNotEmpty and CollectionUtils.isEmpty in agent component, so we should replace them.
> We can see many CollectionUtils call is take too much time in namenode benchmark's result.
> In this patch, I replace almost CollectionUtils functions in agents-common. After adding this patch, in creating file benchmark of hdfs, the performance can improve from creating *7000* files to *7600* files per second.
> By the way, I write a simple test code below. collection.isEmpty is almost take 0 milliseconds, but CollectionUtils.isNotEmpty take 5 milliseconds.
>
> {code:java}
> List<Integer> list = new ArrayList<Integer>();
> for(int i = 0; i<1000; i++)
> list.add(i);
> long startTime = System.currentTimeMillis();
> if(list != null && !list.isEmpty()){
> }
> long endTime = System.currentTimeMillis();
> long totalTime = endTime - startTime;
> System.out.println(totalTime);
> long startTime2 = System.currentTimeMillis();
> CollectionUtils.isNotEmpty(list);
> long endTime2 = System.currentTimeMillis();
> long totalTime2 = endTime2 - startTime2;
> System.out.println(totalTime2);
> {code}
>
> !CollectionUtils.png|width=680,height=324!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)