[jira] [Resolved] (KYLIN-5274) Improve performance of getSubstitutor

["Xiaoxiang Yu (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KYLIN-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Xiaoxiang Yu resolved KYLIN-5274.
---------------------------------
    Resolution: Fixed

> Improve performance of getSubstitutor
> -------------------------------------
>
>                 Key: KYLIN-5274
>                 URL: https://issues.apache.org/jira/browse/KYLIN-5274
>             Project: Kylin
>          Issue Type: Improvement
>            Reporter: Xiaoxiang Yu
>            Assignee: Zhong Yanghong
>            Priority: Major
>             Fix For: 5.0-alpha
>
>         Attachments: image-2022-10-08-17-35-34-500.png, image-2022-10-08-17-50-07-929.png
>
>
> h3. Background
> The following code are called for each call of *_KylinConfig#getOptional_* , which needs to be optimized. In some case, it will improve query performance.
>  
> {code:java}
> tected StrSubstitutor getSubstitutor() {
>     // env > properties
>     final Map<String, Object> all = Maps.newHashMap(); // create a new map every time
>     all.putAll((Map) properties);
>     all.putAll(STATIC_SYSTEM_ENV);
>     return new StrSubstitutor(all);
> } {code}
>  
>  
> h3. How to fix
> 1. Not to create a new map each time.
> 2. Not to use Properties because it extends _Hashtable._
> h3. Micro Benchmark
> Use JMH to show performance(avg time):
>  
> {code:java}
> import org.apache.kylin.common.util.NLocalFileMetadataTestCase;
> import org.openjdk.jmh.annotations.Benchmark;
> import org.openjdk.jmh.annotations.BenchmarkMode;
> import org.openjdk.jmh.annotations.Fork;
> import org.openjdk.jmh.annotations.Measurement;
> import org.openjdk.jmh.annotations.Mode;
> import org.openjdk.jmh.annotations.OutputTimeUnit;
> import org.openjdk.jmh.annotations.Scope;
> import org.openjdk.jmh.annotations.Setup;
> import org.openjdk.jmh.annotations.State;
> import org.openjdk.jmh.annotations.Threads;
> import org.openjdk.jmh.annotations.Warmup;
> import java.util.concurrent.TimeUnit;
> @BenchmarkMode(Mode.AverageTime)
> @OutputTimeUnit(TimeUnit.MILLISECONDS)
> @Warmup(iterations = 1)
> @Measurement(iterations = 10, time = 10, timeUnit = TimeUnit.MILLISECONDS)
> @Threads(1)
> @Fork(value = 1, jvmArgs = {"-Xms2G", "-Xmx2G"})
> @State(Scope.Benchmark)
> public class KylinConfigBenchmark {
>     @Setup
>     public void setUp() throws Exception {
>         NLocalFileMetadataTestCase case1 = new NLocalFileMetadataTestCase();
>         case1.createTestMetadata();
>     }
>     @Benchmark
>     public void getProperty() {
>         KylinConfig config = KylinConfig.getInstanceFromEnv();
>         for(int i = 0; i<= 1000_000; i++ ){
>             config.getJdbcDriverClass();
>         }
>     }
>     public static void main(String[] args) throws Exception {
>         org.openjdk.jmh.Main.main(args);
>     }
> } {code}
>  
>  
> h4. Before Applied
> !image-2022-10-08-17-35-34-500.png!
>  
> h4. After Applied
> !image-2022-10-08-17-50-07-929.png!
>  
> h3. Conclusion
> From JMH, _KylinConfig#getOptional_ will improve about 100 times.
> From loading test, query performance will improve about 2-5%([https://github.com/apache/kylin/pull/1991#issuecomment-1262139953),] eBay says it will improve 30% in their testing.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)