You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kylin.apache.org by "Hao Chen (JIRA)" <ji...@apache.org> on 2016/03/18 07:14:33 UTC

[jira] [Updated] (KYLIN-1506) Refactor time-based filter on resource

     [ https://issues.apache.org/jira/browse/KYLIN-1506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hao Chen updated KYLIN-1506:
----------------------------
    Description: 
H1. Problem

Currently all operations like getJobOutputs/getJobs and so on are use two-times scan to get the response, for example, currently the scan always:
1. Get keys, sort, get first and last key (in fact which is just get by prefix filter) with "store.listResources(resourcePath)"
2. Re-scan the keys with timestamp filter: "store.getAllResources(startKey,endKey,startTime, endTime, Class, Serializer)"

{code}
public List<ExecutableOutputPO> getJobOutputs(long timeStartInMillis, long timeEndInMillis) throws PersistentException {
        try {
            NavigableSet<String> resources = store.listResources(ResourceStore.EXECUTE_OUTPUT_RESOURCE_ROOT);
            if (resources == null || resources.isEmpty()) {
                return Collections.emptyList();
            }
            // Collections.sort(resources);
            String rangeStart = resources.first();
            String rangeEnd = resources.last();
            return store.getAllResources(rangeStart, rangeEnd, timeStartInMillis, timeEndInMillis, ExecutableOutputPO.class, JOB_OUTPUT_SERIALIZER);
        } catch (IOException e) {
            logger.error("error get all Jobs:", e);
            throw new PersistentException(e);
        }
    }
{code}

H2. Solution
In fact we could simply combine the two-times scan into one directly:
{code}
store.getAllResources(resourcePath,startTime, endTime, Class, Serializer)
{code}

  was:
Currently all operations like getJobOutputs/getJobs and so on are use two-times scan to get the response, for example, currently the scan always:
1. Get keys, sort, get first and last key (in fact which is just get by prefix filter)
2. Re-scan the keys with timestamp filter.

{code}
public List<ExecutableOutputPO> getJobOutputs(long timeStartInMillis, long timeEndInMillis) throws PersistentException {
        try {
            NavigableSet<String> resources = store.listResources(ResourceStore.EXECUTE_OUTPUT_RESOURCE_ROOT);
            if (resources == null || resources.isEmpty()) {
                return Collections.emptyList();
            }
            // Collections.sort(resources);
            String rangeStart = resources.first();
            String rangeEnd = resources.last();
            return store.getAllResources(rangeStart, rangeEnd, timeStartInMillis, timeEndInMillis, ExecutableOutputPO.class, JOB_OUTPUT_SERIALIZER);
        } catch (IOException e) {
            logger.error("error get all Jobs:", e);
            throw new PersistentException(e);
        }
    }
{code}




> Refactor time-based filter on resource
> --------------------------------------
>
>                 Key: KYLIN-1506
>                 URL: https://issues.apache.org/jira/browse/KYLIN-1506
>             Project: Kylin
>          Issue Type: Sub-task
>            Reporter: Hao Chen
>
> H1. Problem
> Currently all operations like getJobOutputs/getJobs and so on are use two-times scan to get the response, for example, currently the scan always:
> 1. Get keys, sort, get first and last key (in fact which is just get by prefix filter) with "store.listResources(resourcePath)"
> 2. Re-scan the keys with timestamp filter: "store.getAllResources(startKey,endKey,startTime, endTime, Class, Serializer)"
> {code}
> public List<ExecutableOutputPO> getJobOutputs(long timeStartInMillis, long timeEndInMillis) throws PersistentException {
>         try {
>             NavigableSet<String> resources = store.listResources(ResourceStore.EXECUTE_OUTPUT_RESOURCE_ROOT);
>             if (resources == null || resources.isEmpty()) {
>                 return Collections.emptyList();
>             }
>             // Collections.sort(resources);
>             String rangeStart = resources.first();
>             String rangeEnd = resources.last();
>             return store.getAllResources(rangeStart, rangeEnd, timeStartInMillis, timeEndInMillis, ExecutableOutputPO.class, JOB_OUTPUT_SERIALIZER);
>         } catch (IOException e) {
>             logger.error("error get all Jobs:", e);
>             throw new PersistentException(e);
>         }
>     }
> {code}
> H2. Solution
> In fact we could simply combine the two-times scan into one directly:
> {code}
> store.getAllResources(resourcePath,startTime, endTime, Class, Serializer)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)