You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kylin.apache.org by li...@apache.org on 2018/03/03 04:53:48 UTC
[kylin] branch sync updated (76dc049 -> a05a4d0)
This is an automated email from the ASF dual-hosted git repository.
liyang pushed a change to branch sync
in repository https://gitbox.apache.org/repos/asf/kylin.git.
from 76dc049 minor, fix DefaultSchedulerTest.testMetaStoreRecover()
add a2af43d KYLIN-3205 Allow one column is used for both dimension and precisely count distinct measure
add 4a29d92 KYLIN-2929, speed up dump performance, write dump file to disk in lazy way
add 61f36a2 APACHE-KYLIN-3235: add null check for SQL
add 148f5c6 KYLIN-3069 Add proper time zone support to the WebUI instead of GMT/PST kludge
add d4c8fbc KYLIN-3215
add d31a009 KYLIN-3219
add 5bc5e65 KYLIN-2980, Remove getKey/Value setKey/Value from Kylin's Pair.
add ff6f6c7 KYLIN-3199 The login dialog should be closed when ldap user with no permission login correctly
add ebfce48 KYLIN 3199 code review
add eac03cd KYLIN-3214
add c686b84 KYLIN-2999 fix cube auto-migration
add 0432487 KYLIN-2999 fix ut
add 9347f0b KYLIN-3195 update Billy Liu public keys to KEYS
add 0bc0677 update calcite to 1.13.0-r4
add 56ec6af KYLIN-3243
add ce76ed9 KYLIN-3245 Searching cube support fuzzy search
add 6a558d8 KYLIN-3241
add d06b72e KYLIN-3131
add bc040a9 Minor, make the sample kylin_streaming_cube be consistent with kylin_sales
add 8826e1d KYLIN-3277 Automatically remove the blank at the end of lines in properties files
add 46d4f97 KYLIN-3139 - Failure in map-reduce job due to undefined hdp.version variable when using HDP stack and remote HBase cluster
add 5075f59 KYLIN-3145 Support Kafka JSON message whose property name includes _
add 01bd0d4 KYLIN-3145 Support Kafka JSON message whose property name includes _ (#4028)
add 50435e1 KYLIN-3145 minor, update sample streaming message (#4013)
add 0655dbc KYLIN-3223 - Query for the list of hybrid cubes results in NPE
add 0550cc3 KYLIN-3247
add d4f9d56 KYLIN-3223 fix compile
add 139c303 KYLIN-2884 Add delete segment function for portal
add 65c2315 KYLIN-2672 Only clean necessary cache for CubeMigrationCLI
add 5cecbce KYLIN-2672 minor code review
add 5099066 KYLIN-2672 Add UT
add da0e6b0 KYLIN-3242
add c86f30f KYLIN-3239 Refactor the ACL code about checkPermission and hasPermission
add 82f76db APACHE-KYLIN-2730: add note for some GeneticAlgorithm related lib files that are modified from https://github.com/apache/commons-math
add 3fa11b2 KYLIN-3223 fix adjustment based on new ACL check API added by KYLIN-3239
add 2fd676b KYLIN-3197 When ldap is opened, I use an ignored case user to login, the page does not respond.
add 17435fa KYLIN-3224 Make the column label of pushdown same with query result of cube
add 24c489a KYLIN-3224 code review This closes #100
add f1e9f18 KYLIN-3135 revert first regex too
add 714aee0 revert KYLIN-3197 2fd676b36ec9e0f853d59db31ee62cd90a647641
add e59c33e KYLIN-2960, modify the default LDAP admin group config.
add 99d68c8 minor, update README.md
add c234f8f add coveralls badge
add 519f486 enable coveralls
add 3be7af9 Merge pull request #103 from yiming187/travis
add 21ef4d6 minor update github scm in pom.xml
add 74dff9f KYLIN-3263, bugfix with AbstractExecutable's retry.
add 66ea154 Merge pull request #104 from tttMelody/master
add 23641a2 KYLIN-2884 Add delete segment function for portal - bug fix
add 865e377 Merge pull request #105 from sanjulian/APACHE-KYLIN-2884
add 045b22d KYLIN-3268 Tomcat Security Vulnerability Alert. The version of the tomcat for kylin should upgrade to 7.0.85.
add 2b38b3b ignore checkstyle on external htrace
add 76934db KYLIN-3265 Add jobSearchMode as a filter condition for jobs api
add 31cc006 KYLIN-3265 Add jobSearchMode as a filter condition for jobs UI part
add 6afe713 Merge pull request #109 from chenzhx/master
add 39ef73c KYLIN-3236 The function 'reGenerateAdvancedDict()' has an error logical judgment, which will cause an exception when you edit the cube.
new c5e46bb KYLIN-3234, add a API to ResourceStore that can recursively
new d63160d KYLIN-3233, fix CacheController if cacheKey has "/".
new 4823d8b KYLIN-3234, add HDFS resource store implementation.
new 24d319e KYLIN-3227 Automatically remove the blank at the end of lines in properties files (#4035)
new 4b2ead2 Move out the wrong execute metadata to a test folder
new c6829f1 Fix unstable ut org.apache.kylin.job.impl.threadpool.DefaultSchedulerTest#testMetaStoreRecover
new a040c9a KYLIN-3266 improve kylin CI BuildCubeWithEngine coverage
new 31af5c2 bug fix,fix KylinConfig deadlock
new 0efcf9c KYLIN-3265 update test case to cover it
new 451b7ba KYLIN-3267 MR config override at project/cube level only for mem-hungry build steps
new a05a4d0 KYLIN-3251, add a hook that can customize test_case_data and add MetadataCleanupJob UT.
The 11 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.travis.yml | 5 +-
KEYS | 59 ++++++++
README.md | 3 +-
build/bin/sample.sh | 2 +
build/conf/kylin_job_conf_inmem.xml | 4 +
build/conf/setenv.sh | 18 ++-
build/script/download-tomcat.sh | 4 +-
.../java/org/apache/kylin/common/KylinConfig.java | 4 +-
.../org/apache/kylin/common/KylinConfigBase.java | 31 +++--
.../common/persistence/FileResourceStore.java | 28 +++-
.../common/persistence/HDFSResourceStore.java | 36 ++++-
.../kylin/common/persistence/ResourceStore.java | 12 +-
.../apache/kylin/common/restclient/RestClient.java | 33 ++++-
.../java/org/apache/kylin/common/util/Pair.java | 16 ---
.../src/main/resources/kylin-defaults.properties | 2 +-
.../org/apache/kylin/common/KylinConfigTest.java | 9 ++
.../common/persistence/HDFSResourceStoreTest.java | 77 +++++++++++
.../common/util/LocalFileMetadataTestCase.java | 77 ++++++++---
.../java/org/apache/kylin/cube/CubeManager.java | 3 +-
.../cuboid/algorithm/generic/GeneticAlgorithm.java | 2 +
.../cuboid/algorithm/generic/lib/BitsMutation.java | 2 +
.../cuboid/algorithm/generic/lib/Chromosome.java | 2 +
.../algorithm/generic/lib/ChromosomePair.java | 2 +
.../algorithm/generic/lib/CrossoverPolicy.java | 2 +
.../generic/lib/ElitisticListPopulation.java | 2 +
.../cube/cuboid/algorithm/generic/lib/Fitness.java | 2 +
.../generic/lib/FixedGenerationCount.java | 2 +
.../algorithm/generic/lib/ListPopulation.java | 2 +
.../algorithm/generic/lib/MutationPolicy.java | 2 +
.../algorithm/generic/lib/OnePointCrossover.java | 2 +
.../cuboid/algorithm/generic/lib/Population.java | 2 +
.../algorithm/generic/lib/SelectionPolicy.java | 2 +
.../algorithm/generic/lib/StoppingCondition.java | 2 +
.../algorithm/generic/lib/TournamentSelection.java | 2 +
.../kylin/cube/model/CubeJoinedFlatTableDesc.java | 4 +
.../cube/model/validation/rule/DictionaryRule.java | 6 +-
.../apache/kylin/gridtable/GTAggregateScanner.java | 70 ++++++++--
.../kylin/job/execution/AbstractExecutable.java | 32 ++---
.../apache/kylin/job/ExecutableManagerTest.java | 2 +-
.../apache/kylin/job/RetryableTestExecutable.java | 50 -------
.../job/impl/threadpool/DefaultSchedulerTest.java | 35 +++--
.../kylin/metadata/TableMetadataManager.java | 12 ++
.../org/apache/kylin/metadata/model/TableDesc.java | 2 +-
.../org/apache/kylin/metadata/model/TblColRef.java | 6 +
.../kylin/metadata/project/ProjectL2Cache.java | 4 +
.../kylin/metadata/project/ProjectManager.java | 6 +-
.../org/apache/kylin/metrics/MetricsManager.java | 14 +-
.../java/org/apache/kylin/engine/mr/CubingJob.java | 17 ++-
.../engine/mr/common/MapReduceExecutable.java | 21 ++-
.../engine/mr/common/MapReduceExecutableTest.java | 50 +++++++
.../template/cube_desc/kylin_streaming_cube.json | 4 +-
examples/test_case_data/localmeta/kylin.properties | 6 +-
examples/test_case_data/sandbox/kylin.properties | 2 +-
.../sandbox/kylin_job_conf_inmem.xml | 4 +
external/htrace/pom.xml | 1 +
.../kylin/provision/BuildCubeWithEngine.java | 75 +++++-----
pom.xml | 29 ++--
.../kylin/query/adhoc/PushDownRunnerJdbcImpl.java | 151 ++++++++++++++++++++-
.../org/apache/kylin/query/util/QueryUtil.java | 2 +-
.../org/apache/kylin/query/util/QueryUtilTest.java | 7 +-
.../broadcaster/BroadcasterReceiveServlet.java | 21 ++-
.../kylin/rest/controller/AccessController.java | 10 +-
.../kylin/rest/controller/CacheController.java | 20 +++
.../kylin/rest/controller/CubeController.java | 18 ++-
.../kylin/rest/controller/JobController.java | 14 +-
.../kylin/rest/controller/ProjectController.java | 10 +-
.../kylin/rest/controller/QueryController.java | 2 +-
.../java/org/apache/kylin/rest/msg/Message.java | 4 +
...ybridRequest.java => CubeMigrationRequest.java} | 36 ++---
.../apache/kylin/rest/request/JobListRequest.java | 10 ++
.../rest/security/LDAPAuthoritiesPopulator.java | 1 +
.../apache/kylin/rest/service/CacheService.java | 34 ++++-
.../org/apache/kylin/rest/service/CubeService.java | 82 +++++------
.../apache/kylin/rest/service/HybridService.java | 53 +++++---
.../org/apache/kylin/rest/service/JobService.java | 48 ++++---
.../apache/kylin/rest/service/ModelService.java | 20 ++-
.../apache/kylin/rest/service/ProjectService.java | 10 +-
.../apache/kylin/rest/service/QueryService.java | 76 +++++------
.../org/apache/kylin/rest/util/AclEvaluate.java | 78 +++++++----
.../kylin/rest/controller/CacheControllerTest.java | 54 +++++++-
.../kylin/rest/controller/CubeControllerTest.java | 2 +-
.../kylin/rest/controller/JobControllerTest.java | 21 ++-
.../kylin/rest/service/AdminServiceTest.java | 3 +-
.../apache/kylin/rest/service/JobServiceTest.java | 12 +-
.../execute/d9a2b721-9916-4607-8047-148ceb2473b1 | 0
.../apache/kylin/source/kafka/StreamingParser.java | 6 +-
.../kylin/source/kafka/TimedJsonStreamParser.java | 47 +++++--
.../source/kafka/util/KafkaSampleProducer.java | 1 +
.../source/kafka/TimedJsonStreamParserTest.java | 36 ++++-
source-kafka/src/test/resources/message.json | 2 +-
.../kylin/storage/hbase/HBaseResourceStore.java | 24 ++--
.../org/apache/kylin/tool/CubeMigrationCLI.java | 28 ++--
.../org/apache/kylin/tool/MetadataCleanupJob.java | 56 ++++----
.../tool/metrics/systemcube/CubeDescCreator.java | 8 +-
.../apache/kylin/tool/MetadataCleanupJobTest.java | 76 +++++++++++
.../src/test/resources/test_meta}/UUID | Bin
.../test_meta/cube/ci_inner_join_cube.json | 60 ++++++++
.../test_meta}/cube_desc/ci_inner_join_cube.json | 0
.../04b6db34-b243-4780-855a-4c7ea4cee817.seq | Bin 0 -> 2605183 bytes
.../NAME/b58ce89d-3df6-46d1-8cff-2a9cfc8b7510.dict | Bin 0 -> 823 bytes
.../NAME/64ca8fea-b859-4e63-aea3-bfb4c6ee0c9d.dict | Bin 0 -> 844 bytes
.../execute/d861b8b7-c773-47ab-bb1e-c8782ae8d930 | 41 ++++++
.../d861b8b7-c773-47ab-bb1e-c8782ae8d930 | 13 ++
.../d861b8b7-c773-47ab-bb1e-c8782ae8d930-00 | 13 ++
.../d861b8b7-c773-47ab-bb1e-c8782ae8d930-01 | 13 ++
.../src/test/resources/test_meta/kylin.properties | 0
.../test_meta}/model_desc/ci_inner_join_model.json | 0
.../test/resources/test_meta/project/default.json | 35 +++++
.../test_meta}/table/DEFAULT.TEST_ACCOUNT.json | 0
.../table/DEFAULT.TEST_CATEGORY_GROUPINGS.json | 0
.../test_meta}/table/DEFAULT.TEST_COUNTRY.json | 0
.../test_meta}/table/DEFAULT.TEST_KYLIN_FACT.json | 0
.../test_meta}/table/DEFAULT.TEST_ORDER.json | 0
.../test_meta}/table/EDW.TEST_CAL_DT.json | 0
.../test_meta}/table/EDW.TEST_SELLER_TYPE_DIM.json | 0
.../resources/test_meta}/table/EDW.TEST_SITES.json | 0
.../92456efe-9b79-4385-a5a3-e7f37b677bf7.snapshot | Bin 0 -> 16063 bytes
.../7ecdb07b-a8d0-49d8-892b-fe2dd75512ca.snapshot | Bin 0 -> 16061 bytes
webapp/app/js/controllers/cubeAdvanceSetting.js | 9 +-
webapp/app/js/controllers/cubeEdit.js | 15 +-
webapp/app/js/controllers/cubes.js | 72 +++++++++-
webapp/app/js/controllers/job.js | 5 +-
webapp/app/js/controllers/sourceMeta.js | 11 +-
webapp/app/js/filters/filter.js | 42 ++----
webapp/app/js/model/cubeDescModel.js | 16 ++-
webapp/app/js/model/jobConfig.js | 5 +
webapp/app/js/model/jobListModel.js | 2 +
webapp/app/js/services/cubes.js | 1 +
webapp/app/js/services/kylinProperties.js | 3 +
webapp/app/js/services/users.js | 15 +-
webapp/app/partials/common/access.html | 18 +--
.../cube_delete_segment.html} | 49 +++----
webapp/app/partials/cubes/cube_detail.html | 4 +-
webapp/app/partials/cubes/cubes.html | 9 +-
webapp/app/partials/jobs/jobList.html | 5 +-
webapp/app/partials/query/query.html | 2 +-
136 files changed, 1792 insertions(+), 654 deletions(-)
create mode 100644 core-common/src/test/java/org/apache/kylin/common/persistence/HDFSResourceStoreTest.java
delete mode 100644 core-job/src/test/java/org/apache/kylin/job/RetryableTestExecutable.java
create mode 100644 engine-mr/src/test/java/org/apache/kylin/engine/mr/common/MapReduceExecutableTest.java
copy server-base/src/main/java/org/apache/kylin/rest/request/{HybridRequest.java => CubeMigrationRequest.java} (66%)
rename {examples/test_case_data/localmeta => server/src/test/resources/ut_meta/broken_executable}/execute/d9a2b721-9916-4607-8047-148ceb2473b1 (100%)
create mode 100644 tool/src/test/java/org/apache/kylin/tool/MetadataCleanupJobTest.java
copy {examples/test_case_data/localmeta => tool/src/test/resources/test_meta}/UUID (100%)
create mode 100644 tool/src/test/resources/test_meta/cube/ci_inner_join_cube.json
copy {examples/test_case_data/localmeta => tool/src/test/resources/test_meta}/cube_desc/ci_inner_join_cube.json (100%)
create mode 100644 tool/src/test/resources/test_meta/cube_statistics/kylin_sales_cube/04b6db34-b243-4780-855a-4c7ea4cee817.seq
create mode 100644 tool/src/test/resources/test_meta/dict/DEFAULT.KYLIN_COUNTRY/NAME/b58ce89d-3df6-46d1-8cff-2a9cfc8b7510.dict
create mode 100644 tool/src/test/resources/test_meta/dict/DEFAULT.TEST_COUNTRY/NAME/64ca8fea-b859-4e63-aea3-bfb4c6ee0c9d.dict
create mode 100644 tool/src/test/resources/test_meta/execute/d861b8b7-c773-47ab-bb1e-c8782ae8d930
create mode 100644 tool/src/test/resources/test_meta/execute_output/d861b8b7-c773-47ab-bb1e-c8782ae8d930
create mode 100644 tool/src/test/resources/test_meta/execute_output/d861b8b7-c773-47ab-bb1e-c8782ae8d930-00
create mode 100644 tool/src/test/resources/test_meta/execute_output/d861b8b7-c773-47ab-bb1e-c8782ae8d930-01
copy engine-mr/src/test/resources/multi_file_empty_file/empty.txt => tool/src/test/resources/test_meta/kylin.properties (100%)
copy {examples/test_case_data/localmeta => tool/src/test/resources/test_meta}/model_desc/ci_inner_join_model.json (100%)
create mode 100644 tool/src/test/resources/test_meta/project/default.json
copy {examples/test_case_data/localmeta => tool/src/test/resources/test_meta}/table/DEFAULT.TEST_ACCOUNT.json (100%)
copy {examples/test_case_data/localmeta => tool/src/test/resources/test_meta}/table/DEFAULT.TEST_CATEGORY_GROUPINGS.json (100%)
copy {examples/test_case_data/localmeta => tool/src/test/resources/test_meta}/table/DEFAULT.TEST_COUNTRY.json (100%)
copy {examples/test_case_data/localmeta => tool/src/test/resources/test_meta}/table/DEFAULT.TEST_KYLIN_FACT.json (100%)
copy {examples/test_case_data/localmeta => tool/src/test/resources/test_meta}/table/DEFAULT.TEST_ORDER.json (100%)
copy {examples/test_case_data/localmeta => tool/src/test/resources/test_meta}/table/EDW.TEST_CAL_DT.json (100%)
copy {examples/test_case_data/localmeta => tool/src/test/resources/test_meta}/table/EDW.TEST_SELLER_TYPE_DIM.json (100%)
copy {examples/test_case_data/localmeta => tool/src/test/resources/test_meta}/table/EDW.TEST_SITES.json (100%)
create mode 100644 tool/src/test/resources/test_meta/table_snapshot/DEFAULT.KYLIN_COUNTRY/92456efe-9b79-4385-a5a3-e7f37b677bf7.snapshot
create mode 100644 tool/src/test/resources/test_meta/table_snapshot/DEFAULT.TEST_COUNTRY/7ecdb07b-a8d0-49d8-892b-fe2dd75512ca.snapshot
copy webapp/app/partials/{jobs/job_refresh.html => cubes/cube_delete_segment.html} (59%)
--
To stop receiving notification emails like this one, please contact
liyang@apache.org.
[kylin] 06/11: Fix unstable ut
org.apache.kylin.job.impl.threadpool.DefaultSchedulerTest#testMetaStoreRecover
Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
liyang pushed a commit to branch sync
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit c6829f133f593ce8ee327baf8b797be781b73e27
Author: nichunen <ch...@kyligence.io>
AuthorDate: Fri Feb 9 14:22:25 2018 +0800
Fix unstable ut org.apache.kylin.job.impl.threadpool.DefaultSchedulerTest#testMetaStoreRecover
---
.../job/impl/threadpool/DefaultSchedulerTest.java | 24 +++++++++++-----------
1 file changed, 12 insertions(+), 12 deletions(-)
diff --git a/core-job/src/test/java/org/apache/kylin/job/impl/threadpool/DefaultSchedulerTest.java b/core-job/src/test/java/org/apache/kylin/job/impl/threadpool/DefaultSchedulerTest.java
index 3b24fe6..3c1a7ea 100644
--- a/core-job/src/test/java/org/apache/kylin/job/impl/threadpool/DefaultSchedulerTest.java
+++ b/core-job/src/test/java/org/apache/kylin/job/impl/threadpool/DefaultSchedulerTest.java
@@ -18,6 +18,16 @@
package org.apache.kylin.job.impl.threadpool;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.io.FileNotFoundException;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.TimeUnit;
+
import org.apache.kylin.common.KylinConfig;
import org.apache.kylin.job.BaseTestExecutable;
import org.apache.kylin.job.ErrorTestExecutable;
@@ -39,16 +49,6 @@ import org.junit.rules.ExpectedException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import java.io.FileNotFoundException;
-import java.util.concurrent.CountDownLatch;
-import java.util.concurrent.Executors;
-import java.util.concurrent.ScheduledExecutorService;
-import java.util.concurrent.ScheduledFuture;
-import java.util.concurrent.TimeUnit;
-
-import static org.junit.Assert.assertFalse;
-import static org.junit.Assert.assertTrue;
-
/**
*/
public class DefaultSchedulerTest extends BaseSchedulerTest {
@@ -184,14 +184,14 @@ public class DefaultSchedulerTest extends BaseSchedulerTest {
@Test
public void testMetaStoreRecover() throws Exception {
- logger.info("tesMetaStoreRecover");
+ logger.info("testMetaStoreRecover");
NoErrorStatusExecutable job = new NoErrorStatusExecutable();
ErrorTestExecutable task = new ErrorTestExecutable();
job.addTask(task);
execMgr.addJob(job);
Thread.sleep(2500);
runningJobToError(job.getId());
- Thread.sleep(2500);
+ waitForJobFinish(job.getId(), 10000);
Assert.assertEquals(ExecutableState.ERROR, execMgr.getOutput(job.getId()).getState());
}
--
To stop receiving notification emails like this one, please contact
liyang@apache.org.
[kylin] 04/11: KYLIN-3227 Automatically remove the blank at the end
of lines in properties files (#4035)
Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
liyang pushed a commit to branch sync
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit 24d319eddc58a4ab5a612c11b7a824582afb3b92
Author: GuangYaoLee92 <33...@users.noreply.github.com>
AuthorDate: Thu Feb 8 15:06:59 2018 +0800
KYLIN-3227 Automatically remove the blank at the end of lines in properties files (#4035)
---
core-common/src/test/java/org/apache/kylin/common/KylinConfigTest.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/core-common/src/test/java/org/apache/kylin/common/KylinConfigTest.java b/core-common/src/test/java/org/apache/kylin/common/KylinConfigTest.java
index 8ed1b42..92c730a 100644
--- a/core-common/src/test/java/org/apache/kylin/common/KylinConfigTest.java
+++ b/core-common/src/test/java/org/apache/kylin/common/KylinConfigTest.java
@@ -133,7 +133,7 @@ public class KylinConfigTest extends HotLoadKylinPropertiesTestCase {
}
@Test
- public void testUnexpectedBlankInPro() {
+ public void testUnexpectedBlackInPro() {
KylinConfig conf = KylinConfig.getInstanceFromEnv();
Map<String, String> override = conf.getPropertiesByPrefix("kylin.engine.mr.config-override.");
assertEquals(2, override.size());
--
To stop receiving notification emails like this one, please contact
liyang@apache.org.
[kylin] 10/11: KYLIN-3267 MR config override at project/cube level
only for mem-hungry build steps
Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
liyang pushed a commit to branch sync
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit 451b7bae796e47358900678ac18dce24a2e6d8df
Author: Yifei Wu <va...@gmail.com>
AuthorDate: Tue Feb 27 14:34:23 2018 +0800
KYLIN-3267 MR config override at project/cube level only for mem-hungry build steps
---
build/conf/kylin_job_conf_inmem.xml | 4 ++
.../org/apache/kylin/common/KylinConfigBase.java | 5 +++
.../engine/mr/common/MapReduceExecutable.java | 21 ++++++---
.../engine/mr/common/MapReduceExecutableTest.java | 50 ++++++++++++++++++++++
examples/test_case_data/localmeta/kylin.properties | 2 +
.../sandbox/kylin_job_conf_inmem.xml | 4 ++
6 files changed, 80 insertions(+), 6 deletions(-)
diff --git a/build/conf/kylin_job_conf_inmem.xml b/build/conf/kylin_job_conf_inmem.xml
index d69435a..ddda4dd 100644
--- a/build/conf/kylin_job_conf_inmem.xml
+++ b/build/conf/kylin_job_conf_inmem.xml
@@ -15,6 +15,10 @@
limitations under the License.
-->
<configuration>
+ <property>
+ <name>mapreduce.job.is-mem-hungry</name>
+ <value>true</value>
+ </property>
<property>
<name>mapreduce.job.split.metainfo.maxsize</name>
diff --git a/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java b/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
index eee2d09..7596d2c 100644
--- a/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
+++ b/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
@@ -1079,6 +1079,11 @@ abstract public class KylinConfigBase implements Serializable {
return getPropertiesByPrefix("kylin.engine.mr.config-override.");
}
+ // used for some mem-hungry step
+ public Map<String, String> getMemHungryConfigOverride() {
+ return getPropertiesByPrefix("kylin.engine.mr.mem-hungry-config-override.");
+ }
+
public Map<String, String> getUHCMRConfigOverride() {
return getPropertiesByPrefix("kylin.engine.mr.uhc-config-override.");
}
diff --git a/engine-mr/src/main/java/org/apache/kylin/engine/mr/common/MapReduceExecutable.java b/engine-mr/src/main/java/org/apache/kylin/engine/mr/common/MapReduceExecutable.java
index ca1311d..ae285cc 100755
--- a/engine-mr/src/main/java/org/apache/kylin/engine/mr/common/MapReduceExecutable.java
+++ b/engine-mr/src/main/java/org/apache/kylin/engine/mr/common/MapReduceExecutable.java
@@ -120,7 +120,8 @@ public class MapReduceExecutable extends AbstractExecutable {
job = new Cluster(conf).getJob(JobID.forName(extra.get(ExecutableConstants.MR_JOB_ID)));
logger.info("mr_job_id:" + extra.get(ExecutableConstants.MR_JOB_ID) + " resumed");
} else {
- final Constructor<? extends AbstractHadoopJob> constructor = ClassUtil.forName(mapReduceJobClass, AbstractHadoopJob.class).getConstructor();
+ final Constructor<? extends AbstractHadoopJob> constructor = ClassUtil
+ .forName(mapReduceJobClass, AbstractHadoopJob.class).getConstructor();
final AbstractHadoopJob hadoopJob = constructor.newInstance();
hadoopJob.setConf(conf);
hadoopJob.setAsync(true); // so the ToolRunner.run() returns right away
@@ -154,7 +155,8 @@ public class MapReduceExecutable extends AbstractExecutable {
mgr.updateJobOutput(getId(), ExecutableState.ERROR, hadoopCmdOutput.getInfo(), "killed by admin");
return new ExecuteResult(ExecuteResult.State.FAILED, "killed by admin");
}
- if (status == JobStepStatusEnum.WAITING && (newStatus == JobStepStatusEnum.FINISHED || newStatus == JobStepStatusEnum.ERROR || newStatus == JobStepStatusEnum.RUNNING)) {
+ if (status == JobStepStatusEnum.WAITING && (newStatus == JobStepStatusEnum.FINISHED
+ || newStatus == JobStepStatusEnum.ERROR || newStatus == JobStepStatusEnum.RUNNING)) {
final long waitTime = System.currentTimeMillis() - getStartTime();
setMapReduceWaitTime(waitTime);
}
@@ -262,24 +264,31 @@ public class MapReduceExecutable extends AbstractExecutable {
options.addOption(OPTION_CUBE_NAME);
CustomParser parser = new CustomParser();
CommandLine commandLine = parser.parse(options, jobParams);
-
+
String confFile = commandLine.getOptionValue(BatchConstants.ARG_CONF);
String cubeName = commandLine.getOptionValue(BatchConstants.ARG_CUBE_NAME);
List<String> remainingArgs = Lists.newArrayList();
-
+
if (StringUtils.isNotBlank(confFile)) {
conf.addResource(new Path(confFile));
}
-
+
if (StringUtils.isNotBlank(cubeName)) {
for (Map.Entry<String, String> entry : CubeManager.getInstance(config).getCube(cubeName).getConfig()
.getMRConfigOverride().entrySet()) {
conf.set(entry.getKey(), entry.getValue());
}
+ if (conf.get("mapreduce.job.is-mem-hungry") != null
+ && Boolean.valueOf(conf.get("mapreduce.job.is-mem-hungry"))) {
+ for (Map.Entry<String, String> entry : CubeManager.getInstance(config).getCube(cubeName).getConfig()
+ .getMemHungryConfigOverride().entrySet()) {
+ conf.set(entry.getKey(), entry.getValue());
+ }
+ }
remainingArgs.add("-" + BatchConstants.ARG_CUBE_NAME);
remainingArgs.add(cubeName);
}
-
+
remainingArgs.addAll(parser.getRemainingArgs());
return (String[]) remainingArgs.toArray(new String[remainingArgs.size()]);
}
diff --git a/engine-mr/src/test/java/org/apache/kylin/engine/mr/common/MapReduceExecutableTest.java b/engine-mr/src/test/java/org/apache/kylin/engine/mr/common/MapReduceExecutableTest.java
new file mode 100644
index 0000000..6e4a57d
--- /dev/null
+++ b/engine-mr/src/test/java/org/apache/kylin/engine/mr/common/MapReduceExecutableTest.java
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kylin.engine.mr.common;
+
+import java.lang.reflect.Method;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.kylin.common.HotLoadKylinPropertiesTestCase;
+import org.apache.kylin.common.KylinConfig;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class MapReduceExecutableTest extends HotLoadKylinPropertiesTestCase {
+
+ @Test
+ public void testOverwriteJobConf() throws Exception {
+ MapReduceExecutable executable = new MapReduceExecutable();
+ KylinConfig config = KylinConfig.getInstanceFromEnv();
+
+ Method method = MapReduceExecutable.class.getDeclaredMethod("overwriteJobConf", Configuration.class,
+ KylinConfig.class, new String[] {}.getClass());
+ method.setAccessible(true);
+ Configuration conf = new Configuration();
+ conf.set("mapreduce.job.is-mem-hungry", "true");
+ method.invoke(executable, conf, config, new String[] { "-cubename", "ci_inner_join_cube" });
+ Assert.assertEquals("mem-test1", conf.get("test1"));
+ Assert.assertEquals("mem-test2", conf.get("test2"));
+
+ conf.clear();
+ method.invoke(executable, conf, config, new String[] { "-cubename", "ci_inner_join_cube" });
+ Assert.assertEquals("test1", conf.get("test1"));
+ Assert.assertEquals("test2", conf.get("test2"));
+ }
+}
\ No newline at end of file
diff --git a/examples/test_case_data/localmeta/kylin.properties b/examples/test_case_data/localmeta/kylin.properties
index 257d69c..805432b 100644
--- a/examples/test_case_data/localmeta/kylin.properties
+++ b/examples/test_case_data/localmeta/kylin.properties
@@ -137,6 +137,8 @@ kylin.security.saml.context-path=/kylin
kylin.test.bcc.new.key=some-value
kylin.engine.mr.config-override.test1=test1
kylin.engine.mr.config-override.test2=test2
+kylin.engine.mr.mem-hungry-config-override.test1=mem-test1
+kylin.engine.mr.mem-hungry-config-override.test2=mem-test2
kylin.job.lock=org.apache.kylin.job.lock.MockJobLockDup
kylin.job.lock=org.apache.kylin.job.lock.MockJobLock
diff --git a/examples/test_case_data/sandbox/kylin_job_conf_inmem.xml b/examples/test_case_data/sandbox/kylin_job_conf_inmem.xml
index 5094d24..80b1d4d 100644
--- a/examples/test_case_data/sandbox/kylin_job_conf_inmem.xml
+++ b/examples/test_case_data/sandbox/kylin_job_conf_inmem.xml
@@ -16,6 +16,10 @@
limitations under the License.
-->
<configuration>
+ <property>
+ <name>mapreduce.job.is-mem-hungry</name>
+ <value>true</value>
+ </property>
<property>
<name>mapreduce.job.split.metainfo.maxsize</name>
--
To stop receiving notification emails like this one, please contact
liyang@apache.org.
[kylin] 08/11: bug fix,fix KylinConfig deadlock
Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
liyang pushed a commit to branch sync
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit 31af5c27870dfefb2d1c626a35a4202606faed84
Author: mingming.ge <mi...@kyligence.io>
AuthorDate: Sun Feb 25 19:58:45 2018 +0800
bug fix,fix KylinConfig deadlock
---
core-common/src/main/java/org/apache/kylin/common/KylinConfig.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/core-common/src/main/java/org/apache/kylin/common/KylinConfig.java b/core-common/src/main/java/org/apache/kylin/common/KylinConfig.java
index 959bc14..191d767 100644
--- a/core-common/src/main/java/org/apache/kylin/common/KylinConfig.java
+++ b/core-common/src/main/java/org/apache/kylin/common/KylinConfig.java
@@ -406,7 +406,7 @@ public class KylinConfig extends KylinConfigBase {
if (mgr != null)
return (T) mgr;
- synchronized (this) {
+ synchronized (clz) {
mgr = managersCache.get(clz);
if (mgr != null)
return (T) mgr;
--
To stop receiving notification emails like this one, please contact
liyang@apache.org.
[kylin] 02/11: KYLIN-3233, fix CacheController if cacheKey has "/".
Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
liyang pushed a commit to branch sync
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit d63160d2df0ea7db7f919d10a5f94c3f05bb6653
Author: Jiatao Tao <24...@qq.com>
AuthorDate: Sun Feb 4 21:08:25 2018 +0800
KYLIN-3233, fix CacheController if cacheKey has "/".
---
.../apache/kylin/common/restclient/RestClient.java | 13 +++++++++++--
.../rest/broadcaster/BroadcasterReceiveServlet.java | 21 ++++++++++++++++++---
.../kylin/rest/controller/CacheController.java | 10 ++++++++++
3 files changed, 39 insertions(+), 5 deletions(-)
diff --git a/core-common/src/main/java/org/apache/kylin/common/restclient/RestClient.java b/core-common/src/main/java/org/apache/kylin/common/restclient/RestClient.java
index e1cc13c..93f5e19 100644
--- a/core-common/src/main/java/org/apache/kylin/common/restclient/RestClient.java
+++ b/core-common/src/main/java/org/apache/kylin/common/restclient/RestClient.java
@@ -38,6 +38,7 @@ import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.client.methods.HttpPut;
import org.apache.http.client.methods.HttpRequestBase;
+import org.apache.http.entity.ContentType;
import org.apache.http.entity.StringEntity;
import org.apache.http.impl.client.BasicCredentialsProvider;
import org.apache.http.impl.client.DefaultHttpClient;
@@ -149,8 +150,16 @@ public class RestClient {
}
public void wipeCache(String entity, String event, String cacheKey) throws IOException {
- String url = baseUrl + "/cache/" + entity + "/" + cacheKey + "/" + event;
- HttpPut request = new HttpPut(url);
+ HttpPut request;
+ String url;
+ if (cacheKey.contains("/")) {
+ url = baseUrl + "/cache/" + entity + "/" + event;
+ request = new HttpPut(url);
+ request.setEntity(new StringEntity(cacheKey, ContentType.create("application/json", "UTF-8")));
+ } else {
+ url = baseUrl + "/cache/" + entity + "/" + cacheKey + "/" + event;
+ request = new HttpPut(url);
+ }
HttpResponse response = null;
try {
diff --git a/server-base/src/main/java/org/apache/kylin/rest/broadcaster/BroadcasterReceiveServlet.java b/server-base/src/main/java/org/apache/kylin/rest/broadcaster/BroadcasterReceiveServlet.java
index 0a9c0bf..c6247f0 100644
--- a/server-base/src/main/java/org/apache/kylin/rest/broadcaster/BroadcasterReceiveServlet.java
+++ b/server-base/src/main/java/org/apache/kylin/rest/broadcaster/BroadcasterReceiveServlet.java
@@ -18,7 +18,9 @@
package org.apache.kylin.rest.broadcaster;
+import java.io.BufferedReader;
import java.io.IOException;
+import java.io.InputStreamReader;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
@@ -45,6 +47,7 @@ public class BroadcasterReceiveServlet extends HttpServlet {
}
private static final Pattern PATTERN = Pattern.compile("/(.+)/(.+)/(.+)");
+ private static final Pattern PATTERN2 = Pattern.compile("/(.+)/(.+)");
@Override
protected void doPut(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
@@ -61,14 +64,26 @@ public class BroadcasterReceiveServlet extends HttpServlet {
final String requestURI = req.getRequestURI();
final String substring = requestURI.substring(requestURI.indexOf(startString) + startString.length());
final Matcher matcher = PATTERN.matcher(substring);
+ final Matcher matcher2 = PATTERN2.matcher(substring);
+
if (matcher.matches()) {
String type = matcher.group(1);
- String name = matcher.group(2);
+ String cacheKey = matcher.group(2);
String event = matcher.group(3);
if (handler != null) {
- handler.handle(type, name, event);
+ handler.handle(type, cacheKey, event);
+ }
+ resp.getWriter().write("type:" + type + " name:" + cacheKey + " event:" + event);
+ } else if (matcher2.matches()) {
+ String type = matcher2.group(1);
+ String event = matcher2.group(2);
+ BufferedReader br = new BufferedReader(new InputStreamReader(req.getInputStream(), "utf-8"));
+ String cacheKey = br.readLine();
+ br.close();
+ if (handler != null) {
+ handler.handle(type, cacheKey, event);
}
- resp.getWriter().write("type:" + type + " name:" + name + " event:" + event);
+ resp.getWriter().write("type:" + type + " name:" + cacheKey + " event:" + event);
} else {
resp.getWriter().write("not valid uri");
}
diff --git a/server-base/src/main/java/org/apache/kylin/rest/controller/CacheController.java b/server-base/src/main/java/org/apache/kylin/rest/controller/CacheController.java
index 08b7cc4..ad6e71f 100644
--- a/server-base/src/main/java/org/apache/kylin/rest/controller/CacheController.java
+++ b/server-base/src/main/java/org/apache/kylin/rest/controller/CacheController.java
@@ -69,6 +69,16 @@ public class CacheController extends BasicController {
cacheService.notifyMetadataChange(entity, Broadcaster.Event.getEvent(event), cacheKey);
}
+ /**
+ * If cacheKey has "/", will lead to this method.
+ */
+ @RequestMapping(value = "/{entity}/{event}", method = { RequestMethod.PUT }, produces = { "application/json" })
+ @ResponseBody
+ public void wipeCacheWithRequestBody(@PathVariable String entity, @PathVariable String event,
+ @RequestBody String cacheKey) throws IOException {
+ cacheService.notifyMetadataChange(entity, Broadcaster.Event.getEvent(event), cacheKey);
+ }
+
@RequestMapping(value = "/announce/config", method = { RequestMethod.POST }, produces = { "application/json" })
public void hotLoadKylinConfig() throws IOException {
KylinConfig.getInstanceFromEnv().reloadFromSiteProperties();
--
To stop receiving notification emails like this one, please contact
liyang@apache.org.
[kylin] 07/11: KYLIN-3266 improve kylin CI BuildCubeWithEngine
coverage
Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
liyang pushed a commit to branch sync
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit a040c9a7f419fa1599d35f3becea3f4d67cfcef3
Author: Li Yang <li...@apache.org>
AuthorDate: Thu Feb 22 17:42:55 2018 +0800
KYLIN-3266 improve kylin CI BuildCubeWithEngine coverage
---
.../kylin/provision/BuildCubeWithEngine.java | 75 ++++++++++++----------
1 file changed, 42 insertions(+), 33 deletions(-)
diff --git a/kylin-it/src/test/java/org/apache/kylin/provision/BuildCubeWithEngine.java b/kylin-it/src/test/java/org/apache/kylin/provision/BuildCubeWithEngine.java
index 18a07cf..ce7b892 100644
--- a/kylin-it/src/test/java/org/apache/kylin/provision/BuildCubeWithEngine.java
+++ b/kylin-it/src/test/java/org/apache/kylin/provision/BuildCubeWithEngine.java
@@ -21,6 +21,7 @@ package org.apache.kylin.provision;
import java.io.File;
import java.io.IOException;
import java.lang.reflect.Method;
+import java.text.ParseException;
import java.text.SimpleDateFormat;
import java.util.List;
import java.util.Map;
@@ -107,9 +108,8 @@ public class BuildCubeWithEngine {
logger.info("Adding to classpath: " + new File(confDir).getAbsolutePath());
ClassUtil.addClasspath(new File(confDir).getAbsolutePath());
- String fastModeStr = System.getProperty("fastBuildMode");
- if (fastModeStr != null && fastModeStr.equalsIgnoreCase("true")) {
- fastBuildMode = true;
+ fastBuildMode = isFastBuildMode();
+ if (fastBuildMode) {
logger.info("Will use fast build mode");
} else {
logger.info("Will not use fast build mode");
@@ -145,6 +145,14 @@ public class BuildCubeWithEngine {
}
}
+ private static boolean isFastBuildMode() {
+ String fastModeStr = System.getProperty("fastBuildMode");
+ if (fastModeStr == null)
+ fastModeStr = System.getenv("KYLIN_CI_FASTBUILD");
+
+ return "true".equalsIgnoreCase(fastModeStr);
+ }
+
protected void deployEnv() throws IOException {
DeployUtil.initCliWorkDir();
DeployUtil.deployMetadata();
@@ -268,28 +276,43 @@ public class BuildCubeWithEngine {
private boolean testLeftJoinCube() throws Exception {
String cubeName = "ci_left_join_cube";
clearSegment(cubeName);
- // ci_left_join_cube has percentile which isn't supported by Spark engine now
- // updateCubeEngineType(cubeName);
+
+ // NOTE: ci_left_join_cube has percentile which isn't supported by Spark engine now
+ return doBuildAndMergeOnCube(cubeName);
+ }
+
+ private boolean doBuildAndMergeOnCube(String cubeName) throws ParseException, Exception {
SimpleDateFormat f = new SimpleDateFormat("yyyy-MM-dd");
f.setTimeZone(TimeZone.getTimeZone("GMT"));
long date1 = 0;
- long date2 = f.parse("2013-01-01").getTime();
+ long date2 = f.parse("2012-06-01").getTime();
long date3 = f.parse("2013-07-01").getTime();
long date4 = f.parse("2022-01-01").getTime();
+ long date5 = f.parse("2023-01-01").getTime();
+ long date6 = f.parse("2024-01-01").getTime();
- if (fastBuildMode) {
+ if (fastBuildMode)
return buildSegment(cubeName, date1, date4);
- } else {
- if (buildSegment(cubeName, date1, date2) == true) {
- if (buildSegment(cubeName, date2, date3) == true) {
- if (buildSegment(cubeName, date3, date4) == true) {
- return mergeSegment(cubeName, date1, date3); // don't merge all segments
- }
- }
- }
- }
- return false;
+
+ if (!buildSegment(cubeName, date1, date2))
+ return false;
+ if (!buildSegment(cubeName, date2, date3))
+ return false;
+ if (!buildSegment(cubeName, date3, date4))
+ return false;
+ if (!buildSegment(cubeName, date4, date5)) // one empty segment
+ return false;
+ if (!buildSegment(cubeName, date5, date6)) // another empty segment
+ return false;
+
+ if (!mergeSegment(cubeName, date2, date4)) // merge 2 normal segments
+ return false;
+ if (!mergeSegment(cubeName, date2, date5)) // merge normal and empty
+ return false;
+
+ // now have 2 normal segments [date1, date2) [date2, date5) and 1 empty segment [date5, date6)
+ return true;
}
@SuppressWarnings("unused")
@@ -298,22 +321,8 @@ public class BuildCubeWithEngine {
String cubeName = "ci_inner_join_cube";
clearSegment(cubeName);
- //updateCubeEngineType(cubeName);
-
- SimpleDateFormat f = new SimpleDateFormat("yyyy-MM-dd");
- f.setTimeZone(TimeZone.getTimeZone("GMT"));
- long date1 = 0;
- long date2 = f.parse("2022-07-01").getTime();
- long date3 = f.parse("2023-01-01").getTime();
-
- if (fastBuildMode) {
- return buildSegment(cubeName, date1, date3);
- } else {
- if (buildSegment(cubeName, date1, date2) == true) { // all-in-one build
- return buildSegment(cubeName, date2, date3); // empty segment
- }
- }
- return false;
+
+ return doBuildAndMergeOnCube(cubeName);
}
@SuppressWarnings("unused")
--
To stop receiving notification emails like this one, please contact
liyang@apache.org.
[kylin] 03/11: KYLIN-3234, add HDFS resource store implementation.
Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
liyang pushed a commit to branch sync
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit 4823d8b0f5b344f4b94b82bd7c8bc1d5bc7bc07a
Author: Jiatao Tao <24...@qq.com>
AuthorDate: Wed Feb 7 13:27:59 2018 +0800
KYLIN-3234, add HDFS resource store implementation.
---
.../common/persistence/FileResourceStore.java | 1 -
.../common/persistence/HDFSResourceStore.java | 37 ++++++++---
.../common/persistence/HDFSResourceStoreTest.java | 77 ++++++++++++++++++++++
3 files changed, 105 insertions(+), 10 deletions(-)
diff --git a/core-common/src/main/java/org/apache/kylin/common/persistence/FileResourceStore.java b/core-common/src/main/java/org/apache/kylin/common/persistence/FileResourceStore.java
index 6ac4536..38ccbdd 100644
--- a/core-common/src/main/java/org/apache/kylin/common/persistence/FileResourceStore.java
+++ b/core-common/src/main/java/org/apache/kylin/common/persistence/FileResourceStore.java
@@ -66,7 +66,6 @@ public class FileResourceStore extends ResourceStore {
String[] names = file.list();
// not a directory
if (names == null)
- // fixme should return empty set, like HBase implement.
return null;
String prefix = folderPath.endsWith("/") ? folderPath : folderPath + "/";
if (recursive) {
diff --git a/core-common/src/main/java/org/apache/kylin/common/persistence/HDFSResourceStore.java b/core-common/src/main/java/org/apache/kylin/common/persistence/HDFSResourceStore.java
index 7a3a933..8ad2540 100644
--- a/core-common/src/main/java/org/apache/kylin/common/persistence/HDFSResourceStore.java
+++ b/core-common/src/main/java/org/apache/kylin/common/persistence/HDFSResourceStore.java
@@ -31,7 +31,9 @@ import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.LocatedFileStatus;
import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.RemoteIterator;
import org.apache.kylin.common.KylinConfig;
import org.apache.kylin.common.StorageURL;
import org.apache.kylin.common.util.HadoopUtil;
@@ -89,20 +91,37 @@ public class HDFSResourceStore extends ResourceStore {
@Override
protected NavigableSet<String> listResourcesImpl(String folderPath, boolean recursive) throws IOException {
- if (recursive) {
- throw new IllegalArgumentException("Not support fullPath yet");
- }
Path p = getRealHDFSPath(folderPath);
+ String prefix = folderPath.endsWith("/") ? folderPath : folderPath + "/";
if (!fs.exists(p) || !fs.isDirectory(p)) {
return null;
}
- TreeSet<String> r = new TreeSet<>();
- FileStatus[] statuses = fs.listStatus(p);
- String prefix = folderPath.endsWith("/") ? folderPath : folderPath + "/";
- for (FileStatus status : statuses) {
- r.add(prefix + status.getPath().getName());
+ TreeSet<String> r;
+
+ if (recursive) {
+ r = getAllFilePath(p, prefix);
+ } else {
+ r = getFilePath(p, prefix);
+ }
+ return r.isEmpty() ? null : r;
+ }
+
+ private TreeSet<String> getFilePath(Path p, String prefix) throws IOException {
+ TreeSet<String> fileList = new TreeSet<>();
+ for (FileStatus fileStat : fs.listStatus(p)) {
+ fileList.add(prefix + fileStat.getPath().getName());
+ }
+ return fileList;
+ }
+
+ TreeSet<String> getAllFilePath(Path filePath, String prefix) throws IOException {
+ TreeSet<String> fileList = new TreeSet<>();
+ RemoteIterator<LocatedFileStatus> it = fs.listFiles(filePath, true);
+ while (it.hasNext()) {
+ String[] path = it.next().getPath().toString().split(prefix, 2);
+ fileList.add(prefix + path[1]);
}
- return r;
+ return fileList;
}
@Override
diff --git a/core-common/src/test/java/org/apache/kylin/common/persistence/HDFSResourceStoreTest.java b/core-common/src/test/java/org/apache/kylin/common/persistence/HDFSResourceStoreTest.java
new file mode 100644
index 0000000..e5e26a7
--- /dev/null
+++ b/core-common/src/test/java/org/apache/kylin/common/persistence/HDFSResourceStoreTest.java
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kylin.common.persistence;
+
+import java.io.File;
+import java.lang.reflect.Field;
+import java.util.TreeSet;
+
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.kylin.common.KylinConfig;
+import org.apache.kylin.common.StorageURL;
+import org.apache.kylin.common.util.HadoopUtil;
+import org.apache.kylin.common.util.LocalFileMetadataTestCase;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+public class HDFSResourceStoreTest extends LocalFileMetadataTestCase {
+
+ @Before
+ public void setup() throws Exception {
+ this.createTestMetadata();
+ }
+
+ @After
+ public void after() throws Exception {
+ this.cleanupTestMetadata();
+ }
+
+ @Test
+ public void testListResourcesImpl() throws Exception {
+ String path = "../examples/test_metadata/";
+ String cp = new File(path).getCanonicalFile().getPath();
+ FileSystem fs = HadoopUtil.getFileSystem(cp);
+ HDFSResourceStore store = new HDFSResourceStore(KylinConfig.getInstanceFromEnv(),
+ StorageURL.valueOf("hdfs@hdfs"));
+ Field field = store.getClass().getDeclaredField("fs");
+ field.setAccessible(true);
+ field.set(store, fs);
+
+ File f1 = new File(cp + "/resource/resource/e1.json");
+ File f2 = new File(cp + "/resource/resource/e2.json");
+ if (!f1.getParentFile().exists()) {
+ if (!f1.getParentFile().mkdirs()) {
+ throw new RuntimeException("Can not create dir.");
+ }
+ }
+ if (!(f1.createNewFile() && f2.createNewFile())) {
+ throw new RuntimeException("Can not create file.");
+ }
+
+ Path p = new Path(cp);
+ TreeSet<String> resources = store.getAllFilePath(new Path(p, "resource"), "/resource/");
+ TreeSet<String> expected = new TreeSet<>();
+ expected.add("/resource/resource/e1.json");
+ expected.add("/resource/resource/e2.json");
+ Assert.assertEquals(expected, resources);
+ }
+}
--
To stop receiving notification emails like this one, please contact
liyang@apache.org.
[kylin] 11/11: KYLIN-3251,
add a hook that can customize test_case_data and add
MetadataCleanupJob UT.
Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
liyang pushed a commit to branch sync
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit a05a4d0058976f7a940fdf840643ab7c820ff6e4
Author: Jiatao Tao <24...@qq.com>
AuthorDate: Tue Feb 27 16:56:50 2018 +0800
KYLIN-3251, add a hook that can customize test_case_data and add MetadataCleanupJob UT.
minor, refine test.
minor, refine test.
---
.../common/util/LocalFileMetadataTestCase.java | 77 ++-
.../org/apache/kylin/tool/MetadataCleanupJob.java | 56 +-
.../apache/kylin/tool/MetadataCleanupJobTest.java | 76 +++
tool/src/test/resources/test_meta/UUID | Bin 0 -> 38 bytes
.../test_meta/cube/ci_inner_join_cube.json | 60 +++
.../test_meta/cube_desc/ci_inner_join_cube.json | 574 +++++++++++++++++++++
.../04b6db34-b243-4780-855a-4c7ea4cee817.seq | Bin 0 -> 2605183 bytes
.../NAME/b58ce89d-3df6-46d1-8cff-2a9cfc8b7510.dict | Bin 0 -> 823 bytes
.../NAME/64ca8fea-b859-4e63-aea3-bfb4c6ee0c9d.dict | Bin 0 -> 844 bytes
.../execute/d861b8b7-c773-47ab-bb1e-c8782ae8d930 | 41 ++
.../d861b8b7-c773-47ab-bb1e-c8782ae8d930 | 13 +
.../d861b8b7-c773-47ab-bb1e-c8782ae8d930-00 | 13 +
.../d861b8b7-c773-47ab-bb1e-c8782ae8d930-01 | 13 +
tool/src/test/resources/test_meta/kylin.properties | 0
.../test_meta/model_desc/ci_inner_join_model.json | 231 +++++++++
.../test/resources/test_meta/project/default.json | 35 ++
.../test_meta/table/DEFAULT.TEST_ACCOUNT.json | 36 ++
.../table/DEFAULT.TEST_CATEGORY_GROUPINGS.json | 155 ++++++
.../test_meta/table/DEFAULT.TEST_COUNTRY.json | 26 +
.../test_meta/table/DEFAULT.TEST_KYLIN_FACT.json | 68 +++
.../test_meta/table/DEFAULT.TEST_ORDER.json | 35 ++
.../resources/test_meta/table/EDW.TEST_CAL_DT.json | 413 +++++++++++++++
.../test_meta/table/EDW.TEST_SELLER_TYPE_DIM.json | 46 ++
.../resources/test_meta/table/EDW.TEST_SITES.json | 51 ++
.../92456efe-9b79-4385-a5a3-e7f37b677bf7.snapshot | Bin 0 -> 16063 bytes
.../7ecdb07b-a8d0-49d8-892b-fe2dd75512ca.snapshot | Bin 0 -> 16061 bytes
26 files changed, 1968 insertions(+), 51 deletions(-)
diff --git a/core-common/src/test/java/org/apache/kylin/common/util/LocalFileMetadataTestCase.java b/core-common/src/test/java/org/apache/kylin/common/util/LocalFileMetadataTestCase.java
index 5e6b840..20e1df7 100644
--- a/core-common/src/test/java/org/apache/kylin/common/util/LocalFileMetadataTestCase.java
+++ b/core-common/src/test/java/org/apache/kylin/common/util/LocalFileMetadataTestCase.java
@@ -27,37 +27,40 @@ import org.apache.kylin.common.persistence.ResourceStore;
public class LocalFileMetadataTestCase extends AbstractKylinTestCase {
- public static String LOCALMETA_TEST_DATA = "../examples/test_case_data/localmeta";
- public static String LOCALMETA_TEMP_DATA = "../examples/test_metadata/";
+ public static final String LOCALMETA_TEST_DATA = "../examples/test_case_data/localmeta";
+ public static final String LOCALMETA_TEMP_DATA = "../examples/test_metadata/";
@Override
public void createTestMetadata(String... overlayMetadataDirs) {
- //overlayMetadataDirs is useless yet
- staticCreateTestMetadata(overlayMetadataDirs);
+ staticCreateTestMetadata(true, new OverlayMetaHook(overlayMetadataDirs));
}
public static void staticCreateTestMetadata(String... overlayMetadataDirs) {
- String testDataFolder = LOCALMETA_TEST_DATA;
- KylinConfig.destroyInstance();
+ staticCreateTestMetadata(true, new OverlayMetaHook(overlayMetadataDirs));
+ }
- String tempTestMetadataUrl = LOCALMETA_TEMP_DATA;
+ public static void staticCreateTestMetadata(boolean useTestMeta, MetadataTestCaseHook hook) {
try {
- FileUtils.deleteDirectory(new File(tempTestMetadataUrl));
- FileUtils.copyDirectory(new File(testDataFolder), new File(tempTestMetadataUrl));
+ KylinConfig.destroyInstance();
- for (String overlay : overlayMetadataDirs) {
- FileUtils.copyDirectory(new File(overlay), new File(tempTestMetadataUrl));
+ FileUtils.deleteDirectory(new File(LOCALMETA_TEMP_DATA));
+ if (useTestMeta) {
+ FileUtils.copyDirectory(new File(LOCALMETA_TEST_DATA), new File(LOCALMETA_TEMP_DATA));
}
- } catch (IOException e) {
- e.printStackTrace();
- }
- if (System.getProperty(KylinConfig.KYLIN_CONF) == null && System.getenv(KylinConfig.KYLIN_CONF) == null)
- System.setProperty(KylinConfig.KYLIN_CONF, tempTestMetadataUrl);
+ if (System.getProperty(KylinConfig.KYLIN_CONF) == null && System.getenv(KylinConfig.KYLIN_CONF) == null) {
+ System.setProperty(KylinConfig.KYLIN_CONF, LOCALMETA_TEMP_DATA);
+ }
- KylinConfig config = KylinConfig.getInstanceFromEnv();
- config.setMetadataUrl(tempTestMetadataUrl);
- config.setProperty("kylin.env.hdfs-working-dir", "file:///tmp/kylin");
+ if (hook != null) {
+ hook.hook();
+ }
+ KylinConfig config = KylinConfig.getInstanceFromEnv();
+ config.setMetadataUrl(LOCALMETA_TEMP_DATA);
+ config.setProperty("kylin.env.hdfs-working-dir", "file:///tmp/kylin");
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
}
public static void cleanAfterClass() {
@@ -90,4 +93,40 @@ public class LocalFileMetadataTestCase extends AbstractKylinTestCase {
protected ResourceStore getStore() {
return ResourceStore.getStore(KylinConfig.getInstanceFromEnv());
}
+
+ public interface MetadataTestCaseHook {
+ void hook() throws IOException;
+ }
+
+ public static class OverlayMetaHook implements MetadataTestCaseHook {
+ private String[] overlayMetadataDirs;
+
+ public OverlayMetaHook(String... overlayMetadataDirs) {
+ this.overlayMetadataDirs = overlayMetadataDirs;
+ }
+
+ @Override
+ public void hook() throws IOException {
+ //some test cases may require additional metadata entries besides standard test metadata in test_case_data/localmeta
+ for (String overlay : overlayMetadataDirs) {
+ FileUtils.copyDirectory(new File(overlay), new File(LOCALMETA_TEMP_DATA));
+ }
+ }
+ }
+
+ public static class ExcludeMetaHook implements MetadataTestCaseHook {
+ private String[] excludeMetadataDirs;
+
+ public ExcludeMetaHook(String... excludeMetadataDirs) {
+ this.excludeMetadataDirs = excludeMetadataDirs;
+ }
+
+ @Override
+ public void hook() throws IOException {
+ //some test cases may want exclude metadata entries besides standard test metadata in test_case_data/localmeta
+ for (String exclude : excludeMetadataDirs) {
+ FileUtils.deleteQuietly(new File(exclude));
+ }
+ }
+ }
}
diff --git a/tool/src/main/java/org/apache/kylin/tool/MetadataCleanupJob.java b/tool/src/main/java/org/apache/kylin/tool/MetadataCleanupJob.java
index 8e17645..d1c8423 100644
--- a/tool/src/main/java/org/apache/kylin/tool/MetadataCleanupJob.java
+++ b/tool/src/main/java/org/apache/kylin/tool/MetadataCleanupJob.java
@@ -41,30 +41,23 @@ import org.slf4j.LoggerFactory;
import com.google.common.collect.Lists;
import com.google.common.collect.Sets;
-/**
- */
public class MetadataCleanupJob extends AbstractHadoopJob {
@SuppressWarnings("static-access")
- private static final Option OPTION_DELETE = OptionBuilder.withArgName("delete").hasArg().isRequired(false).withDescription("Delete the unused metadata").create("delete");
+ private static final Option OPTION_DELETE = OptionBuilder.withArgName("delete").hasArg().isRequired(false)
+ .withDescription("Delete the unused metadata").create("delete");
@SuppressWarnings("static-access")
- private static final Option OPTION_THRESHOLD_FOR_JOB = OptionBuilder.withArgName("jobThreshold").hasArg().isRequired(false).withDescription("Specify how many days of job metadata keeping. Default 30 days").create("jobThreshold");
+ private static final Option OPTION_THRESHOLD_FOR_JOB = OptionBuilder.withArgName("jobThreshold").hasArg()
+ .isRequired(false).withDescription("Specify how many days of job metadata keeping. Default 30 days")
+ .create("jobThreshold");
protected static final Logger logger = LoggerFactory.getLogger(MetadataCleanupJob.class);
-
- boolean delete = false;
-
+ private boolean delete = false;
private KylinConfig config = null;
-
private static final long TIME_THREADSHOLD = 1 * 3600 * 1000L; // 1 hour
- private static final int DEFAULT_DAY_THREADSHOLD_FOR_JOB = 30 ; // 30 days
+ private static final int DEFAULT_DAY_THREADSHOLD_FOR_JOB = 30; // 30 days
- /*
- * (non-Javadoc)
- *
- * @see org.apache.hadoop.util.Tool#run(java.lang.String[])
- */
@Override
public int run(String[] args) throws Exception {
Options options = new Options();
@@ -78,11 +71,10 @@ public class MetadataCleanupJob extends AbstractHadoopJob {
logger.info("delete option value: '" + getOptionValue(OPTION_DELETE) + "'");
logger.info("jobThreshold option value: '" + getOptionValue(OPTION_THRESHOLD_FOR_JOB) + "'");
delete = Boolean.parseBoolean(getOptionValue(OPTION_DELETE));
-
- config = KylinConfig.getInstanceFromEnv();
-
- cleanup();
-
+ int jobThresholdDay = optionsHelper.hasOption(OPTION_THRESHOLD_FOR_JOB)
+ ? Integer.valueOf(optionsHelper.getOptionValue(OPTION_THRESHOLD_FOR_JOB))
+ : DEFAULT_DAY_THREADSHOLD_FOR_JOB;
+ cleanup(delete, jobThresholdDay);
return 0;
}
@@ -92,19 +84,17 @@ public class MetadataCleanupJob extends AbstractHadoopJob {
private boolean isOlderThanThreshold(long resourceTime) {
long currentTime = System.currentTimeMillis();
-
- if (currentTime - resourceTime > TIME_THREADSHOLD)
- return true;
- return false;
+ return currentTime - resourceTime > TIME_THREADSHOLD;
}
- public void cleanup() throws Exception {
- CubeManager cubeManager = CubeManager.getInstance(config);
-
+ public List<String> cleanup(boolean delete, int jobThresholdDay) throws Exception {
+ config = KylinConfig.getInstanceFromEnv();
+ CubeManager cubeManager = CubeManager.getInstance(KylinConfig.getInstanceFromEnv());
List<String> toDeleteResource = Lists.newArrayList();
// two level resources, snapshot tables and cube statistics
- for (String resourceRoot : new String[] { ResourceStore.SNAPSHOT_RESOURCE_ROOT, ResourceStore.CUBE_STATISTICS_ROOT }) {
+ for (String resourceRoot : new String[]{ResourceStore.SNAPSHOT_RESOURCE_ROOT,
+ ResourceStore.CUBE_STATISTICS_ROOT}) {
NavigableSet<String> snapshotTables = getStore().listResources(resourceRoot);
if (snapshotTables != null) {
@@ -153,10 +143,11 @@ public class MetadataCleanupJob extends AbstractHadoopJob {
for (ExecutablePO executable : allExecutable) {
long lastModified = executable.getLastModified();
ExecutableOutputPO output = executableDao.getJobOutput(executable.getUuid());
- int jobThresholdDay = optionsHelper.hasOption(OPTION_THRESHOLD_FOR_JOB) ? Integer.valueOf(optionsHelper.getOptionValue(OPTION_THRESHOLD_FOR_JOB)) : DEFAULT_DAY_THREADSHOLD_FOR_JOB;
long jobThresholdTime = jobThresholdDay * 24 * 3600 * 1000L;
- if (System.currentTimeMillis() - lastModified > jobThresholdTime && (ExecutableState.SUCCEED.toString().equals(output.getStatus()) || ExecutableState.DISCARDED.toString().equals(output.getStatus()))) {
+ if (System.currentTimeMillis() - lastModified > jobThresholdTime
+ && (ExecutableState.SUCCEED.toString().equals(output.getStatus())
+ || ExecutableState.DISCARDED.toString().equals(output.getStatus()))) {
toDeleteResource.add(ResourceStore.EXECUTE_RESOURCE_ROOT + "/" + executable.getUuid());
toDeleteResource.add(ResourceStore.EXECUTE_OUTPUT_RESOURCE_ROOT + "/" + executable.getUuid());
@@ -167,18 +158,19 @@ public class MetadataCleanupJob extends AbstractHadoopJob {
}
if (toDeleteResource.size() > 0) {
- logger.info("The following resources have no reference or is too old, will be cleaned from metadata store: \n");
+ logger.info(
+ "The following resources have no reference or is too old, will be cleaned from metadata store: \n");
for (String s : toDeleteResource) {
logger.info(s);
- if (delete == true) {
+ if (delete) {
getStore().deleteResource(s);
}
}
} else {
logger.info("No resource to be cleaned up from metadata store;");
}
-
+ return toDeleteResource;
}
public static void main(String[] args) throws Exception {
diff --git a/tool/src/test/java/org/apache/kylin/tool/MetadataCleanupJobTest.java b/tool/src/test/java/org/apache/kylin/tool/MetadataCleanupJobTest.java
new file mode 100644
index 0000000..ae3bef4
--- /dev/null
+++ b/tool/src/test/java/org/apache/kylin/tool/MetadataCleanupJobTest.java
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kylin.tool;
+
+import static org.apache.kylin.common.util.LocalFileMetadataTestCase.LOCALMETA_TEMP_DATA;
+import static org.apache.kylin.common.util.LocalFileMetadataTestCase.cleanAfterClass;
+import static org.apache.kylin.common.util.LocalFileMetadataTestCase.staticCreateTestMetadata;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.Collection;
+import java.util.List;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.kylin.common.util.LocalFileMetadataTestCase;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class MetadataCleanupJobTest {
+
+ @After
+ public void after() throws Exception {
+ cleanAfterClass();
+ }
+
+ @Test
+ public void testCleanUp() throws Exception {
+ staticCreateTestMetadata(false, new ResetTimeHook(1, "src/test/resources/test_meta"));
+ MetadataCleanupJob metadataCleanupJob = new MetadataCleanupJob();
+ List<String> cleanupList = metadataCleanupJob.cleanup(false, 30);
+ Assert.assertEquals(7, cleanupList.size());
+ }
+
+ @Test
+ public void testNotCleanUp() throws Exception {
+ staticCreateTestMetadata(false, new ResetTimeHook(System.currentTimeMillis(), "src/test/resources/test_meta"));
+ MetadataCleanupJob metadataCleanupJob = new MetadataCleanupJob();
+ List<String> cleanupList = metadataCleanupJob.cleanup(false, 30);
+ Assert.assertEquals(0, cleanupList.size());
+ }
+
+ private class ResetTimeHook extends LocalFileMetadataTestCase.OverlayMetaHook {
+ private long lastModified;
+
+ ResetTimeHook(long lastModified, String... overlayMetadataDirs) {
+ super(overlayMetadataDirs);
+ this.lastModified = lastModified;
+ }
+
+ @Override
+ public void hook() throws IOException {
+ super.hook();
+ Collection<File> files = FileUtils.listFiles(new File(LOCALMETA_TEMP_DATA), null, true);
+ for (File file : files) {
+ file.setLastModified(lastModified);
+ }
+ }
+ }
+}
diff --git a/tool/src/test/resources/test_meta/UUID b/tool/src/test/resources/test_meta/UUID
new file mode 100644
index 0000000..981981c
Binary files /dev/null and b/tool/src/test/resources/test_meta/UUID differ
diff --git a/tool/src/test/resources/test_meta/cube/ci_inner_join_cube.json b/tool/src/test/resources/test_meta/cube/ci_inner_join_cube.json
new file mode 100644
index 0000000..0b074b6
--- /dev/null
+++ b/tool/src/test/resources/test_meta/cube/ci_inner_join_cube.json
@@ -0,0 +1,60 @@
+{
+ "uuid" : "8372c3b7-a33e-4b69-83dd-0bb8b1f8117e",
+ "last_modified" : 0,
+ "name" : "ci_inner_join_cube",
+ "owner" : null,
+ "descriptor" : "ci_inner_join_cube",
+ "display_name" : "ci_inner_join_cube",
+ "cost" : 50,
+ "status" : "READY",
+ "segments" : [{
+ "uuid" : "adb2275e-c1d5-498b-b8ea-0201586868aa",
+ "name" : "19700101000000_20220701000000",
+ "storage_location_identifier" : "KYLIN_J9TE08D9IA",
+ "date_range_start" : 0,
+ "date_range_end" : 1656633600000,
+ "source_offset_start" : 0,
+ "source_offset_end" : 0,
+ "status" : "READY",
+ "size_kb" : 78965,
+ "input_records" : 9911,
+ "input_records_size" : 3955601,
+ "last_build_time" : 1511364122982,
+ "last_build_job_id" : "b58ed952-3041-4431-b560-09b78fa1fca0",
+ "create_time_utc" : 1511363249614,
+ "cuboid_shard_nums" : {
+ "7864320" : 2,
+ "8847375" : 6,
+ "41943504" : 6,
+ "458784" : 2,
+ "3145728" : 2,
+ "8323104" : 4,
+ "2097152" : 2,
+ "8388608" : 2,
+ "16252928" : 6,
+ "25230848" : 6,
+ "3670016" : 2,
+ "67108863" : 6
+ },
+ "total_shards" : 0,
+ "blackout_cuboids" : [ ],
+ "binary_signature" : null,
+ "dictionaries" : {
+ "BUYER_COUNTRY.NAME" : "/dict/DEFAULT.TEST_COUNTRY/NAME/64ca8fea-b859-4e63-aea3-bfb4c6ee0c9d.dict"
+ },
+ "snapshots" : {
+ "DEFAULT.TEST_COUNTRY" : "/table_snapshot/DEFAULT.TEST_COUNTRY/7ecdb07b-a8d0-49d8-892b-fe2dd75512ca.snapshot"
+ },
+ "rowkey_stats" : [ [ "TEST_KYLIN_FACT.LEAF_CATEG_ID", 134, 1 ], [ "TEST_CATEGORY_GROUPINGS.META_CATEG_NAME", 44, 1 ], [ "TEST_CATEGORY_GROUPINGS.CATEG_LVL2_NAME", 94, 1 ], [ "TEST_CATEGORY_GROUPINGS.CATEG_LVL3_NAME", 127, 1 ], [ "TEST_KYLIN_FACT.LSTG_SITE_ID", 8, 1 ], [ "TEST_KYLIN_FACT.SLR_SEGMENT_CD", 8, 1 ], [ "BUYER_ACCOUNT.ACCOUNT_BUYER_LEVEL", 5, 1 ], [ "BUYER_ACCOUNT.ACCOUNT_SELLER_LEVEL", 5, 1 ], [ "BUYER_ACCOUNT.ACCOUNT_COUNTRY", 8, 1 ], [ "BUYER_COUNTRY.NAME", 8, 1 ], [ "SE [...]
+ "additionalInfo" : {
+ "storageType" : "99",
+ "segmentPath" : "file:///Users/imad/mock_kylin/metastore/ci_cube/parquet/8372c3b7-a33e-4b69-83dd-0bb8b1f8117e/adb2275e-c1d5-498b-b8ea-0201586868aa/",
+ "storageFileCount" : "30",
+ "storageSizeBytes" : "80860160",
+ "tableIndexSegmentPath" : "file:///Users/imad/mock_kylin/metastore/ci_cube/parquet/2993d0d2-ee05-4e3a-8247-b121a1bdf7bc/adb2275e-c1d5-498b-b8ea-0201586868aa/",
+ "tableIndexFileCount" : "7",
+ "tableIndexStorageSizeBytes" : "8700293"
+ }
+ }],
+ "create_time" : null
+}
\ No newline at end of file
diff --git a/tool/src/test/resources/test_meta/cube_desc/ci_inner_join_cube.json b/tool/src/test/resources/test_meta/cube_desc/ci_inner_join_cube.json
new file mode 100644
index 0000000..809e188
--- /dev/null
+++ b/tool/src/test/resources/test_meta/cube_desc/ci_inner_join_cube.json
@@ -0,0 +1,574 @@
+{
+ "uuid": "3819ad72-3929-4dff-b59d-cd89a01238af",
+ "name": "ci_inner_join_cube",
+ "model_name": "ci_inner_join_model",
+ "description": null,
+ "dimensions": [
+ {
+ "name": "CAL_DT",
+ "table": "TEST_CAL_DT",
+ "column": "{FK}",
+ "derived": [
+ "WEEK_BEG_DT"
+ ]
+ },
+ {
+ "name": "ORDER_ID",
+ "table": "TEST_KYLIN_FACT",
+ "column": "ORDER_ID"
+ },
+ {
+ "name": "TEST_DATE_ENC",
+ "table": "TEST_ORDER",
+ "column": "TEST_DATE_ENC"
+ },
+ {
+ "name": "TEST_TIME_ENC",
+ "table": "TEST_ORDER",
+ "column": "TEST_TIME_ENC"
+ },
+ {
+ "name": "CATEGORY",
+ "table": "TEST_CATEGORY_GROUPINGS",
+ "column": "{FK}",
+ "derived": [
+ "USER_DEFINED_FIELD1",
+ "USER_DEFINED_FIELD3",
+ "UPD_DATE",
+ "UPD_USER"
+ ]
+ },
+ {
+ "name": "CATEGORY_HIERARCHY",
+ "table": "TEST_CATEGORY_GROUPINGS",
+ "column": "META_CATEG_NAME",
+ "derived": null
+ },
+ {
+ "name": "CATEGORY_HIERARCHY",
+ "table": "TEST_CATEGORY_GROUPINGS",
+ "column": "CATEG_LVL2_NAME",
+ "derived": null
+ },
+ {
+ "name": "CATEGORY_HIERARCHY",
+ "table": "TEST_CATEGORY_GROUPINGS",
+ "column": "CATEG_LVL3_NAME",
+ "derived": null
+ },
+ {
+ "name": "LSTG_FORMAT_NAME",
+ "table": "TEST_KYLIN_FACT",
+ "column": "LSTG_FORMAT_NAME",
+ "derived": null
+ },
+ {
+ "name": "SITE_ID",
+ "table": "TEST_SITES",
+ "column": "{FK}",
+ "derived": [
+ "SITE_NAME",
+ "CRE_USER"
+ ]
+ },
+ {
+ "name": "SELLER_TYPE_CD",
+ "table": "TEST_SELLER_TYPE_DIM",
+ "column": "{FK}",
+ "derived": [
+ "SELLER_TYPE_DESC"
+ ]
+ },
+ {
+ "name": "SELLER_ID",
+ "table": "TEST_KYLIN_FACT",
+ "column": "SELLER_ID"
+ },
+ {
+ "name": "SELLER_BUYER_LEVEL",
+ "table": "SELLER_ACCOUNT",
+ "column": "ACCOUNT_BUYER_LEVEL"
+ },
+ {
+ "name": "SELLER_SELLER_LEVEL",
+ "table": "SELLER_ACCOUNT",
+ "column": "ACCOUNT_SELLER_LEVEL"
+ },
+ {
+ "name": "SELLER_COUNTRY",
+ "table": "SELLER_ACCOUNT",
+ "column": "ACCOUNT_COUNTRY"
+ },
+ {
+ "name": "SELLER_COUNTRY_NAME",
+ "table": "SELLER_COUNTRY",
+ "column": "NAME"
+ },
+ {
+ "name": "BUYER_ID",
+ "table": "TEST_ORDER",
+ "column": "BUYER_ID"
+ },
+ {
+ "name": "BUYER_BUYER_LEVEL",
+ "table": "BUYER_ACCOUNT",
+ "column": "ACCOUNT_BUYER_LEVEL"
+ },
+ {
+ "name": "BUYER_SELLER_LEVEL",
+ "table": "BUYER_ACCOUNT",
+ "column": "ACCOUNT_SELLER_LEVEL"
+ },
+ {
+ "name": "BUYER_COUNTRY",
+ "table": "BUYER_ACCOUNT",
+ "column": "ACCOUNT_COUNTRY"
+ },
+ {
+ "name": "BUYER_COUNTRY_NAME",
+ "table": "BUYER_COUNTRY",
+ "column": "NAME"
+ }
+ ],
+ "measures": [
+ {
+ "name": "TRANS_CNT",
+ "function": {
+ "expression": "COUNT",
+ "parameter": {
+ "type": "constant",
+ "value": "1"
+ },
+ "returntype": "bigint"
+ }
+ },
+ {
+ "name": "ITEM_COUNT_SUM",
+ "function": {
+ "expression": "SUM",
+ "parameter": {
+ "type": "column",
+ "value": "TEST_KYLIN_FACT.ITEM_COUNT"
+ },
+ "returntype": "bigint"
+ }
+ },
+ {
+ "name": "GMV_SUM",
+ "function": {
+ "expression": "SUM",
+ "parameter": {
+ "type": "column",
+ "value": "TEST_KYLIN_FACT.PRICE"
+ },
+ "returntype": "decimal(19,4)"
+ }
+ },
+ {
+ "name": "GMV_MIN",
+ "function": {
+ "expression": "MIN",
+ "parameter": {
+ "type": "column",
+ "value": "TEST_KYLIN_FACT.PRICE"
+ },
+ "returntype": "decimal(19,4)"
+ }
+ },
+ {
+ "name": "GMV_MAX",
+ "function": {
+ "expression": "MAX",
+ "parameter": {
+ "type": "column",
+ "value": "TEST_KYLIN_FACT.PRICE"
+ },
+ "returntype": "decimal(19,4)"
+ }
+ },
+ {
+ "name": "SELLER_HLL",
+ "function": {
+ "expression": "COUNT_DISTINCT",
+ "parameter": {
+ "type": "column",
+ "value": "TEST_KYLIN_FACT.SELLER_ID"
+ },
+ "returntype": "hllc(10)"
+ }
+ },
+ {
+ "name": "SELLER_FORMAT_HLL",
+ "function": {
+ "expression": "COUNT_DISTINCT",
+ "parameter": {
+ "type": "column",
+ "value": "TEST_KYLIN_FACT.LSTG_FORMAT_NAME",
+ "next_parameter": {
+ "type": "column",
+ "value": "TEST_KYLIN_FACT.SELLER_ID"
+ }
+ },
+ "returntype": "hllc(10)"
+ }
+ },
+ {
+ "name": "TOP_SELLER",
+ "function": {
+ "expression": "TOP_N",
+ "parameter": {
+ "type": "column",
+ "value": "TEST_KYLIN_FACT.PRICE",
+ "next_parameter": {
+ "type": "column",
+ "value": "TEST_KYLIN_FACT.SELLER_ID"
+ }
+ },
+ "returntype": "topn(100, 4)",
+ "configuration": {
+ "topn.encoding.TEST_KYLIN_FACT.SELLER_ID": "int:4"
+ }
+ }
+ },
+ {
+ "name": "TEST_COUNT_DISTINCT_BITMAP",
+ "function": {
+ "expression": "COUNT_DISTINCT",
+ "parameter": {
+ "type": "column",
+ "value": "TEST_KYLIN_FACT.TEST_COUNT_DISTINCT_BITMAP"
+ },
+ "returntype": "bitmap"
+ }
+ },
+ {
+ "name": "TEST_EXTENDED_COLUMN",
+ "function": {
+ "expression": "EXTENDED_COLUMN",
+ "parameter": {
+ "type": "column",
+ "value": "TEST_KYLIN_FACT.ORDER_ID",
+ "next_parameter": {
+ "type": "column",
+ "value": "TEST_ORDER.TEST_EXTENDED_COLUMN"
+ }
+ },
+ "returntype": "extendedcolumn(100)"
+ }
+ },
+ {
+ "name": "BUYER_CONTACT",
+ "function": {
+ "expression": "EXTENDED_COLUMN",
+ "parameter": {
+ "type": "column",
+ "value": "TEST_ORDER.BUYER_ID",
+ "next_parameter": {
+ "type": "column",
+ "value": "BUYER_ACCOUNT.ACCOUNT_CONTACT"
+ }
+ },
+ "returntype": "extendedcolumn(100)"
+ }
+ },
+ {
+ "name": "SELLER_CONTACT",
+ "function": {
+ "expression": "EXTENDED_COLUMN",
+ "parameter": {
+ "type": "column",
+ "value": "TEST_KYLIN_FACT.SELLER_ID",
+ "next_parameter": {
+ "type": "column",
+ "value": "SELLER_ACCOUNT.ACCOUNT_CONTACT"
+ }
+ },
+ "returntype": "extendedcolumn(100)"
+ }
+ },
+ {
+ "name": "TRANS_ID_RAW",
+ "function": {
+ "expression": "RAW",
+ "parameter": {
+ "type": "column",
+ "value": "TEST_KYLIN_FACT.TRANS_ID"
+ },
+ "returntype": "raw"
+ }
+ },
+ {
+ "name": "PRICE_RAW",
+ "function": {
+ "expression": "RAW",
+ "parameter": {
+ "type": "column",
+ "value": "TEST_KYLIN_FACT.PRICE"
+ },
+ "returntype": "raw"
+ }
+ },
+ {
+ "name": "CAL_DT_RAW",
+ "function": {
+ "expression": "RAW",
+ "parameter": {
+ "type": "column",
+ "value": "TEST_KYLIN_FACT.CAL_DT"
+ },
+ "returntype": "raw"
+ }
+ }
+ ],
+ "dictionaries": [
+ {
+ "column": "TEST_KYLIN_FACT.TEST_COUNT_DISTINCT_BITMAP",
+ "builder": "org.apache.kylin.dict.global.SegmentAppendTrieDictBuilder"
+ }
+ ],
+ "rowkey": {
+ "rowkey_columns": [
+ {
+ "column": "TEST_KYLIN_FACT.SELLER_ID",
+ "encoding": "int:4"
+ },
+ {
+ "column": "TEST_KYLIN_FACT.ORDER_ID",
+ "encoding": "int:4"
+ },
+ {
+ "column": "TEST_KYLIN_FACT.CAL_DT",
+ "encoding": "dict"
+ },
+ {
+ "column": "TEST_KYLIN_FACT.LEAF_CATEG_ID",
+ "encoding": "dict"
+ },
+ {
+ "column": "TEST_CATEGORY_GROUPINGS.META_CATEG_NAME",
+ "encoding": "dict"
+ },
+ {
+ "column": "TEST_CATEGORY_GROUPINGS.CATEG_LVL2_NAME",
+ "encoding": "dict"
+ },
+ {
+ "column": "TEST_CATEGORY_GROUPINGS.CATEG_LVL3_NAME",
+ "encoding": "dict"
+ },
+ {
+ "column": "TEST_KYLIN_FACT.LSTG_FORMAT_NAME",
+ "encoding": "fixed_length:12"
+ },
+ {
+ "column": "TEST_KYLIN_FACT.LSTG_SITE_ID",
+ "encoding": "dict"
+ },
+ {
+ "column": "TEST_KYLIN_FACT.SLR_SEGMENT_CD",
+ "encoding": "dict"
+ },
+ {
+ "column": "TEST_ORDER.TEST_TIME_ENC",
+ "encoding": "time"
+ },
+ {
+ "column": "TEST_ORDER.TEST_DATE_ENC",
+ "encoding": "date"
+ },
+ {
+ "column": "TEST_ORDER.BUYER_ID",
+ "encoding": "int:4"
+ },
+ {
+ "column": "BUYER_ACCOUNT.ACCOUNT_BUYER_LEVEL",
+ "encoding": "dict"
+ },
+ {
+ "column": "BUYER_ACCOUNT.ACCOUNT_SELLER_LEVEL",
+ "encoding": "dict"
+ },
+ {
+ "column": "BUYER_ACCOUNT.ACCOUNT_COUNTRY",
+ "encoding": "dict"
+ },
+ {
+ "column": "BUYER_COUNTRY.NAME",
+ "encoding": "dict"
+ },
+ {
+ "column": "SELLER_ACCOUNT.ACCOUNT_BUYER_LEVEL",
+ "encoding": "dict"
+ },
+ {
+ "column": "SELLER_ACCOUNT.ACCOUNT_SELLER_LEVEL",
+ "encoding": "dict"
+ },
+ {
+ "column": "SELLER_ACCOUNT.ACCOUNT_COUNTRY",
+ "encoding": "dict"
+ },
+ {
+ "column": "SELLER_COUNTRY.NAME",
+ "encoding": "dict"
+ }
+ ]
+ },
+ "signature": null,
+ "last_modified": 1448959801311,
+ "null_string": null,
+ "hbase_mapping": {
+ "column_family": [
+ {
+ "name": "f1",
+ "columns": [
+ {
+ "qualifier": "m",
+ "measure_refs": [
+ "TRANS_CNT",
+ "ITEM_COUNT_SUM",
+ "GMV_SUM",
+ "GMV_MIN",
+ "GMV_MAX"
+ ]
+ }
+ ]
+ },
+ {
+ "name": "f2",
+ "columns": [
+ {
+ "qualifier": "m",
+ "measure_refs": [
+ "SELLER_HLL",
+ "SELLER_FORMAT_HLL",
+ "TOP_SELLER",
+ "TEST_COUNT_DISTINCT_BITMAP"
+ ]
+ }
+ ]
+ },
+ {
+ "name": "f3",
+ "columns": [
+ {
+ "qualifier": "m",
+ "measure_refs": [
+ "TEST_EXTENDED_COLUMN",
+ "BUYER_CONTACT",
+ "SELLER_CONTACT",
+ "TRANS_ID_RAW",
+ "PRICE_RAW",
+ "CAL_DT_RAW"
+ ]
+ }
+ ]
+ }
+ ]
+ },
+ "aggregation_groups": [
+ {
+ "includes": [
+ "TEST_KYLIN_FACT.CAL_DT",
+ "TEST_KYLIN_FACT.LEAF_CATEG_ID",
+ "TEST_KYLIN_FACT.LSTG_FORMAT_NAME",
+ "TEST_KYLIN_FACT.LSTG_SITE_ID",
+ "TEST_KYLIN_FACT.SLR_SEGMENT_CD",
+ "TEST_CATEGORY_GROUPINGS.META_CATEG_NAME",
+ "TEST_CATEGORY_GROUPINGS.CATEG_LVL2_NAME",
+ "TEST_CATEGORY_GROUPINGS.CATEG_LVL3_NAME"
+ ],
+ "select_rule": {
+ "hierarchy_dims": [
+ [
+ "TEST_CATEGORY_GROUPINGS.META_CATEG_NAME",
+ "TEST_CATEGORY_GROUPINGS.CATEG_LVL2_NAME",
+ "TEST_CATEGORY_GROUPINGS.CATEG_LVL3_NAME",
+ "TEST_KYLIN_FACT.LEAF_CATEG_ID"
+ ]
+ ],
+ "mandatory_dims": [],
+ "joint_dims": [
+ [
+ "TEST_KYLIN_FACT.LSTG_FORMAT_NAME",
+ "TEST_KYLIN_FACT.LSTG_SITE_ID",
+ "TEST_KYLIN_FACT.SLR_SEGMENT_CD"
+ ]
+ ],
+ "dim_cap": 3
+ }
+ },
+ {
+ "includes": [
+ "TEST_KYLIN_FACT.CAL_DT",
+ "TEST_KYLIN_FACT.LEAF_CATEG_ID",
+ "TEST_KYLIN_FACT.LSTG_FORMAT_NAME",
+ "TEST_KYLIN_FACT.LSTG_SITE_ID",
+ "TEST_KYLIN_FACT.SLR_SEGMENT_CD",
+ "TEST_CATEGORY_GROUPINGS.META_CATEG_NAME",
+ "TEST_CATEGORY_GROUPINGS.CATEG_LVL2_NAME",
+ "TEST_CATEGORY_GROUPINGS.CATEG_LVL3_NAME",
+ "TEST_KYLIN_FACT.SELLER_ID",
+ "SELLER_ACCOUNT.ACCOUNT_BUYER_LEVEL",
+ "SELLER_ACCOUNT.ACCOUNT_SELLER_LEVEL",
+ "SELLER_ACCOUNT.ACCOUNT_COUNTRY",
+ "SELLER_COUNTRY.NAME",
+ "TEST_KYLIN_FACT.ORDER_ID",
+ "TEST_ORDER.TEST_DATE_ENC",
+ "TEST_ORDER.TEST_TIME_ENC",
+ "TEST_ORDER.BUYER_ID",
+ "BUYER_ACCOUNT.ACCOUNT_BUYER_LEVEL",
+ "BUYER_ACCOUNT.ACCOUNT_SELLER_LEVEL",
+ "BUYER_ACCOUNT.ACCOUNT_COUNTRY",
+ "BUYER_COUNTRY.NAME"
+ ],
+ "select_rule": {
+ "hierarchy_dims": [],
+ "mandatory_dims": [
+ "TEST_KYLIN_FACT.CAL_DT"
+ ],
+ "joint_dims": [
+ [
+ "TEST_CATEGORY_GROUPINGS.META_CATEG_NAME",
+ "TEST_CATEGORY_GROUPINGS.CATEG_LVL2_NAME",
+ "TEST_CATEGORY_GROUPINGS.CATEG_LVL3_NAME",
+ "TEST_KYLIN_FACT.LEAF_CATEG_ID"
+ ],
+ [
+ "TEST_KYLIN_FACT.LSTG_FORMAT_NAME",
+ "TEST_KYLIN_FACT.LSTG_SITE_ID",
+ "TEST_KYLIN_FACT.SLR_SEGMENT_CD"
+ ],
+ [
+ "TEST_KYLIN_FACT.SELLER_ID",
+ "SELLER_ACCOUNT.ACCOUNT_BUYER_LEVEL",
+ "SELLER_ACCOUNT.ACCOUNT_SELLER_LEVEL",
+ "SELLER_ACCOUNT.ACCOUNT_COUNTRY",
+ "SELLER_COUNTRY.NAME"
+ ],
+ [
+ "TEST_KYLIN_FACT.ORDER_ID",
+ "TEST_ORDER.TEST_DATE_ENC",
+ "TEST_ORDER.TEST_TIME_ENC",
+ "TEST_ORDER.BUYER_ID",
+ "BUYER_ACCOUNT.ACCOUNT_BUYER_LEVEL",
+ "BUYER_ACCOUNT.ACCOUNT_SELLER_LEVEL",
+ "BUYER_ACCOUNT.ACCOUNT_COUNTRY",
+ "BUYER_COUNTRY.NAME"
+ ]
+ ],
+ "dim_cap": 5
+ }
+ }
+ ],
+ "notify_list": null,
+ "status_need_notify": [],
+ "auto_merge_time_ranges": null,
+ "retention_range": 0,
+ "engine_type": 4,
+ "storage_type": 2,
+ "override_kylin_properties": {
+ "kylin.cube.algorithm": "LAYER"
+ },
+ "partition_date_start": 0
+}
diff --git a/tool/src/test/resources/test_meta/cube_statistics/kylin_sales_cube/04b6db34-b243-4780-855a-4c7ea4cee817.seq b/tool/src/test/resources/test_meta/cube_statistics/kylin_sales_cube/04b6db34-b243-4780-855a-4c7ea4cee817.seq
new file mode 100644
index 0000000..5dc08b8
Binary files /dev/null and b/tool/src/test/resources/test_meta/cube_statistics/kylin_sales_cube/04b6db34-b243-4780-855a-4c7ea4cee817.seq differ
diff --git a/tool/src/test/resources/test_meta/dict/DEFAULT.KYLIN_COUNTRY/NAME/b58ce89d-3df6-46d1-8cff-2a9cfc8b7510.dict b/tool/src/test/resources/test_meta/dict/DEFAULT.KYLIN_COUNTRY/NAME/b58ce89d-3df6-46d1-8cff-2a9cfc8b7510.dict
new file mode 100644
index 0000000..7dcca2a
Binary files /dev/null and b/tool/src/test/resources/test_meta/dict/DEFAULT.KYLIN_COUNTRY/NAME/b58ce89d-3df6-46d1-8cff-2a9cfc8b7510.dict differ
diff --git a/tool/src/test/resources/test_meta/dict/DEFAULT.TEST_COUNTRY/NAME/64ca8fea-b859-4e63-aea3-bfb4c6ee0c9d.dict b/tool/src/test/resources/test_meta/dict/DEFAULT.TEST_COUNTRY/NAME/64ca8fea-b859-4e63-aea3-bfb4c6ee0c9d.dict
new file mode 100644
index 0000000..86a5154
Binary files /dev/null and b/tool/src/test/resources/test_meta/dict/DEFAULT.TEST_COUNTRY/NAME/64ca8fea-b859-4e63-aea3-bfb4c6ee0c9d.dict differ
diff --git a/tool/src/test/resources/test_meta/execute/d861b8b7-c773-47ab-bb1e-c8782ae8d930 b/tool/src/test/resources/test_meta/execute/d861b8b7-c773-47ab-bb1e-c8782ae8d930
new file mode 100644
index 0000000..ed6d6fa
--- /dev/null
+++ b/tool/src/test/resources/test_meta/execute/d861b8b7-c773-47ab-bb1e-c8782ae8d930
@@ -0,0 +1,41 @@
+{
+ "uuid" : "d861b8b7-c773-47ab-bb1e-c8782ae8d930",
+ "last_modified" : 0,
+ "version" : "2.3.0.20500",
+ "name" : "BUILD CUBE - ss - FULL_BUILD - GMT+08:00 2017-10-13 10:57:22",
+ "tasks" : [ {
+"uuid" : "d861b8b7-c773-47ab-bb1e-c8782ae8d930-00",
+ "last_modified" : 0,
+ "version" : "2.3.0.20500",
+ "name" : "Create Intermediate Flat Hive Table",
+ "tasks" : null,
+ "type" : "org.apache.kylin.source.hive.CreateFlatHiveTableStep",
+ "params" : {
+ "HiveInit" : "USE default;\n",
+ "HiveRedistributeData" : "DROP TABLE IF EXISTS kylin_intermediate_ss_a110ac52_9a91_49fe_944a_346d61e54115;\nCREATE EXTERNAL TABLE IF NOT EXISTS kylin_intermediate_ss_a110ac52_9a91_49fe_944a_346d61e54115\n(\nKYLIN_SALES_LSTG_FORMAT_NAME string\n,KYLIN_SALES_PRICE decimal(19,4)\n)\nSTORED AS SEQUENCEFILE\nLOCATION 'hdfs://sandbox.hortonworks.com:8020/Users/jiatao.tao/Documents/meta5/kylin-d861b8b7-c773-47ab-bb1e-c8782ae8d930/kylin_intermediate_ss_a110ac52_9a91_49fe_944a_346d61e54115' [...]
+ "cubeName" : "ss"
+ }
+ }, {
+ "uuid" : "d861b8b7-c773-47ab-bb1e-c8782ae8d930-01",
+ "last_modified" : 0,
+ "version" : "2.3.0.20500",
+ "name" : "Redistribute Flat Hive Table",
+ "tasks" : null,
+ "type" : "org.apache.kylin.source.hive.HiveMRInput$RedistributeFlatHiveTableStep",
+ "params" : {
+ "HiveInit" : "USE default;\n",
+ "HiveRedistributeData" : "INSERT OVERWRITE TABLE kylin_intermediate_ss_a110ac52_9a91_49fe_944a_346d61e54115 SELECT * FROM kylin_intermediate_ss_a110ac52_9a91_49fe_944a_346d61e54115 DISTRIBUTE BY RAND();\n",
+ "cubeName" : "ss",
+ "intermediateTable" : "kylin_intermediate_ss_a110ac52_9a91_49fe_944a_346d61e54115"
+ }
+ }],
+ "type" : "org.apache.kylin.engine.mr.CubingJob",
+ "params" : {
+ "submitter" : "ADMIN",
+ "envName" : "DEV",
+ "segmentId" : "a110ac52-9a91-49fe-944a-346d61e54115",
+ "notify_list" : "",
+ "projectName" : "learn_kylin",
+ "cubeName" : "ss"
+ }
+}
\ No newline at end of file
diff --git a/tool/src/test/resources/test_meta/execute_output/d861b8b7-c773-47ab-bb1e-c8782ae8d930 b/tool/src/test/resources/test_meta/execute_output/d861b8b7-c773-47ab-bb1e-c8782ae8d930
new file mode 100644
index 0000000..fab62b5
--- /dev/null
+++ b/tool/src/test/resources/test_meta/execute_output/d861b8b7-c773-47ab-bb1e-c8782ae8d930
@@ -0,0 +1,13 @@
+{
+ "uuid" : "d861b8b7-c773-47ab-bb1e-c8782ae8d930",
+ "last_modified" : 1507863555561,
+ "version" : "2.2.0.20500",
+ "content" : null,
+ "status" : "SUCCEED",
+ "info" : {
+ "startTime" : "1505914710093",
+ "mapReduceWaitTime" : "243712",
+ "algorithm" : "LAYER",
+ "endTime" : "1507863555561"
+ }
+}
\ No newline at end of file
diff --git a/tool/src/test/resources/test_meta/execute_output/d861b8b7-c773-47ab-bb1e-c8782ae8d930-00 b/tool/src/test/resources/test_meta/execute_output/d861b8b7-c773-47ab-bb1e-c8782ae8d930-00
new file mode 100644
index 0000000..19e88f7
--- /dev/null
+++ b/tool/src/test/resources/test_meta/execute_output/d861b8b7-c773-47ab-bb1e-c8782ae8d930-00
@@ -0,0 +1,13 @@
+{
+ "uuid" : "d861b8b7-c773-47ab-bb1e-c8782ae8d930-00",
+ "last_modified" : 1507863555561,
+ "version" : "2.2.0.20500",
+ "content" : null,
+ "status" : "SUCCEED",
+ "info" : {
+ "startTime" : "1505914710093",
+ "mapReduceWaitTime" : "243712",
+ "algorithm" : "LAYER",
+ "endTime" : "1507863555561"
+ }
+}
\ No newline at end of file
diff --git a/tool/src/test/resources/test_meta/execute_output/d861b8b7-c773-47ab-bb1e-c8782ae8d930-01 b/tool/src/test/resources/test_meta/execute_output/d861b8b7-c773-47ab-bb1e-c8782ae8d930-01
new file mode 100644
index 0000000..0759ee3
--- /dev/null
+++ b/tool/src/test/resources/test_meta/execute_output/d861b8b7-c773-47ab-bb1e-c8782ae8d930-01
@@ -0,0 +1,13 @@
+{
+ "uuid" : "d861b8b7-c773-47ab-bb1e-c8782ae8d930-01",
+ "last_modified" : 1507863555561,
+ "version" : "2.2.0.20500",
+ "content" : null,
+ "status" : "SUCCEED",
+ "info" : {
+ "startTime" : "1505914710093",
+ "mapReduceWaitTime" : "243712",
+ "algorithm" : "LAYER",
+ "endTime" : "1507863555561"
+ }
+}
\ No newline at end of file
diff --git a/tool/src/test/resources/test_meta/kylin.properties b/tool/src/test/resources/test_meta/kylin.properties
new file mode 100644
index 0000000..e69de29
diff --git a/tool/src/test/resources/test_meta/model_desc/ci_inner_join_model.json b/tool/src/test/resources/test_meta/model_desc/ci_inner_join_model.json
new file mode 100644
index 0000000..a3c3c05
--- /dev/null
+++ b/tool/src/test/resources/test_meta/model_desc/ci_inner_join_model.json
@@ -0,0 +1,231 @@
+{
+ "uuid": "72ab4ee2-2cdb-4b07-b39e-4c298563ae27",
+ "name": "ci_inner_join_model",
+ "fact_table": "DEFAULT.TEST_KYLIN_FACT",
+ "join_tables": [
+ {
+ "table": "DEFAULT.TEST_ORDER",
+ "kind": "FACT",
+ "join": {
+ "type": "INNER",
+ "primary_key": [
+ "TEST_ORDER.ORDER_ID"
+ ],
+ "foreign_key": [
+ "TEST_KYLIN_FACT.ORDER_ID"
+ ]
+ }
+ },
+ {
+ "table": "DEFAULT.TEST_ACCOUNT",
+ "alias": "BUYER_ACCOUNT",
+ "kind": "FACT",
+ "join": {
+ "type": "INNER",
+ "primary_key": [
+ "BUYER_ACCOUNT.ACCOUNT_ID"
+ ],
+ "foreign_key": [
+ "TEST_ORDER.BUYER_ID"
+ ]
+ }
+ },
+ {
+ "table": "EDW.TEST_CAL_DT",
+ "join": {
+ "type": "INNER",
+ "primary_key": [
+ "TEST_CAL_DT.CAL_DT"
+ ],
+ "foreign_key": [
+ "TEST_KYLIN_FACT.CAL_DT"
+ ]
+ }
+ },
+ {
+ "table": "DEFAULT.TEST_CATEGORY_GROUPINGS",
+ "join": {
+ "type": "INNER",
+ "primary_key": [
+ "TEST_CATEGORY_GROUPINGS.LEAF_CATEG_ID",
+ "TEST_CATEGORY_GROUPINGS.SITE_ID"
+ ],
+ "foreign_key": [
+ "TEST_KYLIN_FACT.LEAF_CATEG_ID",
+ "TEST_KYLIN_FACT.LSTG_SITE_ID"
+ ]
+ }
+ },
+ {
+ "table": "EDW.TEST_SITES",
+ "join": {
+ "type": "INNER",
+ "primary_key": [
+ "TEST_SITES.SITE_ID"
+ ],
+ "foreign_key": [
+ "TEST_KYLIN_FACT.LSTG_SITE_ID"
+ ]
+ }
+ },
+ {
+ "table": "EDW.TEST_SELLER_TYPE_DIM",
+ "join": {
+ "type": "INNER",
+ "primary_key": [
+ "TEST_SELLER_TYPE_DIM.SELLER_TYPE_CD"
+ ],
+ "foreign_key": [
+ "TEST_KYLIN_FACT.SLR_SEGMENT_CD"
+ ]
+ }
+ },
+ {
+ "table": "DEFAULT.TEST_COUNTRY",
+ "alias": "BUYER_COUNTRY",
+ "join": {
+ "type": "INNER",
+ "primary_key": [
+ "BUYER_COUNTRY.COUNTRY"
+ ],
+ "foreign_key": [
+ "BUYER_ACCOUNT.ACCOUNT_COUNTRY"
+ ]
+ }
+ },
+ {
+ "table": "DEFAULT.TEST_COUNTRY",
+ "alias": "SELLER_COUNTRY",
+ "join": {
+ "type": "INNER",
+ "primary_key": [
+ "SELLER_COUNTRY.COUNTRY"
+ ],
+ "foreign_key": [
+ "SELLER_ACCOUNT.ACCOUNT_COUNTRY"
+ ]
+ }
+ },
+ {
+ "table": "DEFAULT.TEST_ACCOUNT",
+ "alias": "SELLER_ACCOUNT",
+ "kind": "FACT",
+ "join": {
+ "type": "INNER",
+ "primary_key": [
+ "SELLER_ACCOUNT.ACCOUNT_ID"
+ ],
+ "foreign_key": [
+ "TEST_KYLIN_FACT.SELLER_ID"
+ ]
+ }
+ }
+ ],
+ "dimensions": [
+ {
+ "table": "TEST_KYLIN_FACT",
+ "columns": [
+ "TRANS_ID",
+ "ORDER_ID",
+ "CAL_DT",
+ "LSTG_FORMAT_NAME",
+ "LSTG_SITE_ID",
+ "LEAF_CATEG_ID",
+ "SLR_SEGMENT_CD",
+ "SELLER_ID",
+ "TEST_COUNT_DISTINCT_BITMAP"
+ ]
+ },
+ {
+ "table": "TEST_ORDER",
+ "columns": [
+ "ORDER_ID",
+ "BUYER_ID",
+ "TEST_DATE_ENC",
+ "TEST_TIME_ENC",
+ "TEST_EXTENDED_COLUMN"
+ ]
+ },
+ {
+ "table": "BUYER_ACCOUNT",
+ "columns": [
+ "ACCOUNT_ID",
+ "ACCOUNT_BUYER_LEVEL",
+ "ACCOUNT_SELLER_LEVEL",
+ "ACCOUNT_COUNTRY",
+ "ACCOUNT_CONTACT"
+ ]
+ },
+ {
+ "table": "SELLER_ACCOUNT",
+ "columns": [
+ "ACCOUNT_ID",
+ "ACCOUNT_BUYER_LEVEL",
+ "ACCOUNT_SELLER_LEVEL",
+ "ACCOUNT_COUNTRY",
+ "ACCOUNT_CONTACT"
+ ]
+ },
+ {
+ "table": "TEST_CATEGORY_GROUPINGS",
+ "columns": [
+ "LEAF_CATEG_ID",
+ "SITE_ID",
+ "META_CATEG_NAME",
+ "CATEG_LVL2_NAME",
+ "CATEG_LVL3_NAME",
+ "USER_DEFINED_FIELD1",
+ "USER_DEFINED_FIELD3",
+ "UPD_DATE",
+ "UPD_USER"
+ ]
+ },
+ {
+ "table": "TEST_SITES",
+ "columns": [
+ "SITE_ID",
+ "SITE_NAME",
+ "CRE_USER"
+ ]
+ },
+ {
+ "table": "TEST_SELLER_TYPE_DIM",
+ "columns": [
+ "SELLER_TYPE_CD",
+ "SELLER_TYPE_DESC"
+ ]
+ },
+ {
+ "table": "TEST_CAL_DT",
+ "columns": [
+ "CAL_DT",
+ "WEEK_BEG_DT"
+ ]
+ },
+ {
+ "table": "BUYER_COUNTRY",
+ "columns": [
+ "COUNTRY",
+ "NAME"
+ ]
+ },
+ {
+ "table": "SELLER_COUNTRY",
+ "columns": [
+ "COUNTRY",
+ "NAME"
+ ]
+ }
+ ],
+ "metrics": [
+ "TEST_KYLIN_FACT.PRICE",
+ "TEST_KYLIN_FACT.ITEM_COUNT"
+ ],
+ "last_modified": 1422435345352,
+ "filter_condition": null,
+ "partition_desc": {
+ "partition_date_column": "DEFAULT.TEST_KYLIN_FACT.CAL_DT",
+ "partition_date_start": 0,
+ "partition_type": "APPEND"
+ }
+}
diff --git a/tool/src/test/resources/test_meta/project/default.json b/tool/src/test/resources/test_meta/project/default.json
new file mode 100644
index 0000000..f2fd757
--- /dev/null
+++ b/tool/src/test/resources/test_meta/project/default.json
@@ -0,0 +1,35 @@
+{
+ "uuid": "1eaca32a-a33e-4b69-83dd-0bb8b1f8c91b",
+ "name": "default",
+ "realizations": [
+ {
+ "name": "ci_inner_join_cube",
+ "type": "CUBE",
+ "realization": "ci_inner_join_cube"
+ }
+ ],
+ "tables": [
+ "DEFAULT.TEST_KYLIN_FACT",
+ "DEFAULT.TEST_ORDER",
+ "DEFAULT.TEST_ACCOUNT",
+ "DEFAULT.TEST_COUNTRY",
+ "EDW.TEST_CAL_DT",
+ "DEFAULT.TEST_CATEGORY_GROUPINGS",
+ "EDW.TEST_SITES",
+ "EDW.TEST_SELLER_TYPE_DIM",
+ "DEFAULT.STREAMING_TABLE",
+ "SSB.CUSTOMER",
+ "SSB.DATES",
+ "SSB.PART",
+ "SSB.SUPPLIER",
+ "SSB.V_LINEORDER",
+ "DEFAULT.FIFTY_DIM",
+ "DEFAULT.WIDE_TABLE"
+ ],
+ "models": [
+ "ci_inner_join_model"
+ ],
+ "override_kylin_properties" :{
+ "kylin.storage.hbase.owner-tag": "kylin@kylin.apache.org"
+ }
+}
diff --git a/tool/src/test/resources/test_meta/table/DEFAULT.TEST_ACCOUNT.json b/tool/src/test/resources/test_meta/table/DEFAULT.TEST_ACCOUNT.json
new file mode 100644
index 0000000..5945713
--- /dev/null
+++ b/tool/src/test/resources/test_meta/table/DEFAULT.TEST_ACCOUNT.json
@@ -0,0 +1,36 @@
+{
+ "uuid" : "f386e39e-40d7-44c2-9eb3-41b365632231",
+
+ "name" : "TEST_ACCOUNT",
+ "data_gen" : "2000",
+
+ "columns" : [ {
+ "id" : "1",
+ "name" : "ACCOUNT_ID",
+ "datatype" : "bigint",
+ "data_gen" : "ID|10000000",
+ "index": "T"
+ }, {
+ "id" : "2",
+ "name" : "ACCOUNT_BUYER_LEVEL",
+ "datatype" : "int",
+ "data_gen" : "RAND||0|5"
+ }, {
+ "id" : "3",
+ "name" : "ACCOUNT_SELLER_LEVEL",
+ "datatype" : "int",
+ "data_gen" : "RAND||0|5"
+ }, {
+ "id" : "4",
+ "name" : "ACCOUNT_COUNTRY",
+ "datatype" : "string",
+ "data_gen" : "CN|FR|GB|GE|JP|IT|RU|US",
+ "index": "T"
+ }, {
+ "id" : "5",
+ "name" : "ACCOUNT_CONTACT",
+ "datatype" : "string"
+ } ],
+ "database" : "DEFAULT",
+ "last_modified" : 0
+}
\ No newline at end of file
diff --git a/tool/src/test/resources/test_meta/table/DEFAULT.TEST_CATEGORY_GROUPINGS.json b/tool/src/test/resources/test_meta/table/DEFAULT.TEST_CATEGORY_GROUPINGS.json
new file mode 100644
index 0000000..adaf987
--- /dev/null
+++ b/tool/src/test/resources/test_meta/table/DEFAULT.TEST_CATEGORY_GROUPINGS.json
@@ -0,0 +1,155 @@
+{
+
+ "uuid" : "952d11b5-69d9-45d1-92af-227489485e3f",
+ "name" : "TEST_CATEGORY_GROUPINGS",
+ "columns" : [ {
+ "id" : "1",
+ "name" : "LEAF_CATEG_ID",
+ "datatype" : "bigint",
+ "index": "T"
+ }, {
+ "id" : "2",
+ "name" : "LEAF_CATEG_NAME",
+ "datatype" : "string",
+ "index": "T"
+ }, {
+ "id" : "3",
+ "name" : "SITE_ID",
+ "datatype" : "int",
+ "index": "T"
+ }, {
+ "id" : "4",
+ "name" : "CATEG_BUSN_MGR",
+ "datatype" : "string"
+ }, {
+ "id" : "5",
+ "name" : "CATEG_BUSN_UNIT",
+ "datatype" : "string"
+ }, {
+ "id" : "6",
+ "name" : "REGN_CATEG",
+ "datatype" : "string"
+ }, {
+ "id" : "7",
+ "name" : "USER_DEFINED_FIELD1",
+ "datatype" : "string"
+ }, {
+ "id" : "8",
+ "name" : "USER_DEFINED_FIELD3",
+ "datatype" : "string"
+ }, {
+ "id" : "9",
+ "name" : "GROUPINGS_CRE_DATE",
+ "datatype" : "string"
+ }, {
+ "id" : "10",
+ "name" : "UPD_DATE",
+ "datatype" : "string"
+ }, {
+ "id" : "11",
+ "name" : "GROUPINGS_CRE_USER",
+ "datatype" : "string"
+ }, {
+ "id" : "12",
+ "name" : "UPD_USER",
+ "datatype" : "string"
+ }, {
+ "id" : "13",
+ "name" : "META_CATEG_ID",
+ "datatype" : "decimal"
+ }, {
+ "id" : "14",
+ "name" : "META_CATEG_NAME",
+ "datatype" : "string"
+ }, {
+ "id" : "15",
+ "name" : "CATEG_LVL2_ID",
+ "datatype" : "decimal"
+ }, {
+ "id" : "16",
+ "name" : "CATEG_LVL3_ID",
+ "datatype" : "decimal"
+ }, {
+ "id" : "17",
+ "name" : "CATEG_LVL4_ID",
+ "datatype" : "decimal"
+ }, {
+ "id" : "18",
+ "name" : "CATEG_LVL5_ID",
+ "datatype" : "decimal"
+ }, {
+ "id" : "19",
+ "name" : "CATEG_LVL6_ID",
+ "datatype" : "decimal"
+ }, {
+ "id" : "20",
+ "name" : "CATEG_LVL7_ID",
+ "datatype" : "decimal"
+ }, {
+ "id" : "21",
+ "name" : "CATEG_LVL2_NAME",
+ "datatype" : "string"
+ }, {
+ "id" : "22",
+ "name" : "CATEG_LVL3_NAME",
+ "datatype" : "string"
+ }, {
+ "id" : "23",
+ "name" : "CATEG_LVL4_NAME",
+ "datatype" : "string"
+ }, {
+ "id" : "24",
+ "name" : "CATEG_LVL5_NAME",
+ "datatype" : "string"
+ }, {
+ "id" : "25",
+ "name" : "CATEG_LVL6_NAME",
+ "datatype" : "string"
+ }, {
+ "id" : "26",
+ "name" : "CATEG_LVL7_NAME",
+ "datatype" : "string"
+ }, {
+ "id" : "27",
+ "name" : "CATEG_FLAGS",
+ "datatype" : "decimal"
+ }, {
+ "id" : "28",
+ "name" : "ADULT_CATEG_YN",
+ "datatype" : "string"
+ }, {
+ "id" : "29",
+ "name" : "DOMAIN_ID",
+ "datatype" : "decimal"
+ }, {
+ "id" : "30",
+ "name" : "USER_DEFINED_FIELD5",
+ "datatype" : "string"
+ }, {
+ "id" : "31",
+ "name" : "VCS_ID",
+ "datatype" : "decimal"
+ }, {
+ "id" : "32",
+ "name" : "GCS_ID",
+ "datatype" : "decimal"
+ }, {
+ "id" : "33",
+ "name" : "MOVE_TO",
+ "datatype" : "decimal"
+ }, {
+ "id" : "34",
+ "name" : "SAP_CATEGORY_ID",
+ "datatype" : "decimal"
+ }, {
+ "id" : "35",
+ "name" : "SRC_ID",
+ "datatype" : "tinyint"
+ }, {
+ "id" : "36",
+ "name" : "BSNS_VRTCL_NAME",
+ "datatype" : "string"
+ } ],
+ "database" : "DEFAULT",
+ "last_modified" : 0
+}
\ No newline at end of file
diff --git a/tool/src/test/resources/test_meta/table/DEFAULT.TEST_COUNTRY.json b/tool/src/test/resources/test_meta/table/DEFAULT.TEST_COUNTRY.json
new file mode 100644
index 0000000..870cb2d
--- /dev/null
+++ b/tool/src/test/resources/test_meta/table/DEFAULT.TEST_COUNTRY.json
@@ -0,0 +1,26 @@
+{
+ "uuid" : "e286e39e-40d7-44c2-8fa2-41b365632882",
+
+ "name" : "TEST_COUNTRY",
+ "columns" : [ {
+ "id" : "1",
+ "name" : "COUNTRY",
+ "datatype" : "string",
+ "index": "T"
+ }, {
+ "id" : "2",
+ "name" : "LATITUDE",
+ "datatype" : "double"
+ }, {
+ "id" : "3",
+ "name" : "LONGITUDE",
+ "datatype" : "double"
+ }, {
+ "id" : "4",
+ "name" : "NAME",
+ "datatype" : "string",
+ "index": "T"
+ } ],
+ "database" : "DEFAULT",
+ "last_modified" : 0
+}
\ No newline at end of file
diff --git a/tool/src/test/resources/test_meta/table/DEFAULT.TEST_KYLIN_FACT.json b/tool/src/test/resources/test_meta/table/DEFAULT.TEST_KYLIN_FACT.json
new file mode 100644
index 0000000..4be05f2
--- /dev/null
+++ b/tool/src/test/resources/test_meta/table/DEFAULT.TEST_KYLIN_FACT.json
@@ -0,0 +1,68 @@
+{
+ "uuid" : "e286e39e-40d7-44c2-8fa2-41b365522771",
+ "name" : "TEST_KYLIN_FACT",
+ "data_gen" : "1",
+ "columns" : [ {
+ "id" : "1",
+ "name" : "TRANS_ID",
+ "datatype" : "bigint",
+ "data_gen" : "ID"
+ }, {
+ "id" : "2",
+ "name" : "ORDER_ID",
+ "datatype" : "bigint",
+ "index": "T"
+ }, {
+ "id" : "3",
+ "name" : "CAL_DT",
+ "datatype" : "date",
+ "data_gen" : "FK,order",
+ "index": "T"
+ }, {
+ "id" : "4",
+ "name" : "LSTG_FORMAT_NAME",
+ "datatype" : "string",
+ "data_gen" : "FP-GTC|FP-non GTC|ABIN|Auction|Others",
+ "index": "T"
+ }, {
+ "id" : "5",
+ "name" : "LEAF_CATEG_ID",
+ "datatype" : "bigint",
+ "data_gen" : "FK,null,nullstr=0",
+ "index": "T"
+ }, {
+ "id" : "6",
+ "name" : "LSTG_SITE_ID",
+ "datatype" : "int",
+ "index": "T"
+ }, {
+ "id" : "7",
+ "name" : "SLR_SEGMENT_CD",
+ "datatype" : "smallint",
+ "data_gen" : "FK,pk=EDW.TEST_SELLER_TYPE_DIM_TABLE.SELLER_TYPE_CD",
+ "index": "T"
+ }, {
+ "id" : "8",
+ "name" : "SELLER_ID",
+ "datatype" : "int",
+ "data_gen" : "RAND||10000000|10001000",
+ "index": "T"
+ }, {
+ "id" : "9",
+ "name" : "PRICE",
+ "datatype" : "decimal",
+ "data_gen" : "RAND|.##|-100|1000"
+ }, {
+ "id" : "10",
+ "name" : "ITEM_COUNT",
+ "datatype" : "int",
+ "data_gen" : "RAND"
+ }, {
+ "id" : "11",
+ "name" : "TEST_COUNT_DISTINCT_BITMAP",
+ "datatype" : "string",
+ "data_gen" : "RAND"
+ } ],
+ "database" : "DEFAULT",
+ "last_modified" : 0
+}
diff --git a/tool/src/test/resources/test_meta/table/DEFAULT.TEST_ORDER.json b/tool/src/test/resources/test_meta/table/DEFAULT.TEST_ORDER.json
new file mode 100644
index 0000000..7232492
--- /dev/null
+++ b/tool/src/test/resources/test_meta/table/DEFAULT.TEST_ORDER.json
@@ -0,0 +1,35 @@
+{
+ "uuid" : "f216e39e-40d7-44c2-8fa2-41b3656398ae",
+ "name" : "TEST_ORDER",
+ "data_gen" : "0.5",
+ "columns" : [ {
+ "id" : "1",
+ "name" : "ORDER_ID",
+ "datatype" : "bigint",
+ "data_gen" : "ID",
+ "index": "T"
+ }, {
+ "id" : "2",
+ "name" : "BUYER_ID",
+ "datatype" : "bigint",
+ "data_gen" : "RAND||10000500|10001500",
+ "index": "T"
+ }, {
+ "id" : "3",
+ "name" : "TEST_DATE_ENC",
+ "datatype" : "date",
+ "data_gen" : "RAND"
+ }, {
+ "id" : "4",
+ "name" : "TEST_TIME_ENC",
+ "datatype" : "timestamp",
+ "data_gen" : "RAND"
+ }, {
+ "id" : "5",
+ "name" : "TEST_EXTENDED_COLUMN",
+ "datatype" : "string",
+ "data_gen" : "RAND"
+ } ],
+ "database" : "DEFAULT",
+ "last_modified" : 0
+}
\ No newline at end of file
diff --git a/tool/src/test/resources/test_meta/table/EDW.TEST_CAL_DT.json b/tool/src/test/resources/test_meta/table/EDW.TEST_CAL_DT.json
new file mode 100644
index 0000000..1230649
--- /dev/null
+++ b/tool/src/test/resources/test_meta/table/EDW.TEST_CAL_DT.json
@@ -0,0 +1,413 @@
+{
+
+ "uuid" : "0ff420eb-79ad-40bd-bca9-12d8cd05c60a",
+ "name" : "TEST_CAL_DT",
+ "columns" : [ {
+ "id" : "1",
+ "name" : "CAL_DT",
+ "datatype" : "date",
+ "index": "T"
+ }, {
+ "id" : "2",
+ "name" : "YEAR_BEG_DT",
+ "datatype" : "date",
+ "index": "T"
+ }, {
+ "id" : "3",
+ "name" : "QTR_BEG_DT",
+ "datatype" : "date",
+ "index": "T"
+ }, {
+ "id" : "4",
+ "name" : "MONTH_BEG_DT",
+ "datatype" : "date",
+ "index": "T"
+ }, {
+ "id" : "5",
+ "name" : "WEEK_BEG_DT",
+ "datatype" : "date",
+ "index": "T"
+ }, {
+ "id" : "6",
+ "name" : "AGE_FOR_YEAR_ID",
+ "datatype" : "smallint"
+ }, {
+ "id" : "7",
+ "name" : "AGE_FOR_QTR_ID",
+ "datatype" : "smallint"
+ }, {
+ "id" : "8",
+ "name" : "AGE_FOR_MONTH_ID",
+ "datatype" : "smallint"
+ }, {
+ "id" : "9",
+ "name" : "AGE_FOR_WEEK_ID",
+ "datatype" : "smallint"
+ }, {
+ "id" : "10",
+ "name" : "AGE_FOR_DT_ID",
+ "datatype" : "smallint"
+ }, {
+ "id" : "11",
+ "name" : "AGE_FOR_RTL_YEAR_ID",
+ "datatype" : "smallint"
+ }, {
+ "id" : "12",
+ "name" : "AGE_FOR_RTL_QTR_ID",
+ "datatype" : "smallint"
+ }, {
+ "id" : "13",
+ "name" : "AGE_FOR_RTL_MONTH_ID",
+ "datatype" : "smallint"
+ }, {
+ "id" : "14",
+ "name" : "AGE_FOR_RTL_WEEK_ID",
+ "datatype" : "smallint"
+ }, {
+ "id" : "15",
+ "name" : "AGE_FOR_CS_WEEK_ID",
+ "datatype" : "smallint"
+ }, {
+ "id" : "16",
+ "name" : "DAY_OF_CAL_ID",
+ "datatype" : "int"
+ }, {
+ "id" : "17",
+ "name" : "DAY_OF_YEAR_ID",
+ "datatype" : "smallint"
+ }, {
+ "id" : "18",
+ "name" : "DAY_OF_QTR_ID",
+ "datatype" : "smallint"
+ }, {
+ "id" : "19",
+ "name" : "DAY_OF_MONTH_ID",
+ "datatype" : "smallint"
+ }, {
+ "id" : "20",
+ "name" : "DAY_OF_WEEK_ID",
+ "datatype" : "int"
+ }, {
+ "id" : "21",
+ "name" : "WEEK_OF_YEAR_ID",
+ "datatype" : "tinyint"
+ }, {
+ "id" : "22",
+ "name" : "WEEK_OF_CAL_ID",
+ "datatype" : "int"
+ }, {
+ "id" : "23",
+ "name" : "MONTH_OF_QTR_ID",
+ "datatype" : "tinyint"
+ }, {
+ "id" : "24",
+ "name" : "MONTH_OF_YEAR_ID",
+ "datatype" : "tinyint"
+ }, {
+ "id" : "25",
+ "name" : "MONTH_OF_CAL_ID",
+ "datatype" : "smallint"
+ }, {
+ "id" : "26",
+ "name" : "QTR_OF_YEAR_ID",
+ "datatype" : "tinyint"
+ }, {
+ "id" : "27",
+ "name" : "QTR_OF_CAL_ID",
+ "datatype" : "smallint"
+ }, {
+ "id" : "28",
+ "name" : "YEAR_OF_CAL_ID",
+ "datatype" : "smallint"
+ }, {
+ "id" : "29",
+ "name" : "YEAR_END_DT",
+ "datatype" : "string"
+ }, {
+ "id" : "30",
+ "name" : "QTR_END_DT",
+ "datatype" : "string"
+ }, {
+ "id" : "31",
+ "name" : "MONTH_END_DT",
+ "datatype" : "string"
+ }, {
+ "id" : "32",
+ "name" : "WEEK_END_DT",
+ "datatype" : "string"
+ }, {
+ "id" : "33",
+ "name" : "CAL_DT_NAME",
+ "datatype" : "string"
+ }, {
+ "id" : "34",
+ "name" : "CAL_DT_DESC",
+ "datatype" : "string"
+ }, {
+ "id" : "35",
+ "name" : "CAL_DT_SHORT_NAME",
+ "datatype" : "string"
+ }, {
+ "id" : "36",
+ "name" : "YTD_YN_ID",
+ "datatype" : "tinyint"
+ }, {
+ "id" : "37",
+ "name" : "QTD_YN_ID",
+ "datatype" : "tinyint"
+ }, {
+ "id" : "38",
+ "name" : "MTD_YN_ID",
+ "datatype" : "tinyint"
+ }, {
+ "id" : "39",
+ "name" : "WTD_YN_ID",
+ "datatype" : "tinyint"
+ }, {
+ "id" : "40",
+ "name" : "SEASON_BEG_DT",
+ "datatype" : "string"
+ }, {
+ "id" : "41",
+ "name" : "DAY_IN_YEAR_COUNT",
+ "datatype" : "smallint"
+ }, {
+ "id" : "42",
+ "name" : "DAY_IN_QTR_COUNT",
+ "datatype" : "tinyint"
+ }, {
+ "id" : "43",
+ "name" : "DAY_IN_MONTH_COUNT",
+ "datatype" : "tinyint"
+ }, {
+ "id" : "44",
+ "name" : "DAY_IN_WEEK_COUNT",
+ "datatype" : "tinyint"
+ }, {
+ "id" : "45",
+ "name" : "RTL_YEAR_BEG_DT",
+ "datatype" : "string"
+ }, {
+ "id" : "46",
+ "name" : "RTL_QTR_BEG_DT",
+ "datatype" : "string"
+ }, {
+ "id" : "47",
+ "name" : "RTL_MONTH_BEG_DT",
+ "datatype" : "string"
+ }, {
+ "id" : "48",
+ "name" : "RTL_WEEK_BEG_DT",
+ "datatype" : "string"
+ }, {
+ "id" : "49",
+ "name" : "CS_WEEK_BEG_DT",
+ "datatype" : "string"
+ }, {
+ "id" : "50",
+ "name" : "CAL_DATE",
+ "datatype" : "string"
+ }, {
+ "id" : "51",
+ "name" : "DAY_OF_WEEK",
+ "datatype" : "string"
+ }, {
+ "id" : "52",
+ "name" : "MONTH_ID",
+ "datatype" : "string"
+ }, {
+ "id" : "53",
+ "name" : "PRD_DESC",
+ "datatype" : "string"
+ }, {
+ "id" : "54",
+ "name" : "PRD_FLAG",
+ "datatype" : "string"
+ }, {
+ "id" : "55",
+ "name" : "PRD_ID",
+ "datatype" : "string"
+ }, {
+ "id" : "56",
+ "name" : "PRD_IND",
+ "datatype" : "string"
+ }, {
+ "id" : "57",
+ "name" : "QTR_DESC",
+ "datatype" : "string"
+ }, {
+ "id" : "58",
+ "name" : "QTR_ID",
+ "datatype" : "string"
+ }, {
+ "id" : "59",
+ "name" : "QTR_IND",
+ "datatype" : "string"
+ }, {
+ "id" : "60",
+ "name" : "RETAIL_WEEK",
+ "datatype" : "string"
+ }, {
+ "id" : "61",
+ "name" : "RETAIL_YEAR",
+ "datatype" : "string"
+ }, {
+ "id" : "62",
+ "name" : "RETAIL_START_DATE",
+ "datatype" : "string"
+ }, {
+ "id" : "63",
+ "name" : "RETAIL_WK_END_DATE",
+ "datatype" : "string"
+ }, {
+ "id" : "64",
+ "name" : "WEEK_IND",
+ "datatype" : "string"
+ }, {
+ "id" : "65",
+ "name" : "WEEK_NUM_DESC",
+ "datatype" : "string"
+ }, {
+ "id" : "66",
+ "name" : "WEEK_BEG_DATE",
+ "datatype" : "string"
+ }, {
+ "id" : "67",
+ "name" : "WEEK_END_DATE",
+ "datatype" : "string"
+ }, {
+ "id" : "68",
+ "name" : "WEEK_IN_YEAR_ID",
+ "datatype" : "string"
+ }, {
+ "id" : "69",
+ "name" : "WEEK_ID",
+ "datatype" : "string"
+ }, {
+ "id" : "70",
+ "name" : "WEEK_BEG_END_DESC_MDY",
+ "datatype" : "string"
+ }, {
+ "id" : "71",
+ "name" : "WEEK_BEG_END_DESC_MD",
+ "datatype" : "string"
+ }, {
+ "id" : "72",
+ "name" : "YEAR_ID",
+ "datatype" : "string"
+ }, {
+ "id" : "73",
+ "name" : "YEAR_IND",
+ "datatype" : "string"
+ }, {
+ "id" : "74",
+ "name" : "CAL_DT_MNS_1YEAR_DT",
+ "datatype" : "string"
+ }, {
+ "id" : "75",
+ "name" : "CAL_DT_MNS_2YEAR_DT",
+ "datatype" : "string"
+ }, {
+ "id" : "76",
+ "name" : "CAL_DT_MNS_1QTR_DT",
+ "datatype" : "string"
+ }, {
+ "id" : "77",
+ "name" : "CAL_DT_MNS_2QTR_DT",
+ "datatype" : "string"
+ }, {
+ "id" : "78",
+ "name" : "CAL_DT_MNS_1MONTH_DT",
+ "datatype" : "string"
+ }, {
+ "id" : "79",
+ "name" : "CAL_DT_MNS_2MONTH_DT",
+ "datatype" : "string"
+ }, {
+ "id" : "80",
+ "name" : "CAL_DT_MNS_1WEEK_DT",
+ "datatype" : "string"
+ }, {
+ "id" : "81",
+ "name" : "CAL_DT_MNS_2WEEK_DT",
+ "datatype" : "string"
+ }, {
+ "id" : "82",
+ "name" : "CURR_CAL_DT_MNS_1YEAR_YN_ID",
+ "datatype" : "tinyint"
+ }, {
+ "id" : "83",
+ "name" : "CURR_CAL_DT_MNS_2YEAR_YN_ID",
+ "datatype" : "tinyint"
+ }, {
+ "id" : "84",
+ "name" : "CURR_CAL_DT_MNS_1QTR_YN_ID",
+ "datatype" : "tinyint"
+ }, {
+ "id" : "85",
+ "name" : "CURR_CAL_DT_MNS_2QTR_YN_ID",
+ "datatype" : "tinyint"
+ }, {
+ "id" : "86",
+ "name" : "CURR_CAL_DT_MNS_1MONTH_YN_ID",
+ "datatype" : "tinyint"
+ }, {
+ "id" : "87",
+ "name" : "CURR_CAL_DT_MNS_2MONTH_YN_ID",
+ "datatype" : "tinyint"
+ }, {
+ "id" : "88",
+ "name" : "CURR_CAL_DT_MNS_1WEEK_YN_IND",
+ "datatype" : "tinyint"
+ }, {
+ "id" : "89",
+ "name" : "CURR_CAL_DT_MNS_2WEEK_YN_IND",
+ "datatype" : "tinyint"
+ }, {
+ "id" : "90",
+ "name" : "RTL_MONTH_OF_RTL_YEAR_ID",
+ "datatype" : "string"
+ }, {
+ "id" : "91",
+ "name" : "RTL_QTR_OF_RTL_YEAR_ID",
+ "datatype" : "tinyint"
+ }, {
+ "id" : "92",
+ "name" : "RTL_WEEK_OF_RTL_YEAR_ID",
+ "datatype" : "tinyint"
+ }, {
+ "id" : "93",
+ "name" : "SEASON_OF_YEAR_ID",
+ "datatype" : "tinyint"
+ }, {
+ "id" : "94",
+ "name" : "YTM_YN_ID",
+ "datatype" : "tinyint"
+ }, {
+ "id" : "95",
+ "name" : "YTQ_YN_ID",
+ "datatype" : "tinyint"
+ }, {
+ "id" : "96",
+ "name" : "YTW_YN_ID",
+ "datatype" : "tinyint"
+ }, {
+ "id" : "97",
+ "name" : "CAL_DT_CRE_DATE",
+ "datatype" : "string"
+ }, {
+ "id" : "98",
+ "name" : "CAL_DT_CRE_USER",
+ "datatype" : "string"
+ }, {
+ "id" : "99",
+ "name" : "CAL_DT_UPD_DATE",
+ "datatype" : "string"
+ }, {
+ "id" : "100",
+ "name" : "CAL_DT_UPD_USER",
+ "datatype" : "string"
+ } ],
+ "database" : "edw",
+ "last_modified" : 0
+}
\ No newline at end of file
diff --git a/tool/src/test/resources/test_meta/table/EDW.TEST_SELLER_TYPE_DIM.json b/tool/src/test/resources/test_meta/table/EDW.TEST_SELLER_TYPE_DIM.json
new file mode 100644
index 0000000..136ae57
--- /dev/null
+++ b/tool/src/test/resources/test_meta/table/EDW.TEST_SELLER_TYPE_DIM.json
@@ -0,0 +1,46 @@
+{
+ "uuid" : "9ecc90c4-55df-436f-8602-2fbd4bca72e1",
+ "name" : "TEST_SELLER_TYPE_DIM",
+ "columns" : [ {
+ "id" : "1",
+ "name" : "SELLER_TYPE_CD",
+ "datatype" : "smallint",
+ "index": "T"
+ }, {
+ "id" : "2",
+ "name" : "SELLER_TYPE_DESC",
+ "datatype" : "string"
+ }, {
+ "id" : "3",
+ "name" : "GLBL_RPRT_SLR_SGMNT_CD",
+ "datatype" : "tinyint"
+ }, {
+ "id" : "4",
+ "name" : "SELLER_GROUP_CD",
+ "datatype" : "tinyint",
+ "index": "T"
+ }, {
+ "id" : "5",
+ "name" : "SELLER_GROUP_DESC",
+ "datatype" : "string"
+ }, {
+ "id" : "6",
+ "name" : "DIM_CRE_DATE",
+ "datatype" : "string"
+ }, {
+ "id" : "7",
+ "name" : "DIM_CRE_USER",
+ "datatype" : "string"
+ }, {
+ "id" : "8",
+ "name" : "DIM_UPD_DATE",
+ "datatype" : "string"
+ }, {
+ "id" : "9",
+ "name" : "DIM_UPD_USER",
+ "datatype" : "string"
+ } ],
+ "database" : "edw",
+ "table_type" : "VIRTUAL_VIEW",
+ "last_modified" : 0
+}
\ No newline at end of file
diff --git a/tool/src/test/resources/test_meta/table/EDW.TEST_SITES.json b/tool/src/test/resources/test_meta/table/EDW.TEST_SITES.json
new file mode 100644
index 0000000..4c2d727
--- /dev/null
+++ b/tool/src/test/resources/test_meta/table/EDW.TEST_SITES.json
@@ -0,0 +1,51 @@
+{
+
+ "uuid" : "338a3325-a947-46d1-9ece-e079b3b8d4a6",
+ "name" : "TEST_SITES",
+ "columns" : [ {
+ "id" : "1",
+ "name" : "SITE_ID",
+ "datatype" : "int",
+ "index": "T"
+ }, {
+ "id" : "2",
+ "name" : "SITE_NAME",
+ "datatype" : "string",
+ "index": "T"
+ }, {
+ "id" : "3",
+ "name" : "SITE_DOMAIN_CODE",
+ "datatype" : "string"
+ }, {
+ "id" : "4",
+ "name" : "DFAULT_LSTG_CURNCY",
+ "datatype" : "int"
+ }, {
+ "id" : "5",
+ "name" : "EOA_EMAIL_CSTMZBL_SITE_YN_ID",
+ "datatype" : "tinyint"
+ }, {
+ "id" : "6",
+ "name" : "SITE_CNTRY_ID",
+ "datatype" : "int",
+ "index": "T"
+ }, {
+ "id" : "7",
+ "name" : "CRE_DATE",
+ "datatype" : "string"
+ }, {
+ "id" : "8",
+ "name" : "SITES_UPD_DATE",
+ "datatype" : "string"
+ }, {
+ "id" : "9",
+ "name" : "CRE_USER",
+ "datatype" : "string"
+ }, {
+ "id" : "10",
+ "name" : "SITES_UPD_USER",
+ "datatype" : "string"
+ } ],
+ "database" : "edw",
+ "last_modified" : 0
+}
\ No newline at end of file
diff --git a/tool/src/test/resources/test_meta/table_snapshot/DEFAULT.KYLIN_COUNTRY/92456efe-9b79-4385-a5a3-e7f37b677bf7.snapshot b/tool/src/test/resources/test_meta/table_snapshot/DEFAULT.KYLIN_COUNTRY/92456efe-9b79-4385-a5a3-e7f37b677bf7.snapshot
new file mode 100644
index 0000000..58ca104
Binary files /dev/null and b/tool/src/test/resources/test_meta/table_snapshot/DEFAULT.KYLIN_COUNTRY/92456efe-9b79-4385-a5a3-e7f37b677bf7.snapshot differ
diff --git a/tool/src/test/resources/test_meta/table_snapshot/DEFAULT.TEST_COUNTRY/7ecdb07b-a8d0-49d8-892b-fe2dd75512ca.snapshot b/tool/src/test/resources/test_meta/table_snapshot/DEFAULT.TEST_COUNTRY/7ecdb07b-a8d0-49d8-892b-fe2dd75512ca.snapshot
new file mode 100644
index 0000000..4f24fe8
Binary files /dev/null and b/tool/src/test/resources/test_meta/table_snapshot/DEFAULT.TEST_COUNTRY/7ecdb07b-a8d0-49d8-892b-fe2dd75512ca.snapshot differ
--
To stop receiving notification emails like this one, please contact
liyang@apache.org.
[kylin] 01/11: KYLIN-3234,
add a API to ResourceStore that can recursively
Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
liyang pushed a commit to branch sync
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit c5e46bbf93e6ba6092cab3bd364f19680d23ac30
Author: Jiatao Tao <24...@qq.com>
AuthorDate: Sat Mar 3 09:25:01 2018 +0800
KYLIN-3234, add a API to ResourceStore that can recursively
---
.../common/persistence/FileResourceStore.java | 29 ++++++++++++++++------
.../common/persistence/HDFSResourceStore.java | 5 +++-
.../kylin/common/persistence/ResourceStore.java | 12 +++++++--
.../kylin/storage/hbase/HBaseResourceStore.java | 24 ++++++++++++------
4 files changed, 52 insertions(+), 18 deletions(-)
diff --git a/core-common/src/main/java/org/apache/kylin/common/persistence/FileResourceStore.java b/core-common/src/main/java/org/apache/kylin/common/persistence/FileResourceStore.java
index 12c8aba..6ac4536 100644
--- a/core-common/src/main/java/org/apache/kylin/common/persistence/FileResourceStore.java
+++ b/core-common/src/main/java/org/apache/kylin/common/persistence/FileResourceStore.java
@@ -25,11 +25,14 @@ import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
+import java.util.Collection;
import java.util.Collections;
import java.util.List;
import java.util.NavigableSet;
import java.util.TreeSet;
+import com.google.common.base.Preconditions;
+import org.apache.commons.io.FileUtils;
import org.apache.commons.io.IOUtils;
import org.apache.kylin.common.KylinConfig;
import org.slf4j.Logger;
@@ -56,16 +59,28 @@ public class FileResourceStore extends ResourceStore {
}
@Override
- protected NavigableSet<String> listResourcesImpl(String folderPath) throws IOException {
+ protected NavigableSet<String> listResourcesImpl(String folderPath, boolean recursive) throws IOException {
synchronized (FileResourceStore.class) {
- String[] names = file(folderPath).list();
- if (names == null) // not a directory
- return null;
-
TreeSet<String> r = new TreeSet<>();
+ File file = file(folderPath);
+ String[] names = file.list();
+ // not a directory
+ if (names == null)
+ // fixme should return empty set, like HBase implement.
+ return null;
String prefix = folderPath.endsWith("/") ? folderPath : folderPath + "/";
- for (String n : names) {
- r.add(prefix + n);
+ if (recursive) {
+ Collection<File> files = FileUtils.listFiles(file, null, true);
+ for (File f : files) {
+ String path = f.getAbsolutePath();
+ String[] split = path.split(prefix);
+ Preconditions.checkArgument(split.length == 2);
+ r.add(prefix + split[1]);
+ }
+ } else {
+ for (String n : names) {
+ r.add(prefix + n);
+ }
}
return r;
}
diff --git a/core-common/src/main/java/org/apache/kylin/common/persistence/HDFSResourceStore.java b/core-common/src/main/java/org/apache/kylin/common/persistence/HDFSResourceStore.java
index bfc3a72..7a3a933 100644
--- a/core-common/src/main/java/org/apache/kylin/common/persistence/HDFSResourceStore.java
+++ b/core-common/src/main/java/org/apache/kylin/common/persistence/HDFSResourceStore.java
@@ -88,7 +88,10 @@ public class HDFSResourceStore extends ResourceStore {
}
@Override
- protected NavigableSet<String> listResourcesImpl(String folderPath) throws IOException {
+ protected NavigableSet<String> listResourcesImpl(String folderPath, boolean recursive) throws IOException {
+ if (recursive) {
+ throw new IllegalArgumentException("Not support fullPath yet");
+ }
Path p = getRealHDFSPath(folderPath);
if (!fs.exists(p) || !fs.isDirectory(p)) {
return null;
diff --git a/core-common/src/main/java/org/apache/kylin/common/persistence/ResourceStore.java b/core-common/src/main/java/org/apache/kylin/common/persistence/ResourceStore.java
index 629e12e..a6b3337 100644
--- a/core-common/src/main/java/org/apache/kylin/common/persistence/ResourceStore.java
+++ b/core-common/src/main/java/org/apache/kylin/common/persistence/ResourceStore.java
@@ -128,13 +128,21 @@ abstract public class ResourceStore {
*/
final public NavigableSet<String> listResources(String folderPath) throws IOException {
String path = norm(folderPath);
- return listResourcesImpl(path);
+ return listResourcesImpl(path, false);
+ }
+
+ /**
+ * List resources and its full path, only support HBase now.
+ */
+ final public NavigableSet<String> listResourcesRecursively(String folderPath) throws IOException {
+ String path = norm(folderPath);
+ return listResourcesImpl(path, true);
}
/**
* return null if given path is not a folder or not exists
*/
- abstract protected NavigableSet<String> listResourcesImpl(String folderPath) throws IOException;
+ abstract protected NavigableSet<String> listResourcesImpl(String folderPath, boolean recursive) throws IOException;
protected String createMetaStoreUUID() throws IOException {
return UUID.randomUUID().toString();
diff --git a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/HBaseResourceStore.java b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/HBaseResourceStore.java
index 1f83e3e..3762437 100644
--- a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/HBaseResourceStore.java
+++ b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/HBaseResourceStore.java
@@ -118,15 +118,23 @@ public class HBaseResourceStore extends ResourceStore {
}
@Override
- protected NavigableSet<String> listResourcesImpl(String folderPath) throws IOException {
+ protected NavigableSet<String> listResourcesImpl(String folderPath, boolean recursive) throws IOException {
final TreeSet<String> result = new TreeSet<>();
-
- visitFolder(folderPath, new KeyOnlyFilter(), new FolderVisitor() {
- @Override
- public void visit(String childPath, String fullPath, Result hbaseResult) {
- result.add(childPath);
- }
- });
+ if (recursive) {
+ visitFolder(folderPath, new KeyOnlyFilter(), new FolderVisitor() {
+ @Override
+ public void visit(String childPath, String fullPath, Result hbaseResult) {
+ result.add(fullPath);
+ }
+ });
+ } else {
+ visitFolder(folderPath, new KeyOnlyFilter(), new FolderVisitor() {
+ @Override
+ public void visit(String childPath, String fullPath, Result hbaseResult) {
+ result.add(childPath);
+ }
+ });
+ }
// return null to indicate not a folder
return result.isEmpty() ? null : result;
}
--
To stop receiving notification emails like this one, please contact
liyang@apache.org.
[kylin] 05/11: Move out the wrong execute metadata to a test folder
Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
liyang pushed a commit to branch sync
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit 4b2ead2c5d3b9f465a166416477972741a016b5c
Author: nichunen <ch...@kyligence.io>
AuthorDate: Thu Feb 8 18:54:51 2018 +0800
Move out the wrong execute metadata to a test folder
---
.../org/apache/kylin/common/util/LocalFileMetadataTestCase.java | 6 +++++-
.../src/test/java/org/apache/kylin/job/ExecutableManagerTest.java | 2 +-
.../broken_executable}/execute/d9a2b721-9916-4607-8047-148ceb2473b1 | 0
3 files changed, 6 insertions(+), 2 deletions(-)
diff --git a/core-common/src/test/java/org/apache/kylin/common/util/LocalFileMetadataTestCase.java b/core-common/src/test/java/org/apache/kylin/common/util/LocalFileMetadataTestCase.java
index 94c014e..5e6b840 100644
--- a/core-common/src/test/java/org/apache/kylin/common/util/LocalFileMetadataTestCase.java
+++ b/core-common/src/test/java/org/apache/kylin/common/util/LocalFileMetadataTestCase.java
@@ -33,7 +33,7 @@ public class LocalFileMetadataTestCase extends AbstractKylinTestCase {
@Override
public void createTestMetadata(String... overlayMetadataDirs) {
//overlayMetadataDirs is useless yet
- staticCreateTestMetadata();
+ staticCreateTestMetadata(overlayMetadataDirs);
}
public static void staticCreateTestMetadata(String... overlayMetadataDirs) {
@@ -44,6 +44,10 @@ public class LocalFileMetadataTestCase extends AbstractKylinTestCase {
try {
FileUtils.deleteDirectory(new File(tempTestMetadataUrl));
FileUtils.copyDirectory(new File(testDataFolder), new File(tempTestMetadataUrl));
+
+ for (String overlay : overlayMetadataDirs) {
+ FileUtils.copyDirectory(new File(overlay), new File(tempTestMetadataUrl));
+ }
} catch (IOException e) {
e.printStackTrace();
}
diff --git a/core-job/src/test/java/org/apache/kylin/job/ExecutableManagerTest.java b/core-job/src/test/java/org/apache/kylin/job/ExecutableManagerTest.java
index 73f0410..3535d37 100644
--- a/core-job/src/test/java/org/apache/kylin/job/ExecutableManagerTest.java
+++ b/core-job/src/test/java/org/apache/kylin/job/ExecutableManagerTest.java
@@ -45,7 +45,7 @@ public class ExecutableManagerTest extends LocalFileMetadataTestCase {
@Before
public void setup() throws Exception {
- createTestMetadata();
+ createTestMetadata("../server/src/test/resources/ut_meta/broken_executable");
service = ExecutableManager.getInstance(KylinConfig.getInstanceFromEnv());
}
diff --git a/examples/test_case_data/localmeta/execute/d9a2b721-9916-4607-8047-148ceb2473b1 b/server/src/test/resources/ut_meta/broken_executable/execute/d9a2b721-9916-4607-8047-148ceb2473b1
similarity index 100%
rename from examples/test_case_data/localmeta/execute/d9a2b721-9916-4607-8047-148ceb2473b1
rename to server/src/test/resources/ut_meta/broken_executable/execute/d9a2b721-9916-4607-8047-148ceb2473b1
--
To stop receiving notification emails like this one, please contact
liyang@apache.org.
[kylin] 09/11: KYLIN-3265 update test case to cover it
Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
liyang pushed a commit to branch sync
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit 0efcf9c9e66879bbee633d9799aa529aefcde543
Author: shaofengshi <sh...@apache.org>
AuthorDate: Mon Feb 26 17:53:01 2018 +0800
KYLIN-3265 update test case to cover it
---
.../kylin/rest/controller/JobControllerTest.java | 21 ++++++++++++++++-----
1 file changed, 16 insertions(+), 5 deletions(-)
diff --git a/server/src/test/java/org/apache/kylin/rest/controller/JobControllerTest.java b/server/src/test/java/org/apache/kylin/rest/controller/JobControllerTest.java
index e87cecb..4364743 100644
--- a/server/src/test/java/org/apache/kylin/rest/controller/JobControllerTest.java
+++ b/server/src/test/java/org/apache/kylin/rest/controller/JobControllerTest.java
@@ -18,11 +18,6 @@
package org.apache.kylin.rest.controller;
-import static org.junit.Assert.assertNotNull;
-
-import java.io.IOException;
-import java.util.Date;
-
import org.apache.kylin.common.KylinConfig;
import org.apache.kylin.cube.CubeDescManager;
import org.apache.kylin.cube.CubeInstance;
@@ -43,6 +38,11 @@ import org.junit.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Qualifier;
+import java.io.IOException;
+import java.util.Date;
+
+import static org.junit.Assert.assertNotNull;
+
/**
* @author xduo
*/
@@ -97,6 +97,17 @@ public class JobControllerTest extends ServiceTestBase {
jobRequest.setTimeFilter(4);
Assert.assertNotNull(jobSchedulerController.list(jobRequest));
+ jobRequest.setJobSearchMode("ALL");
+ Assert.assertNotNull(jobSchedulerController.list(jobRequest));
+
+
+ jobRequest.setJobSearchMode("");
+ Assert.assertNotNull(jobSchedulerController.list(jobRequest));
+
+
+ jobRequest.setJobSearchMode("wrong-input");
+ Assert.assertNotNull(jobSchedulerController.list(jobRequest));
+
JobBuildRequest jobBuildRequest = new JobBuildRequest();
jobBuildRequest.setBuildType("BUILD");
jobBuildRequest.setStartTime(0L);
--
To stop receiving notification emails like this one, please contact
liyang@apache.org.