You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kylin.apache.org by GitBox <gi...@apache.org> on 2020/01/13 11:41:42 UTC
[GitHub] [kylin] wangxiaojing123 opened a new pull request #1056: KYLIN-4336
Global domain dict for MR build engine
wangxiaojing123 opened a new pull request #1056: KYLIN-4336 Global domain dict for MR build engine
URL: https://github.com/apache/kylin/pull/1056
issue link: https://issues.apache.org/jira/browse/KYLIN-4336
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [kylin] nichunen closed pull request #1056: KYLIN-4336 Global
domain dict for MR build engine
Posted by GitBox <gi...@apache.org>.
nichunen closed pull request #1056: KYLIN-4336 Global domain dict for MR build engine
URL: https://github.com/apache/kylin/pull/1056
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [kylin] codecov-io commented on issue #1056: KYLIN-4336 Global
domain dict for MR build engine
Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #1056: KYLIN-4336 Global domain dict for MR build engine
URL: https://github.com/apache/kylin/pull/1056#issuecomment-582375163
# [Codecov](https://codecov.io/gh/apache/kylin/pull/1056?src=pr&el=h1) Report
> Merging [#1056](https://codecov.io/gh/apache/kylin/pull/1056?src=pr&el=desc) into [master](https://codecov.io/gh/apache/kylin/commit/37770217b2a9d7a62a40acae74fe77ea1d170b2a?src=pr&el=desc) will **increase** coverage by `0.02%`.
> The diff coverage is `10%`.
[](https://codecov.io/gh/apache/kylin/pull/1056?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #1056 +/- ##
============================================
+ Coverage 25% 25.03% +0.02%
- Complexity 6180 6225 +45
============================================
Files 1438 1441 +3
Lines 87669 87972 +303
Branches 12256 12305 +49
============================================
+ Hits 21926 22026 +100
- Misses 63593 63779 +186
- Partials 2150 2167 +17
```
| [Impacted Files](https://codecov.io/gh/apache/kylin/pull/1056?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...be/model/CubeDescTiretreeGlobalDomainDictUtil.java](https://codecov.io/gh/apache/kylin/pull/1056/diff?src=pr&el=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9jdWJlL21vZGVsL0N1YmVEZXNjVGlyZXRyZWVHbG9iYWxEb21haW5EaWN0VXRpbC5qYXZh) | `0% <0%> (ø)` | `0 <0> (?)` | |
| [...apache/kylin/measure/bitmap/BitmapMeasureType.java](https://codecov.io/gh/apache/kylin/pull/1056/diff?src=pr&el=tree#diff-Y29yZS1tZXRhZGF0YS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUva3lsaW4vbWVhc3VyZS9iaXRtYXAvQml0bWFwTWVhc3VyZVR5cGUuamF2YQ==) | `18.86% <0%> (ø)` | `4 <0> (ø)` | :arrow_down: |
| [...ache/kylin/engine/mr/common/AbstractHadoopJob.java](https://codecov.io/gh/apache/kylin/pull/1056/diff?src=pr&el=tree#diff-ZW5naW5lLW1yL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9lbmdpbmUvbXIvY29tbW9uL0Fic3RyYWN0SGFkb29wSm9iLmphdmE=) | `15.21% <0%> (-0.04%)` | `2 <0> (ø)` | |
| [...va/org/apache/kylin/cube/model/DictionaryDesc.java](https://codecov.io/gh/apache/kylin/pull/1056/diff?src=pr&el=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9jdWJlL21vZGVsL0RpY3Rpb25hcnlEZXNjLmphdmE=) | `65.38% <10%> (-34.62%)` | `8 <1> (+1)` | |
| [...ain/java/org/apache/kylin/cube/model/CubeDesc.java](https://codecov.io/gh/apache/kylin/pull/1056/diff?src=pr&el=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9jdWJlL21vZGVsL0N1YmVEZXNjLmphdmE=) | `63.56% <20%> (-0.57%)` | `182 <1> (+1)` | |
| [...org/apache/kylin/metadata/model/DataModelDesc.java](https://codecov.io/gh/apache/kylin/pull/1056/diff?src=pr&el=tree#diff-Y29yZS1tZXRhZGF0YS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUva3lsaW4vbWV0YWRhdGEvbW9kZWwvRGF0YU1vZGVsRGVzYy5qYXZh) | `59.21% <25%> (-0.58%)` | `72 <1> (-1)` | |
| [...c/main/java/org/apache/kylin/cube/CubeManager.java](https://codecov.io/gh/apache/kylin/pull/1056/diff?src=pr&el=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9jdWJlL0N1YmVNYW5hZ2VyLmphdmE=) | `36.52% <50%> (+0.02%)` | `44 <0> (ø)` | :arrow_down: |
| [...lin/cube/model/validation/rule/DictionaryRule.java](https://codecov.io/gh/apache/kylin/pull/1056/diff?src=pr&el=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9jdWJlL21vZGVsL3ZhbGlkYXRpb24vcnVsZS9EaWN0aW9uYXJ5UnVsZS5qYXZh) | `86.84% <60%> (-4.59%)` | `16 <1> (+1)` | |
| [...engine/spark/util/PercentileCounterSerializer.java](https://codecov.io/gh/apache/kylin/pull/1056/diff?src=pr&el=tree#diff-ZW5naW5lLXNwYXJrL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9lbmdpbmUvc3BhcmsvdXRpbC9QZXJjZW50aWxlQ291bnRlclNlcmlhbGl6ZXIuamF2YQ==) | `90.47% <0%> (-9.53%)` | `3% <0%> (ø)` | |
| [.../org/apache/kylin/dict/lookup/SnapshotManager.java](https://codecov.io/gh/apache/kylin/pull/1056/diff?src=pr&el=tree#diff-Y29yZS1kaWN0aW9uYXJ5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9kaWN0L2xvb2t1cC9TbmFwc2hvdE1hbmFnZXIuamF2YQ==) | `68.5% <0%> (-3.82%)` | `30% <0%> (+3%)` | |
| ... and [55 more](https://codecov.io/gh/apache/kylin/pull/1056/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/kylin/pull/1056?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/kylin/pull/1056?src=pr&el=footer). Last update [3777021...1c1fbe5](https://codecov.io/gh/apache/kylin/pull/1056?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [kylin] nichunen commented on a change in pull request #1056:
KYLIN-4336 Global domain dict for MR build engine
Posted by GitBox <gi...@apache.org>.
nichunen commented on a change in pull request #1056: KYLIN-4336 Global domain dict for MR build engine
URL: https://github.com/apache/kylin/pull/1056#discussion_r374521549
##########
File path: core-cube/src/main/java/org/apache/kylin/cube/model/CubeDesc.java
##########
@@ -1375,12 +1376,37 @@ public void setParentForward(int parentForward) {
result.remove(dictDesc.getColumnRef());
result.add(dictDesc.getResuseColumnRef());
}
+
+ //tiretree global domain dic
+ if (Objects.isNull(dictDesc.getResuseColumnRef()) && Objects.nonNull(dictDesc.getReuseColumn())) {
+ logger.info("tiretree global domain dic : column {} use tiretree global domain dic, reuse column {} ", dictDesc.getColumnRef(), dictDesc.getReuseColumn());
Review comment:
I think it's better to put this into the scope of the previous conditional expression "if (dictDesc.getResuseColumnRef() != null) "
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [kylin] wangxiaojing123 commented on a change in pull request
#1056: KYLIN-4336 Global domain dict for MR build engine
Posted by GitBox <gi...@apache.org>.
wangxiaojing123 commented on a change in pull request #1056: KYLIN-4336 Global domain dict for MR build engine
URL: https://github.com/apache/kylin/pull/1056#discussion_r375105245
##########
File path: core-cube/src/main/java/org/apache/kylin/cube/model/CubeDesc.java
##########
@@ -1375,12 +1376,37 @@ public void setParentForward(int parentForward) {
result.remove(dictDesc.getColumnRef());
result.add(dictDesc.getResuseColumnRef());
}
+
+ //tiretree global domain dic
+ if (Objects.isNull(dictDesc.getResuseColumnRef()) && Objects.nonNull(dictDesc.getReuseColumn())) {
+ logger.info("tiretree global domain dic : column {} use tiretree global domain dic, reuse column {} ", dictDesc.getColumnRef(), dictDesc.getReuseColumn());
Review comment:
Thanks for coding review .The previous conditional expression use for reuse col of the same cube (Existing functions of kylin), and dictDesc.getResuseColumnRef() is not null, but the new global domain dic's dictDesc.getResuseColumnRef() is null . After the expression is satisfied, the content to be processed is different too， the global domain dic only result.remove(dictDesc.getColumnRef()) no result.add(dictDesc.getResuseColumnRef()).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [kylin] wangxiaojing123 commented on a change in pull request
#1056: KYLIN-4336 Global domain dict for MR build engine
Posted by GitBox <gi...@apache.org>.
wangxiaojing123 commented on a change in pull request #1056: KYLIN-4336 Global domain dict for MR build engine
URL: https://github.com/apache/kylin/pull/1056#discussion_r375105626
##########
File path: core-cube/src/main/java/org/apache/kylin/cube/model/CubeDescTiretreeGlobalDomainDictUtil.java
##########
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+*/
+
+package org.apache.kylin.cube.model;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.kylin.common.KylinConfig;
+import org.apache.kylin.cube.CubeInstance;
+import org.apache.kylin.cube.CubeManager;
+import org.apache.kylin.cube.CubeSegment;
+import org.apache.kylin.metadata.model.DataModelDesc;
+import org.apache.kylin.metadata.model.DataModelManager;
+import org.apache.kylin.metadata.model.TableDesc;
+import org.apache.kylin.metadata.model.TableRef;
+import org.apache.kylin.metadata.model.TblColRef;
+import org.apache.kylin.source.SourceManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+
+public class CubeDescTiretreeGlobalDomainDictUtil {
+ private static final Logger logger = LoggerFactory.getLogger(CubeDescTiretreeGlobalDomainDictUtil.class);
+
+ /**
+ * get reuse global tiretree global dic path
+ * @param tblColRef
+ * @param cubeDesc
+ * @return
+ */
+ public static String globalReuseDictPath(KylinConfig config, TblColRef tblColRef, CubeDesc cubeDesc) {
+ String globalResumeDictPath = null;
+ List<GlobalDict> globalDicts = cubeDesc.listDomainDict();
+ DataModelManager metadataManager = DataModelManager.getInstance(config);
+ CubeManager cubeManager = CubeManager.getInstance(KylinConfig.getInstanceFromEnv());
+ for (GlobalDict dict : globalDicts) {
+ if (dict.getSrc().getIdentity().equalsIgnoreCase(tblColRef.getIdentity())) {
+ String model = dict.getModel();
+ String cube = dict.getCube();
+ logger.info("cube:{} column:{} tiretree global domain dic reuse model:{} cube{} column:{} ", cubeDesc.getName() , tblColRef.getName(), model, cube, dict.getDesc());
+
+ DataModelDesc dataModel = metadataManager.getDataModelDesc(model);
+ if (Objects.isNull(dataModel)) {
+ logger.error("get cube:{} column:{} tiretree global domain dic reuse DataModelDesc error", cubeDesc.getName(), tblColRef.getName());
+ return null;
+ }
+
+ CubeInstance cubeInstance = cubeManager.getCube(cube);
+ CubeSegment cubeSegment = cubeInstance.getLatestReadySegment();
+
+ TblColRef colRef = dataModel.findColumn(dict.getDesc());
+ if(Objects.isNull(colRef)){
+ logger.error("get cube:{} column:{} tiretree global domain dic TblColRef error");
+ return null;
+ }
+
+ globalResumeDictPath = cubeSegment.getDictResPath(colRef);
+
+ if (StringUtils.isBlank(globalResumeDictPath)) {
+ logger.error("get cube:{} column:{} tiretree global domain dic resume dict path error");
+ }
+ logger.error("get cube:{} column:{} tiretree global domain dic resume dict path is {}", globalResumeDictPath);
+ break;
+ }
+ }
+ return globalResumeDictPath;
+ }
+
+
+
+
+ /**
+ * add resuce global tiretree global dic for baseid job
+ * @param cubeDesc
+ * @param dumpList
+ */
+ public static void cuboidJob(CubeDesc cubeDesc, Set<String> dumpList) {
+ logger.info("cube {} start to add global domain dic", cubeDesc.getName());
+ CubeManager cubeManager = CubeManager.getInstance(KylinConfig.getInstanceFromEnv());
+ DataModelManager metadataManager =DataModelManager.getInstance(KylinConfig.getInstanceFromEnv());
+
+ cubeManager.getCube(cubeDesc.getName());
Review comment:
Thanks ,I will fix it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [kylin] nichunen commented on a change in pull request #1056:
KYLIN-4336 Global domain dict for MR build engine
Posted by GitBox <gi...@apache.org>.
nichunen commented on a change in pull request #1056: KYLIN-4336 Global domain dict for MR build engine
URL: https://github.com/apache/kylin/pull/1056#discussion_r374570739
##########
File path: core-cube/src/main/java/org/apache/kylin/cube/model/CubeDescTiretreeGlobalDomainDictUtil.java
##########
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+*/
+
+package org.apache.kylin.cube.model;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.kylin.common.KylinConfig;
+import org.apache.kylin.cube.CubeInstance;
+import org.apache.kylin.cube.CubeManager;
+import org.apache.kylin.cube.CubeSegment;
+import org.apache.kylin.metadata.model.DataModelDesc;
+import org.apache.kylin.metadata.model.DataModelManager;
+import org.apache.kylin.metadata.model.TableDesc;
+import org.apache.kylin.metadata.model.TableRef;
+import org.apache.kylin.metadata.model.TblColRef;
+import org.apache.kylin.source.SourceManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+
+public class CubeDescTiretreeGlobalDomainDictUtil {
+ private static final Logger logger = LoggerFactory.getLogger(CubeDescTiretreeGlobalDomainDictUtil.class);
+
+ /**
+ * get reuse global tiretree global dic path
+ * @param tblColRef
+ * @param cubeDesc
+ * @return
+ */
+ public static String globalReuseDictPath(KylinConfig config, TblColRef tblColRef, CubeDesc cubeDesc) {
+ String globalResumeDictPath = null;
+ List<GlobalDict> globalDicts = cubeDesc.listDomainDict();
+ DataModelManager metadataManager = DataModelManager.getInstance(config);
+ CubeManager cubeManager = CubeManager.getInstance(KylinConfig.getInstanceFromEnv());
+ for (GlobalDict dict : globalDicts) {
+ if (dict.getSrc().getIdentity().equalsIgnoreCase(tblColRef.getIdentity())) {
+ String model = dict.getModel();
+ String cube = dict.getCube();
+ logger.info("cube:{} column:{} tiretree global domain dic reuse model:{} cube{} column:{} ", cubeDesc.getName() , tblColRef.getName(), model, cube, dict.getDesc());
+
+ DataModelDesc dataModel = metadataManager.getDataModelDesc(model);
+ if (Objects.isNull(dataModel)) {
+ logger.error("get cube:{} column:{} tiretree global domain dic reuse DataModelDesc error", cubeDesc.getName(), tblColRef.getName());
+ return null;
+ }
+
+ CubeInstance cubeInstance = cubeManager.getCube(cube);
+ CubeSegment cubeSegment = cubeInstance.getLatestReadySegment();
+
+ TblColRef colRef = dataModel.findColumn(dict.getDesc());
+ if(Objects.isNull(colRef)){
+ logger.error("get cube:{} column:{} tiretree global domain dic TblColRef error");
+ return null;
+ }
+
+ globalResumeDictPath = cubeSegment.getDictResPath(colRef);
+
+ if (StringUtils.isBlank(globalResumeDictPath)) {
+ logger.error("get cube:{} column:{} tiretree global domain dic resume dict path error");
+ }
+ logger.error("get cube:{} column:{} tiretree global domain dic resume dict path is {}", globalResumeDictPath);
+ break;
+ }
+ }
+ return globalResumeDictPath;
+ }
+
+
+
+
+ /**
+ * add resuce global tiretree global dic for baseid job
+ * @param cubeDesc
+ * @param dumpList
+ */
+ public static void cuboidJob(CubeDesc cubeDesc, Set<String> dumpList) {
Review comment:
The method name is not clear enough to clarify its function
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [kylin] nichunen commented on issue #1056: KYLIN-4336 Global domain
dict for MR build engine
Posted by GitBox <gi...@apache.org>.
nichunen commented on issue #1056: KYLIN-4336 Global domain dict for MR build engine
URL: https://github.com/apache/kylin/pull/1056#issuecomment-590746655
Merged in PR https://github.com/apache/kylin/pull/1058
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [kylin] wangxiaojing123 commented on a change in pull request
#1056: KYLIN-4336 Global domain dict for MR build engine
Posted by GitBox <gi...@apache.org>.
wangxiaojing123 commented on a change in pull request #1056: KYLIN-4336 Global domain dict for MR build engine
URL: https://github.com/apache/kylin/pull/1056#discussion_r375107473
##########
File path: core-cube/src/main/java/org/apache/kylin/cube/model/CubeDescTiretreeGlobalDomainDictUtil.java
##########
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+*/
+
+package org.apache.kylin.cube.model;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.kylin.common.KylinConfig;
+import org.apache.kylin.cube.CubeInstance;
+import org.apache.kylin.cube.CubeManager;
+import org.apache.kylin.cube.CubeSegment;
+import org.apache.kylin.metadata.model.DataModelDesc;
+import org.apache.kylin.metadata.model.DataModelManager;
+import org.apache.kylin.metadata.model.TableDesc;
+import org.apache.kylin.metadata.model.TableRef;
+import org.apache.kylin.metadata.model.TblColRef;
+import org.apache.kylin.source.SourceManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+
+public class CubeDescTiretreeGlobalDomainDictUtil {
+ private static final Logger logger = LoggerFactory.getLogger(CubeDescTiretreeGlobalDomainDictUtil.class);
+
+ /**
+ * get reuse global tiretree global dic path
+ * @param tblColRef
+ * @param cubeDesc
+ * @return
+ */
+ public static String globalReuseDictPath(KylinConfig config, TblColRef tblColRef, CubeDesc cubeDesc) {
+ String globalResumeDictPath = null;
+ List<GlobalDict> globalDicts = cubeDesc.listDomainDict();
+ DataModelManager metadataManager = DataModelManager.getInstance(config);
+ CubeManager cubeManager = CubeManager.getInstance(KylinConfig.getInstanceFromEnv());
+ for (GlobalDict dict : globalDicts) {
+ if (dict.getSrc().getIdentity().equalsIgnoreCase(tblColRef.getIdentity())) {
+ String model = dict.getModel();
+ String cube = dict.getCube();
+ logger.info("cube:{} column:{} tiretree global domain dic reuse model:{} cube{} column:{} ", cubeDesc.getName() , tblColRef.getName(), model, cube, dict.getDesc());
+
+ DataModelDesc dataModel = metadataManager.getDataModelDesc(model);
+ if (Objects.isNull(dataModel)) {
+ logger.error("get cube:{} column:{} tiretree global domain dic reuse DataModelDesc error", cubeDesc.getName(), tblColRef.getName());
+ return null;
+ }
+
+ CubeInstance cubeInstance = cubeManager.getCube(cube);
+ CubeSegment cubeSegment = cubeInstance.getLatestReadySegment();
+
+ TblColRef colRef = dataModel.findColumn(dict.getDesc());
+ if(Objects.isNull(colRef)){
+ logger.error("get cube:{} column:{} tiretree global domain dic TblColRef error");
+ return null;
+ }
+
+ globalResumeDictPath = cubeSegment.getDictResPath(colRef);
+
+ if (StringUtils.isBlank(globalResumeDictPath)) {
+ logger.error("get cube:{} column:{} tiretree global domain dic resume dict path error");
+ }
+ logger.error("get cube:{} column:{} tiretree global domain dic resume dict path is {}", globalResumeDictPath);
+ break;
+ }
+ }
+ return globalResumeDictPath;
+ }
+
+
+
+
+ /**
+ * add resuce global tiretree global dic for baseid job
+ * @param cubeDesc
+ * @param dumpList
+ */
+ public static void cuboidJob(CubeDesc cubeDesc, Set<String> dumpList) {
Review comment:
OK, I will fix it, thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [kylin] coveralls commented on issue #1056: KYLIN-4336 Global
domain dict for MR build engine
Posted by GitBox <gi...@apache.org>.
coveralls commented on issue #1056: KYLIN-4336 Global domain dict for MR build engine
URL: https://github.com/apache/kylin/pull/1056#issuecomment-573682139
## Pull Request Test Coverage Report for [Build 5515](https://coveralls.io/builds/28069899)
* **19** of **111** **(17.12%)** changed or added relevant lines in **8** files are covered.
* **8** unchanged lines in **3** files lost coverage.
* Overall coverage decreased (**-0.02%**) to **27.502%**
---
| Changes Missing Coverage | Covered Lines | Changed/Added Lines | % |
| :-----|--------------|--------|---: |
| [core-cube/src/main/java/org/apache/kylin/cube/CubeManager.java](https://coveralls.io/builds/28069899/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2FCubeManager.java#L1196) | 5 | 6 | 83.33%
| [core-cube/src/main/java/org/apache/kylin/cube/model/validation/rule/DictionaryRule.java](https://coveralls.io/builds/28069899/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Fmodel%2Fvalidation%2Frule%2FDictionaryRule.java#L88) | 4 | 5 | 80.0%
| [engine-mr/src/main/java/org/apache/kylin/engine/mr/common/AbstractHadoopJob.java](https://coveralls.io/builds/28069899/source?filename=engine-mr%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fengine%2Fmr%2Fcommon%2FAbstractHadoopJob.java#L591) | 0 | 1 | 0.0%
| [core-metadata/src/main/java/org/apache/kylin/metadata/model/DataModelDesc.java](https://coveralls.io/builds/28069899/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmetadata%2Fmodel%2FDataModelDesc.java#L296) | 5 | 8 | 62.5%
| [core-metadata/src/main/java/org/apache/kylin/measure/bitmap/BitmapMeasureType.java](https://coveralls.io/builds/28069899/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmeasure%2Fbitmap%2FBitmapMeasureType.java#L112) | 0 | 4 | 0.0%
| [core-cube/src/main/java/org/apache/kylin/cube/model/CubeDesc.java](https://coveralls.io/builds/28069899/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Fmodel%2FCubeDesc.java#L1381) | 3 | 10 | 30.0%
| [core-cube/src/main/java/org/apache/kylin/cube/model/DictionaryDesc.java](https://coveralls.io/builds/28069899/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Fmodel%2FDictionaryDesc.java#L82) | 2 | 10 | 20.0%
| [core-cube/src/main/java/org/apache/kylin/cube/model/CubeDescTiretreeGlobalDomainDictUtil.java](https://coveralls.io/builds/28069899/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Fmodel%2FCubeDescTiretreeGlobalDomainDictUtil.java#L41) | 0 | 67 | 0.0%
<!-- | **Total:** | **19** | **111** | **17.12%** | -->
| Files with Coverage Reduction | New Missed Lines | % |
| :-----|--------------|--: |
| [core-cube/src/main/java/org/apache/kylin/cube/model/CubeDesc.java](https://coveralls.io/builds/28069899/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Fmodel%2FCubeDesc.java#L1386) | 1 | 69.81% |
| [core-job/src/main/java/org/apache/kylin/job/impl/threadpool/DefaultScheduler.java](https://coveralls.io/builds/28069899/source?filename=core-job%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fjob%2Fimpl%2Fthreadpool%2FDefaultScheduler.java#L194) | 2 | 80.23% |
| [core-cube/src/main/java/org/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://coveralls.io/builds/28069899/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Finmemcubing%2FMemDiskStore.java#L439) | 5 | 77.81% |
<!-- | **Total:** | **8** | | -->
| Totals | [](https://coveralls.io/builds/28069899) |
| :-- | --: |
| Change from base [Build 5512](https://coveralls.io/builds/28061631): | -0.02% |
| Covered Lines: | 24139 |
| Relevant Lines: | 87771 |
---
##### 💛 - [Coveralls](https://coveralls.io)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [kylin] nichunen commented on a change in pull request #1056:
KYLIN-4336 Global domain dict for MR build engine
Posted by GitBox <gi...@apache.org>.
nichunen commented on a change in pull request #1056: KYLIN-4336 Global domain dict for MR build engine
URL: https://github.com/apache/kylin/pull/1056#discussion_r374557840
##########
File path: core-cube/src/main/java/org/apache/kylin/cube/model/CubeDescTiretreeGlobalDomainDictUtil.java
##########
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+*/
+
+package org.apache.kylin.cube.model;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.kylin.common.KylinConfig;
+import org.apache.kylin.cube.CubeInstance;
+import org.apache.kylin.cube.CubeManager;
+import org.apache.kylin.cube.CubeSegment;
+import org.apache.kylin.metadata.model.DataModelDesc;
+import org.apache.kylin.metadata.model.DataModelManager;
+import org.apache.kylin.metadata.model.TableDesc;
+import org.apache.kylin.metadata.model.TableRef;
+import org.apache.kylin.metadata.model.TblColRef;
+import org.apache.kylin.source.SourceManager;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+
+public class CubeDescTiretreeGlobalDomainDictUtil {
+ private static final Logger logger = LoggerFactory.getLogger(CubeDescTiretreeGlobalDomainDictUtil.class);
+
+ /**
+ * get reuse global tiretree global dic path
+ * @param tblColRef
+ * @param cubeDesc
+ * @return
+ */
+ public static String globalReuseDictPath(KylinConfig config, TblColRef tblColRef, CubeDesc cubeDesc) {
+ String globalResumeDictPath = null;
+ List<GlobalDict> globalDicts = cubeDesc.listDomainDict();
+ DataModelManager metadataManager = DataModelManager.getInstance(config);
+ CubeManager cubeManager = CubeManager.getInstance(KylinConfig.getInstanceFromEnv());
+ for (GlobalDict dict : globalDicts) {
+ if (dict.getSrc().getIdentity().equalsIgnoreCase(tblColRef.getIdentity())) {
+ String model = dict.getModel();
+ String cube = dict.getCube();
+ logger.info("cube:{} column:{} tiretree global domain dic reuse model:{} cube{} column:{} ", cubeDesc.getName() , tblColRef.getName(), model, cube, dict.getDesc());
+
+ DataModelDesc dataModel = metadataManager.getDataModelDesc(model);
+ if (Objects.isNull(dataModel)) {
+ logger.error("get cube:{} column:{} tiretree global domain dic reuse DataModelDesc error", cubeDesc.getName(), tblColRef.getName());
+ return null;
+ }
+
+ CubeInstance cubeInstance = cubeManager.getCube(cube);
+ CubeSegment cubeSegment = cubeInstance.getLatestReadySegment();
+
+ TblColRef colRef = dataModel.findColumn(dict.getDesc());
+ if(Objects.isNull(colRef)){
+ logger.error("get cube:{} column:{} tiretree global domain dic TblColRef error");
+ return null;
+ }
+
+ globalResumeDictPath = cubeSegment.getDictResPath(colRef);
+
+ if (StringUtils.isBlank(globalResumeDictPath)) {
+ logger.error("get cube:{} column:{} tiretree global domain dic resume dict path error");
+ }
+ logger.error("get cube:{} column:{} tiretree global domain dic resume dict path is {}", globalResumeDictPath);
+ break;
+ }
+ }
+ return globalResumeDictPath;
+ }
+
+
+
+
+ /**
+ * add resuce global tiretree global dic for baseid job
+ * @param cubeDesc
+ * @param dumpList
+ */
+ public static void cuboidJob(CubeDesc cubeDesc, Set<String> dumpList) {
+ logger.info("cube {} start to add global domain dic", cubeDesc.getName());
+ CubeManager cubeManager = CubeManager.getInstance(KylinConfig.getInstanceFromEnv());
+ DataModelManager metadataManager =DataModelManager.getInstance(KylinConfig.getInstanceFromEnv());
+
+ cubeManager.getCube(cubeDesc.getName());
Review comment:
useless
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [kylin] coveralls edited a comment on issue #1056: KYLIN-4336
Global domain dict for MR build engine
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on issue #1056: KYLIN-4336 Global domain dict for MR build engine
URL: https://github.com/apache/kylin/pull/1056#issuecomment-573682139
## Pull Request Test Coverage Report for [Build 5595](https://coveralls.io/builds/28541529)
* **19** of **110** **(17.27%)** changed or added relevant lines in **8** files are covered.
* **1725** unchanged lines in **46** files lost coverage.
* Overall coverage decreased (**-0.02%**) to **27.504%**
---
| Changes Missing Coverage | Covered Lines | Changed/Added Lines | % |
| :-----|--------------|--------|---: |
| [core-cube/src/main/java/org/apache/kylin/cube/CubeManager.java](https://coveralls.io/builds/28541529/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2FCubeManager.java#L1196) | 5 | 6 | 83.33%
| [core-cube/src/main/java/org/apache/kylin/cube/model/validation/rule/DictionaryRule.java](https://coveralls.io/builds/28541529/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Fmodel%2Fvalidation%2Frule%2FDictionaryRule.java#L88) | 4 | 5 | 80.0%
| [engine-mr/src/main/java/org/apache/kylin/engine/mr/common/AbstractHadoopJob.java](https://coveralls.io/builds/28541529/source?filename=engine-mr%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fengine%2Fmr%2Fcommon%2FAbstractHadoopJob.java#L591) | 0 | 1 | 0.0%
| [core-metadata/src/main/java/org/apache/kylin/metadata/model/DataModelDesc.java](https://coveralls.io/builds/28541529/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmetadata%2Fmodel%2FDataModelDesc.java#L296) | 5 | 8 | 62.5%
| [core-metadata/src/main/java/org/apache/kylin/measure/bitmap/BitmapMeasureType.java](https://coveralls.io/builds/28541529/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmeasure%2Fbitmap%2FBitmapMeasureType.java#L112) | 0 | 4 | 0.0%
| [core-cube/src/main/java/org/apache/kylin/cube/model/CubeDesc.java](https://coveralls.io/builds/28541529/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Fmodel%2FCubeDesc.java#L1381) | 3 | 10 | 30.0%
| [core-cube/src/main/java/org/apache/kylin/cube/model/DictionaryDesc.java](https://coveralls.io/builds/28541529/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Fmodel%2FDictionaryDesc.java#L82) | 2 | 10 | 20.0%
| [core-cube/src/main/java/org/apache/kylin/cube/model/CubeDescTiretreeGlobalDomainDictUtil.java](https://coveralls.io/builds/28541529/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Fmodel%2FCubeDescTiretreeGlobalDomainDictUtil.java#L41) | 0 | 66 | 0.0%
<!-- | **Total:** | **19** | **110** | **17.27%** | -->
| Files with Coverage Reduction | New Missed Lines | % |
| :-----|--------------|--: |
| [core-common/src/main/java/org/apache/kylin/common/threadlocal/InternalThreadLocal.java](https://coveralls.io/builds/28541529/source?filename=core-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcommon%2Fthreadlocal%2FInternalThreadLocal.java#L148) | 1 | 61.76% |
| [core-cube/src/main/java/org/apache/kylin/cube/model/CubeDesc.java](https://coveralls.io/builds/28541529/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Fmodel%2FCubeDesc.java#L1386) | 1 | 69.81% |
| [core-metadata/src/main/java/org/apache/kylin/metadata/badquery/BadQueryHistoryManager.java](https://coveralls.io/builds/28541529/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmetadata%2Fbadquery%2FBadQueryHistoryManager.java#L92) | 1 | 83.33% |
| [stream-coordinator/src/main/java/org/apache/kylin/stream/coordinator/client/HttpCoordinatorClient.java](https://coveralls.io/builds/28541529/source?filename=stream-coordinator%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fstream%2Fcoordinator%2Fclient%2FHttpCoordinatorClient.java#L216) | 1 | 0% |
| [core-cube/src/main/java/org/apache/kylin/cube/cuboid/TreeCuboidScheduler.java](https://coveralls.io/builds/28541529/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Fcuboid%2FTreeCuboidScheduler.java#L124) | 2 | 68.46% |
| [core-metadata/src/main/java/org/apache/kylin/measure/topn/DoubleDeltaSerializer.java](https://coveralls.io/builds/28541529/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmeasure%2Ftopn%2FDoubleDeltaSerializer.java#L143) | 2 | 97.92% |
| [core-metadata/src/main/java/org/apache/kylin/metadata/TableMetadataManager.java](https://coveralls.io/builds/28541529/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmetadata%2FTableMetadataManager.java#L488) | 2 | 62.19% |
| [core-common/src/main/java/org/apache/kylin/common/util/DecimalUtil.java](https://coveralls.io/builds/28541529/source?filename=core-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcommon%2Futil%2FDecimalUtil.java#L35) | 3 | 0% |
| [core-dictionary/src/main/java/org/apache/kylin/dict/NumberDictionary.java](https://coveralls.io/builds/28541529/source?filename=core-dictionary%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fdict%2FNumberDictionary.java#L34) | 3 | 30.0% |
| [core-metadata/src/main/java/org/apache/kylin/measure/percentile/PercentileSerializer.java](https://coveralls.io/builds/28541529/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmeasure%2Fpercentile%2FPercentileSerializer.java#L38) | 3 | 85.0% |
<!-- | **Total:** | **1725** | | -->
| Totals | [](https://coveralls.io/builds/28541529) |
| :-- | --: |
| Change from base [Build 5512](https://coveralls.io/builds/28061631): | -0.02% |
| Covered Lines: | 24196 |
| Relevant Lines: | 87972 |
---
##### 💛 - [Coveralls](https://coveralls.io)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [kylin] lgtm-com[bot] commented on issue #1056: KYLIN-4336 Global
domain dict for MR build engine
Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on issue #1056: KYLIN-4336 Global domain dict for MR build engine
URL: https://github.com/apache/kylin/pull/1056#issuecomment-582374943
This pull request **introduces 4 alerts** when merging 1c1fbe5370f9b188a7b72af43fff5df6ad8f6ed9 into c51b708000354a1b16650ddd730b6728cad75fd4 - [view on LGTM.com](https://lgtm.com/projects/g/apache/kylin/rev/pr-a8f937ba22103b306c46a961004747d2678e3a2a)
**new alerts:**
* 3 for Missing format argument
* 1 for Unused format argument
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services