You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kylin.apache.org by GitBox <gi...@apache.org> on 2023/01/09 11:23:05 UTC
[GitHub] [kylin] liukun4515 opened a new pull request, #2072: [KYLIN-5387]: migrate cube planner1 to kylin5
liukun4515 opened a new pull request, #2072:
URL: https://github.com/apache/kylin/pull/2072
## Proposed changes
Describe the big picture of your changes here to communicate to the maintainers why we should accept this pull request. If it fixes a bug or resolves a feature request, be sure to link to that issue.
## Branch to commit
- [ ] Branch **kylin3** for v2.x to v3.x
- [ ] Branch **kylin4** for v4.x
- [ ] Branch **kylin5** for v5.x
## Types of changes
What types of changes does your code introduce to Kylin?
_Put an `x` in the boxes that apply_
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
- [ ] Documentation Update (if none of the other choices apply)
## Checklist
_Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._
- [ ] I have created an issue on [Kylin's jira](https://issues.apache.org/jira/browse/KYLIN), and have described the bug/feature there in detail
- [ ] Commit messages in my PR start with the related jira ID, like "KYLIN-0000 Make Kylin project open-source"
- [ ] Compiling and unit tests pass locally with my changes
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] I have added necessary documentation (if appropriate)
- [ ] Any dependent changes have been merged
## Further comments
If this is a relatively large or complex change, kick off the discussion at user@kylin.apache.org or dev@kylin.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kylin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] liukun4515 commented on a diff in pull request #2072: [KYLIN-5387]: migrate cube planner1 to kylin5
Posted by "liukun4515 (via GitHub)" <gi...@apache.org>.
liukun4515 commented on code in PR #2072:
URL: https://github.com/apache/kylin/pull/2072#discussion_r1106538769
##########
src/spark-project/engine-spark/src/main/scala/org/apache/kylin/engine/spark/job/stage/build/CostBasedPlanner.scala:
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kylin.engine.spark.job.stage.build
+
+import org.apache.kylin.engine.spark.job.SegmentJob
+import org.apache.kylin.engine.spark.job.stage.BuildParam
+import org.apache.kylin.engine.spark.model.SegmentFlatTableDesc
+import org.apache.kylin.engine.spark.smarter.IndexDependencyParser
+import org.apache.kylin.metadata.cube.cuboid.AdaptiveSpanningTree
+import org.apache.kylin.metadata.cube.cuboid.AdaptiveSpanningTree.AdaptiveTreeBuilder
+import org.apache.kylin.metadata.cube.model.NDataSegment
+
+class CostBasedPlanner(jobContext: SegmentJob, dataSegment: NDataSegment, buildParam: BuildParam)
+ extends FlatTableAndDictBase(jobContext, dataSegment, buildParam) {
+ override def execute(): Unit = {
+ val (cost, sourceCount) = generateCostTable()
+ getRecommendedLayoutAndUpdateMetadata(cost, sourceCount)
+
+ val result = jobContext.updateIndexPlanIfNeed()
+ if (result) {
+ // update span tree and table desc with the new build job layouts
Review Comment:
partition and no-partition have different logic and different constructor for the `AdaptiveSpanningTree` and `SegmentFlatTableDesc`, they are hard to be combined together.
On the original codebase, the code for partition and no-partition are separated, such as `PartitionMaterializedFactTableView` and `MaterializedFactTableView`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kylin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus commented on pull request #2072: [KYLIN-5387]: migrate cube planner1 to kylin5
Posted by "hit-lacus (via GitHub)" <gi...@apache.org>.
hit-lacus commented on PR #2072:
URL: https://github.com/apache/kylin/pull/2072#issuecomment-1446048731
Close this because it is merged in https://github.com/apache/kylin/pull/2089
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kylin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] liukun4515 commented on a diff in pull request #2072: [KYLIN-5387]: migrate cube planner1 to kylin5
Posted by "liukun4515 (via GitHub)" <gi...@apache.org>.
liukun4515 commented on code in PR #2072:
URL: https://github.com/apache/kylin/pull/2072#discussion_r1106541531
##########
src/spark-project/engine-spark/src/main/scala/org/apache/kylin/engine/spark/job/stage/build/CostBasedPlanner.scala:
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kylin.engine.spark.job.stage.build
+
+import org.apache.kylin.engine.spark.job.SegmentJob
+import org.apache.kylin.engine.spark.job.stage.BuildParam
+import org.apache.kylin.engine.spark.model.SegmentFlatTableDesc
+import org.apache.kylin.engine.spark.smarter.IndexDependencyParser
+import org.apache.kylin.metadata.cube.cuboid.AdaptiveSpanningTree
+import org.apache.kylin.metadata.cube.cuboid.AdaptiveSpanningTree.AdaptiveTreeBuilder
+import org.apache.kylin.metadata.cube.model.NDataSegment
+
+class CostBasedPlanner(jobContext: SegmentJob, dataSegment: NDataSegment, buildParam: BuildParam)
+ extends FlatTableAndDictBase(jobContext, dataSegment, buildParam) {
+ override def execute(): Unit = {
+ val (cost, sourceCount) = generateCostTable()
+ getRecommendedLayoutAndUpdateMetadata(cost, sourceCount)
+
+ val result = jobContext.updateIndexPlanIfNeed()
+ if (result) {
+ // update span tree and table desc with the new build job layouts
Review Comment:
If you want to combine the logic for partition and no-partition, we can do this in the next follow-up PR.
But I find the logics in the `build`, `merge` and other `job` are all divided into partition and non-partition
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kylin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] liukun4515 commented on a diff in pull request #2072: [KYLIN-5387]: migrate cube planner1 to kylin5
Posted by GitBox <gi...@apache.org>.
liukun4515 commented on code in PR #2072:
URL: https://github.com/apache/kylin/pull/2072#discussion_r1064537167
##########
src/core-metadata/src/main/java/org/apache/kylin/metadata/cube/planner/algorithm/AbstractRecommendAlgorithm.java:
##########
@@ -0,0 +1,82 @@
+/*
Review Comment:
The files in this package are migrated from kylin3.1, the codes have not been changed.
##########
src/core-metadata/src/main/java/org/apache/kylin/metadata/cube/planner/algorithm/genetic/BitsChromosome.java:
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
Review Comment:
The files in this package are migrated from kylin3.1, the codes have not been changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kylin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] liukun4515 commented on pull request #2072: [KYLIN-5387]: migrate cube planner1 to kylin5
Posted by GitBox <gi...@apache.org>.
liukun4515 commented on PR #2072:
URL: https://github.com/apache/kylin/pull/2072#issuecomment-1381277815
@hit-lacus add the google doc to describe what I do in this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kylin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] sonarcloud[bot] commented on pull request #2072: [KYLIN-5387]: migrate cube planner1 to kylin5
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #2072:
URL: https://github.com/apache/kylin/pull/2072#issuecomment-1430719716
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache_kylin&pullRequest=2072)
[](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_kylin&pullRequest=2072&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_kylin&pullRequest=2072&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_kylin&pullRequest=2072&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=CODE_SMELL) [82 Code Smells](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_kylin&pullRequest=2072&metric=new_coverage&view=list) [72.3% Coverage](https://sonarcloud.io/component_measures?id=apache_kylin&pullRequest=2072&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache_kylin&pullRequest=2072&metric=new_duplicated_lines_density&view=list) [0.3% Duplication](https://sonarcloud.io/component_measures?id=apache_kylin&pullRequest=2072&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kylin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] liukun4515 commented on a diff in pull request #2072: [KYLIN-5387]: migrate cube planner1 to kylin5
Posted by "liukun4515 (via GitHub)" <gi...@apache.org>.
liukun4515 commented on code in PR #2072:
URL: https://github.com/apache/kylin/pull/2072#discussion_r1106538769
##########
src/spark-project/engine-spark/src/main/scala/org/apache/kylin/engine/spark/job/stage/build/CostBasedPlanner.scala:
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kylin.engine.spark.job.stage.build
+
+import org.apache.kylin.engine.spark.job.SegmentJob
+import org.apache.kylin.engine.spark.job.stage.BuildParam
+import org.apache.kylin.engine.spark.model.SegmentFlatTableDesc
+import org.apache.kylin.engine.spark.smarter.IndexDependencyParser
+import org.apache.kylin.metadata.cube.cuboid.AdaptiveSpanningTree
+import org.apache.kylin.metadata.cube.cuboid.AdaptiveSpanningTree.AdaptiveTreeBuilder
+import org.apache.kylin.metadata.cube.model.NDataSegment
+
+class CostBasedPlanner(jobContext: SegmentJob, dataSegment: NDataSegment, buildParam: BuildParam)
+ extends FlatTableAndDictBase(jobContext, dataSegment, buildParam) {
+ override def execute(): Unit = {
+ val (cost, sourceCount) = generateCostTable()
+ getRecommendedLayoutAndUpdateMetadata(cost, sourceCount)
+
+ val result = jobContext.updateIndexPlanIfNeed()
+ if (result) {
+ // update span tree and table desc with the new build job layouts
Review Comment:
partition and no-partition have different logic and different constructor for the `AdaptiveSpanningTree` and `SegmentFlatTableDesc`, they are hard to merge together.
On the original codebase, the code for partition and no-partition are separated, such as `PartitionMaterializedFactTableView` and `MaterializedFactTableView`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kylin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] liukun4515 commented on pull request #2072: [KYLIN-5387]: migrate cube planner1 to kylin5
Posted by "liukun4515 (via GitHub)" <gi...@apache.org>.
liukun4515 commented on PR #2072:
URL: https://github.com/apache/kylin/pull/2072#issuecomment-1420342804
@hit-lacus please help to reset the ci
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kylin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] sonarcloud[bot] commented on pull request #2072: [KYLIN-5387]: migrate cube planner1 to kylin5
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #2072:
URL: https://github.com/apache/kylin/pull/2072#issuecomment-1425133569
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache_kylin&pullRequest=2072)
[](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=BUG) [2 Bugs](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_kylin&pullRequest=2072&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_kylin&pullRequest=2072&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_kylin&pullRequest=2072&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=CODE_SMELL) [79 Code Smells](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_kylin&pullRequest=2072&metric=new_coverage&view=list) [55.4% Coverage](https://sonarcloud.io/component_measures?id=apache_kylin&pullRequest=2072&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache_kylin&pullRequest=2072&metric=new_duplicated_lines_density&view=list) [0.3% Duplication](https://sonarcloud.io/component_measures?id=apache_kylin&pullRequest=2072&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kylin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] sonarcloud[bot] commented on pull request #2072: [KYLIN-5387]: migrate cube planner1 to kylin5
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #2072:
URL: https://github.com/apache/kylin/pull/2072#issuecomment-1425412938
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache_kylin&pullRequest=2072)
[](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=BUG) [2 Bugs](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_kylin&pullRequest=2072&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_kylin&pullRequest=2072&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_kylin&pullRequest=2072&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=CODE_SMELL) [82 Code Smells](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_kylin&pullRequest=2072&metric=new_coverage&view=list) [72.3% Coverage](https://sonarcloud.io/component_measures?id=apache_kylin&pullRequest=2072&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache_kylin&pullRequest=2072&metric=new_duplicated_lines_density&view=list) [0.3% Duplication](https://sonarcloud.io/component_measures?id=apache_kylin&pullRequest=2072&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kylin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] liukun4515 commented on a diff in pull request #2072: [KYLIN-5387]: migrate cube planner1 to kylin5
Posted by "liukun4515 (via GitHub)" <gi...@apache.org>.
liukun4515 commented on code in PR #2072:
URL: https://github.com/apache/kylin/pull/2072#discussion_r1105225759
##########
src/spark-project/spark-it/src/test/scala/org/apache/kylin/it/TestCubePlanner.scala:
##########
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kylin.it
+
+import java.util
+import java.util.TimeZone
+
+import org.apache.kylin.common._
+import org.apache.kylin.metadata.cube.model.NDataflowManager.NDataflowUpdater
+import org.apache.kylin.metadata.cube.model._
+import org.apache.kylin.metadata.realization.RealizationStatusEnum
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.SparderEnv
+import org.apache.spark.sql.common.{LocalMetadata, SparderBaseFunSuite}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.execution.utils.SchemaProcessor
+
+class TestCubePlanner extends SparderBaseFunSuite
Review Comment:
This is IT for cube planner:
1. basic test case
2. corner test case: delete one dimension after building one segment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kylin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] sonarcloud[bot] commented on pull request #2072: [KYLIN-5387]: migrate cube planner1 to kylin5
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #2072:
URL: https://github.com/apache/kylin/pull/2072#issuecomment-1425581112
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache_kylin&pullRequest=2072)
[](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_kylin&pullRequest=2072&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_kylin&pullRequest=2072&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_kylin&pullRequest=2072&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=CODE_SMELL) [81 Code Smells](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_kylin&pullRequest=2072&metric=new_coverage&view=list) [72.2% Coverage](https://sonarcloud.io/component_measures?id=apache_kylin&pullRequest=2072&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache_kylin&pullRequest=2072&metric=new_duplicated_lines_density&view=list) [0.3% Duplication](https://sonarcloud.io/component_measures?id=apache_kylin&pullRequest=2072&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kylin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] liukun4515 commented on a diff in pull request #2072: [KYLIN-5387]: migrate cube planner1 to kylin5
Posted by GitBox <gi...@apache.org>.
liukun4515 commented on code in PR #2072:
URL: https://github.com/apache/kylin/pull/2072#discussion_r1065294892
##########
src/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java:
##########
@@ -3673,4 +3679,53 @@ public int getSecondStorageWaitLockTimeout() {
public boolean getDDLEnabled(){
return Boolean.parseBoolean(getOptional("kylin.source.ddl.enabled", FALSE));
}
+
+ // ============================================================================
+ // Cost case cuboid Planner
+ // ============================================================================
+
+ public boolean enableCostBasedIndexPlanner() {
+ // If we enable the cost base cuboid planner, we will recommend subset of layouts from the rule.
+ return Boolean.parseBoolean(getOptional("kylin.index.costbased.enabled", FALSE));
Review Comment:
The default of cost based index is not enabled
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kylin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] sonarcloud[bot] commented on pull request #2072: [KYLIN-5387]: migrate cube planner1 to kylin5
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #2072:
URL: https://github.com/apache/kylin/pull/2072#issuecomment-1378863845
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache_kylin&pullRequest=2072)
[](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=BUG) [2 Bugs](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_kylin&pullRequest=2072&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_kylin&pullRequest=2072&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_kylin&pullRequest=2072&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=CODE_SMELL) [79 Code Smells](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_kylin&pullRequest=2072&metric=new_coverage&view=list) [7.0% Coverage](https://sonarcloud.io/component_measures?id=apache_kylin&pullRequest=2072&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache_kylin&pullRequest=2072&metric=new_duplicated_lines_density&view=list) [0.3% Duplication](https://sonarcloud.io/component_measures?id=apache_kylin&pullRequest=2072&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kylin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus commented on pull request #2072: [KYLIN-5387]: migrate cube planner1 to kylin5
Posted by GitBox <gi...@apache.org>.
hit-lacus commented on PR #2072:
URL: https://github.com/apache/kylin/pull/2072#issuecomment-1378753390
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kylin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] liukun4515 commented on a diff in pull request #2072: [KYLIN-5387]: migrate cube planner1 to kylin5
Posted by GitBox <gi...@apache.org>.
liukun4515 commented on code in PR #2072:
URL: https://github.com/apache/kylin/pull/2072#discussion_r1065302713
##########
src/kylin-it/src/test/java/org/apache/kylin/metadata/cube/planner/algorithm/ITAlgorithmTestBase.java:
##########
@@ -0,0 +1,567 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
Review Comment:
The test cases are migrated from kylin3.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kylin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] liukun4515 commented on a diff in pull request #2072: [KYLIN-5387]: migrate cube planner1 to kylin5
Posted by "liukun4515 (via GitHub)" <gi...@apache.org>.
liukun4515 commented on code in PR #2072:
URL: https://github.com/apache/kylin/pull/2072#discussion_r1106538769
##########
src/spark-project/engine-spark/src/main/scala/org/apache/kylin/engine/spark/job/stage/build/CostBasedPlanner.scala:
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kylin.engine.spark.job.stage.build
+
+import org.apache.kylin.engine.spark.job.SegmentJob
+import org.apache.kylin.engine.spark.job.stage.BuildParam
+import org.apache.kylin.engine.spark.model.SegmentFlatTableDesc
+import org.apache.kylin.engine.spark.smarter.IndexDependencyParser
+import org.apache.kylin.metadata.cube.cuboid.AdaptiveSpanningTree
+import org.apache.kylin.metadata.cube.cuboid.AdaptiveSpanningTree.AdaptiveTreeBuilder
+import org.apache.kylin.metadata.cube.model.NDataSegment
+
+class CostBasedPlanner(jobContext: SegmentJob, dataSegment: NDataSegment, buildParam: BuildParam)
+ extends FlatTableAndDictBase(jobContext, dataSegment, buildParam) {
+ override def execute(): Unit = {
+ val (cost, sourceCount) = generateCostTable()
+ getRecommendedLayoutAndUpdateMetadata(cost, sourceCount)
+
+ val result = jobContext.updateIndexPlanIfNeed()
+ if (result) {
+ // update span tree and table desc with the new build job layouts
Review Comment:
partition and no-partition have different logic and different construct for the `AdaptiveSpanningTree` and `SegmentFlatTableDesc`, they are hard to be combined together.
On the original codebase, the code for partition and no-partition are separated, such as `PartitionMaterializedFactTableView` and `MaterializedFactTableView`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kylin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] sonarcloud[bot] commented on pull request #2072: [KYLIN-5387]: migrate cube planner1 to kylin5
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #2072:
URL: https://github.com/apache/kylin/pull/2072#issuecomment-1427568486
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache_kylin&pullRequest=2072)
[](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_kylin&pullRequest=2072&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_kylin&pullRequest=2072&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_kylin&pullRequest=2072&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=CODE_SMELL) [82 Code Smells](https://sonarcloud.io/project/issues?id=apache_kylin&pullRequest=2072&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_kylin&pullRequest=2072&metric=new_coverage&view=list) [72.3% Coverage](https://sonarcloud.io/component_measures?id=apache_kylin&pullRequest=2072&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache_kylin&pullRequest=2072&metric=new_duplicated_lines_density&view=list) [0.3% Duplication](https://sonarcloud.io/component_measures?id=apache_kylin&pullRequest=2072&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kylin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] dethrive commented on a diff in pull request #2072: [KYLIN-5387]: migrate cube planner1 to kylin5
Posted by "dethrive (via GitHub)" <gi...@apache.org>.
dethrive commented on code in PR #2072:
URL: https://github.com/apache/kylin/pull/2072#discussion_r1105990882
##########
src/spark-project/engine-spark/src/main/scala/org/apache/kylin/engine/spark/job/stage/build/CostBasedPlanner.scala:
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kylin.engine.spark.job.stage.build
+
+import org.apache.kylin.engine.spark.job.SegmentJob
+import org.apache.kylin.engine.spark.job.stage.BuildParam
+import org.apache.kylin.engine.spark.model.SegmentFlatTableDesc
+import org.apache.kylin.engine.spark.smarter.IndexDependencyParser
+import org.apache.kylin.metadata.cube.cuboid.AdaptiveSpanningTree
+import org.apache.kylin.metadata.cube.cuboid.AdaptiveSpanningTree.AdaptiveTreeBuilder
+import org.apache.kylin.metadata.cube.model.NDataSegment
+
+class CostBasedPlanner(jobContext: SegmentJob, dataSegment: NDataSegment, buildParam: BuildParam)
+ extends FlatTableAndDictBase(jobContext, dataSegment, buildParam) {
+ override def execute(): Unit = {
+ val (cost, sourceCount) = generateCostTable()
+ getRecommendedLayoutAndUpdateMetadata(cost, sourceCount)
+
+ val result = jobContext.updateIndexPlanIfNeed()
+ if (result) {
+ // update span tree and table desc with the new build job layouts
Review Comment:
重复代码需要抽出来，否则以后维护越来越难
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kylin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kylin] hit-lacus closed pull request #2072: [KYLIN-5387]: migrate cube planner1 to kylin5
Posted by "hit-lacus (via GitHub)" <gi...@apache.org>.
hit-lacus closed pull request #2072: [KYLIN-5387]: migrate cube planner1 to kylin5
URL: https://github.com/apache/kylin/pull/2072
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kylin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org