You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kylin.apache.org by "kangkaisen (JIRA)" <ji...@apache.org> on 2017/07/14 03:26:00 UTC
[jira] [Commented] (KYLIN-2575) Experimental feature: Computed
Column
[ https://issues.apache.org/jira/browse/KYLIN-2575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16086788#comment-16086788 ]
kangkaisen commented on KYLIN-2575:
-----------------------------------
This commit https://github.com/apache/kylin/commit/d61f867ff9fa170e155310edeb03db6059433f2f maybe broken the Spark Cubing. I have registered ComputedColumnDesc to kyro and changed the engine_type of ci_inner_join_cube to 4.
After I annotated this code: CalciteParser.ensureNoTableNameExists(expression)， Spark Cubing has run successfully.
{code:java}
17/07/13 21:33:45 INFO CubeManager: Loading Cube from folder kylin_metadata(key='/cube')@kylin_metadata@hbase
17/07/13 21:33:45 INFO CubeDescManager: Initializing CubeDescManager with config kylin_metadata@hbase
17/07/13 21:33:45 INFO CubeDescManager: Reloading Cube Metadata from folder kylin_metadata(key='/cube_desc')@kylin_metadata@hbase
17/07/13 21:33:45 INFO ProjectManager: Initializing ProjectManager with metadata url kylin_metadata@hbase
17/07/13 21:33:45 INFO MeasureTypeFactory: Checking custom measure types from kylin config
17/07/13 21:33:45 INFO MeasureTypeFactory: registering COUNT_DISTINCT(hllc), class org.apache.kylin.measure.hllc.HLLCMeasureType$Factory
17/07/13 21:33:45 INFO MeasureTypeFactory: registering COUNT_DISTINCT(bitmap), class org.apache.kylin.measure.bitmap.BitmapMeasureType$Factory
17/07/13 21:33:45 INFO MeasureTypeFactory: registering TOP_N(topn), class org.apache.kylin.measure.topn.TopNMeasureType$Factory
17/07/13 21:33:45 INFO MeasureTypeFactory: registering RAW(raw), class org.apache.kylin.measure.raw.RawMeasureType$Factory
17/07/13 21:33:45 INFO MeasureTypeFactory: registering EXTENDED_COLUMN(extendedcolumn), class org.apache.kylin.measure.extendedcolumn.ExtendedC
olumnMeasureType$Factory
17/07/13 21:33:45 INFO MeasureTypeFactory: registering PERCENTILE(percentile), class org.apache.kylin.measure.percentile.PercentileMeasureType$
Factory
17/07/13 21:33:46 INFO MetadataManager: Reloading data model at /model_desc/ci_inner_join_model.json
Exception in thread "main" java.lang.NoSuchFieldError: names
at org.apache.kylin.metadata.model.tool.CalciteParser$1.visit(CalciteParser.java:140)
at org.apache.calcite.sql.SqlIdentifier.accept(SqlIdentifier.java:271)
at org.apache.calcite.sql.SqlOperator.acceptCall(SqlOperator.java:670)
at org.apache.calcite.sql.util.SqlBasicVisitor.visit(SqlBasicVisitor.java:45)
at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:130)
at org.apache.kylin.metadata.model.tool.CalciteParser.ensureNoTableNameExists(CalciteParser.java:147)
at org.apache.kylin.metadata.model.ComputedColumnDesc.init(ComputedColumnDesc.java:49)
at org.apache.kylin.metadata.model.DataModelDesc.initComputedColumns(DataModelDesc.java:490)
at org.apache.kylin.metadata.model.DataModelDesc.init(DataModelDesc.java:355)
at org.apache.kylin.metadata.MetadataManager.reloadDataModelDescAt(MetadataManager.java:595)
at org.apache.kylin.metadata.MetadataManager.reloadAllDataModel(MetadataManager.java:579)
at org.apache.kylin.metadata.MetadataManager.init(MetadataManager.java:295)
at org.apache.kylin.metadata.MetadataManager.<init>(MetadataManager.java:122)
at org.apache.kylin.metadata.MetadataManager.getInstance(MetadataManager.java:87)
at org.apache.kylin.cube.model.CubeDesc.init(CubeDesc.java:589)
at org.apache.kylin.cube.CubeDescManager.loadCubeDesc(CubeDescManager.java:196)
at org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc(CubeDescManager.java:319)
at org.apache.kylin.cube.CubeDescManager.<init>(CubeDescManager.java:114)
at org.apache.kylin.cube.CubeDescManager.getInstance(CubeDescManager.java:81)
at org.apache.kylin.cube.CubeManager.reloadCubeLocalAt(CubeManager.java:809)
at org.apache.kylin.cube.CubeManager.loadAllCubeInstance(CubeManager.java:787)
at org.apache.kylin.cube.CubeManager.<init>(CubeManager.java:147)
at org.apache.kylin.cube.CubeManager.getInstance(CubeManager.java:105)
at org.apache.kylin.engine.spark.SparkCubingByLayer.execute(SparkCubingByLayer.java:160)
at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37)
at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
{code}
> Experimental feature: Computed Column
> -------------------------------------
>
> Key: KYLIN-2575
> URL: https://issues.apache.org/jira/browse/KYLIN-2575
> Project: Kylin
> Issue Type: New Feature
> Reporter: hongbin ma
> Assignee: hongbin ma
> Fix For: v2.1.0
>
>
> Computed column is a virtual column that is calculated from an expression of existing columns. For example, TAX is computed from PRICE * TAX_RATE; TX_YEAR is from EXTRACT(year from TX_DATE).
> Currently user have to create a view to enrich these computed columns, then feed the view to cube. This has two inconvenience:
> Create a view is not easy.
> The query has to be rewritten to use view instead of the original table.
> Let Kylin/KAP directly support computed column will be a big step forward.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)