You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kylin.apache.org by "vinoyang (JIRA)" <ji...@apache.org> on 2019/04/20 02:40:00 UTC

[jira] [Updated] (KYLIN-3963) Provide a custom serializer for PercentileCounter

     [ https://issues.apache.org/jira/browse/KYLIN-3963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

vinoyang updated KYLIN-3963:
----------------------------
    Description: 
{{PercentileCounter}} contains an instance field of Type {{TDigest}} which is not serializable, so it is marked as {{transient}}. By default, it triggers a NPE when use Flink cube engine:
{code:java}
Caused by: org.apache.flink.client.program.ProgramInvocationException: Job failed. (JobID: 8077ff0e0ff01a26f62b1babe90a34b7) at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:268) at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:487) at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:475) at org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:62) at org.apache.kylin.engine.flink.FlinkCubingByLayer.execute(FlinkCubingByLayer.java:196) at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37) ... 18 more Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed. at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:146) at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:265) ... 23 more Caused by: java.lang.NullPointerException at org.apache.kylin.measure.percentile.PercentileCounter.writeRegisters(PercentileCounter.java:68) at org.apache.kylin.measure.percentile.PercentileSerializer.serialize(PercentileSerializer.java:70) at org.apache.kylin.measure.percentile.PercentileSerializer.serialize(PercentileSerializer.java:26) at org.apache.kylin.measure.MeasureCodec.encode(MeasureCodec.java:76) at org.apache.kylin.measure.BufferedMeasureCodec.encode(BufferedMeasureCodec.java:93) at org.apache.kylin.engine.flink.FlinkCubingByLayer$1.map(FlinkCubingByLayer.java:234) at org.apache.kylin.engine.flink.FlinkCubingByLayer$1.map(FlinkCubingByLayer.java:218) at org.apache.flink.runtime.operators.MapDriver.run(MapDriver.java:103) at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:503) at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:368) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:704) at java.lang.Thread.run(Thread.java:745)
{code}
So we should provide a customized serializer for {{PercentileCounter}}.

  was:
{{PercentileCounter}} contains an instance field of Type {{TDigest which is not serializable, so it is marked as {{}}transient}}. By default, it triggers a NPE when use Flink cube engine:
{code:java}
Caused by: org.apache.flink.client.program.ProgramInvocationException: Job failed. (JobID: 8077ff0e0ff01a26f62b1babe90a34b7) at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:268) at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:487) at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:475) at org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:62) at org.apache.kylin.engine.flink.FlinkCubingByLayer.execute(FlinkCubingByLayer.java:196) at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37) ... 18 more Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed. at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:146) at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:265) ... 23 more Caused by: java.lang.NullPointerException at org.apache.kylin.measure.percentile.PercentileCounter.writeRegisters(PercentileCounter.java:68) at org.apache.kylin.measure.percentile.PercentileSerializer.serialize(PercentileSerializer.java:70) at org.apache.kylin.measure.percentile.PercentileSerializer.serialize(PercentileSerializer.java:26) at org.apache.kylin.measure.MeasureCodec.encode(MeasureCodec.java:76) at org.apache.kylin.measure.BufferedMeasureCodec.encode(BufferedMeasureCodec.java:93) at org.apache.kylin.engine.flink.FlinkCubingByLayer$1.map(FlinkCubingByLayer.java:234) at org.apache.kylin.engine.flink.FlinkCubingByLayer$1.map(FlinkCubingByLayer.java:218) at org.apache.flink.runtime.operators.MapDriver.run(MapDriver.java:103) at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:503) at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:368) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:704) at java.lang.Thread.run(Thread.java:745)
{code}
So we should provide a customized serializer for {{PercentileCounter}}.


> Provide a custom serializer for PercentileCounter
> -------------------------------------------------
>
>                 Key: KYLIN-3963
>                 URL: https://issues.apache.org/jira/browse/KYLIN-3963
>             Project: Kylin
>          Issue Type: Sub-task
>          Components: Flink Engine
>            Reporter: vinoyang
>            Assignee: vinoyang
>            Priority: Major
>
> {{PercentileCounter}} contains an instance field of Type {{TDigest}} which is not serializable, so it is marked as {{transient}}. By default, it triggers a NPE when use Flink cube engine:
> {code:java}
> Caused by: org.apache.flink.client.program.ProgramInvocationException: Job failed. (JobID: 8077ff0e0ff01a26f62b1babe90a34b7) at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:268) at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:487) at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:475) at org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:62) at org.apache.kylin.engine.flink.FlinkCubingByLayer.execute(FlinkCubingByLayer.java:196) at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37) ... 18 more Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed. at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:146) at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:265) ... 23 more Caused by: java.lang.NullPointerException at org.apache.kylin.measure.percentile.PercentileCounter.writeRegisters(PercentileCounter.java:68) at org.apache.kylin.measure.percentile.PercentileSerializer.serialize(PercentileSerializer.java:70) at org.apache.kylin.measure.percentile.PercentileSerializer.serialize(PercentileSerializer.java:26) at org.apache.kylin.measure.MeasureCodec.encode(MeasureCodec.java:76) at org.apache.kylin.measure.BufferedMeasureCodec.encode(BufferedMeasureCodec.java:93) at org.apache.kylin.engine.flink.FlinkCubingByLayer$1.map(FlinkCubingByLayer.java:234) at org.apache.kylin.engine.flink.FlinkCubingByLayer$1.map(FlinkCubingByLayer.java:218) at org.apache.flink.runtime.operators.MapDriver.run(MapDriver.java:103) at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:503) at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:368) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:704) at java.lang.Thread.run(Thread.java:745)
> {code}
> So we should provide a customized serializer for {{PercentileCounter}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)