You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Sangeeta Gulia <sa...@knoldus.in> on 2017/04/17 07:09:51 UTC
Issue while creating connector
Hi Team,
I am trying to create a connector for carbondata. Currently i am working on
creating OutputFormat for carbondata. For that i need to create a
carboncontext which i initialise in configure method. Internally
CarbonContext uses sparkContext.
I am able to create carbonContext through my code but when it run statement
"executionEnvironment.execute()
Exception in thread "main" org.apache.flink.optimizer.CompilerException:
Error translating node 'Data Sink
"org.apache.flink.CarbonDataOutputFormat@76225435" : NONE [[
GlobalProperties [partitioning=RANDOM_PARTITIONED] ]] [[ LocalProperties
[ordering=null, grouped=null, unique=null] ]]': Could not write the user
code wrapper class
org.apache.flink.api.common.operators.util.UserCodeObjectWrapper :
java.io.NotSerializableException:
org.apache.spark.sql.hive.client.ClientWrapper
at
org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit(JobGraphGenerator.java:381)
at
org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit(JobGraphGenerator.java:106)
at
org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:198)
at
org.apache.flink.optimizer.plan.OptimizedPlan.accept(OptimizedPlan.java:128)
at
org.apache.flink.optimizer.plantranslate.JobGraphGenerator.compileJobGraph(JobGraphGenerator.java:192)
at org.apache.flink.client.LocalExecutor.executePlan(LocalExecutor.java:188)
at
org.apache.flink.api.java.LocalEnvironment.execute(LocalEnvironment.java:91)
at
org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:926)
at CarbonJDBCExample.main(CarbonJDBCExample.java:86)
Caused by:
org.apache.flink.runtime.operators.util.CorruptConfigurationException:
Could not write the user code wrapper class
org.apache.flink.api.common.operators.util.UserCodeObjectWrapper :
java.io.NotSerializableException:
org.apache.spark.sql.hive.client.ClientWrapper
at
org.apache.flink.runtime.operators.util.TaskConfig.setStubWrapper(TaskConfig.java:281)
at
org.apache.flink.optimizer.plantranslate.JobGraphGenerator.createDataSinkVertex(JobGraphGenerator.java:903)
at
org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit(JobGraphGenerator.java:278)
... 8 more
Caused by: java.io.NotSerializableException:
org.apache.spark.sql.hive.client.ClientWrapper
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
at
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
at
org.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:317)
at
org.apache.flink.util.InstantiationUtil.writeObjectToConfig(InstantiationUtil.java:254)
at
org.apache.flink.runtime.operators.util.TaskConfig.setStubWrapper(TaskConfig.java:279)
... 10 more
--
Warm Regards,
Sangeeta Gulia
Software Consultant
Knoldus Software LLP
Re: Issue while creating connector
Posted by Chesnay Schepler <ch...@apache.org>.
As the exceptions says your outputformat isn't serializable; is the
carboncontext marked as transient?
On 17.04.2017 09:12, Sangeeta Gulia wrote:
> Hi Team,
>
> Please ignore the previous mail.
>
> I am trying to create a connector for carbondata. Currently i am working on
> creating OutputFormat for carbondata. For that i need to create a
> carboncontext which i initialise in configure method. Internally
> CarbonContext uses sparkContext.
>
> I am able to create carbonContext through my code but when it run statement
> "executionEnvironment.execute()
> it throws the below error:
>
> Exception in thread "main" org.apache.flink.optimizer.CompilerException:
> Error translating node 'Data Sink
> "org.apache.flink.CarbonDataOutputFormat@76225435"
> : NONE [[ GlobalProperties [partitioning=RANDOM_PARTITIONED] ]] [[
> LocalProperties [ordering=null, grouped=null, unique=null] ]]': Could not
> write the user code wrapper class org.apache.flink.api.common.
> operators.util.UserCodeObjectWrapper : java.io.NotSerializableException:
> org.apache.spark.sql.hive.client.ClientWrapper
> at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit(
> JobGraphGenerator.java:381)
> at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit(
> JobGraphGenerator.java:106)
> at org.apache.flink.optimizer.plan.SingleInputPlanNode.
> accept(SingleInputPlanNode.java:198)
> at org.apache.flink.optimizer.plan.OptimizedPlan.accept(
> OptimizedPlan.java:128)
> at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.
> compileJobGraph(JobGraphGenerator.java:192)
> at org.apache.flink.client.LocalExecutor.executePlan(LocalExecutor.java:188)
> at org.apache.flink.api.java.LocalEnvironment.execute(
> LocalEnvironment.java:91)
> at org.apache.flink.api.java.ExecutionEnvironment.execute(
> ExecutionEnvironment.java:926)
> at CarbonJDBCExample.main(CarbonJDBCExample.java:86)
> Caused by: org.apache.flink.runtime.operators.util.CorruptConfigurationException:
> Could not write the user code wrapper class org.apache.flink.api.common.
> operators.util.UserCodeObjectWrapper : java.io.NotSerializableException:
> org.apache.spark.sql.hive.client.ClientWrapper
> at org.apache.flink.runtime.operators.util.TaskConfig.
> setStubWrapper(TaskConfig.java:281)
> at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.
> createDataSinkVertex(JobGraphGenerator.java:903)
> at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit(
> JobGraphGenerator.java:278)
> ... 8 more
> Caused by: java.io.NotSerializableException: org.apache.spark.sql.hive.
> client.ClientWrapper
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
> at java.io.ObjectOutputStream.defaultWriteFields(
> ObjectOutputStream.java:1548)
> at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
> at java.io.ObjectOutputStream.writeOrdinaryObject(
> ObjectOutputStream.java:1432)
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
> at java.io.ObjectOutputStream.defaultWriteFields(
> ObjectOutputStream.java:1548)
> at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
> at java.io.ObjectOutputStream.writeOrdinaryObject(
> ObjectOutputStream.java:1432)
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
> at java.io.ObjectOutputStream.defaultWriteFields(
> ObjectOutputStream.java:1548)
> at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
> at java.io.ObjectOutputStream.writeOrdinaryObject(
> ObjectOutputStream.java:1432)
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
> at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
> at org.apache.flink.util.InstantiationUtil.serializeObject(
> InstantiationUtil.java:317)
> at org.apache.flink.util.InstantiationUtil.writeObjectToConfig(
> InstantiationUtil.java:254)
> at org.apache.flink.runtime.operators.util.TaskConfig.
> setStubWrapper(TaskConfig.java:279)
> ... 10 more
>
> Help me with the same.
>
> On Mon, Apr 17, 2017 at 12:39 PM, Sangeeta Gulia <sa...@knoldus.in>
> wrote:
>
>> Hi Team,
>>
>> I am trying to create a connector for carbondata. Currently i am working
>> on creating OutputFormat for carbondata. For that i need to create a
>> carboncontext which i initialise in configure method. Internally
>> CarbonContext uses sparkContext.
>>
>> I am able to create carbonContext through my code but when it run statement
>> "executionEnvironment.execute()
>>
>> Exception in thread "main" org.apache.flink.optimizer.CompilerException:
>> Error translating node 'Data Sink "org.apache.flink.
>> CarbonDataOutputFormat@76225435" : NONE [[ GlobalProperties
>> [partitioning=RANDOM_PARTITIONED] ]] [[ LocalProperties [ordering=null,
>> grouped=null, unique=null] ]]': Could not write the user code wrapper class
>> org.apache.flink.api.common.operators.util.UserCodeObjectWrapper : java.io
>> .NotSerializableException: org.apache.spark.sql.hive.client.ClientWrapper
>> at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit(
>> JobGraphGenerator.java:381)
>> at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit(
>> JobGraphGenerator.java:106)
>> at org.apache.flink.optimizer.plan.SingleInputPlanNode.
>> accept(SingleInputPlanNode.java:198)
>> at org.apache.flink.optimizer.plan.OptimizedPlan.accept(
>> OptimizedPlan.java:128)
>> at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.
>> compileJobGraph(JobGraphGenerator.java:192)
>> at org.apache.flink.client.LocalExecutor.executePlan(
>> LocalExecutor.java:188)
>> at org.apache.flink.api.java.LocalEnvironment.execute(
>> LocalEnvironment.java:91)
>> at org.apache.flink.api.java.ExecutionEnvironment.execute(
>> ExecutionEnvironment.java:926)
>> at CarbonJDBCExample.main(CarbonJDBCExample.java:86)
>> Caused by: org.apache.flink.runtime.operators.util.CorruptConfigurationException:
>> Could not write the user code wrapper class org.apache.flink.api.common.
>> operators.util.UserCodeObjectWrapper : java.io.NotSerializableException:
>> org.apache.spark.sql.hive.client.ClientWrapper
>> at org.apache.flink.runtime.operators.util.TaskConfig.
>> setStubWrapper(TaskConfig.java:281)
>> at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.
>> createDataSinkVertex(JobGraphGenerator.java:903)
>> at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit(
>> JobGraphGenerator.java:278)
>> ... 8 more
>> Caused by: java.io.NotSerializableException: org.apache.spark.sql.hive.
>> client.ClientWrapper
>> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
>> at java.io.ObjectOutputStream.defaultWriteFields(
>> ObjectOutputStream.java:1548)
>> at java.io.ObjectOutputStream.writeSerialData(
>> ObjectOutputStream.java:1509)
>> at java.io.ObjectOutputStream.writeOrdinaryObject(
>> ObjectOutputStream.java:1432)
>> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
>> at java.io.ObjectOutputStream.defaultWriteFields(
>> ObjectOutputStream.java:1548)
>> at java.io.ObjectOutputStream.writeSerialData(
>> ObjectOutputStream.java:1509)
>> at java.io.ObjectOutputStream.writeOrdinaryObject(
>> ObjectOutputStream.java:1432)
>> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
>> at java.io.ObjectOutputStream.defaultWriteFields(
>> ObjectOutputStream.java:1548)
>> at java.io.ObjectOutputStream.writeSerialData(
>> ObjectOutputStream.java:1509)
>> at java.io.ObjectOutputStream.writeOrdinaryObject(
>> ObjectOutputStream.java:1432)
>> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
>> at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
>> at org.apache.flink.util.InstantiationUtil.serializeObject(
>> InstantiationUtil.java:317)
>> at org.apache.flink.util.InstantiationUtil.writeObjectToConfig(
>> InstantiationUtil.java:254)
>> at org.apache.flink.runtime.operators.util.TaskConfig.
>> setStubWrapper(TaskConfig.java:279)
>> ... 10 more
>>
>>
>>
>> --
>> Warm Regards,
>>
>> Sangeeta Gulia
>> Software Consultant
>> Knoldus Software LLP
>>
>
>
Re: Issue while creating connector
Posted by Sangeeta Gulia <sa...@knoldus.in>.
Hi Team,
Please ignore the previous mail.
I am trying to create a connector for carbondata. Currently i am working on
creating OutputFormat for carbondata. For that i need to create a
carboncontext which i initialise in configure method. Internally
CarbonContext uses sparkContext.
I am able to create carbonContext through my code but when it run statement
"executionEnvironment.execute()
it throws the below error:
Exception in thread "main" org.apache.flink.optimizer.CompilerException:
Error translating node 'Data Sink
"org.apache.flink.CarbonDataOutputFormat@76225435"
: NONE [[ GlobalProperties [partitioning=RANDOM_PARTITIONED] ]] [[
LocalProperties [ordering=null, grouped=null, unique=null] ]]': Could not
write the user code wrapper class org.apache.flink.api.common.
operators.util.UserCodeObjectWrapper : java.io.NotSerializableException:
org.apache.spark.sql.hive.client.ClientWrapper
at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit(
JobGraphGenerator.java:381)
at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit(
JobGraphGenerator.java:106)
at org.apache.flink.optimizer.plan.SingleInputPlanNode.
accept(SingleInputPlanNode.java:198)
at org.apache.flink.optimizer.plan.OptimizedPlan.accept(
OptimizedPlan.java:128)
at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.
compileJobGraph(JobGraphGenerator.java:192)
at org.apache.flink.client.LocalExecutor.executePlan(LocalExecutor.java:188)
at org.apache.flink.api.java.LocalEnvironment.execute(
LocalEnvironment.java:91)
at org.apache.flink.api.java.ExecutionEnvironment.execute(
ExecutionEnvironment.java:926)
at CarbonJDBCExample.main(CarbonJDBCExample.java:86)
Caused by: org.apache.flink.runtime.operators.util.CorruptConfigurationException:
Could not write the user code wrapper class org.apache.flink.api.common.
operators.util.UserCodeObjectWrapper : java.io.NotSerializableException:
org.apache.spark.sql.hive.client.ClientWrapper
at org.apache.flink.runtime.operators.util.TaskConfig.
setStubWrapper(TaskConfig.java:281)
at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.
createDataSinkVertex(JobGraphGenerator.java:903)
at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit(
JobGraphGenerator.java:278)
... 8 more
Caused by: java.io.NotSerializableException: org.apache.spark.sql.hive.
client.ClientWrapper
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
at java.io.ObjectOutputStream.defaultWriteFields(
ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(
ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.defaultWriteFields(
ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(
ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.defaultWriteFields(
ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(
ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
at org.apache.flink.util.InstantiationUtil.serializeObject(
InstantiationUtil.java:317)
at org.apache.flink.util.InstantiationUtil.writeObjectToConfig(
InstantiationUtil.java:254)
at org.apache.flink.runtime.operators.util.TaskConfig.
setStubWrapper(TaskConfig.java:279)
... 10 more
Help me with the same.
On Mon, Apr 17, 2017 at 12:39 PM, Sangeeta Gulia <sa...@knoldus.in>
wrote:
> Hi Team,
>
> I am trying to create a connector for carbondata. Currently i am working
> on creating OutputFormat for carbondata. For that i need to create a
> carboncontext which i initialise in configure method. Internally
> CarbonContext uses sparkContext.
>
> I am able to create carbonContext through my code but when it run statement
> "executionEnvironment.execute()
>
> Exception in thread "main" org.apache.flink.optimizer.CompilerException:
> Error translating node 'Data Sink "org.apache.flink.
> CarbonDataOutputFormat@76225435" : NONE [[ GlobalProperties
> [partitioning=RANDOM_PARTITIONED] ]] [[ LocalProperties [ordering=null,
> grouped=null, unique=null] ]]': Could not write the user code wrapper class
> org.apache.flink.api.common.operators.util.UserCodeObjectWrapper : java.io
> .NotSerializableException: org.apache.spark.sql.hive.client.ClientWrapper
> at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit(
> JobGraphGenerator.java:381)
> at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit(
> JobGraphGenerator.java:106)
> at org.apache.flink.optimizer.plan.SingleInputPlanNode.
> accept(SingleInputPlanNode.java:198)
> at org.apache.flink.optimizer.plan.OptimizedPlan.accept(
> OptimizedPlan.java:128)
> at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.
> compileJobGraph(JobGraphGenerator.java:192)
> at org.apache.flink.client.LocalExecutor.executePlan(
> LocalExecutor.java:188)
> at org.apache.flink.api.java.LocalEnvironment.execute(
> LocalEnvironment.java:91)
> at org.apache.flink.api.java.ExecutionEnvironment.execute(
> ExecutionEnvironment.java:926)
> at CarbonJDBCExample.main(CarbonJDBCExample.java:86)
> Caused by: org.apache.flink.runtime.operators.util.CorruptConfigurationException:
> Could not write the user code wrapper class org.apache.flink.api.common.
> operators.util.UserCodeObjectWrapper : java.io.NotSerializableException:
> org.apache.spark.sql.hive.client.ClientWrapper
> at org.apache.flink.runtime.operators.util.TaskConfig.
> setStubWrapper(TaskConfig.java:281)
> at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.
> createDataSinkVertex(JobGraphGenerator.java:903)
> at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit(
> JobGraphGenerator.java:278)
> ... 8 more
> Caused by: java.io.NotSerializableException: org.apache.spark.sql.hive.
> client.ClientWrapper
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
> at java.io.ObjectOutputStream.defaultWriteFields(
> ObjectOutputStream.java:1548)
> at java.io.ObjectOutputStream.writeSerialData(
> ObjectOutputStream.java:1509)
> at java.io.ObjectOutputStream.writeOrdinaryObject(
> ObjectOutputStream.java:1432)
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
> at java.io.ObjectOutputStream.defaultWriteFields(
> ObjectOutputStream.java:1548)
> at java.io.ObjectOutputStream.writeSerialData(
> ObjectOutputStream.java:1509)
> at java.io.ObjectOutputStream.writeOrdinaryObject(
> ObjectOutputStream.java:1432)
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
> at java.io.ObjectOutputStream.defaultWriteFields(
> ObjectOutputStream.java:1548)
> at java.io.ObjectOutputStream.writeSerialData(
> ObjectOutputStream.java:1509)
> at java.io.ObjectOutputStream.writeOrdinaryObject(
> ObjectOutputStream.java:1432)
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
> at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
> at org.apache.flink.util.InstantiationUtil.serializeObject(
> InstantiationUtil.java:317)
> at org.apache.flink.util.InstantiationUtil.writeObjectToConfig(
> InstantiationUtil.java:254)
> at org.apache.flink.runtime.operators.util.TaskConfig.
> setStubWrapper(TaskConfig.java:279)
> ... 10 more
>
>
>
> --
> Warm Regards,
>
> Sangeeta Gulia
> Software Consultant
> Knoldus Software LLP
>
--
Warm Regards,
Sangeeta Gulia
Software Consultant
Knoldus Software LLP