[jira] [Created] (BEAM-3482) Java serialiazation exception when using BigQueryIO

["Shashank Prabhakara (JIRA)" <ji...@apache.org>]

Shashank Prabhakara created BEAM-3482:
-----------------------------------------

             Summary: Java serialiazation exception when using BigQueryIO
                 Key: BEAM-3482
                 URL: https://issues.apache.org/jira/browse/BEAM-3482
             Project: Beam
          Issue Type: Bug
          Components: sdk-java-gcp
    Affects Versions: 2.3.0
            Reporter: Shashank Prabhakara
            Assignee: Chamikara Jayalath


When writing data to BQ using BigQueryIO, we get the following exception when checkpointing. To reproduce, use BigQueryIO.writeTableRows() using sufficiently large dataset with latest code from master branch.
The following stack trace is when using using ApexRunner:

ERROR com.datatorrent.stram.engine.StreamingContainer: Operator set [OperatorDeployInfo[id=20,name=BigQueryIO.Write/PrepareWrite/ParDo(Anonymous)/ParMultiDo(Anonymous),type=GENERIC,checkpoint=\{ffffffffffffffff, 0, 0},inputs=[OperatorDeployInfo.Inpu
tDeployInfo[portName=input,streamId=stream9,sourceNodeId=19,sourcePortName=output,locality=CONTAINER_LOCAL,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream56,bufferServer=<null>]]]] stopped running due to
an exception.
com.esotericsoftware.kryo.KryoException: Error during Java serialization.
Serialization trace:
doFn (org.apache.beam.runners.apex.translation.operators.ApexParDoOperator)
        at com.esotericsoftware.kryo.serializers.JavaSerializer.write(JavaSerializer.java:33)
        at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
        at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
        at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
        at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:599)
        at com.datatorrent.common.util.FSStorageAgent.store(FSStorageAgent.java:190)
        at com.datatorrent.common.util.AsyncFSStorageAgent.save(AsyncFSStorageAgent.java:101)
        at com.datatorrent.stram.engine.Node.checkpoint(Node.java:521)
        at com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:461)
        at com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1428)
Caused by: java.io.NotSerializableException: org.apache.beam.sdk.io.gcp.bigquery.DynamicDestinations$SideInputAccessorViaProcessContext
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
        at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
        at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
        at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
        at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
        at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
        at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
        at com.esotericsoftware.kryo.serializers.JavaSerializer.write(JavaSerializer.java:30)
        ... 9 more



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)