[GitHub] [hudi] poiyyq commented on issue #1971: Schema evoluation causes issue when using kafka source in hudi deltastreamer

[GitBox <gi...@apache.org>]

poiyyq commented on issue #1971:
URL: https://github.com/apache/hudi/issues/1971#issuecomment-678086882


   Hi, I face the same problem with you.    
   
   this exception happened on "reader.read(null, decoder);"
   
   Convert serialized bytes back into avro record,  it's a bug ? how to fix it?
   `
     /**
      * Convert serialized bytes back into avro record.
      */
     public static GenericRecord bytesToAvro(byte[] bytes, Schema schema) throws IOException {
       BinaryDecoder decoder = DecoderFactory.get().binaryDecoder(bytes, reuseDecoder.get());
       reuseDecoder.set(decoder);
       GenericDatumReader<GenericRecord> reader = new GenericDatumReader<>(schema);
       return reader.read(null, decoder);
     }`
   
   `
   581408 [pool-17-thread-2] ERROR org.apache.hudi.io.HoodieWriteHandle  - Error writing record HoodieRecord{key=HoodieKey { recordKey=10001 partitionPath=driver001}, currentLocation='null', newLocation='null'}
   java.io.EOFException
   	at org.apache.avro.io.BinaryDecoder.ensureBounds(BinaryDecoder.java:473)
   	at org.apache.avro.io.BinaryDecoder.readDouble(BinaryDecoder.java:243)
   	at org.apache.avro.io.ResolvingDecoder.readDouble(ResolvingDecoder.java:190)
   	at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:186)
   	at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
   	at org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:232)
   	at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:222)
   	at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:175)
   	at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
   	at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:145)
   	at org.apache.hudi.common.util.HoodieAvroUtils.bytesToAvro(HoodieAvroUtils.java:89)
   	at org.apache.hudi.common.model.OverwriteWithLatestAvroPayload.getInsertValue(OverwriteWithLatestAvroPayload.java:80)
   	at org.apache.hudi.execution.CopyOnWriteLazyInsertIterable$HoodieInsertValueGenResult.<init>(CopyOnWriteLazyInsertIterable.java:74)
   	at org.apache.hudi.execution.CopyOnWriteLazyInsertIterable.lambda$getTransformFunction$0(CopyOnWriteLazyInsertIterable.java:87)
   	at org.apache.hudi.common.util.queue.BoundedInMemoryQueue.insertRecord(BoundedInMemoryQueue.java:170)
   	at org.apache.hudi.common.util.queue.IteratorBasedQueueProducer.produce(IteratorBasedQueueProducer.java:46)
   	at org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.lambda$null$0(BoundedInMemoryExecutor.java:92)
   	at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
   	at java.util.concurrent.FutureTask.run(FutureTask.java)
   	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   	at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
   	at java.util.concurrent.FutureTask.run(FutureTask.java)
   	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   	at java.lang.Thread.run(Thread.java:748)
   `


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org