You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "sivabalan narayanan (Jira)" <ji...@apache.org> on 2020/05/23 21:35:00 UTC
[jira] [Commented] (HUDI-721) AvroConversionUtils is broken for
complex types in 0.6
[ https://issues.apache.org/jira/browse/HUDI-721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17114954#comment-17114954 ]
sivabalan narayanan commented on HUDI-721:
------------------------------------------
[~uditme]: did you get a chance to follow up on this ticket.
> AvroConversionUtils is broken for complex types in 0.6
> ------------------------------------------------------
>
> Key: HUDI-721
> URL: https://issues.apache.org/jira/browse/HUDI-721
> Project: Apache Hudi
> Issue Type: Bug
> Components: Common Core
> Reporter: Alexander Filipchik
> Assignee: Udit Mehrotra
> Priority: Major
> Labels: bug-bash-0.6.0
> Fix For: 0.6.0
>
>
> hi,
> was working on the upgrade from 0.5 to 0.6 and hit a bug in AvroConversionUtils. I originally blames it on Spark parquet to avro schema generator (convertStructTypeToAvroSchema method), but after some debugging I'm pretty sure the issue is somewhere in the: AvroConversionHelper.
> What happens: when complexes type is extracted using SqlTransformer (using select bla fro <SRC>) where bla is complex type with arrays of struct, Kryo serialization breaks with :
>
> {code:java}
> 28701 [dag-scheduler-event-loop] INFO org.apache.spark.scheduler.DAGScheduler - ResultStage 1 (isEmpty at DeltaSync.java:337) failed in 12.146 s due to Job aborted due to stage failure: Task 0 in stage 1.0 failed 1 times, most recent failure: Lost task 0.0 in stage 1.0 (TID 1, localhost, executor driver): org.apache.avro.UnresolvedUnionException: Not in union
> at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:740)
> at org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:205)
> at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:123)
> at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:75)
> at org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:166)
> at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:156)
> at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:118)
> at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:75)
> at org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:192)
> at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:120)
> at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:75)
> at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:125)
> at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:75)
> at org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:166)
> at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:156)
> at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:118)
> at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:75)
> at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:125)
> at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:75)
> at org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:166)
> at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:156)
> at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:118)
> at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:75)
> at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:125)
> at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:75)
> at org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:166)
> at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:156)
> at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:118)
> at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:75)
> at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:62)
> at org.apache.spark.serializer.GenericAvroSerializer.serializeDatum(GenericAvroSerializer.scala:125)
> at org.apache.spark.serializer.GenericAvroSerializer.write(GenericAvroSerializer.scala:159)
> at org.apache.spark.serializer.GenericAvroSerializer.write(GenericAvroSerializer.scala:47)
> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:651)
> at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:361)
> at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:302)
> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:651)
> at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:351)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:456)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)Driver stacktrace:
> 28702 [main] INFO org.apache.spark.scheduler.DAGScheduler - Job 1 failed: isEmpty at DeltaSync.java:337, took 12.149897 s
> 28702 [main] ERROR org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer - Got error running delta sync once. Shutting down
> org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 1 times, most recent failure: Lost task 0.0 in stage 1.0 (TID 1, localhost, executor driver): org.apache.avro.UnresolvedUnionException: Not in union [{"type":"record","name":"order_item_detail","namespace":"hoodie.source.hoodie_source.order.customer_order.customer_items","fields":[{"name":"external_id","type":[{"type":"record","name":"external_id","namespace":"hoodie.source.hoodie_source.order.customer_order.customer_items.order_item_detail","fields":[{"name":"id","type":["string","null"]},{"name":"display_id","type":["string","null"]},{"name":"exist","type":["boolean","null"]}]},"null"]},{"name":"name","type":["string","null"]},{"name":"sale_price","type":[{"type":"record","name":"sale_price","namespace":"hoodie.source.hoodie_source.order.customer_order.customer_items.order_item_detail","fields":[{"name":"currency_code","type":["string","null"]},{"name":"units","type":["long","null"]},{"name":"nanos","type":["int","null"]},{"name":"exist","type":["boolean","null"]}]},"null"]},{"name":"quantity","type":["int","null"]},{"name":"note","type":["string","null"]},{"name":"customer_item_id","type":["string","null"]},{"name":"menu_customer_item_id","type":["string","null"]},{"name":"entity_path","type":[{"type":"record","name":"entity_path","namespace":"hoodie.source.hoodie_source.order.customer_order.customer_items.order_item_detail","fields":[{"name":"path_nodes","type":[{"type":"array","items":{"type":"record","name":"path_nodes","namespace":"hoodie.source.hoodie_source.order.customer_order.customer_items.order_item_detail.entity_path","fields":[{"name":"id","type":["string","null"]},{"name":"type","type":["string","null"]},{"name":"exist","type":["boolean","null"]}]}},"null"]},{"name":"exist","type":["boolean","null"]}]},"null"]},{"name":"exist","type":["boolean","null"]}]},"null"]: {"external_id": null, "name": "Item 0", "sale_price": {"currency_code": "KRW", "units": 900, "nanos": 0, "exist": null}, "quantity": 1, "note": "Item 0 note", "customer_item_id": "37a49c46-42dd-4306-8ea5-e542bdfc0b0c", "menu_customer_item_id": "", "entity_path": null, "exist": null}
> at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:740)
> at org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:205)
> at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:123)
> at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:75)
> at org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:166)
> at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:156)
> at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:118)
> at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:75)
> at org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:192)
> at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:120)
> at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:75)
> at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:125)
> at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:75)
> at org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:166)
> at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:156)
> at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:118)
> at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:75)
> at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:125)
> at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:75)
> at org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:166)
> at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:156)
> at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:118)
> at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:75)
> at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:125)
> at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:75)
> at org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:166)
> at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:156)
> at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:118)
> at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:75)
> at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:62)
> at org.apache.spark.serializer.GenericAvroSerializer.serializeDatum(GenericAvroSerializer.scala:125)
> at org.apache.spark.serializer.GenericAvroSerializer.write(GenericAvroSerializer.scala:159)
> at org.apache.spark.serializer.GenericAvroSerializer.write(GenericAvroSerializer.scala:47)
> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:651)
> at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:361)
> at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:302)
> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:651)
> at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:351)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:456)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
>
> For a problematic pyload:
> Integer i = union.getIndexNamed(getSchemaName(datum))
> breakes to:
> union.getIndexNamed(getSchemaName(datum)) returns null.
> getSchemaName(datum) returns: hoodie.source.hoodie_source.order.customer_items.customer_items.order_item_detail
> but union's schema:
> {code:java}
> {"type":"record","name":"order_item_detail",
> "namespace":"hoodie.source.hoodie_source.order.customer_order.customer_items"
> {code}
> customer_items.customer_items is repeated in the result of getSchemaName.
> union.getIndexNamed("hoodie.source.hoodie_source.order.customer_order.customer_items.order_item_detail")
> returns proper index
--
This message was sent by Atlassian Jira
(v8.3.4#803005)