You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "ROOBALJINDAL (via GitHub)" <gi...@apache.org> on 2025/11/20 03:57:39 UTC
[I] Hudi write succeeds for the first time but fails second time: Can't redefine: Pay [hudi]
ROOBALJINDAL opened a new issue, #14306:
URL: https://github.com/apache/hudi/issues/14306
### Bug Description
**What happened:**
Schema:
```
root
|-- TotalPay: struct (nullable = true)
| |-- Insurance: struct (nullable = true)
| | |-- Amount: double (nullable = true)
| | |-- Basis: string (nullable = true)
| | |-- Rate: double (nullable = true)
| |-- Pay: struct (nullable = true)
| | |-- Amount: decimal(20,5) (nullable = true)
| | |-- Basis: string (nullable = true)
| | |-- Rate: decimal(20,5) (nullable = true)
|-- ProductLine: struct (nullable = true)
| |-- ProductPrice: struct (nullable = true)
| | |-- BasisCalculation: array (nullable = true)
| | | |-- element: struct (containsNull = true)
| | | | |-- Insurance: double (nullable = true)
| | | | |-- Pay: decimal(20,5) (nullable = true)
```
Hudi write works for the first time but fails for the subsequent writes.
**What you expected:**
I expected it to have a successful save run second time too
**Steps to reproduce:**
1. Ingest some data, it succeeds first time
2. Again ingest new data to hudi table, it fails
Dont know if its a similar issue or not, its still open: https://github.com/apache/hudi/issues/10983
### Environment
**Hudi version:** 0.14.0-amzn-0
**Query engine:** (Spark/Flink/Trino etc) Trino
Using EMR 6.15, even tried on latest EMR 7.10 but still the issue remains the same
### Logs and Stack Trace
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 92.0 failed 4 times, most recent failure: Lost task 0.3 in stage 92.0 (TID 889) ([2600:1f14:7b3:b702:11fd:2236:6cdf:6263] executor 4): org.apache.hudi.exception.HoodieUpsertException: Error upserting bucketType UPDATE for partition :0
at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpsertPartition(BaseSparkCommitActionExecutor.java:342)
at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.lambda$mapPartitionsAsRDD$a3ab3c4$1(BaseSparkCommitActionExecutor.java:257)
at org.apache.spark.api.java.JavaRDDLike.$anonfun$mapPartitionsWithIndex$1(JavaRDDLike.scala:102)
at org.apache.spark.api.java.JavaRDDLike.$anonfun$mapPartitionsWithIndex$1$adapted(JavaRDDLike.scala:102)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsWithIndex$2(RDD.scala:905)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsWithIndex$2$adapted(RDD.scala:905)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:328)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)
at org.apache.spark.rdd.RDD.$anonfun$getOrCompute$1(RDD.scala:377)
at org.apache.spark.storage.BlockManager.$anonfun$doPutIterator$1(BlockManager.scala:1552)
at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$doPut(BlockManager.scala:1462)
at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1526)
at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:1349)
at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:375)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:326)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:328)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:92)
at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
at org.apache.spark.scheduler.Task.run(Task.scala:141)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:563)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1541)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:566)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.hudi.exception.HoodieException: org.apache.avro.SchemaParseException: Can't redefine: Pay
at org.apache.hudi.table.action.commit.HoodieMergeHelper.runMerge(HoodieMergeHelper.java:149)
at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpdateInternal(BaseSparkCommitActionExecutor.java:387)
at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpdate(BaseSparkCommitActionExecutor.java:369)
at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpsertPartition(BaseSparkCommitActionExecutor.java:335)
... 29 more
Caused by: org.apache.avro.SchemaParseException: Can't redefine: Pay
at org.apache.avro.Schema$Names.put(Schema.java:1586)
at org.apache.avro.Schema$NamedSchema.writeNameRef(Schema.java:844)
at org.apache.avro.Schema$FixedSchema.toJson(Schema.java:1315)
at org.apache.avro.Schema$UnionSchema.toJson(Schema.java:1278)
at org.apache.avro.Schema$RecordSchema.fieldsToJson(Schema.java:1039)
at org.apache.avro.Schema$RecordSchema.toJson(Schema.java:1023)
at org.apache.avro.Schema$ArraySchema.toJson(Schema.java:1173)
at org.apache.avro.Schema$UnionSchema.toJson(Schema.java:1278)
at org.apache.avro.Schema$RecordSchema.fieldsToJson(Schema.java:1039)
at org.apache.avro.Schema$RecordSchema.toJson(Schema.java:1023)
at org.apache.avro.Schema$UnionSchema.toJson(Schema.java:1278)
at org.apache.avro.Schema$RecordSchema.fieldsToJson(Schema.java:1039)
at org.apache.avro.Schema$RecordSchema.toJson(Schema.java:1023)
at org.apache.avro.Schema$ArraySchema.toJson(Schema.java:1173)
at org.apache.avro.Schema$UnionSchema.toJson(Schema.java:1278)
at org.apache.avro.Schema$RecordSchema.fieldsToJson(Schema.java:1039)
at org.apache.avro.Schema$RecordSchema.toJson(Schema.java:1023)
at org.apache.avro.Schema.toString(Schema.java:433)
at org.apache.avro.Schema.toString(Schema.java:405)
at org.apache.avro.Schema.toString(Schema.java:396)
at org.apache.parquet.avro.AvroReadSupport.setAvroReadSchema(AvroReadSupport.java:73)
at org.apache.hudi.io.storage.HoodieAvroParquetReader.getIndexedRecordIteratorInternal(HoodieAvroParquetReader.java:162)
at org.apache.hudi.io.storage.HoodieAvroParquetReader.getIndexedRecordIterator(HoodieAvroParquetReader.java:94)
at org.apache.hudi.io.storage.HoodieAvroParquetReader.getRecordIterator(HoodieAvroParquetReader.java:73)
at org.apache.hudi.table.action.commit.HoodieMergeHelper.runMerge(HoodieMergeHelper.java:126)
... 32 more
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2974)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2910)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2909)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2909)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1263)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1263)
at scala.Option.foreach(Option.scala:407)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1263)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:3173)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:3112)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:3101)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:1028)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2271)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2292)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2311)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2336)
at org.apache.spark.rdd.RDD.count(RDD.scala:1266)
at org.apache.hudi.HoodieSparkSqlWriter$.commitAndPerformPostOperations(HoodieSparkSqlWriter.scala:1050)
at org.apache.hudi.HoodieSparkSqlWriter$.writeInternal(HoodieSparkSqlWriter.scala:441)
at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:132)
at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:150)
at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:47)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:84)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:104)
at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)
at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:250)
at org.apache.spark.sql.execution.SQLExecution$.executeQuery$1(SQLExecution.scala:123)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$9(SQLExecution.scala:160)
at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)
at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:250)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$8(SQLExecution.scala:160)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:271)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:159)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:827)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:69)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:101)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:97)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:554)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:107)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:554)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:32)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:32)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:32)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:530)
at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:97)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:84)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:82)
at org.apache.spark.sql.execution.QueryExecution.assertCommandExecuted(QueryExecution.scala:142)
at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:856)
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:387)
at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:360)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:239)
at com.edifecs.em.cloud.ecf.ECFSaveComplexData.execute(ECFSaveComplexData.java:38)
at com.edifecs.em.cloud.ecf.ECFInvoker.execute(ECFInvoker.java:17)
at com.edifecs.em.cloud.ecf.util.CommonUtil.saveData(CommonUtil.java:98)
at com.edifecs.em.cloud.ecf.util.ECFProccessUtil.processAndSaveHierarchicalData(ECFProccessUtil.java:124)
at com.edifecs.em.cloud.ecf.ECFExtractETL.lambda$main$0(ECFExtractETL.java:456)
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290)
at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool.helpComplete(ForkJoinPool.java:1870)
at java.util.concurrent.ForkJoinPool.externalHelpComplete(ForkJoinPool.java:2467)
at java.util.concurrent.ForkJoinTask.externalAwaitDone(ForkJoinTask.java:324)
at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:405)
at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734)
at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159)
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650)
at com.edifecs.em.cloud.ecf.ECFExtractETL.main(ECFExtractETL.java:449)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1066)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1158)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1167)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.hudi.exception.HoodieUpsertException: Error upserting bucketType UPDATE for partition :0
at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpsertPartition(BaseSparkCommitActionExecutor.java:342)
at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.lambda$mapPartitionsAsRDD$a3ab3c4$1(BaseSparkCommitActionExecutor.java:257)
at org.apache.spark.api.java.JavaRDDLike.$anonfun$mapPartitionsWithIndex$1(JavaRDDLike.scala:102)
at org.apache.spark.api.java.JavaRDDLike.$anonfun$mapPartitionsWithIndex$1$adapted(JavaRDDLike.scala:102)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsWithIndex$2(RDD.scala:905)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsWithIndex$2$adapted(RDD.scala:905)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:328)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)
at org.apache.spark.rdd.RDD.$anonfun$getOrCompute$1(RDD.scala:377)
at org.apache.spark.storage.BlockManager.$anonfun$doPutIterator$1(BlockManager.scala:1552)
at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$doPut(BlockManager.scala:1462)
at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1526)
at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:1349)
at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:375)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:326)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:328)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:92)
at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
at org.apache.spark.scheduler.Task.run(Task.scala:141)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:563)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1541)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:566)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.hudi.exception.HoodieException: org.apache.avro.SchemaParseException: Can't redefine: Pay
at org.apache.hudi.table.action.commit.HoodieMergeHelper.runMerge(HoodieMergeHelper.java:149)
at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpdateInternal(BaseSparkCommitActionExecutor.java:387)
at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpdate(BaseSparkCommitActionExecutor.java:369)
at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpsertPartition(BaseSparkCommitActionExecutor.java:335)
... 29 more
Caused by: org.apache.avro.SchemaParseException: Can't redefine: Pay
at org.apache.avro.Schema$Names.put(Schema.java:1586)
at org.apache.avro.Schema$NamedSchema.writeNameRef(Schema.java:844)
at org.apache.avro.Schema$FixedSchema.toJson(Schema.java:1315)
at org.apache.avro.Schema$UnionSchema.toJson(Schema.java:1278)
at org.apache.avro.Schema$RecordSchema.fieldsToJson(Schema.java:1039)
at org.apache.avro.Schema$RecordSchema.toJson(Schema.java:1023)
at org.apache.avro.Schema$ArraySchema.toJson(Schema.java:1173)
at org.apache.avro.Schema$UnionSchema.toJson(Schema.java:1278)
at org.apache.avro.Schema$RecordSchema.fieldsToJson(Schema.java:1039)
at org.apache.avro.Schema$RecordSchema.toJson(Schema.java:1023)
at org.apache.avro.Schema$UnionSchema.toJson(Schema.java:1278)
at org.apache.avro.Schema$RecordSchema.fieldsToJson(Schema.java:1039)
at org.apache.avro.Schema$RecordSchema.toJson(Schema.java:1023)
at org.apache.avro.Schema$ArraySchema.toJson(Schema.java:1173)
at org.apache.avro.Schema$UnionSchema.toJson(Schema.java:1278)
at org.apache.avro.Schema$RecordSchema.fieldsToJson(Schema.java:1039)
at org.apache.avro.Schema$RecordSchema.toJson(Schema.java:1023)
at org.apache.avro.Schema.toString(Schema.java:433)
at org.apache.avro.Schema.toString(Schema.java:405)
at org.apache.avro.Schema.toString(Schema.java:396)
at org.apache.parquet.avro.AvroReadSupport.setAvroReadSchema(AvroReadSupport.java:73)
at org.apache.hudi.io.storage.HoodieAvroParquetReader.getIndexedRecordIteratorInternal(HoodieAvroParquetReader.java:162)
at org.apache.hudi.io.storage.HoodieAvroParquetReader.getIndexedRecordIterator(HoodieAvroParquetReader.java:94)
at org.apache.hudi.io.storage.HoodieAvroParquetReader.getRecordIterator(HoodieAvroParquetReader.java:73)
at org.apache.hudi.table.action.commit.HoodieMergeHelper.runMerge(HoodieMergeHelper.java:126)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Hudi write succeeds for the first time but fails second time: Can't redefine: Pay [hudi]
Posted by "deepakpanda93 (via GitHub)" <gi...@apache.org>.
deepakpanda93 commented on issue #14306:
URL: https://github.com/apache/hudi/issues/14306#issuecomment-3616832970
Hello @ROOBALJINDAL
Thank you for sharing more information.
I tried with xml files too. Still not able to reproduce the issue. Below are the schema/input file and the sample code I have used for the test.
[input11.xml](https://github.com/user-attachments/files/23961541/input11.xml)
[input22.xml](https://github.com/user-attachments/files/23961542/input22.xml)
[repro_hudi_xml_bug_v1.py](https://github.com/user-attachments/files/23961543/repro_hudi_xml_bug_v1.py)
[schema1.xml](https://github.com/user-attachments/files/23961540/schema1.xml)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Hudi write succeeds for the first time but fails second time: Can't redefine: Pay [hudi]
Posted by "ROOBALJINDAL (via GitHub)" <gi...@apache.org>.
ROOBALJINDAL commented on issue #14306:
URL: https://github.com/apache/hudi/issues/14306#issuecomment-3668626825
@deepakpanda93 great. We are using EMR 6.15, even we migrated to 7.10 EMR which uses 0.15.0-amzn-7 that also didnt work. Since we are relying on EMR releases, do you have any work-around to fix this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Hudi write succeeds for the first time but fails second time: Can't redefine: Pay [hudi]
Posted by "ROOBALJINDAL (via GitHub)" <gi...@apache.org>.
ROOBALJINDAL commented on issue #14306:
URL: https://github.com/apache/hudi/issues/14306#issuecomment-3664777036
Strange. I see 1.0E15 value in your case instead of 999999999999999.99999, probably due to double datatype.
Can you try passing in same static hardcoded schema but with decimal(20,5) type only for Services.BasisCalculation.Pay? It should be reproducible with that?
@deepakpanda93
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Hudi write succeeds for the first time but fails second time: Can't redefine: Pay [hudi]
Posted by "deepakpanda93 (via GitHub)" <gi...@apache.org>.
deepakpanda93 commented on issue #14306:
URL: https://github.com/apache/hudi/issues/14306#issuecomment-3670078356
@ROOBALJINDAL allow me sometime to check this internally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Hudi write succeeds for the first time but fails second time: Can't redefine: Pay [hudi]
Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy commented on issue #14306:
URL: https://github.com/apache/hudi/issues/14306#issuecomment-3815385825
Hi @ROOBALJINDAL
Did you try the suggestion from @deepakpanda93 above to resolve your issue? If not, please give it a try and let me know if you require any further help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Hudi write succeeds for the first time but fails second time: Can't redefine: Pay [hudi]
Posted by "ROOBALJINDAL (via GitHub)" <gi...@apache.org>.
ROOBALJINDAL commented on issue #14306:
URL: https://github.com/apache/hudi/issues/14306#issuecomment-3833491179
@deepakpanda93 @rangareddy we have made a change in spark-xml library to rename decimals and it worked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Hudi write succeeds for the first time but fails second time: Can't redefine: Pay [hudi]
Posted by "deepakpanda93 (via GitHub)" <gi...@apache.org>.
deepakpanda93 commented on issue #14306:
URL: https://github.com/apache/hudi/issues/14306#issuecomment-3562333908
Hello @ROOBALJINDAL
I tested the Hudi Avro "Can’t redefine field" issue with my dataset and Hudi 0.14 + Spark 3.4, and the second write worked successfully without any errors. I have attached my sample code below for reference.
Please review and let me know if you would like me to run it in any other specific way.
[SchemaParseException.py](https://github.com/user-attachments/files/23673289/SchemaParseException.py)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Hudi write succeeds for the first time but fails second time: Can't redefine: Pay [hudi]
Posted by "deepakpanda93 (via GitHub)" <gi...@apache.org>.
deepakpanda93 commented on issue #14306:
URL: https://github.com/apache/hudi/issues/14306#issuecomment-3664750132
@ROOBALJINDAL I am nowhere specifying double for pay field. I have used the below for the schema/input files.
```
<Company>
<TotalPay>
<Pay>
<Rate value="999999999999999.99999"/>
<Basis value="twenty"/>
</Pay>
</TotalPay>
<Services>
<BasisCalculation>
<Pay>999999999999999.99999</Pay>
</BasisCalculation>
</Services>
</Company>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Hudi write succeeds for the first time but fails second time: Can't redefine: Pay [hudi]
Posted by "deepakpanda93 (via GitHub)" <gi...@apache.org>.
deepakpanda93 commented on issue #14306:
URL: https://github.com/apache/hudi/issues/14306#issuecomment-3688929353
@ROOBALJINDAL I tried an alternative approach by forcing the `Pay` field to be of type String, and it worked. This was tested using Hudi 0.15 with Spark 3.4.
If aggregations or mathematical calculations are required on the `Pay` field, it can be explicitly type-cast at query time before performing those operations.
Schema:
```
root
|-- _hoodie_commit_time: string (nullable = true)
|-- _hoodie_commit_seqno: string (nullable = true)
|-- _hoodie_record_key: string (nullable = true)
|-- _hoodie_partition_path: string (nullable = true)
|-- _hoodie_file_name: string (nullable = true)
|-- Services: struct (nullable = true)
| |-- BasisCalculation: struct (nullable = true)
| | |-- Pay: string (nullable = true)
|-- TotalPay: struct (nullable = true)
| |-- Pay: struct (nullable = true)
| | |-- Rate: struct (nullable = true)
| | | |-- _value: double (nullable = true)
| | | |-- defvalue: string (nullable = true)
| | |-- Basis: struct (nullable = true)
| | | |-- _value: string (nullable = true)
| | | |-- defvalue: string (nullable = true)
|-- record_key: string (nullable = false)
```
Output:
```
+-------------------+---------------------+------------------+----------------------+------------------------------------------------------------------------+-------------------------+--------------------------------------+----------+
|_hoodie_commit_time|_hoodie_commit_seqno |_hoodie_record_key|_hoodie_partition_path|_hoodie_file_name |Services |TotalPay |record_key|
+-------------------+---------------------+------------------+----------------------+------------------------------------------------------------------------+-------------------------+--------------------------------------+----------+
|20251224062441786 |20251224062441786_0_0|1 | |7b74de72-2a28-4464-b19d-e3894e895c59-0_0-52-76_20251224062441786.parquet|{{999999999999999.99999}}|{{{1.0E15, null}, {twentyFive, null}}}|1 |
+-------------------+---------------------+------------------+----------------------+------------------------------------------------------------------------+-------------------------+--------------------------------------+----------+
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Hudi write succeeds for the first time but fails second time: Can't redefine: Pay [hudi]
Posted by "ROOBALJINDAL (via GitHub)" <gi...@apache.org>.
ROOBALJINDAL commented on issue #14306:
URL: https://github.com/apache/hudi/issues/14306#issuecomment-3663766990
@deepakpanda93 did you check? were you able to reproduce it now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Hudi write succeeds for the first time but fails second time: Can't redefine: Pay [hudi]
Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy closed issue #14306: Hudi write succeeds for the first time but fails second time: Can't redefine: Pay
URL: https://github.com/apache/hudi/issues/14306
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Hudi write succeeds for the first time but fails second time: Can't redefine: Pay [hudi]
Posted by "deepakpanda93 (via GitHub)" <gi...@apache.org>.
deepakpanda93 commented on issue #14306:
URL: https://github.com/apache/hudi/issues/14306#issuecomment-3664648803
@ROOBALJINDAL
I used the same sample code with your updated schema/input file. Below is the outcome.
```
=== Loaded schema ===
{"fields":[{"metadata":{},"name":"Services","nullable":true,"type":{"fields":[{"metadata":{},"name":"BasisCalculation","nullable":true,"type":{"fields":[{"metadata":{},"name":"Pay","nullable":true,"type":"double"}],"type":"struct"}}],"type":"struct"}},{"metadata":{},"name":"TotalPay","nullable":true,"type":{"fields":[{"metadata":{},"name":"Pay","nullable":true,"type":{"fields":[{"metadata":{},"name":"Basis","nullable":true,"type":{"fields":[{"metadata":{},"name":"_value","nullable":true,"type":"string"},{"metadata":{},"name":"defvalue","nullable":true,"type":"string"}],"type":"struct"}},{"metadata":{},"name":"Rate","nullable":true,"type":{"fields":[{"metadata":{},"name":"_value","nullable":true,"type":"double"},{"metadata":{},"name":"defvalue","nullable":true,"type":"string"}],"type":"struct"}}],"type":"struct"}}],"type":"struct"}}],"type":"struct"}
25/12/17 10:06:02 INFO FileInputFormat: Total input files to process : 1
=== FIRST HUDI WRITE (should succeed) ===
=== SECOND HUDI WRITE (expected to FAIL) ===
root
|-- _hoodie_commit_time: string (nullable = true)
|-- _hoodie_commit_seqno: string (nullable = true)
|-- _hoodie_record_key: string (nullable = true)
|-- _hoodie_partition_path: string (nullable = true)
|-- _hoodie_file_name: string (nullable = true)
|-- Services: struct (nullable = true)
| |-- BasisCalculation: struct (nullable = true)
| | |-- Pay: double (nullable = true)
|-- TotalPay: struct (nullable = true)
| |-- Pay: struct (nullable = true)
| | |-- Basis: struct (nullable = true)
| | | |-- _value: string (nullable = true)
| | | |-- defvalue: string (nullable = true)
| | |-- Rate: struct (nullable = true)
| | | |-- _value: double (nullable = true)
| | | |-- defvalue: string (nullable = true)
|-- record_key: string (nullable = false)
+-------------------+---------------------+------------------+----------------------+------------------------------------------------------------------------+----------+----------------------------------+----------+
|_hoodie_commit_time|_hoodie_commit_seqno |_hoodie_record_key|_hoodie_partition_path|_hoodie_file_name |Services |TotalPay |record_key|
+-------------------+---------------------+------------------+----------------------+------------------------------------------------------------------------+----------+----------------------------------+----------+
|20251217100606010 |20251217100606010_0_0|1 | |b9d5ba29-0478-49c3-b8b4-a4595e4bf455-0_0-52-76_20251217100606010.parquet|{{1.0E15}}|{{{twenty, null}, {1.0E15, null}}}|1 |
+-------------------+---------------------+------------------+----------------------+------------------------------------------------------------------------+----------+----------------------------------+----------+
```
I have checked the basepath as well. It has 2 parquet files for 2 commits.
```
root@6ba4293d5eb5:/opt# ls -l /tmp/hudi_xml_bug_table
total 856
-rw-r--r-- 1 root root 436514 Dec 17 10:06 b9d5ba29-0478-49c3-b8b4-a4595e4bf455-0_0-24-37_20251217100602427.parquet
-rw-r--r-- 1 root root 436509 Dec 17 10:06 b9d5ba29-0478-49c3-b8b4-a4595e4bf455-0_0-52-76_20251217100606010.parquet
```
Please do review my sample code (available in the previous comment) and let me know in case any modification. I am using Spark 3.4 with Hudi0.14.
```
spark-submit --packages org.apache.hudi:hudi-spark3.4-bundle_2.12:0.14.0,com.databricks:spark-xml_2.12:0.17.0 --conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer' --conf 'spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog' --conf 'spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension' --conf 'spark.kryo.registrator=org.apache.spark.HoodieSparkKryoRegistrar' repro_hudi_bug_xml.py
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Hudi write succeeds for the first time but fails second time: Can't redefine: Pay [hudi]
Posted by "ROOBALJINDAL (via GitHub)" <gi...@apache.org>.
ROOBALJINDAL commented on issue #14306:
URL: https://github.com/apache/hudi/issues/14306#issuecomment-3664689514
@deepakpanda93 reason its not reproducible for you is because it has created 'pay' of double under
```
|-- Services: struct (nullable = true)
| |-- BasisCalculation: struct (nullable = true)
| | |-- Pay: double (nullable = true)
```
Dont know why its interpreted as a double, not decimal. Issue is reproducible with a double and a struct with same name but under different nodes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Hudi write succeeds for the first time but fails second time: Can't redefine: Pay [hudi]
Posted by "ROOBALJINDAL (via GitHub)" <gi...@apache.org>.
ROOBALJINDAL commented on issue #14306:
URL: https://github.com/apache/hudi/issues/14306#issuecomment-3611733758
Defining steps to reproduce it.
Please note, we are ingesting xmls, hence we are using databricks spark-xml library : https://github.com/databricks/spark-xml/tree/v0.17.0
**schema.xml**
```
<?xml version="1.0" encoding="UTF-8"?>
<echcf:Company xmlns:echcf="http://xml.edifecs.com/schema/ECHCF" xmlns:ucf="http://xml.edifecs.com/schema/UCF" xmlns:ucfd="http://xml.edifecs.com/schema/UCFD" xmlns:hcfd="http://xml.edifecs.com/schema/HCFD" version="1.1.1.4">
<echcf:TotalPay>
<hcfd:Pay>
<hcfd:Rate value="999999999999999.99999"/>
<hcfd:Basis value="sqiqspuaxn"/>
</hcfd:Pay>
<hcfd:BasisCalculation>
<hcfd:Pay>999999999999999.99999</hcfd:Pay>
</hcfd:BasisCalculation>
</echcf:TotalPay>
</echcf:Company>
```
**input1.xml**
```
<?xml version="1.0" encoding="UTF-8"?>
<echcf:Company xmlns:echcf="http://xml.edifecs.com/schema/ECHCF" xmlns:ucf="http://xml.edifecs.com/schema/UCF" xmlns:ucfd="http://xml.edifecs.com/schema/UCFD" xmlns:hcfd="http://xml.edifecs.com/schema/HCFD" version="1.1.1.4">
<echcf:TotalPay>
<hcfd:Pay>
<hcfd:Rate value="999999999999999.99999"/>
<hcfd:Basis value="sqiqspuaxn"/>
</hcfd:Pay>
<hcfd:BasisCalculation>
<hcfd:Pay>999999999999999.99999</hcfd:Pay>
</hcfd:BasisCalculation>
</echcf:TotalPay>
</echcf:Company>
```
**input2.xml**
```
<?xml version="1.0" encoding="UTF-8"?>
<echcf:Company xmlns:echcf="http://xml.edifecs.com/schema/ECHCF" xmlns:ucf="http://xml.edifecs.com/schema/UCF" xmlns:ucfd="http://xml.edifecs.com/schema/UCFD" xmlns:hcfd="http://xml.edifecs.com/schema/HCFD" version="1.1.1.4">
<echcf:TotalPay>
<hcfd:Pay>
<hcfd:Rate value="999999999999999.99999"/>
<hcfd:Basis value="sqiqspuaxn"/>
</hcfd:Pay>
<hcfd:BasisCalculation>
<hcfd:Pay>999999999999999.99999</hcfd:Pay>
</hcfd:BasisCalculation>
</echcf:TotalPay>
</echcf:Company>
```
High level code:
```
Dataset<Row> cloudDf = spark.read().format("com.databricks.spark.xml")
.....
.option("rowTag", "Company")
.option("valueTag", "defvalue")
.load(schema.xml);
StructType schema = cloudDf.schema();
DataFrameReader sparkReader = sparkReader.option("rowTag", ecfExtractETL.getRowTag())
.schema(schema)
.option("rowTag", "Company")
.option("valueTag", "defvalue")
.option("awsRegion", "us-west-2")
.option("compression", "gzip");
Dataset<Row> dfInitial = finalSparkReader.load("input1.xml");
//save dataset to hudi with UPSERT :: successful
Dataset<Row> dfInitial = finalSparkReader.load("input2.xml");
//save dataset to hudi with UPSERT :: failed
```
First ingestion is successful but second fails with error:
`Caused by: org.apache.avro.SchemaParseException: Can't redefine: Pay
Caused by: org.apache.hudi.exception.HoodieException:
org.apache.avro.SchemaParseException: Can't redefine: Pay
Caused by: org.apache.hudi.exception.HoodieUpsertException: Error upserting bucketType UPDATE for partition :0`
We can see the first record in hive as :
```
Totalpay:
{basiscalculation={pay=999999999999999.99999}, pay={basis={_value=sqiqspuaxn, defvalue=null}, rate={_value=999999999999999.99999, defvalue=null}}}
```
Schema in trino/hive:
<img width="918" height="59" alt="Image" src="https://github.com/user-attachments/assets/9b85b95a-a0b4-4108-8788-ce14dc983d75" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Hudi write succeeds for the first time but fails second time: Can't redefine: Pay [hudi]
Posted by "deepakpanda93 (via GitHub)" <gi...@apache.org>.
deepakpanda93 commented on issue #14306:
URL: https://github.com/apache/hudi/issues/14306#issuecomment-3665233083
@ROOBALJINDAL Now I am able to reproduce the issue with `Pay: decimal(20,5)`
```
25/12/17 12:22:57 ERROR BaseSparkCommitActionExecutor: Error upserting bucketType UPDATE for partition :0
org.apache.hudi.exception.HoodieException: org.apache.avro.SchemaParseException: Can't redefine: Pay
at org.apache.hudi.table.action.commit.HoodieMergeHelper.runMerge(HoodieMergeHelper.java:149)
at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpdateInternal(BaseSparkCommitActionExecutor.java:387)
at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpdate(BaseSparkCommitActionExecutor.java:369)
at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpsertPartition(BaseSparkCommitActionExecutor.java:335)
at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$doPut(BlockManager.scala:1458)
at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1522)
at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:1349)
at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:378)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:329)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.apache.avro.SchemaParseException: Can't redefine: Pay
at org.apache.avro.Schema$Names.put(Schema.java:1586)
at org.apache.avro.Schema$NamedSchema.writeNameRef(Schema.java:844)
at org.apache.avro.Schema$RecordSchema.toJson(Schema.java:1011)
```
Schema:
```
root
|-- _hoodie_commit_time: string (nullable = true)
|-- _hoodie_commit_seqno: string (nullable = true)
|-- _hoodie_record_key: string (nullable = true)
|-- _hoodie_partition_path: string (nullable = true)
|-- _hoodie_file_name: string (nullable = true)
|-- Services: struct (nullable = true)
| |-- BasisCalculation: struct (nullable = true)
| | |-- Pay: decimal(20,5) (nullable = true)
|-- TotalPay: struct (nullable = true)
| |-- Pay: struct (nullable = true)
| | |-- Rate: struct (nullable = true)
| | | |-- _value: double (nullable = true)
| | | |-- defvalue: string (nullable = true)
| | |-- Basis: struct (nullable = true)
| | | |-- _value: string (nullable = true)
| | | |-- defvalue: string (nullable = true)
|-- record_key: string (nullable = false)
```
I tested the same with Hudi 1.1.0, and it is working without any issues.
```
Schema:
root
|-- _hoodie_commit_time: string (nullable = true)
|-- _hoodie_commit_seqno: string (nullable = true)
|-- _hoodie_record_key: string (nullable = true)
|-- _hoodie_partition_path: string (nullable = true)
|-- _hoodie_file_name: string (nullable = true)
|-- Services: struct (nullable = true)
| |-- BasisCalculation: struct (nullable = true)
| | |-- Pay: decimal(20,5) (nullable = true)
|-- TotalPay: struct (nullable = true)
| |-- Pay: struct (nullable = true)
| | |-- Rate: struct (nullable = true)
| | | |-- _value: double (nullable = true)
| | | |-- defvalue: string (nullable = true)
| | |-- Basis: struct (nullable = true)
| | | |-- _value: string (nullable = true)
| | | |-- defvalue: string (nullable = true)
|-- record_key: string (nullable = false)
+-------------------+---------------------+------------------+----------------------+------------------------------------------------------------------------+-------------------------+----------------------------------+----------+
|_hoodie_commit_time|_hoodie_commit_seqno |_hoodie_record_key|_hoodie_partition_path|_hoodie_file_name |Services |TotalPay |record_key|
+-------------------+---------------------+------------------+----------------------+------------------------------------------------------------------------+-------------------------+----------------------------------+----------+
|20251217124148081 |20251217124148081_0_0|1 | |2503c99d-d6d8-4173-bd32-5811e9db0c57-0_0-47-82_20251217124148081.parquet|{{999999999999999.99999}}|{{{1.0E15, null}, {twenty, null}}}|1 |
+-------------------+---------------------+------------------+----------------------+------------------------------------------------------------------------+-------------------------+----------------------------------+----------+
root@6ba4293d5eb5:/opt# ls -l /tmp/hudi_xml_bug_table_110/
total 856
-rw-r--r-- 1 root root 436705 Dec 17 12:41 2503c99d-d6d8-4173-bd32-5811e9db0c57-0_0-26-40_20251217124145257.parquet
-rw-r--r-- 1 root root 436705 Dec 17 12:41 2503c99d-d6d8-4173-bd32-5811e9db0c57-0_0-47-82_20251217124148081.parquet
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Hudi write succeeds for the first time but fails second time: Can't redefine: Pay [hudi]
Posted by "ROOBALJINDAL (via GitHub)" <gi...@apache.org>.
ROOBALJINDAL commented on issue #14306:
URL: https://github.com/apache/hudi/issues/14306#issuecomment-3641477241
@deepakpanda93 sorry for the back and forth, It will be reproduced with the following
```
<Company>
<TotalPay>
<Pay>
<Rate value="999999999999999.99999"/>
<Basis value="twenty"/>
</Pay>
</TotalPay>
<Services>
<BasisCalculation>
<Pay>999999999999999.99999</Pay>
</BasisCalculation>
</Services>
</Company>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Hudi write succeeds for the first time but fails second time: Can't redefine: Pay [hudi]
Posted by "ROOBALJINDAL (via GitHub)" <gi...@apache.org>.
ROOBALJINDAL commented on issue #14306:
URL: https://github.com/apache/hudi/issues/14306#issuecomment-3686695280
@deepakpanda93 let me know if you find something
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Hudi write succeeds for the first time but fails second time: Can't redefine: Pay [hudi]
Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy commented on issue #14306:
URL: https://github.com/apache/hudi/issues/14306#issuecomment-3845272660
Thanks @ROOBALJINDAL for the update. I am closing this issue now. Please feel free to open a new one if you have any further questions or concerns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Hudi write succeeds for the first time but fails second time: Can't redefine: Pay [hudi]
Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy commented on issue #14306:
URL: https://github.com/apache/hudi/issues/14306#issuecomment-3570481595
Hi @ROOBALJINDAL @deepakpanda93 provided a script to reproduce this issue, but reproduction was unsuccessful. Could you please update the script accordingly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Hudi write succeeds for the first time but fails second time: Can't redefine: Pay [hudi]
Posted by "ROOBALJINDAL (via GitHub)" <gi...@apache.org>.
ROOBALJINDAL commented on issue #14306:
URL: https://github.com/apache/hudi/issues/14306#issuecomment-3555658296
fyi.. @ad1happy2go @nsivabalan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Hudi write succeeds for the first time but fails second time: Can't redefine: Pay [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #14306:
URL: https://github.com/apache/hudi/issues/14306#issuecomment-3560087827
hey @rangareddy : can you follow up here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org