You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "yangZhengW (via GitHub)" <gi...@apache.org> on 2024/06/08 22:47:28 UTC
[I] [SUPPORT] hoodie.datasource.write.precombine.field is invalid [hudi]
yangZhengW opened a new issue, #11421:
URL: https://github.com/apache/hudi/issues/11421
When I try using hoodie.datasource.write.precombine.field + upsert to duplicate the record key, I found it's valid
Using spark-shell command line execute below code:
val schema = StructType(
List(
StructField("id", IntegerType, nullable = false),
StructField("value", StringType, nullable = true),
StructField("timestamp", IntegerType, nullable = true),
StructField("paritionpath", StringType, nullable = true)
)
)
val data2 = Seq(
Row(1, "a108", 96, "a"),
Row(2, "b108", 96, "b"),
Row(5, "c108", 990, "c")
)
val df2 = spark.createDataFrame(spark.sparkContext.parallelize(data2), schema)
df2.write.format("hudi").option("hoodie.index.type","SIMPLE")
.option("hoodie.datasource.write.partitionpath.field", "paritionpath")
.option("hoodie.datasource.write.keygenerator.class", "org.apache.hudi.keygen.SimpleKeyGenerator")
.option("hoodie.table.name", "table_name")
.option("hoodie.datasource.write.recordkey.field", "id")
.option("hoodie.datasource.write.precombine.field", "timestamp")
.option("hoodie.datasource.write.table.type", "MERGE_ON_READ")
.option("hoodie.datasource.write.operation", "upsert")
.mode(SaveMode.Append)
.save("bathpath")
val showDf = spark.read.format("hudi").load("basepath")
showDf.show()
+-------------------+--------------------+------------------+----------------------+--------------------+---+-----+---------+------------+
|_hoodie_commit_time|_hoodie_commit_seqno|_hoodie_record_key|_hoodie_partition_path| _hoodie_file_name| id|value|timestamp|paritionpath|
+-------------------+--------------------+------------------+----------------------+--------------------+---+-----+---------+------------+
| 20240608221958570|20240608221958570...| 2| b|dc45a52c-14f2-4a2...| 2| b109| 96| b|
| 20240608221958570|20240608221958570...| 5| c|528fcffd-5094-4f0...| 5| c109| 990| c|
| 20240608221958570|20240608221958570...| 1| a|2e3e5726-87bd-423...| 1| a109| 96| a|
+-------------------+--------------------+------------------+----------------------+--------------------+---+-----+---------+------------+
Then I modify the data2 as bellow
val data2 = Seq(
Row(1, "a108", 95, "a"),
Row(2, "b108", 95, "b"),
Row(5, "c108", 989, "c")
)
In my option, The result should be ignore, because the precombine key is lesser then current content.
when I run the same code:
val df2 = spark.createDataFrame(spark.sparkContext.parallelize(data2), schema)
df2.write.format("hudi").option("hoodie.index.type","SIMPLE")
.option("hoodie.datasource.write.partitionpath.field", "paritionpath")
.option("hoodie.datasource.write.keygenerator.class", "org.apache.hudi.keygen.SimpleKeyGenerator")
.option("hoodie.database.name", "gwatch_data")
.option("hoodie.table.name", "table_name")
.option("hoodie.datasource.write.recordkey.field", "id")
.option("hoodie.datasource.write.precombine.field", "timestamp")
.option("hoodie.datasource.write.table.type", "MERGE_ON_READ")
.option("hoodie.datasource.write.operation", "upsert")
.mode(SaveMode.Append)
.save("bath path")
val showDf = spark.read.format("hudi").load("basepath")
showDf.show()
I got result as below
scala> showDf.show()
+-------------------+--------------------+------------------+----------------------+--------------------+---+-----+---------+------------+
|_hoodie_commit_time|_hoodie_commit_seqno|_hoodie_record_key|_hoodie_partition_path| _hoodie_file_name| id|value|timestamp|paritionpath|
+-------------------+--------------------+------------------+----------------------+--------------------+---+-----+---------+------------+
| 20240608222058375|20240608222058375...| 2| b|dc45a52c-14f2-4a2...| 2| b108| 95| b|
| 20240608222058375|20240608222058375...| 5| c|528fcffd-5094-4f0...| 5| c108| 989| c|
| 20240608222058375|20240608222058375...| 1| a|2e3e5726-87bd-423...| 1| a108| 95| a|
+-------------------+--------------------+------------------+----------------------+--------------------+---+-----+---------+------------+
* Hudi version :0.14
* Spark version : 3.4.1
* Storage (HDFS/S3/GCS..) : S3
Why this invalid, Could you help me see it? some configuration is missing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] hoodie.datasource.write.precombine.field is invalid [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #11421:
URL: https://github.com/apache/hudi/issues/11421#issuecomment-2165419618
Thanks @yangZhengW Closing this issue.
You can also read this to understand more- https://www.google.com/search?client=safari&rls=en&q=medium+curious+case+of+DefaultAvroPayload&ie=UTF-8&oe=UTF-8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] hoodie.datasource.write.precombine.field is invalid [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #11421:
URL: https://github.com/apache/hudi/issues/11421#issuecomment-2156254116
did you try the `DefaultAvroPayload` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] hoodie.datasource.write.precombine.field is invalid [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope closed issue #11421: [SUPPORT] hoodie.datasource.write.precombine.field is invalid
URL: https://github.com/apache/hudi/issues/11421
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] hoodie.datasource.write.precombine.field is invalid [hudi]
Posted by "yangZhengW (via GitHub)" <gi...@apache.org>.
yangZhengW commented on issue #11421:
URL: https://github.com/apache/hudi/issues/11421#issuecomment-2156274391
> did you try the `DefaultAvroPayload` ?
It's valid. thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org