You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/05/04 15:39:33 UTC
[GitHub] [incubator-hudi] AnkitMishra3220 opened a new issue #1591: Skip Null Value on Upsert
AnkitMishra3220 opened a new issue #1591:
URL: https://github.com/apache/incubator-hudi/issues/1591
**_Tips before filing an issue_**
- Have you gone through our [FAQs](https://cwiki.apache.org/confluence/display/HUDI/FAQ)? Yes
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
**Describe the problem you faced**
My use-case is to design data lakes from like events which I am getting from Kafka. It also requires
functionality '**Skip Null Value on Upsert**' because we have data for fields specific to events.
Can we skip null values on Upsert ? I have not found any switch for this functionality.
A clear and concise description of the problem.
Want to achieve **Skip null value on upsert** functionality.
**To Reproduce**
Steps to reproduce the behavior:
1. Insert Data to Hudi Table hudi_trips_snapshot
df.write.format("hudi").options(getQuickstartWriteConfigs).option(PRECOMBINE_FIELD_OPT_KEY, "ts").option(RECORDKEY_FIELD_OPT_KEY, "uuid").option(PARTITIONPATH_FIELD_OPT_KEY, "partitionpath").option(TABLE_NAME, tableName). mode(Overwrite).save(basePath)
val tripsSnapshotDF = spark.read.format("hudi").load(basePath + "/*/*/*/*")
tripsSnapshotDF.createOrReplaceTempView("hudi_trips_snapshot")
<img width="1234" alt="Screenshot 2020-05-04 at 8 22 55 PM" src="https://user-images.githubusercontent.com/40383808/80984339-59e39580-8e4b-11ea-8f29-640bb1a6dd09.png">
2. Create a Dataframe which contain null values for one column and updated values for another column.
<img width="1164" alt="Screenshot 2020-05-04 at 8 22 43 PM" src="https://user-images.githubusercontent.com/40383808/80984400-6ff15600-8e4b-11ea-9058-96a58c597a76.png">
3. Update the Hudi Table with new dataframe.
updateDf.write.format("org.apache.hudi").options(getQuickstartWriteConfigs).option(PRECOMBINE_FIELD_OPT_KEY, "ts").option(RECORDKEY_FIELD_OPT_KEY, "uuid").option(PARTITIONPATH_FIELD_OPT_KEY, "partitionpath").option(TABLE_NAME, tableName).mode(Append).save(basePath)
4. Check updated table. It is providing updated values but not skipping null values on Upsert.
<img width="1184" alt="Screenshot 2020-05-04 at 8 22 06 PM" src="https://user-images.githubusercontent.com/40383808/80984454-813a6280-8e4b-11ea-8b4d-aefdcefecaa8.png">
**Expected behavior**
There should be some switch/property present for functionality '**Skip Null Value on Upsert**'
**Environment Description**
* Hudi version : 0.5.2-incubating
* Spark version : 2.4.4
* Hive version : 3.1.2
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) : No
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-hudi] AnkitMishra3220 commented on issue #1591: Skip Null Value on Upsert
Posted by GitBox <gi...@apache.org>.
AnkitMishra3220 commented on issue #1591:
URL: https://github.com/apache/incubator-hudi/issues/1591#issuecomment-624056620
Thanks @pratyakshsharma ! I will check second option and try to implement.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-hudi] pratyakshsharma commented on issue #1591: Skip Null Value on Upsert
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on issue #1591:
URL: https://github.com/apache/incubator-hudi/issues/1591#issuecomment-624106521
Sounds good. Let us know if you face any more issues :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-hudi] pratyakshsharma commented on issue #1591: Skip Null Value on Upsert
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on issue #1591:
URL: https://github.com/apache/incubator-hudi/issues/1591#issuecomment-623693070
Hi @AnkitMishra3220 IIUC you want the records to be updated but want to retain the value of the field which has updated value as null. Unfortunately there is no such switch to skip null values on upsert as of now. You can either try to populate default values for fields with null values or try to do some workarounds using transformers (but I feel that will not be a good way :( ).
Another way to handle this can be comparing old and new parquet files after every hudi commit, and try to do one round of clean up to suit your use case. But this would need dev cycles from your side.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org