You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/04/21 14:33:14 UTC
[GitHub] [hudi] p-powell opened a new issue, #5385: _hoodie_is_deleted Column
p-powell opened a new issue, #5385:
URL: https://github.com/apache/hudi/issues/5385
Just a general question about the _hoodie_is_deleted column. Do we need to include the _hoodie_is_deleted column in the initial load of data to a hudi table? Is it expected to be there?
Environment Description
Hudi version : 0.8.0 / 0.9.0
Spark version : 3.2
Hive version :
Hadoop version : 3.3.1
Storage (HDFS/S3/GCS..) :
Running on Docker? (yes/no) : yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #5385: _hoodie_is_deleted Column
Posted by GitBox <gi...@apache.org>.
codope commented on issue #5385:
URL: https://github.com/apache/hudi/issues/5385#issuecomment-1109727087
@p-powell It's not strictly necessary. If your use case is to delete records, then Hudi supports two kinds of deletes: soft deletes (just null out the values) and hard deletes (physically remove, no trace of record). For the latter, you would need top include _hoodie_is_deleted. You can read more here: https://hudi.apache.org/docs/writing_data/#deletes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] p-powell commented on issue #5385: _hoodie_is_deleted Column
Posted by GitBox <gi...@apache.org>.
p-powell commented on issue #5385:
URL: https://github.com/apache/hudi/issues/5385#issuecomment-1110394248
@yihua what do you mean by schema update? Is the modifying existing hudi table before upsert? Or you talking about just adding _hoodie_is_deleted to the data set used for the upsert?
Also is _hoodie_is_deleted a system field? If so why isnt it always present?
Is there a good example of this?
tia
Peter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pratyakshsharma commented on issue #5385: _hoodie_is_deleted Column
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on issue #5385:
URL: https://github.com/apache/hudi/issues/5385#issuecomment-1111347648
it is not necessary to add the column first before actually writing data with that column. Hudi can implicitly take care of that provided the schema is evolved in backward compatible ways. Hope that helps.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] p-powell closed issue #5385: _hoodie_is_deleted Column
Posted by GitBox <gi...@apache.org>.
p-powell closed issue #5385: _hoodie_is_deleted Column
URL: https://github.com/apache/hudi/issues/5385
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on issue #5385: _hoodie_is_deleted Column
Posted by GitBox <gi...@apache.org>.
yihua commented on issue #5385:
URL: https://github.com/apache/hudi/issues/5385#issuecomment-1111537861
> The above test case does not add _hoodie_is_deleted to the existing hudi table before writing a dataset with _hoodie_is_deleted column.
In that case, the dataset to be written has the new schema, and Hudi automatically picks that up and evolves the table schema when writing the data.
> Also, what is the best way to modify the schema of an existing hudi table?
As @pratyakshsharma mentioned, you don't explicitly modify the schema of an existing Hudi table manually. Hudi takes care of it in a backward-compatible way. You only need to provide the new schema or the data with new schema. For the example I gave, the changed schema is passed to Deltastreamer as an argument so that the deltasteamer can pick that up, and update the table with the right schema. For Spark datasource (e.g., `df.write`), you don't even need to change the schema, since the dataframe to write has embedded schema with the field `_hoodie_is_deleted`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on issue #5385: _hoodie_is_deleted Column
Posted by GitBox <gi...@apache.org>.
yihua commented on issue #5385:
URL: https://github.com/apache/hudi/issues/5385#issuecomment-1110592113
> what do you mean by schema update? Is the modifying existing hudi table before upsert? Or you talking about just adding _hoodie_is_deleted to the data set used for the upsert?
So you need to change the schema by adding the `_hoodie_is_deleted` to schema before the next upsert. Then, for the upsert, you need to have the field `_hoodie_is_deleted` for the batch and set the `_hoodie_is_deleted` to true for the records to be deleted. You can find a concrete example below derived from the [Deletes docs](https://hudi.apache.org/docs/writing_data/#deletes).
> Also is _hoodie_is_deleted a system field? If so why isnt it always present?
`_hoodie_is_deleted` is used by the write client internally to identify deletes. Only when you want to support both inserts, updates, and deletes in the same batch with UPSERT operation, you need this field. We're going to relax this requirement of adding `_hoodie_is_deleted` in the future. This field is not required in all cases, e.g., the DELETE operation, normal inserts, upserts without deletes.
> Is there a good example of this?
You can check [Deletes docs](https://hudi.apache.org/docs/writing_data/#deletes):
- Using DataSource or DeltaStreamer to delete records: add a column named `_hoodie_is_deleted` to DataSet. The value of this column must be set to `true` for all the records to be deleted and either `false` or left `null` for any records which are to be upserted.
Let's say the original schema is:
```
{
"type":"record",
"name":"example_tbl",
"fields":[{
"name": "uuid",
"type": "String"
}, {
"name": "ts",
"type": "string"
}, {
"name": "partitionPath",
"type": "string"
}, {
"name": "rank",
"type": "long"
}
]}
```
Make sure you add `_hoodie_is_deleted` column:
```
{
"type":"record",
"name":"example_tbl",
"fields":[{
"name": "uuid",
"type": "String"
}, {
"name": "ts",
"type": "string"
}, {
"name": "partitionPath",
"type": "string"
}, {
"name": "rank",
"type": "long"
}, {
"name" : "_hoodie_is_deleted",
"type" : "boolean",
"default" : false
}
]}
```
Then any record you want to delete you can mark `_hoodie_is_deleted` as true:
```
{"ts": 0.0, "uuid": "19tdb048-c93e-4532-adf9-f61ce6afe10", "rank": 1045, "partitionpath": "americas/brazil/sao_paulo", "_hoodie_is_deleted" : true}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on issue #5385: _hoodie_is_deleted Column
Posted by GitBox <gi...@apache.org>.
yihua commented on issue #5385:
URL: https://github.com/apache/hudi/issues/5385#issuecomment-1113636680
@p-powell let me know if you have more questions. Feel free to close this issue if all good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] p-powell commented on issue #5385: _hoodie_is_deleted Column
Posted by GitBox <gi...@apache.org>.
p-powell commented on issue #5385:
URL: https://github.com/apache/hudi/issues/5385#issuecomment-1110126327
@codope Thanks for the response. Bit confused because in this [test case](https://github.com/apache/hudi/pull/3541/commits/694300477f61a9169c15cac1ddf67368dbf5dd1b) _hoodie_is_deleted is only being added to the update dataframe. And the record to be deleted is set to True. On the initial save to the hudi table the _hoodie_is_deleted is not being included. I am sure this test case is passing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on issue #5385: _hoodie_is_deleted Column
Posted by GitBox <gi...@apache.org>.
yihua commented on issue #5385:
URL: https://github.com/apache/hudi/issues/5385#issuecomment-1110355105
@p-powell Yes, to answer your question directly, the initial insert to the table does not need `_hoodie_is_deleted` field and that field can be added later to the new batch with schema update when doing deletes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] p-powell commented on issue #5385: _hoodie_is_deleted Column
Posted by GitBox <gi...@apache.org>.
p-powell commented on issue #5385:
URL: https://github.com/apache/hudi/issues/5385#issuecomment-1111048795
@yihua Thanks for the response. Does that mean that this test case fails then? https://github.com/apache/hudi/pull/3541/commits/694300477f61a9169c15cac1ddf67368dbf5dd1b
The above test case does not add _hoodie_is_deleted to the existing hudi table before writing a dataset with _hoodie_is_deleted column.
Also, what is the best way to modify the schema of an existing hudi table?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org