You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "CompassChengHan (via GitHub)" <gi...@apache.org> on 2024/12/13 04:14:20 UTC
[I] [Feature] TableWrite skip null fields check, when RowKind is DELETE [paimon]
CompassChengHan opened a new issue, #4702:
URL: https://github.com/apache/paimon/issues/4702
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Motivation
When deleting a row from PrimaryKey Table, we can set the RowKind type to RowKind.DELETE.
Currently, the TableWriteImpl class will always invoke checkNullability method, ensure all fileds comply with null constraint,
We suggest only verify primary keys null constraint if RowKind is DELETE, then when delete a row, only need provide primary keys.
### Solution
_No response_
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] TableWrite skip null fields check, when RowKind is DELETE [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi closed issue #4702: [Feature] TableWrite skip null fields check, when RowKind is DELETE
URL: https://github.com/apache/paimon/issues/4702
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] TableWrite skip null fields check, when RowKind is DELETE [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on issue #4702:
URL: https://github.com/apache/paimon/issues/4702#issuecomment-2544461140
> Actually, the data will be completed even though only pk provided. And since deletion is rare comparing to insert and update, ignoring other field check wont save too much time.
He is not trying to save time, he can only provide the primary key field and no other fields, in order to avoid errors
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] TableWrite skip null fields check, when RowKind is DELETE [paimon]
Posted by "JackeyLee007 (via GitHub)" <gi...@apache.org>.
JackeyLee007 commented on issue #4702:
URL: https://github.com/apache/paimon/issues/4702#issuecomment-2543876512
Actually, the data will be completed even though only pk provided. And since deletion is rare comparing to insert and update, ignoring other field check wont save too much time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] TableWrite skip null fields check, when RowKind is DELETE [paimon]
Posted by "JackeyLee007 (via GitHub)" <gi...@apache.org>.
JackeyLee007 commented on issue #4702:
URL: https://github.com/apache/paimon/issues/4702#issuecomment-2545544234
But as I tried, it does need only the primary key when deleting a record, eg.
```sql
delete from some_table where id = 123;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] TableWrite skip null fields check, when RowKind is DELETE [paimon]
Posted by "esnhysythh (via GitHub)" <gi...@apache.org>.
esnhysythh commented on issue #4702:
URL: https://github.com/apache/paimon/issues/4702#issuecomment-4955571652
> But as I tried, it does need only the primary key when deleting a record, eg.
>
> delete from some_table where id = 123;
Yes, this works in `paimon-flink-connector`, but the reason is that it first performs a query to fetch the existing row values, and then writes a row with the same column values but with `RowKind.DELETE`.
The relevant implementation is here:
`paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/action/DeleteAction.java`
So all columns are filled with their previous values from the Paimon table. For `NOT NULL` columns, they will naturally not be null, so the validation issue does not occur in this path.
However, I do not think this necessarily means the current behavior is reasonable. The extra query cost is not negligible, especially when Paimon is used as a lake table and the data may be stored on remote OSS/S3/HDFS.
In my understanding, the main impact of this issue is on changelog generation: the generated delete changelog may no longer contain the previous values. This does not affect the final table semantics, but it does lose some information in the changelog.
For tables using the `deduplicate` merge engine, I think this validation is unnecessary.
So I believe this is indeed a real issue.
---
是的，在 `paimon-flink-connector` 中，这种写法是可以工作的。但原因是它会先执行一次查询，查出待删除行的已有列值，然后再写入一行列值相同、但 `RowKind` 为 `DELETE` 的数据。
相关实现可以参考：
`paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/action/DeleteAction.java`
也就是说，所有列都会被填充为 Paimon 表中已有的值。对于 `NOT NULL` 列来说，它们自然不会为空，因此在这个路径下不会触发校验问题。
但是，我认为这并不说明当前行为一定是合理的。额外的一次查询成本并不可忽略，尤其是当 Paimon 作为数据湖表使用，并且数据存储在远端 OSS/S3/HDFS 上时。
我的理解是，这个问题主要影响 changelog 生成：生成的 delete changelog 可能不再包含删除前的旧值。这不会影响最终表语义，但确实会让 changelog 丢失部分信息。
对于使用 `deduplicate` merge engine 的表，我认为这个校验是不必要的。
因此，我认为这里确实存在一个实际问题。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org