You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/08/10 01:07:01 UTC
[GitHub] [pinot] huaheh opened a new issue, #9188: Support to merge the existing column value and the incoming event
huaheh opened a new issue, #9188:
URL: https://github.com/apache/pinot/issues/9188
In a partial upsert table, we can define an upsert strategy for each column, for example, OVERWRITE. This leads to the column's value always being overwritten by the next incoming event. The suggestion is to have control over merging the existing column value and the field value of the incoming event.
For example, a table has two columns, col1 and col2, both having OVERWRITE upsert strategy. With incoming event {"f1":"v1", "f2": null}, the expected transformation is like
```
{
"columnName": "col1",
"transformFunction": "OVERWRITE_IF(f1 != null, f1, col1)"
},
{
"columnName": "col2",
"transformFunction": "OVERWRITE_IF(f2 != null, f2, col2)"
},
```
This way col1 will be updated with v1 while col2 will remain its old value. This flexibility allows us to ingest some fields of an event and drop the other fields, and not have to drop the whole event if we merely don't want some columns to be overwritten.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #9188: Support to merge the existing column value and the incoming event
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #9188:
URL: https://github.com/apache/pinot/issues/9188#issuecomment-1216922614
I'll close the issue for now. @huaheh Let me know if you run into issues when trying it out
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] huaheh commented on issue #9188: Support to merge the existing column value and the incoming event
Posted by GitBox <gi...@apache.org>.
huaheh commented on issue #9188:
URL: https://github.com/apache/pinot/issues/9188#issuecomment-1215992249
Thanks, the "null ignoring" behavior is very useful and currently satisfies our needs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #9188: Support to merge the existing column value and the incoming event
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #9188:
URL: https://github.com/apache/pinot/issues/9188#issuecomment-1215967161
@huaheh Yes. The `partialUpsertStrategies` will only be applied when the existing and new values are both non-null.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] huaheh commented on issue #9188: Support to merge the existing column value and the incoming event
Posted by GitBox <gi...@apache.org>.
huaheh commented on issue #9188:
URL: https://github.com/apache/pinot/issues/9188#issuecomment-1213827773
Hi @Jackie-Jiang, do you mean `null` is always ignored regardless of the column's partialUpsertStrategies?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] huaheh commented on issue #9188: Support to merge the existing column value and the incoming event
Posted by GitBox <gi...@apache.org>.
huaheh commented on issue #9188:
URL: https://github.com/apache/pinot/issues/9188#issuecomment-1213636001
Not just null, also apply the merge on non-null values.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #9188: Support to merge the existing column value and the incoming event
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #9188:
URL: https://github.com/apache/pinot/issues/9188#issuecomment-1211215470
Partial upsert will always ignore the `null` values, so for your use case, if the new record has {"col1": "v1", "col2": null} with OVERRIDE strategy, it should achieve what you want.
cc @deemoliu seems this behavior is not clearly documented
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang closed issue #9188: Support to merge the existing column value and the incoming event
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang closed issue #9188: Support to merge the existing column value and the incoming event
URL: https://github.com/apache/pinot/issues/9188
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org