You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "gortiz (via GitHub)" <gi...@apache.org> on 2023/12/20 12:46:21 UTC
[I] Allow to add derived columns that depends on new columns [pinot]
gortiz opened a new issue, #12186:
URL: https://github.com/apache/pinot/issues/12186
Right now Pinot fails when a table is modified in such a way that two columns are added and one of them depend on the other.
More specifically, assuming table T has columns A1, A2 and A3, it should be possible to add columns B and C such as:
```js
//table config
"ingestionConfig": {
"transformConfigs": [
{
"columnName": "C",
"transformFunction": "B + 1"
}
]
}
```
But right now this fails with:
```
Caused by: java.lang.RuntimeException: Failed to create derived column: C because argument: B does not exist in the segment
```
This error is thrown in https://github.com/apache/pinot/blob/85b5779cff8d33412937ceb6d4dfc972b6c8bd97/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/defaultcolumn/BaseDefaultColumnHandler.java#L374 when at least one of the columns used in the transform function is not included in the columns already stored in the segment.
This is not a blocker because the same change can be done by first adding column B, reload and then modify the table again to add column D and reload.
It is important to note that at the same time we add this feature we should add checks to avoid cycle dependencies. Specifically, it should be forbidden to add columns `E1, E2, ..., En` when `E1` depends on `En` and `Ek` depends on `E(k-1)` for all k in [1, n].
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Allow to add derived columns that depends on new columns [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #12186:
URL: https://github.com/apache/pinot/issues/12186#issuecomment-5321773591
This issue has been automatically marked as stale because it has not had any activity in the last 150 days. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Allow to add derived columns that depends on new columns [pinot]
Posted by "npawar (via GitHub)" <gi...@apache.org>.
npawar commented on issue #12186:
URL: https://github.com/apache/pinot/issues/12186#issuecomment-2170965322
Chatting with Siddharth Dharm offline, who'd been trying this issue out. As per him this doesn't reproduce with these steps. I tried just now, and it indeed doesn’t reproduce that way. I see both B and C get created.
BUt what I did notice in the process, was that C did not use the function to calculate it's value after B. So both B and C had value Integrer.MIN_INT. This is more apparent when you set a default value to B.
For instance:
I added 2 columns E and F, where F = E + 1. And I put defaultNullValue of 10 for E. But when I query after reload, I see E has value 10 and F has Integer.MIN_INT instead of 11.
<img width="565" alt="Screen Shot 2024-06-15 at 4 03 02 PM" src="https://github.com/apache/pinot/assets/19693933/fa9c13cf-763d-40bb-b077-ac6c480aafe5">
<img width="397" alt="Screen Shot 2024-06-15 at 4 03 19 PM" src="https://github.com/apache/pinot/assets/19693933/fa719557-c335-4b39-9c55-ab418d1c42b0">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Allow to add derived columns that depends on new columns [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #12186:
URL: https://github.com/apache/pinot/issues/12186#issuecomment-5786775626
This issue has been closed because it has been inactive for 180 days. If you believe this issue is still relevant, please feel free to reopen it or create a new issue. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Allow to add derived columns that depends on new columns [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #12186: Allow to add derived columns that depends on new columns
URL: https://github.com/apache/pinot/issues/12186
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org