You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/06/13 18:36:06 UTC
[GitHub] [pinot] bdstuart opened a new issue, #8886: Add support for skipping/defaulting columns during managed offline flow rollup/dedup process
bdstuart opened a new issue, #8886:
URL: https://github.com/apache/pinot/issues/8886
Here is what I said in the pinot troubleshooting channel:
If this works as I think it might I could have the best of both worlds maybe. A certain amount of my data is in realtime table w/ the event_id for potential auditing, then as I move to offline table I default event_id to 0 and get good rollup.
To which Jackie repsonded:
It is absolutely reasonable. We don't support it currently, but it is doable. Essentially we need to add a new task config to skip some columns when running the task in ROLLUP or DEDUP mode. Internally we will fill default values to these columns so that they won't be considered. Can you please help file a github issue describing the requirements?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] snleee commented on issue #8886: Add support for skipping/defaulting columns during managed offline flow rollup/dedup process
Posted by GitBox <gi...@apache.org>.
snleee commented on issue #8886:
URL: https://github.com/apache/pinot/issues/8886#issuecomment-1154407952
- I think that we need to add the transformation function support on `SegmentProcessorFramework`. We can do sth similar to the ingestion config. Currently, map phase hardcodes to initialize the default composite transformer.
- On the merge/roll-up task level, we need to decide whether we want to expose the entire transformation layer or we just want to expose the config to skip columns to satisfy this requirement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] npawar commented on issue #8886: Add support for skipping/defaulting columns during managed offline flow rollup/dedup process
Posted by GitBox <gi...@apache.org>.
npawar commented on issue #8886:
URL: https://github.com/apache/pinot/issues/8886#issuecomment-1169216438
@snleee are you planning to pick this up?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #8886: Add support for skipping/defaulting columns during managed offline flow rollup/dedup process
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #8886:
URL: https://github.com/apache/pinot/issues/8886#issuecomment-1156811751
For this feature request, another solution is to just not read the value for the skipped columns. The existing transformer can handle the filling of default values.
@snleee Do you see other use cases where we want custom transform other than the one in the ingestion config?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org