You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "t0mpere (via GitHub)" <gi...@apache.org> on 2025/10/14 11:18:40 UTC
[I] `output.segment.dir.uri` is only read on ClusterInfo configs and not from TaskConfig in MergeRollupTask [pinot]
t0mpere opened a new issue, #17014:
URL: https://github.com/apache/pinot/issues/17014
Based on this [discussion](https://apache-pinot.slack.com/archives/C011C9JHN7R/p1759937177991909) on Slack.
I've found a bug where `output.segment.dir.uri` is never read from the task config and always from the controller config. This leads to an edge case where if the deep store is not configured globally, it's impossible to run a `metadata` job.
The function `getPushTaskConfig` should prioritise taskConfig over global controllerConfig.
I will open a PR to fix and refactor the function if we agree on the behaviour.
Example:
#### Controller config
```
controller.data.dir=/var/pinot/controller/data
```
#### Task config
```
"MergeRollupTask": {
"1day.mergeType": "concat",
"1day.bucketTimePeriod": "1d",
"1day.bufferTimePeriod": "1d",
"1day.maxNumRecordsPerSegment": "100000",
"1day.maxNumRecordsPerTask": "500000",
"1day.maxNumParallelBuckets": "10",
"minionInstanceTag": "merge",
"push.mode": "METADATA",
"output.segment.dir.uri": "gs://my-bucket/LOADED_HOURLY/merged",
"schedule": "0 1 * * * ?"
}
```
#### Result
```
{
"configs": {
"push.mode": "TAR",
...
"output.segment.dir.uri": "/var/pinot/controller/data/LOADED_HOURLY",
...
},
"tableName": "LOADED_HOURLY_OFFLINE",
"taskId": "Task_MergeRollupTask_5cd6364b-7012-4f60-8e8b-58ee4a2196c1_1759943074336_0",
"taskType": "MergeRollupTask"
}
```
#### Expected
```
{
"configs": {
"push.mode": "METADATA",
...
"output.segment.dir.uri": "gs://my-bucket/LOADED_HOURLY/merged",
...
},
"tableName": "LOADED_HOURLY_OFFLINE",
"taskId": "Task_MergeRollupTask_5cd6364b-7012-4f60-8e8b-58ee4a2196c1_1759943074336_0",
"taskType": "MergeRollupTask"
}
```
cc: @shounakmk219
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] `output.segment.dir.uri` is only read on ClusterInfo configs and not from TaskConfig in MergeRollupTask [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang closed issue #17014: `output.segment.dir.uri` is only read on ClusterInfo configs and not from TaskConfig in MergeRollupTask
URL: https://github.com/apache/pinot/issues/17014
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] `output.segment.dir.uri` is only read on ClusterInfo configs and not from TaskConfig in MergeRollupTask [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #17014:
URL: https://github.com/apache/pinot/issues/17014#issuecomment-3403723156
Thanks for raising the issue. I'd count this as an enhancement where we allow user to override `output.segment.dir.uri` through task config
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org