You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "Jackie-Jiang (via GitHub)" <gi...@apache.org> on 2025/07/10 01:18:57 UTC
[I] Ingestion aggregations does not handle data type conversion for input value [pinot]
Jackie-Jiang opened a new issue, #16317:
URL: https://github.com/apache/pinot/issues/16317
For ingestion aggregations (doc: https://docs.pinot.apache.org/developers/advanced/ingestion-level-aggregations), when the aggregated column is not the same as original input column (e.g. aggregate `SUM(col1)` as `col2`), since `col1` won't be configured in the schema, Pinot won't convert the data type for it in `DataTypeTransformer`.
`ValueAggregator` assumes the input value to be in certain data types (most `ValueAggregator` requires `Number` as input). The input is in other types, `ValueAggregator` can throw `ClassCastException` and leave segment in a corrupted state described in #16316.
We need to also handle the data type conversion for input column of ingestion aggregations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Ingestion aggregations does not handle data type conversion for input value [pinot]
Posted by "KKcorps (via GitHub)" <gi...@apache.org>.
KKcorps commented on issue #16317:
URL: https://github.com/apache/pinot/issues/16317#issuecomment-3060769569
Is doing it implicitly the right path forward? Imo, we should actually enforce that users add columns to the schema which are being used in aggregation or other methods.
the reason is that we have multiple transformers and we should actually expect our values to go through them all before being picked up for any functions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Ingestion aggregations does not handle data type conversion for input value [pinot]
Posted by "Vamsi-klu (via GitHub)" <gi...@apache.org>.
Vamsi-klu commented on issue #16317:
URL: https://github.com/apache/pinot/issues/16317#issuecomment-5077283974
Opened PR to address this: https://github.com/apache/pinot/pull/19087
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Ingestion aggregations does not handle data type conversion for input value [pinot]
Posted by "noob-se7en (via GitHub)" <gi...@apache.org>.
noob-se7en commented on issue #16317:
URL: https://github.com/apache/pinot/issues/16317#issuecomment-3320799533
I guess its more for improving the usability as code-wise its just same.
somewhat like: https://github.com/apache/pinot/pull/16868/files
@Jackie-Jiang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Ingestion aggregations does not handle data type conversion for input value [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #16317:
URL: https://github.com/apache/pinot/issues/16317#issuecomment-3071174212
> Imo, we should actually enforce that users add columns to the schema which are being used in aggregation or other methods.
This might not always be possible. One example would be: the original value is number; the aggregated value is bytes (e.g. HyperLogLog). We cannot store the aggregated value as number, and we cannot keep the original value because it needs to be aggregated at ingestion time.
I do agree ideally user should config all columns referenced during ingestion in schema. We can consider adding a new field type, which is only used for data type conversion, but not for column definition in the table post ingestion.
Similarly (out of the scope of this issue), we can add a field type for deleted column. See #15931
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Ingestion aggregations does not handle data type conversion for input value [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang closed issue #16317: Ingestion aggregations does not handle data type conversion for input value
URL: https://github.com/apache/pinot/issues/16317
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Ingestion aggregations does not handle data type conversion for input value [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #16317: Ingestion aggregations does not handle data type conversion for input value
URL: https://github.com/apache/pinot/issues/16317
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Ingestion aggregations does not handle data type conversion for input value [pinot]
Posted by "noob-se7en (via GitHub)" <gi...@apache.org>.
noob-se7en commented on issue #16317:
URL: https://github.com/apache/pinot/issues/16317#issuecomment-3241400691
@Jackie-Jiang
From this PR: https://github.com/apache/pinot/pull/16519
> The long term fix should be to allow user to specify the data type for the input column if they are not defined in the schema
For the supported aggregations types: MAX,MIN,SUM,COUNT,SUMPRECISION I don't see input data other then Number or String and the current code already handles it.
For the DISTINCTCOUNTHLL, DISTINCTCOUNTHLLPLUS I see they can work with any input data using based on:
https://github.com/apache/pinot/blob/ed5e306c0fd006c0d52e993d4f7c5cb2b11dcab9/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/aggregator/DistinctCountHLLValueAggregator.java#L80-L91
So do we really need long term solution where we have new config for input data type:
```
"aggregationConfigs": [
{
"columnName": "total_sales",
"aggregationFunction": "SUM(price)",
"inputDataType": "STRING",
},
]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Ingestion aggregations does not handle data type conversion for input value [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #16317:
URL: https://github.com/apache/pinot/issues/16317#issuecomment-3304550464
@noob-se7en I think we still want this functionality. E.g. in the future we support MIN/MAX with STRING input, we need to differentiate number and string
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Ingestion aggregations does not handle data type conversion for input value [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #16317:
URL: https://github.com/apache/pinot/issues/16317#issuecomment-4113146827
This issue has been automatically marked as stale because it has not had any activity in the last 150 days. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Ingestion aggregations does not handle data type conversion for input value [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #16317:
URL: https://github.com/apache/pinot/issues/16317#issuecomment-4320903644
This issue has been closed because it has been inactive for 180 days. If you believe this issue is still relevant, please feel free to reopen it or create a new issue. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Ingestion aggregations does not handle data type conversion for input value [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #16317:
URL: https://github.com/apache/pinot/issues/16317#issuecomment-5109947368
This is fixed with #18816
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org