You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/03/24 16:58:43 UTC
[GitHub] [pinot] saumya2700 opened a new issue #8399: Persist raw data from kafka topic as it is
saumya2700 opened a new issue #8399:
URL: https://github.com/apache/pinot/issues/8399
**Feature request:**
To save raw data from topic in pinot table.
**Use case :**
We have lots of complex schemas and we are using pinot for saving and retreiving topic data with times stamp and some other fields.
We do not want to map all nested columns from complex schema and create pinot schema and use lots of transformation functions. There are some places we want raw data as it is in pinot table.
**Sample data :**
`{
"header": {
"tid": "12wee",
"rid": 1,
"timestamp": 1647347092337
},
"status": "200_SUCCESS",
"jasData": {
"sdata": -22.89122,
"cnn": 0.823469,
"kli": 2.238848,
"olp": [
{
"ovPerc": 0.032486767,
"hg": 30.0,
"abshi": 6.661863
}
],
"terrkl": {
"ovPerc": 0.9675132,
"dist": [
-25.17232,
-25.17232,
-25.130081
]
},
"bcut": 2.77
},
"rgData": {
"pre": 102033.33,
"pv": 0.16,
"t": 287.36,
"timestamp": 1647347069000
},
"timestamp": 1647347092337
}
}`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] saumya2700 commented on issue #8399: Persist raw data from kafka topic as it is
Posted by GitBox <gi...@apache.org>.
saumya2700 commented on issue #8399:
URL: https://github.com/apache/pinot/issues/8399#issuecomment-1080263644
@Jackie-Jiang we want whole record to be stored as json or bytes. Latter one with jsonformat is working but our use case is to store whole message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #8399: Persist raw data from kafka topic as it is
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #8399:
URL: https://github.com/apache/pinot/issues/8399#issuecomment-1077922356
Do you want the whole record to be stored as a json field, or a field to be stored as a json field? Pinot can achieve the latter by the `jsonFormat` ingestion transform function (e.g. `jsonFormat(jasData)` to store the `jasData` as a json). Storing whole record as json is not supported currently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #8399: Persist raw data from kafka topic as it is
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #8399:
URL: https://github.com/apache/pinot/issues/8399#issuecomment-1082330899
We may consider adding a new config in the `IngestionConfig` to store the json string of the record into a field. The logic needs to be implemented into the `RecordExtractor`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org