You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2021/11/17 23:36:56 UTC
[GitHub] [pinot] mneedham opened a new issue #7790: [Docs]
mneedham opened a new issue #7790:
URL: https://github.com/apache/pinot/issues/7790
This is a note of something that I need to document
```
Jonathan Meyer 12 hours ago
Hello :slightly_smiling_face:
Quick question regarding ingestionConfig on REALTIME tables
Is there any way to jsonPathString + further process the result with Groovy in transformConfig ? (edited)
Jonathan Meyer 12 hours ago
Say we have
```
{"data": {
"someKey": <needs_processing>
}
}
```
I'd like to process data.someKey with Groovy
It's possible from the query side with :
```
groovy('{"returnType":"LONG","isSingleValue":true}', 'Long.valueOf((arg0.substring(0, 8) + arg0.substring(18)), 16)', userId)
```
Basically need to do this ahead of time in the transformConfig of a REALTIME table
```
Working example:
```
Sure, here it is:
"ingestionConfig": {
"transformConfigs": [
{
"columnName": "userOid",
"transformFunction": "jsonPathString(data, '$.userId')"
},
{
"columnName": "userId",
"transformFunction": "Groovy({Long.valueOf((userOid.substring(0, 8) + userOid.substring(18)), 16)}, userOid)"
}
]
}
```
```
Jonathan Meyer 8 hours ago
As we can see, to populate userId we use column userOid which is itself the output of a transformation
Jonathan Meyer 8 hours ago
Note that we are using 0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed
And I haven't tested any other combinations of transformConfigs, so maybe some limitations still apply (e.g. order in which they are listed, maybe ?) (edited)
Neha Pawar 6 hours ago
No limitations should apply.. we added support for chaining, early this year. Thanks for pointing out!
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mneedham commented on issue #7790: [Docs] Chaining transformations
Posted by GitBox <gi...@apache.org>.
mneedham commented on issue #7790:
URL: https://github.com/apache/pinot/issues/7790#issuecomment-972765111
Have updated the docs here - https://docs.pinot.apache.org/developers/advanced/ingestion-level-transformations#chaining-transformations. Thanks to Jonathan for the example!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mneedham closed issue #7790: [Docs] Chaining transformations
Posted by GitBox <gi...@apache.org>.
mneedham closed issue #7790:
URL: https://github.com/apache/pinot/issues/7790
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org