You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/04/27 12:17:05 UTC
[GitHub] [pinot] saumya2700 opened a new issue, #8604: Getting Jackson infinite recursion error after sometime when usingcomplexTypeConfig( FiledsTounnest)
saumya2700 opened a new issue, #8604:
URL: https://github.com/apache/pinot/issues/8604
Getting **Jackson infinite recursion error** after sometime and table segments went offline now that table is completely off and not ingesting any data. When I have created that table everything was fine data was going in this table, after 2,3 days saw that data is not coming to this particular table. During this period I have only add some indexes in table config apart from that nothing is changed.
Not sure what caused this issue , I created same table again with different name just to test that theory that at starting it is working fine and yes the with the same table config new table started working but again after 3,4 days it stopped working and from external view i can see all segments are off and getting same jackson infinite recursion error from logs.
[sampledataTableSchema (3).txt](https://github.com/apache/pinot/files/8571897/sampledataTableSchema.3.txt)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] saurabhd336 commented on issue #8604: Getting Jackson infinite recursion error after sometime when using complexTypeConfig( FiledsTounnest)
Posted by GitBox <gi...@apache.org>.
saurabhd336 commented on issue #8604:
URL: https://github.com/apache/pinot/issues/8604#issuecomment-1117144025
@saumya2700 just trying to understand the usecase here, why'd you want to unnest non array fields? Ideally filedsToUnnest should have fields that are array of objects. This issue is being caused due to non array fields being unnested that isn't a valid usecase.
Can you describe your usecase you're trying to solve with unnesting these fields? There should be some alternative ways to solve for it. I feel JSON indexing / transformConfigs should suffice.
cc: @npawar
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] saumya2700 commented on issue #8604: Getting Jackson infinite recursion error after sometime when using complexTypeConfig( FiledsTounnest)
Posted by GitBox <gi...@apache.org>.
saumya2700 commented on issue #8604:
URL: https://github.com/apache/pinot/issues/8604#issuecomment-1125633524
> @saumya2700 You don't need to add the transform configs for every field. As i've pasted in my message above, all you need to do is remove the non-array fields from the fieldsToUnnest. Regarding the transform functions you're trying out, it will not work for the existing rows, because those rows don't have any way of knowing what "header" is. That information was only in the source data. To make it work in a backward compatible manner, you'll have to provide a "defaultValue" to the JSONPATHSTRING function as, JSONPATHSTRING(header, '$.trId', ''). It would be best to 1) try with the complex config or 2) reingest the data
YEs it will not work for existing rows , but it is not working for new rows even. I dont want backward compatible just wanted to get data for new records in this new column.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] npawar commented on issue #8604: Getting Jackson infinite recursion error after sometime when using complexTypeConfig( FiledsTounnest)
Posted by GitBox <gi...@apache.org>.
npawar commented on issue #8604:
URL: https://github.com/apache/pinot/issues/8604#issuecomment-1129264331
what is the latest set of configs you're trying with? Have you added default value to the JSONPATHSTRING functions? When you reload segments, the transform function will break on older records, so you need to have a default value set.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] saumya2700 commented on issue #8604: Getting Jackson infinite recursion error after sometime when using complexTypeConfig( FiledsTounnest)
Posted by GitBox <gi...@apache.org>.
saumya2700 commented on issue #8604:
URL: https://github.com/apache/pinot/issues/8604#issuecomment-1113095534
More logs around this error :
[detailLogsInfiniteRecursion.txt](https://github.com/apache/pinot/files/8589707/detailLogsInfiniteRecursion.txt)
and I am using version 0.10.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] saumya2700 commented on issue #8604: Getting Jackson infinite recursion error after sometime when using complexTypeConfig( FiledsTounnest)
Posted by GitBox <gi...@apache.org>.
saumya2700 commented on issue #8604:
URL: https://github.com/apache/pinot/issues/8604#issuecomment-1120610187
> @saumya2700 just trying to understand the usecase here, why'd you want to unnest non array fields? Ideally filedsToUnnest should have fields that are array of objects. This issue is being caused due to non array fields being unnested that isn't a valid usecase. Can you describe your usecase you're trying to solve with unnesting these fields? There should be some alternative ways to solve for it. I feel JSON indexing / transformConfigs should suffice. cc: @npawar
yes okay sure I will try this and let you know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] npawar commented on issue #8604: Getting Jackson infinite recursion error after sometime when using complexTypeConfig( FiledsTounnest)
Posted by GitBox <gi...@apache.org>.
npawar commented on issue #8604:
URL: https://github.com/apache/pinot/issues/8604#issuecomment-1125569145
@saumya2700
You don't need to add the transform configs for every field. As i've pasted in my message above, all you need to do is remove the non-array fields from the fieldsToUnnest.
Regarding the transform functions you're trying out, it will not work for the existing rows, because those rows don't have any way of knowing what "header" is. That information was only in the source data. To make it work in a backward compatible manner, you'll have to provide a "defaultValue" to the JSONPATHSTRING function as, JSONPATHSTRING(header, '$.trId', '<default value>').
It would be best to 1) try with the complex config or 2) reingest the data
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] npawar commented on issue #8604: Getting Jackson infinite recursion error after sometime when using complexTypeConfig( FiledsTounnest)
Posted by GitBox <gi...@apache.org>.
npawar commented on issue #8604:
URL: https://github.com/apache/pinot/issues/8604#issuecomment-1118106277
@saumya2700 , as Saurabh has pointed out, you need to remove all the non-array fields from fieldsToUnnest. Here's what works. Schema should have, instead of just calActivity.metadata,
```
{
"name": "calActivity.metadata.activityType",
"dataType": "STRING"
},
{
"name": "calActivity.metadata.conf",
"dataType": "FLOAT"
},
```
Ingestion config should just have
```
"complexTypeConfig": {
"fieldsToUnnest": ["calActivity.metadata"]
}
```
You dont need any transform functions, just this is enough.
Please try this out and let us know
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] saurabhd336 commented on issue #8604: Getting Jackson infinite recursion error after sometime when using complexTypeConfig( FiledsTounnest)
Posted by GitBox <gi...@apache.org>.
saurabhd336 commented on issue #8604:
URL: https://github.com/apache/pinot/issues/8604#issuecomment-1117195673
I was able to successfully ingest (without the infinite nesting) using transformConfigs as follows
`"ingestionConfig": {
"transformConfigs": [
{
"columnName": "header.trId",
"transformFunction": "JSONPATHSTRING(header, '$.trId')"
}
]
}`
Similar transformConfigs can be added for all the nested fields to be able to extract them during ingestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] saumya2700 commented on issue #8604: Getting Jackson infinite recursion error after sometime when using complexTypeConfig( FiledsTounnest)
Posted by GitBox <gi...@apache.org>.
saumya2700 commented on issue #8604:
URL: https://github.com/apache/pinot/issues/8604#issuecomment-1124916600
Hi @npawar , I have tried to do the same in existing table, but existing table is not showing any value:
"ingestionConfig": { "transformConfigs": [ { "columnName": "header.trId", "transformFunction": "JSONPATHSTRING(header, '$.trId')" } ] }, created new column as header_trId in existing schema and table, but for new messages header_trId is always coming as null. I reloaded the segments also but no luck.
Tried same table config and created new table and it worked, is it not gonna work with existing table, if this the case how we will update or add schema if need to add new fields to the schema. Please suggest am I missing any steps in between.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] npawar commented on issue #8604: Getting Jackson infinite recursion error after sometime when using complexTypeConfig( FiledsTounnest)
Posted by GitBox <gi...@apache.org>.
npawar commented on issue #8604:
URL: https://github.com/apache/pinot/issues/8604#issuecomment-1111313199
thanks for the details @saumya2700 , will take a look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] saurabhd336 commented on issue #8604: Getting Jackson infinite recursion error after sometime when using complexTypeConfig( FiledsTounnest)
Posted by GitBox <gi...@apache.org>.
saurabhd336 commented on issue #8604:
URL: https://github.com/apache/pinot/issues/8604#issuecomment-1112185068
@saumya2700 Would you happen to have to full exception? Also it'll help if you could share the pinot version you're running.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org