You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "ege-st (via GitHub)" <gi...@apache.org> on 2023/07/07 18:25:35 UTC
[GitHub] [pinot] ege-st opened a new issue, #11061: Stackoverflow risk when flattening a nested document
ege-st opened a new issue, #11061:
URL: https://github.com/apache/pinot/issues/11061
In the function which takes a nested document (such as a JSON document) and flattens a specific field, it's possible for a sufficiently nested document to cause a stack overflow. This is because there is no depth check at the point of recursion, so if an incoming document is structured like `{f1: {f2: f3: {...}}}` and it's deep enough, then `flattenMap` could keep recursing until there's a stack overflow.
Suggested solution: add in a depth check that will cause `flattenMap` to fault when the depth limit is reached and communicate an error message back to the user. It looks like Pinot has a standard practice of skipping incoming documents if they are malformed, so this can probably just log the error and increment the necessary meters and move to the next document.
One of the recursive calls which can lead to a stack overflow (last line in code block):
https://github.com/apache/pinot/blob/2fabbe40057d99afa497071f7083b4f16e40137a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/recordtransformer/ComplexTypeTransformer.java#L256-L270
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] ege-st commented on issue #11061: Stackoverflow risk when flattening a nested document
Posted by "ege-st (via GitHub)" <gi...@apache.org>.
ege-st commented on issue #11061:
URL: https://github.com/apache/pinot/issues/11061#issuecomment-1631024215
> @ege-st Can you please assign this to me?
Sure, but first I'm going to create a sample JSON document that can create the stack overflow and post it in a gist.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] s0nskar commented on issue #11061: Stackoverflow risk when flattening a nested document
Posted by "s0nskar (via GitHub)" <gi...@apache.org>.
s0nskar commented on issue #11061:
URL: https://github.com/apache/pinot/issues/11061#issuecomment-1627468685
@ege-st Can you please assign this to me?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] navina commented on issue #11061: Stackoverflow risk when flattening a nested document
Posted by "navina (via GitHub)" <gi...@apache.org>.
navina commented on issue #11061:
URL: https://github.com/apache/pinot/issues/11061#issuecomment-1625850481
Label: `bug`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org