You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "jackylee-ch (via GitHub)" <gi...@apache.org> on 2025/12/24 17:47:13 UTC
[PR] [GLUTEN-3034][CORE] Fix allocating negative buffer size [incubator-gluten]
jackylee-ch opened a new pull request, #3035:
URL: https://github.com/apache/incubator-gluten/pull/3035
## What changes were proposed in this pull request?
Fixes: #3034
## How was this patch tested?
unit tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-3034][CORE] Fix allocating negative buffer size [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW closed pull request #3035: [GLUTEN-3034][CORE] Fix allocating negative buffer size
URL: https://github.com/apache/incubator-gluten/pull/3035
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-3034][CORE] Fix allocating negative buffer size [incubator-gluten]
Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on PR #3035:
URL: https://github.com/apache/incubator-gluten/pull/3035#issuecomment-3692783686
Hi, @abbas-gadhia. We previously encountered the issue with negative values. In my PR, I updated the type to Long, but realized it could then exceed the intended maximum. To address this, I’ve added a limit constraint to ensure the value stays within the required range. More detail can be found in #3034.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-3034][CORE] Fix allocating negative buffer size [incubator-gluten]
Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on PR #3035:
URL: https://github.com/apache/incubator-gluten/pull/3035#issuecomment-3693072761
@abbas-gadhia Yes, you are right. For your problem, we should focus more on why the incoming `neededSize` became a negative value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-3034][CORE] Fix allocating negative buffer size [gluten]
Posted by "abbas-gadhia (via GitHub)" <gi...@apache.org>.
abbas-gadhia commented on PR #3035:
URL: https://github.com/apache/gluten/pull/3035#issuecomment-4261767342
FYI. This turned out to be a non-issue in OSS gluten. It was something related to a data type mismatch issue in my code that led to a timestamp_ntz data type being interpreted as a string type
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-3034][CORE] Fix allocating negative buffer size [incubator-gluten]
Posted by "abbas-gadhia (via GitHub)" <gi...@apache.org>.
abbas-gadhia commented on PR #3035:
URL: https://github.com/apache/incubator-gluten/pull/3035#issuecomment-3692767029
I noticed that the title of the PR is about a negative buffer size but it seems that the fix is related to the "exceeds size limitation" error.
Is this about the former error i.e negative size or size limitation?
We seem to be facing the negative size error with the following stacktrace
```
Reason: Operator::getOutput failed for [operator: ValueStream, plan node ID: 0]: Error during calling Java code from native code: java.lang.IllegalArgumentException: Cannot grow BufferHolder by size -588050728 because the size is negative
at org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder.grow(BufferHolder.java:67)
at org.apache.spark.sql.catalyst.expressions.codegen.UnsafeWriter.grow(UnsafeWriter.java:63)
at org.apache.spark.sql.catalyst.expressions.codegen.UnsafeWriter.writeUnalignedBytes(UnsafeWriter.java:127)
at org.apache.spark.sql.catalyst.expressions.codegen.UnsafeWriter.write(UnsafeWriter.java:110)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.writeFields_0_3$(Unknown Source)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
at org.apache.gluten.execution.RowToVeloxColumnarExec$$anon$1.next(RowToVeloxColumnarExec.scala:159)
at org.apache.gluten.execution.RowToVeloxColumnarExec$$anon$1.next(RowToVeloxColumnarExec.scala:139)
```
I think the negative size error originates from velox i.e the `serializeVariableWidth` method, but i still need to confirm this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-3034][CORE] Fix allocating negative buffer size [incubator-gluten]
Posted by "abbas-gadhia (via GitHub)" <gi...@apache.org>.
abbas-gadhia commented on PR #3035:
URL: https://github.com/apache/incubator-gluten/pull/3035#issuecomment-3711418812
I'm trying to see if UnsafeRowFast.cpp#serializeRow can cause this overflow, specifically this code
`uint64_t sizeAndOffset = variableWidthOffset << 32 | size;`
It seems it 'might' overflow in the case of many varchar columns with large data > 125 columns. Still trying to confirm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-3034][CORE] Fix allocating negative buffer size [incubator-gluten]
Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on PR #3035:
URL: https://github.com/apache/incubator-gluten/pull/3035#issuecomment-3690736911
Hi, @FelixYBW . Did you reproduce the same issue on your side?
I remember tuning the batch size could mitigate it to some extent; if this PR is still required for a stable fix, I’m happy to take another look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-3034][CORE] Fix allocating negative buffer size [incubator-gluten]
Posted by "abbas-gadhia (via GitHub)" <gi...@apache.org>.
abbas-gadhia commented on PR #3035:
URL: https://github.com/apache/incubator-gluten/pull/3035#issuecomment-3693002137
Hi @jackylee-ch
Correct me if i'm wrong, but it seems from the changes in the PR that it would reduce the number of rows to keep the buffer size in check.
This differs from the negative size error i mentioned above, since it occurs when the BufferHolder tries to grow itself for a UTF8String and the numBytes of that string is negative.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-3034][CORE] Fix allocating negative buffer size [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on PR #3035:
URL: https://github.com/apache/incubator-gluten/pull/3035#issuecomment-3711406805
@jackylee-ch thank you for your info. I reopen it per @abbas-gadhia's request. Since it's different issue. Let's close it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org