You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by GitBox <gi...@apache.org> on 2021/02/26 14:00:01 UTC
[GitHub] [drill] oleg-zinovev opened a new pull request #2183: DRILL-6547: IllegalStateException: Tried to remove unmanaged buffer in concat function
oleg-zinovev opened a new pull request #2183:
URL: https://github.com/apache/drill/pull/2183
# [DRILL-6547](https://issues.apache.org/jira/browse/DRILL-6547): IllegalStateException: Tried to remove unmanaged buffer in concat function
## Description
Certain overloads of the **concat** function do not override the current working buffer when calling **reallocIfNeeded**. Because of this, repeated calls to reallocIfNeeded on this buffer could lead to an error **Tried to remove unmanaged buffer**
## Testing
TestBugFixes#testDRILL6547 has been added
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] paul-rogers commented on pull request #2183: DRILL-6547: IllegalStateException: Tried to remove unmanaged buffer in concat function
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on pull request #2183:
URL: https://github.com/apache/drill/pull/2183#issuecomment-811694253
@oleg-zinovev, sorry for the delay. For some reason, Github does not notify me of new comments, even though I'm a "watcher." Odd.
Thanks also for the explanation and generated code. I agree with your analysis. If we ever do a "revised edition" of the Drill book, we should explain this detail.
I wonder: is this the only remaining function that has this problem? When I quickly glanced through the UDFs, it seemed some used the pattern like the one you are using here, others use the "wrong" pattern. Should we fix all of them to avoid having to fix them one by one?
Also, the reason that I said that one of the VARCHAR fields would probably be null is because of the name of the function:
``` java
public static class ConcatRightNullInput implements DrillSimpleFunc {
```
We seem to also have:
``` java
public static class ConcatLeftNullInput implements DrillSimpleFunc {
```
Which, by the way, has the same bug you are fixing here.
There is also:
``` java
public static class ConcatBothNullInput implements DrillSimpleFunc {
```
For which, presumably, both inputs are NULL (zero length), but we still allocate a buffer, and that code has the bug you are fixing. Of course, if both inputs are NULL, then there length is undefined (NULL says to ignore the value; we cannot assume that the string value is the empty string or that the start/end offsets are valid.)
Finally, we also have:
``` java
public static class ConcatOperator implements DrillSimpleFunc {
```
Which does use both inputs, and which already has the fix you propose here.
This is what I meant: we have multiple variations of the same function, some are fixed, some are still broken. The variations claim that one or both sides are NULL, but we are still working with the (possibly invalid) VARCHARs in those cases, when we should not.
Now, because things are not complex enough, we have another issue. If either of the arguments of `CONCAT(left, right)` are the SQL NULL value, then the output should be... the non-null argument? Or, NULL? At least according to [SQL Server](https://docs.microsoft.com/en-us/sql/t-sql/functions/concat-transact-sql?view=sql-server-ver15), the rule is that a NULL value is treated as an empty string. We should thus modify the three `Null` variations to do that: use an empty string, *not* the value of the null argument. That is, for `ConcatRightNullInput`, return the left input. For `ConcatLeftNullInput`, return the right input. For `ConcatBothNullInput`, return an empty (zero-length) string.
I hope this clarifies things a bit!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] oleg-zinovev edited a comment on pull request #2183: DRILL-6547: IllegalStateException: Tried to remove unmanaged buffer in concat function
Posted by GitBox <gi...@apache.org>.
oleg-zinovev edited a comment on pull request #2183:
URL: https://github.com/apache/drill/pull/2183#issuecomment-791903981
@paul-rogers
Thank you so much for reviewing the fix.
1. Here are the code generated to perform the projection:
```java
public class ProjectorGen0
extends ProjectorTemplate
{
DrillBuf work0;
NullableVarCharHolder constant2;
NullableVarCharVector vv3;
VarCharVector vv8;
public ProjectorGen0() {
try {
__DRILL_INIT__();
} catch (SchemaChangeException e) {
throw new UnsupportedOperationException(e);
}
}
public void doEval(int inIndex, int outIndex)
throws SchemaChangeException
{
{
NullableVarCharHolder out6 = new NullableVarCharHolder();
{
out6 .isSet = vv3 .getAccessor().isSet((inIndex));
if (out6 .isSet == 1) {
out6 .buffer = vv3 .getBuffer();
long startEnd = vv3 .getAccessor().getStartEnd((inIndex));
out6 .start = ((int) startEnd);
out6 .end = ((int)(startEnd >> 32));
}
}
//---- start of eval portion of concat function. ----//
VarCharHolder out7 = new VarCharHolder();
{
final VarCharHolder out = new VarCharHolder();
NullableVarCharHolder left = constant2;
NullableVarCharHolder right = out6;
DrillBuf buffer = work0;
StringFunctions$ConcatBothNullInput_eval:
{
out.buffer = buffer = buffer.reallocIfNeeded((left.end - left.start) + (right.end - right.start));
out.start = out.end = 0;
int id = 0;
if (left.isSet == 1) {
for (id = left.start; id < left.end; id++) {
out.buffer.setByte(out.end++, left.buffer.getByte(id));
}
}
if (right.isSet == 1) {
for (id = right.start; id < right.end; id++) {
out.buffer.setByte(out.end++, right.buffer.getByte(id));
}
}
}
work0 = buffer;
out7 .start = out.start;
out7 .end = out.end;
out7 .buffer = out.buffer;
}
//---- end of eval portion of concat function. ----//
vv8 .getMutator().setSafe((outIndex), out7 .start, out7 .end, out7 .buffer);
}
}
public void doSetup(FragmentContext context, RecordBatch incoming, RecordBatch outgoing)
throws SchemaChangeException
{
{
work0 = (incoming).getContext().getManagedBuffer();
NullableVarCharHolder out1 = new NullableVarCharHolder();
constant2 = new NullableVarCharHolder();
constant2 .isSet = out1 .isSet;
constant2 .start = out1 .start;
constant2 .end = out1 .end;
constant2 .buffer = out1 .buffer;
int[] fieldIds4 = new int[ 1 ] ;
fieldIds4 [ 0 ] = 0;
Object tmp5 = (incoming).getValueAccessorById(NullableVarCharVector.class, fieldIds4).getValueVector();
if (tmp5 == null) {
throw new SchemaChangeException("Failure while loading vector vv3 with id: TypedFieldId [fieldIds=[0], remainder=null].");
}
vv3 = ((NullableVarCharVector) tmp5);
/** start SETUP for function concat **/
{
NullableVarCharHolder left = constant2;
DrillBuf buffer = work0;
StringFunctions$ConcatBothNullInput_setup:
{}
work0 = buffer;
}
/** end SETUP for function concat **/
int[] fieldIds9 = new int[ 1 ] ;
fieldIds9 [ 0 ] = 0;
Object tmp10 = (outgoing).getValueAccessorById(VarCharVector.class, fieldIds9).getValueVector();
if (tmp10 == null) {
throw new SchemaChangeException("Failure while loading vector vv8 with id: TypedFieldId [fieldIds=[0], remainder=null].");
}
vv8 = ((VarCharVector) tmp10);
}
}
public void __DRILL_INIT__()
throws SchemaChangeException
{
}
}
```
As you can see, the buffer is requested from the fragment context in the doSetup method (which in turn is called within setup). This happens once for each new schema in **ProjectRecordBatch#setupNewSchemaFromInput**
When calculating the function body, the following buffer usage pattern is used:
- Extract the current buffer from the class field to a local variable
- Execute the function body
- Save the value of the local buffer variable in the class field
In the current implementation, managed buffer is "reused" for each subsequent row. Therefore, the result of reallocIfNeeded must be stored in the buffer field declared within the function.
2. You are not quite right when you say that the result of left. end - left. start will always be 0. For example, the same implementation of the function will be used in the case when the first argument is the nullable field of the parquet file, in which not all values will be null.
P.S. Perhaps I did not understand you correctly
3. Unfortunately, I can't use the sample request given in the bug, as it uses a file that is missing in the Drill project. In addition, this file is uploaded to MapR jira, where I do not have access. If you can attach a file to the specified Drill bug, I will happily replace my request with the request given in the bug.
P.S. I'm sorry for my English
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] paul-rogers commented on a change in pull request #2183: DRILL-6547: IllegalStateException: Tried to remove unmanaged buffer in concat function
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2183:
URL: https://github.com/apache/drill/pull/2183#discussion_r612022037
##########
File path: exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/StringFunctions.java
##########
@@ -1425,7 +1419,7 @@ public void eval() {
out.buffer = buffer = buffer.reallocIfNeeded((left.end - left.start) + (right.end - right.start));
out.start = out.end = 0;
- int id = 0;
+ int id;
Review comment:
Nit: thanks for removing the unneeded initializer. Might as well move the type into the `for` loop:
```
for (int id =...
```
Here and below.
##########
File path: exec/java-exec/src/test/java/org/apache/drill/TestBugFixes.java
##########
@@ -24,8 +24,10 @@
import java.util.List;
Review comment:
I wonder if the tests should move to a file that is a bit more descriptive? Do we have tests for string functions? Else, future folks won't know that the code came from a "bug fix" and to look in this file.
Also, should we have tests for all the functions that were fixed? Presumably those tests don't exist because otherwise we would have found the bugs sooner.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] oleg-zinovev edited a comment on pull request #2183: DRILL-6547: IllegalStateException: Tried to remove unmanaged buffer in concat function
Posted by GitBox <gi...@apache.org>.
oleg-zinovev edited a comment on pull request #2183:
URL: https://github.com/apache/drill/pull/2183#issuecomment-791903981
@paul-rogers
Thank you so much for reviewing the fix.
1. Here are the code generated to perform the projection:
```java
public class ProjectorGen0
extends ProjectorTemplate
{
DrillBuf work0;
NullableVarCharHolder constant2;
NullableVarCharVector vv3;
VarCharVector vv8;
public ProjectorGen0() {
try {
__DRILL_INIT__();
} catch (SchemaChangeException e) {
throw new UnsupportedOperationException(e);
}
}
public void doEval(int inIndex, int outIndex)
throws SchemaChangeException
{
{
NullableVarCharHolder out6 = new NullableVarCharHolder();
{
out6 .isSet = vv3 .getAccessor().isSet((inIndex));
if (out6 .isSet == 1) {
out6 .buffer = vv3 .getBuffer();
long startEnd = vv3 .getAccessor().getStartEnd((inIndex));
out6 .start = ((int) startEnd);
out6 .end = ((int)(startEnd >> 32));
}
}
//---- start of eval portion of concat function. ----//
VarCharHolder out7 = new VarCharHolder();
{
final VarCharHolder out = new VarCharHolder();
NullableVarCharHolder left = constant2;
NullableVarCharHolder right = out6;
DrillBuf buffer = work0;
StringFunctions$ConcatBothNullInput_eval:
{
out.buffer = buffer = buffer.reallocIfNeeded((left.end - left.start) + (right.end - right.start));
out.start = out.end = 0;
int id = 0;
if (left.isSet == 1) {
for (id = left.start; id < left.end; id++) {
out.buffer.setByte(out.end++, left.buffer.getByte(id));
}
}
if (right.isSet == 1) {
for (id = right.start; id < right.end; id++) {
out.buffer.setByte(out.end++, right.buffer.getByte(id));
}
}
}
work0 = buffer;
out7 .start = out.start;
out7 .end = out.end;
out7 .buffer = out.buffer;
}
//---- end of eval portion of concat function. ----//
vv8 .getMutator().setSafe((outIndex), out7 .start, out7 .end, out7 .buffer);
}
}
public void doSetup(FragmentContext context, RecordBatch incoming, RecordBatch outgoing)
throws SchemaChangeException
{
{
work0 = (incoming).getContext().getManagedBuffer();
NullableVarCharHolder out1 = new NullableVarCharHolder();
constant2 = new NullableVarCharHolder();
constant2 .isSet = out1 .isSet;
constant2 .start = out1 .start;
constant2 .end = out1 .end;
constant2 .buffer = out1 .buffer;
int[] fieldIds4 = new int[ 1 ] ;
fieldIds4 [ 0 ] = 0;
Object tmp5 = (incoming).getValueAccessorById(NullableVarCharVector.class, fieldIds4).getValueVector();
if (tmp5 == null) {
throw new SchemaChangeException("Failure while loading vector vv3 with id: TypedFieldId [fieldIds=[0], remainder=null].");
}
vv3 = ((NullableVarCharVector) tmp5);
/** start SETUP for function concat **/
{
NullableVarCharHolder left = constant2;
DrillBuf buffer = work0;
StringFunctions$ConcatBothNullInput_setup:
{}
work0 = buffer;
}
/** end SETUP for function concat **/
int[] fieldIds9 = new int[ 1 ] ;
fieldIds9 [ 0 ] = 0;
Object tmp10 = (outgoing).getValueAccessorById(VarCharVector.class, fieldIds9).getValueVector();
if (tmp10 == null) {
throw new SchemaChangeException("Failure while loading vector vv8 with id: TypedFieldId [fieldIds=[0], remainder=null].");
}
vv8 = ((VarCharVector) tmp10);
}
}
public void __DRILL_INIT__()
throws SchemaChangeException
{
}
}
```
As you can see, the buffer is requested from the fragment context in the doSetup method (which in turn is called within setup). This happens once for each new schema in **ProjectRecordBatch#setupNewSchemaFromInput**
When calculating the function body, the following buffer usage pattern is used:
- Extract the current buffer from the class field to a local variable (**DrillBuf buffer = work0;**)
- Execute the function body
- Save the value of the local buffer variable in the class field (**work0 = buffer;**)
In the current implementation, managed buffer is "reused" for each subsequent row. Therefore, the result of reallocIfNeeded must be stored in the buffer field declared within the function.
2. You are not quite right when you say that the result of left. end - left. start will always be 0. For example, the same implementation of the function will be used in the case when the first argument is the nullable field of the parquet file, in which not all values will be null.
P.S. Perhaps I did not understand you correctly
3. Unfortunately, I can't use the sample request given in the bug, as it uses a file that is missing in the Drill project. In addition, this file is uploaded to MapR jira, where I do not have access. If you can attach a file to the specified Drill bug, I will happily replace my request with the request given in the bug.
P.S. I'm sorry for my English
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] cgivre commented on pull request #2183: DRILL-6547: IllegalStateException: Tried to remove unmanaged buffer in concat function
Posted by GitBox <gi...@apache.org>.
cgivre commented on pull request #2183:
URL: https://github.com/apache/drill/pull/2183#issuecomment-815852309
@paul-rogers @oleg-zinovev ,
Hello all. I was on vacation last week, but is this PR ready to be merged?
Thx!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] paul-rogers commented on pull request #2183: DRILL-6547: IllegalStateException: Tried to remove unmanaged buffer in concat function
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on pull request #2183:
URL: https://github.com/apache/drill/pull/2183#issuecomment-818314527
I would suggest scanning as many functions as you can for this error and fixing them all here. We know about the problem; it is a systematic oversight of the earlier authors, and so we might as well fix the whole thing rather than doing each case one PR at a time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] oleg-zinovev edited a comment on pull request #2183: DRILL-6547: IllegalStateException: Tried to remove unmanaged buffer in concat function
Posted by GitBox <gi...@apache.org>.
oleg-zinovev edited a comment on pull request #2183:
URL: https://github.com/apache/drill/pull/2183#issuecomment-791903981
@paul-rogers
Thank you so much for reviewing the fix.
1. Here are the code generated to perform the projection:
```java
public class ProjectorGen0
extends ProjectorTemplate
{
DrillBuf work0;
NullableVarCharHolder constant2;
NullableVarCharVector vv3;
VarCharVector vv8;
public ProjectorGen0() {
try {
__DRILL_INIT__();
} catch (SchemaChangeException e) {
throw new UnsupportedOperationException(e);
}
}
public void doEval(int inIndex, int outIndex)
throws SchemaChangeException
{
{
NullableVarCharHolder out6 = new NullableVarCharHolder();
{
out6 .isSet = vv3 .getAccessor().isSet((inIndex));
if (out6 .isSet == 1) {
out6 .buffer = vv3 .getBuffer();
long startEnd = vv3 .getAccessor().getStartEnd((inIndex));
out6 .start = ((int) startEnd);
out6 .end = ((int)(startEnd >> 32));
}
}
//---- start of eval portion of concat function. ----//
VarCharHolder out7 = new VarCharHolder();
{
final VarCharHolder out = new VarCharHolder();
NullableVarCharHolder left = constant2;
NullableVarCharHolder right = out6;
DrillBuf buffer = work0;
StringFunctions$ConcatBothNullInput_eval:
{
out.buffer = buffer = buffer.reallocIfNeeded((left.end - left.start) + (right.end - right.start));
out.start = out.end = 0;
int id = 0;
if (left.isSet == 1) {
for (id = left.start; id < left.end; id++) {
out.buffer.setByte(out.end++, left.buffer.getByte(id));
}
}
if (right.isSet == 1) {
for (id = right.start; id < right.end; id++) {
out.buffer.setByte(out.end++, right.buffer.getByte(id));
}
}
}
work0 = buffer;
out7 .start = out.start;
out7 .end = out.end;
out7 .buffer = out.buffer;
}
//---- end of eval portion of concat function. ----//
vv8 .getMutator().setSafe((outIndex), out7 .start, out7 .end, out7 .buffer);
}
}
public void doSetup(FragmentContext context, RecordBatch incoming, RecordBatch outgoing)
throws SchemaChangeException
{
{
work0 = (incoming).getContext().getManagedBuffer();
NullableVarCharHolder out1 = new NullableVarCharHolder();
constant2 = new NullableVarCharHolder();
constant2 .isSet = out1 .isSet;
constant2 .start = out1 .start;
constant2 .end = out1 .end;
constant2 .buffer = out1 .buffer;
int[] fieldIds4 = new int[ 1 ] ;
fieldIds4 [ 0 ] = 0;
Object tmp5 = (incoming).getValueAccessorById(NullableVarCharVector.class, fieldIds4).getValueVector();
if (tmp5 == null) {
throw new SchemaChangeException("Failure while loading vector vv3 with id: TypedFieldId [fieldIds=[0], remainder=null].");
}
vv3 = ((NullableVarCharVector) tmp5);
/** start SETUP for function concat **/
{
NullableVarCharHolder left = constant2;
DrillBuf buffer = work0;
StringFunctions$ConcatBothNullInput_setup:
{}
work0 = buffer;
}
/** end SETUP for function concat **/
int[] fieldIds9 = new int[ 1 ] ;
fieldIds9 [ 0 ] = 0;
Object tmp10 = (outgoing).getValueAccessorById(VarCharVector.class, fieldIds9).getValueVector();
if (tmp10 == null) {
throw new SchemaChangeException("Failure while loading vector vv8 with id: TypedFieldId [fieldIds=[0], remainder=null].");
}
vv8 = ((VarCharVector) tmp10);
}
}
public void __DRILL_INIT__()
throws SchemaChangeException
{
}
}
```
As you can see, the buffer is requested from the fragment context in the doSetup method (which in turn is called within setup). This happens once for each new schema in **ProjectRecordBatch#setupNewSchemaFromInput**
When calculating the function body, the buffer usage pattern is as follows:
1) Extract the current buffer from the class field to a local variable
2) Execute the function body
3) Save the value of the local buffer variable in the class field
In the current implementation, managed buffer is "reused" for each subsequent row. Therefore, the result of reallocIfNeeded must be stored in the buffer field declared within the function.
2. You are not quite right when you say that the result of left. end - left. start will always be 0. For example, the same implementation of the function will be used in the case when the first argument is the nullable field of the parquet file, in which not all values will be null.
P.S. Perhaps I did not understand you correctly
3. Unfortunately, I can't use the sample request given in the bug, as it uses a file that is missing from the drill project. In addition, this file is uploaded to MapR jira, where I do not have access. If you can attach a file to the specified Drill bug, I will happily replace my request with the request given in the bug.
P.S. I'm sorry for my English
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] oleg-zinovev edited a comment on pull request #2183: DRILL-6547: IllegalStateException: Tried to remove unmanaged buffer in concat function
Posted by GitBox <gi...@apache.org>.
oleg-zinovev edited a comment on pull request #2183:
URL: https://github.com/apache/drill/pull/2183#issuecomment-815969446
@paul-rogers , @luocooong , @cgivre
Thank you for the additional explanations. I have changed the **ConcatLeftNullInput**, **ConcatBothNullInput** and **ConcatRightNullInput** functions based on your comments.
I have also fixed the same error when using a buffer in the **binary_string** and **string_binary** functions.
I have removed the buffer in the **right**, **regexp_matches**, **substring**, **left** functions, since it was not used in this functions bodies.
P.S. I have found buffer overflow error in **replace** function. Should I open another PR, or I can add fix in current one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] luocooong commented on pull request #2183: DRILL-6547: IllegalStateException: Tried to remove unmanaged buffer in concat function
Posted by GitBox <gi...@apache.org>.
luocooong commented on pull request #2183:
URL: https://github.com/apache/drill/pull/2183#issuecomment-812961546
I agree. @oleg-zinovev Thanks for the PR. Then, Is it possible to check all the buffer issues in the file that contains the concat functions? thanks again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] cgivre commented on pull request #2183: DRILL-6547: IllegalStateException: Tried to remove unmanaged buffer in concat function
Posted by GitBox <gi...@apache.org>.
cgivre commented on pull request #2183:
URL: https://github.com/apache/drill/pull/2183#issuecomment-845243809
@paul-rogers @luocooong
Can we merge this?
To Paul's point, I'm not sure that we have unit tests for this specific case, but most of the functions do have specific unit tests to verify that they at least do what they purport to do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] luocooong commented on pull request #2183: DRILL-6547: IllegalStateException: Tried to remove unmanaged buffer in concat function
Posted by GitBox <gi...@apache.org>.
luocooong commented on pull request #2183:
URL: https://github.com/apache/drill/pull/2183#issuecomment-816669211
@oleg-zinovev Thanks. I recommend that you can create a new PR for the `replace` function, only put the `[Minor Update]` in the front of the PR title.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] oleg-zinovev commented on a change in pull request #2183: DRILL-6547: IllegalStateException: Tried to remove unmanaged buffer in concat function
Posted by GitBox <gi...@apache.org>.
oleg-zinovev commented on a change in pull request #2183:
URL: https://github.com/apache/drill/pull/2183#discussion_r631950582
##########
File path: exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/StringFunctions.java
##########
@@ -1425,7 +1419,7 @@ public void eval() {
out.buffer = buffer = buffer.reallocIfNeeded((left.end - left.start) + (right.end - right.start));
out.start = out.end = 0;
- int id = 0;
+ int id;
Review comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] oleg-zinovev edited a comment on pull request #2183: DRILL-6547: IllegalStateException: Tried to remove unmanaged buffer in concat function
Posted by GitBox <gi...@apache.org>.
oleg-zinovev edited a comment on pull request #2183:
URL: https://github.com/apache/drill/pull/2183#issuecomment-791903981
@paul-rogers
Thank you so much for reviewing the fix.
1. Here are the code generated to perform the projection:
```java
public class ProjectorGen0
extends ProjectorTemplate
{
DrillBuf work0;
NullableVarCharHolder constant2;
NullableVarCharVector vv3;
VarCharVector vv8;
public ProjectorGen0() {
try {
__DRILL_INIT__();
} catch (SchemaChangeException e) {
throw new UnsupportedOperationException(e);
}
}
public void doEval(int inIndex, int outIndex)
throws SchemaChangeException
{
{
NullableVarCharHolder out6 = new NullableVarCharHolder();
{
out6 .isSet = vv3 .getAccessor().isSet((inIndex));
if (out6 .isSet == 1) {
out6 .buffer = vv3 .getBuffer();
long startEnd = vv3 .getAccessor().getStartEnd((inIndex));
out6 .start = ((int) startEnd);
out6 .end = ((int)(startEnd >> 32));
}
}
//---- start of eval portion of concat function. ----//
VarCharHolder out7 = new VarCharHolder();
{
final VarCharHolder out = new VarCharHolder();
NullableVarCharHolder left = constant2;
NullableVarCharHolder right = out6;
DrillBuf buffer = work0;
StringFunctions$ConcatBothNullInput_eval:
{
out.buffer = buffer = buffer.reallocIfNeeded((left.end - left.start) + (right.end - right.start));
out.start = out.end = 0;
int id = 0;
if (left.isSet == 1) {
for (id = left.start; id < left.end; id++) {
out.buffer.setByte(out.end++, left.buffer.getByte(id));
}
}
if (right.isSet == 1) {
for (id = right.start; id < right.end; id++) {
out.buffer.setByte(out.end++, right.buffer.getByte(id));
}
}
}
work0 = buffer;
out7 .start = out.start;
out7 .end = out.end;
out7 .buffer = out.buffer;
}
//---- end of eval portion of concat function. ----//
vv8 .getMutator().setSafe((outIndex), out7 .start, out7 .end, out7 .buffer);
}
}
public void doSetup(FragmentContext context, RecordBatch incoming, RecordBatch outgoing)
throws SchemaChangeException
{
{
work0 = (incoming).getContext().getManagedBuffer();
NullableVarCharHolder out1 = new NullableVarCharHolder();
constant2 = new NullableVarCharHolder();
constant2 .isSet = out1 .isSet;
constant2 .start = out1 .start;
constant2 .end = out1 .end;
constant2 .buffer = out1 .buffer;
int[] fieldIds4 = new int[ 1 ] ;
fieldIds4 [ 0 ] = 0;
Object tmp5 = (incoming).getValueAccessorById(NullableVarCharVector.class, fieldIds4).getValueVector();
if (tmp5 == null) {
throw new SchemaChangeException("Failure while loading vector vv3 with id: TypedFieldId [fieldIds=[0], remainder=null].");
}
vv3 = ((NullableVarCharVector) tmp5);
/** start SETUP for function concat **/
{
NullableVarCharHolder left = constant2;
DrillBuf buffer = work0;
StringFunctions$ConcatBothNullInput_setup:
{}
work0 = buffer;
}
/** end SETUP for function concat **/
int[] fieldIds9 = new int[ 1 ] ;
fieldIds9 [ 0 ] = 0;
Object tmp10 = (outgoing).getValueAccessorById(VarCharVector.class, fieldIds9).getValueVector();
if (tmp10 == null) {
throw new SchemaChangeException("Failure while loading vector vv8 with id: TypedFieldId [fieldIds=[0], remainder=null].");
}
vv8 = ((VarCharVector) tmp10);
}
}
public void __DRILL_INIT__()
throws SchemaChangeException
{
}
}
```
As you can see, the buffer is requested from the fragment context in the doSetup method (which in turn is called within setup). This happens once for each new schema in **ProjectRecordBatch#setupNewSchemaFromInput**
When calculating the function body, the following buffer usage pattern is used:
- Extract the current buffer from the class field to a local variable (**DrillBuf buffer = work0;**)
- Execute the function body
- Save the value of the local buffer variable in the class field (**work0 = buffer;**)
In the current implementation, managed buffer is "reused" for each subsequent row. Therefore, the result of reallocIfNeeded must be stored in the buffer field declared within the function.
2. You are not quite right when you say that the result of left. end - left. start will always be 0. For example, the same implementation of the function will be used in the case when the first argument is the nullable field of the parquet file, in which not all values will be null.
P.S. Perhaps I did not understand you correctly
3. Unfortunately, I can't use the sample query given in the bug, as it uses a file that is missing in the Drill project. In addition, this file is uploaded to MapR jira, where I do not have access. If you can attach a file to the specified Drill bug, I will happily replace my query with the query given in the bug.
P.S. I'm sorry for my English
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] paul-rogers commented on pull request #2183: DRILL-6547: IllegalStateException: Tried to remove unmanaged buffer in concat function
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on pull request #2183:
URL: https://github.com/apache/drill/pull/2183#issuecomment-791072405
Thanks for fixing this.
I suspect the problem here runs deeper than this particular fix. This is Drill: nothing is simple, things build on hacks which build on undocumented complexities. I suspect that is the issue here.
In particular, we need to know the rules for buffer ownership in UDFs which is, of course, not documented. The only reasonable way to learn this is to run a query, save the generated Java code and inspect the calls. Warning: the generated code is a huge mess.
The key question is this: when you declare an `@Inject` variable (`buffer` in your case), does the runtime provide an initial buffer? My guess is that it does in the form of a "managed" buffer. (This just means that there is some auto-free mechanism.) I believe that the buffer is allocated on each call, but the generated code will say for sure. (The UDF code is copied into the generated code.)
The function declares an `@Output`, in this case `out`. If this variable provides a buffer, it would seem that ownership has to pass to the caller, since the caller will copy that buffer into a value vector. My guess is that if the buffer is "managed", it will be freed automatically at some point.
(Note that managed buffers can cause an OOM because of when they are freed: I think it is during fragment shutdown, IIRC. I had to fix a reader bug related to this issue.)
To summarize, it appears that the proper pattern in UDFs is:
* Declare an `@Inject` buffer which the runtime will provide.
* Resize the buffer if needed, using the return value from `reallocIfNeeded`.
* Use that buffer as the output buffer by setting it to `out.buffer` where needed.
* Managed buffers are freed automatically, at some unknown point.
If this is true, there is no reason to update `buffer`, the proper pattern is shown in [DecimalAggrTypeFunctions1](https://github.com/apache/drill/blob/2eb6979206ad1b6f830a070dffa822edbfac6189/exec/java-exec/src/main/codegen/templates/Decimal/DecimalAggrTypeFunctions1.java#L113) (with my own comments):
``` java
// Inject means that the runtime will provide a default-size buffer on each call
@Inject DrillBuf buffer;
// Holder for output. NOT initialized b the runtime.
@Output ${type.outputType}Holder out;
...
// Resize the provided buffer, replacing if necessary, and assigning that
// to the output variable.
out.buffer = buffer.reallocIfNeeded(len);
// Using the now-valid output buffer, copy in any needed data.
out.buffer.setBytes(0, bytes);
// Upon exit, is the output buffer is now owned by caller? Must be, or the output
// data would be immediately freed if the caller were to release the injected
// buffer, and all hell would break loose.
```
As usual, there is no documentation of this, you just gotta know (or figure it out every time.)
The buffer management happens in `BufferManagerImpl`, as shown in the JIRA stack trace, which is called by `reallocIfNeeded`:
```
public DrillBuf replace(DrillBuf old, int newSize) {
if (managedBuffers.remove(old.memoryAddress()) == null) {
throw new IllegalStateException("Tried to remove unmanaged buffer.");
}
old.release(1);
return getManagedBuffer(newSize);
}
```
This is saying, "Hey, that buffer you gave me? It ain't one of mine." The question is why. The theory in this bug is that we forgot to replace our working buffer. Let's check that by doing a [search for `reallocIfNeeded`](https://github.com/apache/drill/search?p=2&q=reallocIfNeeded). We find many cases where we don't save the reallocated buffer back into the `@Inject` variable. For [example](https://github.com/apache/drill/blob/2eb6979206ad1b6f830a070dffa822edbfac6189/exec/java-exec/src/main/codegen/templates/Decimal/DecimalAggrTypeFunctions1.java#L113):
``` java
out.buffer = buffer.reallocIfNeeded(len);
out.buffer.setBytes(0, bytes);
```
I *suspect*, but am not sure without running the code, that every call to the UDF is issued a new managed `buffer`. So, I suspect that the problem will only occur if, within a single function call, we:
* Call `reallocIfNeeded` and get a new buffer.
* Call `reallocIfNeeded` on the original (now unregistered) buffer, but only if the previous call actually resized the buffer.
Looking at the code in this function, there are no loops, so there is no ability to make the above mistake. Yet, many other functions in the file changed here have the exact same code pattern introduced in this fix. Why does the code work so often if many other functions *do not* follow the pattern adopted here?
There are two possibilities:
* The generated code has a bug: it is not providing a new buffer on each function invocation.
* The problem is not where we think it is: it is elsewhere. (The stack trace in the JIRA ticket shows generated code, so we can't be sure which bit of code is actually making the call.)
In particular, I'm not convinced that the unit test reproduces the error. The unit test is different from the query in the JIRA ticket. A good practice is to run the proposed test *before* any fix to make sure we can reproduce the issue.
The proposed test (simplified):
``` sql
select concat(cast(null as varchar), EXPR$0) as c1
"from (values('foo'), ('bar'))
```
User's query (also simplified):
``` sql
select concat(BinaryValue, 'aaa') NewVarcharValue from ...;
```
What is different? It may not be obvious, but in the first query, we have a `NULL` const and a column (value vector, which happens to be defined by a `VALUES` clause with a constant). In the second, we have a column value and a string constant. That string constant is not stored in a value vector, it is in a temporary buffer (IIRC).
My hunch is that the problem is with the constant: it is causing something to break. This is why so many of the functions in `StringFunctions.java` have code like the proposed fix: they are also likely tested with constants. If the problem is with constants, then it means that all other UDFs that work with strings, but don't have the proposed fix, will fail when used with constants.
That is, sounds like we got us a bit of a problem; this fix is just the tip of the iceberg.
How should we proceed? We can do just this fix, and fix other issues as people encounter and report them. Or, we can figure out what is *really* going on by looking at the generated code. To do that, modify the [Project operator](https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/project/ProjectRecordBatch.java#L279):
``` java
boolean saveCode = true; \\ change from false in the master branch
```
Set the variable to `true` as shown above (it will be `false` by default.) Run a query like the user's. Code will be generated. By default I think it is in `/tmp/drill/codegen/...`. Analyze what's going on with the `@Inject` buffer.
Pay particular attention to how constants are handled. What is causing us to try to reallocate a non-managed buffer in this case?
Then, we can decide how to proceed:
* If the fix here is the correct one, make it in all UDFs that have the wrong pattern. (See the search above, there are **many** such occurrences.) Add a comment to the `reallocIfNeeded` method to explain the proper usage. Provide a code snippet to show what should be done.
* If the problem is with constant handling, work out what should be done. Fix it. Revert the `out.buffer = buffer = buffer.ralloc...` bits of code in `StringFunctions.java` and retest to ensure constants work correctly.
BTW, in researching these issues, I found a number of other problems.
If the proposed fix is correct, then a particularly nasty buf is [here](https://github.com/apache/drill/blob/2eb6979206ad1b6f830a070dffa822edbfac6189/exec/vector/src/main/java/org/apache/drill/exec/vector/ValueHolderHelper.java#L121) where the new buffer is created internally and never returned.
The function we're trying to fix, `ConcatLeftNullInput`, has a bug:
``` java
public static class ConcatLeftNullInput implements DrillSimpleFunc {
...
@Override
public void eval() {
out.buffer = buffer.reallocIfNeeded( (left.end - left.start) + (right.end - right.start));
```
The function is `...LeftNullInput` and, yet, we are asking for the `left.end` and `left.start`. By definition these are zero. Why? Because the left is `NULL`. In fact, if the left value is `NULL`, I'm not sure the `start` and `end` values are even defined to be any particular value. (Amazing this stuff works as well as it does.)
In the same file, this code is *just plain wrong*:
``` java
public static class Replace implements DrillSimpleFunc {
...
@Inject DrillBuf buffer;
@Output VarCharHolder out;
@Override
public void setup() {
buffer = buffer.reallocIfNeeded(8000);
}
```
See the bug? What if the new string is larger than 8K? Kaboom. What if the string is 10 characters? Huge waste of memory. The above is what we call "crude and not quite effective", it works most of the time, except when it doesn't.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] paul-rogers commented on pull request #2183: DRILL-6547: IllegalStateException: Tried to remove unmanaged buffer in concat function
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on pull request #2183:
URL: https://github.com/apache/drill/pull/2183#issuecomment-812925090
@luocooong, thanks for the question. I raised several issues. I'll leave it to the team to decide whether to worry about those issues, or just wait for someone to report new bugs. My tendency is, once I become aware of an issue, to fix it if doing so is quick; I find that more efficient than waiting for some poor user to hit the issue, debug the problem, report the bug, get the bug prioritized, and go through another code review cycle. But, that's just me.
My suggestion would be to inspect and, if necessary, fix all the buffer issues in the file that contains the concat functions. An "extra for experts" is to change the code in the concat functions that works with null values to *not* inspect the null value, as, in the general case, it can be invalid. Of course, we don't know that these are problems, probably because we have no good tests. So, if you are of the "test driven development" persuasion, I'd encourage writing a few tests to verify behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] cgivre commented on pull request #2183: DRILL-6547: IllegalStateException: Tried to remove unmanaged buffer in concat function
Posted by GitBox <gi...@apache.org>.
cgivre commented on pull request #2183:
URL: https://github.com/apache/drill/pull/2183#issuecomment-840692812
> > I would suggest scanning as many functions as you can for this error and fixing them all here. We know about the problem; it is a systematic oversight of the earlier authors, and so we might as well fix the whole thing rather than doing each case one PR at a time.
>
> Fixed all uses of reallocIfNeeded that I could find
Thanks for doing this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] luocooong commented on pull request #2183: DRILL-6547: IllegalStateException: Tried to remove unmanaged buffer in concat function
Posted by GitBox <gi...@apache.org>.
luocooong commented on pull request #2183:
URL: https://github.com/apache/drill/pull/2183#issuecomment-845565607
@cgivre Yes, It is time to merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] oleg-zinovev commented on a change in pull request #2183: DRILL-6547: IllegalStateException: Tried to remove unmanaged buffer in concat function
Posted by GitBox <gi...@apache.org>.
oleg-zinovev commented on a change in pull request #2183:
URL: https://github.com/apache/drill/pull/2183#discussion_r631771430
##########
File path: exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/StringFunctions.java
##########
@@ -1425,7 +1419,7 @@ public void eval() {
out.buffer = buffer = buffer.reallocIfNeeded((left.end - left.start) + (right.end - right.start));
out.start = out.end = 0;
- int id = 0;
+ int id;
Review comment:
In this case, the id variable is also used in the next loop, as a pointer to where to continue writing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] paul-rogers edited a comment on pull request #2183: DRILL-6547: IllegalStateException: Tried to remove unmanaged buffer in concat function
Posted by GitBox <gi...@apache.org>.
paul-rogers edited a comment on pull request #2183:
URL: https://github.com/apache/drill/pull/2183#issuecomment-791072405
Thanks for fixing this.
I suspect the problem here runs deeper than this particular fix. This is Drill: nothing is simple, things build on hacks which build on undocumented complexities. I suspect that is the issue here.
In particular, we need to know the rules for buffer ownership in UDFs which is, of course, not documented. The only reasonable way to learn this is to run a query, save the generated Java code and inspect the calls. Warning: the generated code is a huge mess.
The key question is this: when you declare an `@Inject` variable (`buffer` in your case), does the runtime provide an initial buffer? My guess is that it does in the form of a "managed" buffer. (This just means that there is some auto-free mechanism.) I believe that the buffer is allocated on each call, but the generated code will say for sure. (The UDF code is copied into the generated code.)
The function declares an `@Output`, in this case `out`. If this variable provides a buffer, it would seem that ownership has to pass to the caller, since the caller will copy that buffer into a value vector. My guess is that if the buffer is "managed", it will be freed automatically at some point.
(Note that managed buffers can cause an OOM because of when they are freed: I think it is during fragment shutdown, IIRC. I had to fix a reader bug related to this issue.)
To summarize, it appears that the proper pattern in UDFs is:
* Declare an `@Inject` buffer which the runtime will provide.
* Resize the buffer if needed, using the return value from `reallocIfNeeded`.
* Use that buffer as the output buffer by setting it to `out.buffer` where needed.
* Managed buffers are freed automatically, at some unknown point.
If this is true, there is no reason to update `buffer`, the proper pattern is shown in [DecimalAggrTypeFunctions1](https://github.com/apache/drill/blob/2eb6979206ad1b6f830a070dffa822edbfac6189/exec/java-exec/src/main/codegen/templates/Decimal/DecimalAggrTypeFunctions1.java#L113) (with my own comments):
``` java
// Inject means that the runtime will provide a default-size buffer on each call
@Inject DrillBuf buffer;
// Holder for output. NOT initialized b the runtime.
@Output ${type.outputType}Holder out;
...
// Resize the provided buffer, replacing if necessary, and assigning that
// to the output variable.
out.buffer = buffer.reallocIfNeeded(len);
// Using the now-valid output buffer, copy in any needed data.
out.buffer.setBytes(0, bytes);
// Upon exit, is the output buffer is now owned by caller? Must be, or the output
// data would be immediately freed if the caller were to release the injected
// buffer, and all hell would break loose.
```
As usual, there is no documentation of this, you just gotta know (or figure it out every time.)
The buffer management happens in `BufferManagerImpl`, as shown in the JIRA stack trace, which is called by `reallocIfNeeded`:
```
public DrillBuf replace(DrillBuf old, int newSize) {
if (managedBuffers.remove(old.memoryAddress()) == null) {
throw new IllegalStateException("Tried to remove unmanaged buffer.");
}
old.release(1);
return getManagedBuffer(newSize);
}
```
This is saying, "Hey, that buffer you gave me? It ain't one of mine." The question is why. The theory in this bug is that we forgot to replace our working buffer. Let's check that by doing a [search for `reallocIfNeeded`](https://github.com/apache/drill/search?p=2&q=reallocIfNeeded). We find many cases where we don't save the reallocated buffer back into the `@Inject` variable. For [example](https://github.com/apache/drill/blob/2eb6979206ad1b6f830a070dffa822edbfac6189/exec/java-exec/src/main/codegen/templates/Decimal/DecimalAggrTypeFunctions1.java#L113):
``` java
out.buffer = buffer.reallocIfNeeded(len);
out.buffer.setBytes(0, bytes);
```
I *suspect*, but am not sure without running the code, that every call to the UDF is issued a new managed `buffer`. So, I suspect that the problem will only occur if, within a single function call, we:
* Call `reallocIfNeeded` and get a new buffer.
* Call `reallocIfNeeded` on the original (now unregistered) buffer, but only if the previous call actually resized the buffer.
Looking at the code in this function, there are no loops, so there is no ability to make the above mistake. Yet, many other functions in the file changed here have the exact same code pattern introduced in this fix. Why does the code work so often if many other functions *do not* follow the pattern adopted here?
There are two possibilities:
* The generated code has a bug: it is not providing a new buffer on each function invocation.
* The problem is not where we think it is: it is elsewhere. (The stack trace in the JIRA ticket shows generated code, so we can't be sure which bit of code is actually making the call.)
In particular, I'm not convinced that the unit test reproduces the error. The unit test is different from the query in the JIRA ticket. A good practice is to run the proposed test *before* any fix to make sure we can reproduce the issue.
The proposed test (simplified):
``` sql
select concat(cast(null as varchar), EXPR$0) as c1
"from (values('foo'), ('bar'))
```
User's query (also simplified):
``` sql
select concat(BinaryValue, 'aaa') NewVarcharValue from ...;
```
What is different? It may not be obvious, but in the first query, we have a `NULL` const and a column (value vector, which happens to be defined by a `VALUES` clause with a constant). In the second, we have a column value and a string constant. That string constant is not stored in a value vector, it is in a temporary buffer (IIRC).
My hunch is that the problem is with the constant: it is causing something to break. This is why so many of the functions in `StringFunctions.java` have code like the proposed fix: they are also likely tested with constants. If the problem is with constants, then it means that all other UDFs that work with strings, but don't have the proposed fix, will fail when used with constants.
That is, sounds like we got us a bit of a problem; this fix is just the tip of the iceberg.
How should we proceed? We can do just this fix, and fix other issues as people encounter and report them. Or, we can figure out what is *really* going on by looking at the generated code. To do that, modify the [Project operator](https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/project/ProjectRecordBatch.java#L279):
``` java
boolean saveCode = true; \\ change from false in the master branch
```
Set the variable to `true` as shown above (it will be `false` by default.) Run a query like the user's. Code will be generated. By default I think it is in `/tmp/drill/codegen/...`. Analyze what's going on with the `@Inject` buffer.
Pay particular attention to how constants are handled. What is causing us to try to reallocate a non-managed buffer in this case?
As it turns out, with this trick, you can actually *debug* the generated code. A bit after the above line is one that generates the file. Set a breakpoint after that line. Then, go hunt down the generated file. Set a breakpoint somewhere. Let the code run. It will stop at your breakpoint in the generated code. This is easy in Eclipse. IIRC, IntelliJ users have to do some trick to put the code in a path that IntelliJ likes. Jinfeng may have left a note somewhere from how he made it work many years ago.
Then, we can decide how to proceed:
* If the fix here is the correct one, make it in all UDFs that have the wrong pattern. (See the search above, there are **many** such occurrences.) Add a comment to the `reallocIfNeeded` method to explain the proper usage. Provide a code snippet to show what should be done.
* If the problem is with constant handling, work out what should be done. Fix it. Revert the `out.buffer = buffer = buffer.ralloc...` bits of code in `StringFunctions.java` and retest to ensure constants work correctly.
BTW, in researching these issues, I found a number of other problems.
If the proposed fix is correct, then a particularly nasty buf is [here](https://github.com/apache/drill/blob/2eb6979206ad1b6f830a070dffa822edbfac6189/exec/vector/src/main/java/org/apache/drill/exec/vector/ValueHolderHelper.java#L121) where the new buffer is created internally and never returned.
The function we're trying to fix, `ConcatLeftNullInput`, has a bug:
``` java
public static class ConcatLeftNullInput implements DrillSimpleFunc {
...
@Override
public void eval() {
out.buffer = buffer.reallocIfNeeded( (left.end - left.start) + (right.end - right.start));
```
The function is `...LeftNullInput` and, yet, we are asking for the `left.end` and `left.start`. By definition these are zero. Why? Because the left is `NULL`. In fact, if the left value is `NULL`, I'm not sure the `start` and `end` values are even defined to be any particular value. (Amazing this stuff works as well as it does.)
In the same file, this code is *just plain wrong*:
``` java
public static class Replace implements DrillSimpleFunc {
...
@Inject DrillBuf buffer;
@Output VarCharHolder out;
@Override
public void setup() {
buffer = buffer.reallocIfNeeded(8000);
}
```
See the bug? What if the new string is larger than 8K? Kaboom. What if the string is 10 characters? Huge waste of memory. The above is what we call "crude and not quite effective", it works most of the time, except when it doesn't.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] cgivre merged pull request #2183: DRILL-6547: IllegalStateException: Tried to remove unmanaged buffer in concat function
Posted by GitBox <gi...@apache.org>.
cgivre merged pull request #2183:
URL: https://github.com/apache/drill/pull/2183
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] oleg-zinovev commented on pull request #2183: DRILL-6547: IllegalStateException: Tried to remove unmanaged buffer in concat function
Posted by GitBox <gi...@apache.org>.
oleg-zinovev commented on pull request #2183:
URL: https://github.com/apache/drill/pull/2183#issuecomment-791903981
@paul-rogers
Thank you so much for parsing the fix.
1. Here are the code generated to perform the projection:
```java
public class ProjectorGen0
extends ProjectorTemplate
{
DrillBuf work0;
NullableVarCharHolder constant2;
NullableVarCharVector vv3;
VarCharVector vv8;
public ProjectorGen0() {
try {
__DRILL_INIT__();
} catch (SchemaChangeException e) {
throw new UnsupportedOperationException(e);
}
}
public void doEval(int inIndex, int outIndex)
throws SchemaChangeException
{
{
NullableVarCharHolder out6 = new NullableVarCharHolder();
{
out6 .isSet = vv3 .getAccessor().isSet((inIndex));
if (out6 .isSet == 1) {
out6 .buffer = vv3 .getBuffer();
long startEnd = vv3 .getAccessor().getStartEnd((inIndex));
out6 .start = ((int) startEnd);
out6 .end = ((int)(startEnd >> 32));
}
}
//---- start of eval portion of concat function. ----//
VarCharHolder out7 = new VarCharHolder();
{
final VarCharHolder out = new VarCharHolder();
NullableVarCharHolder left = constant2;
NullableVarCharHolder right = out6;
DrillBuf buffer = work0;
StringFunctions$ConcatBothNullInput_eval:
{
out.buffer = buffer = buffer.reallocIfNeeded((left.end - left.start) + (right.end - right.start));
out.start = out.end = 0;
int id = 0;
if (left.isSet == 1) {
for (id = left.start; id < left.end; id++) {
out.buffer.setByte(out.end++, left.buffer.getByte(id));
}
}
if (right.isSet == 1) {
for (id = right.start; id < right.end; id++) {
out.buffer.setByte(out.end++, right.buffer.getByte(id));
}
}
}
work0 = buffer;
out7 .start = out.start;
out7 .end = out.end;
out7 .buffer = out.buffer;
}
//---- end of eval portion of concat function. ----//
vv8 .getMutator().setSafe((outIndex), out7 .start, out7 .end, out7 .buffer);
}
}
public void doSetup(FragmentContext context, RecordBatch incoming, RecordBatch outgoing)
throws SchemaChangeException
{
{
work0 = (incoming).getContext().getManagedBuffer();
NullableVarCharHolder out1 = new NullableVarCharHolder();
constant2 = new NullableVarCharHolder();
constant2 .isSet = out1 .isSet;
constant2 .start = out1 .start;
constant2 .end = out1 .end;
constant2 .buffer = out1 .buffer;
int[] fieldIds4 = new int[ 1 ] ;
fieldIds4 [ 0 ] = 0;
Object tmp5 = (incoming).getValueAccessorById(NullableVarCharVector.class, fieldIds4).getValueVector();
if (tmp5 == null) {
throw new SchemaChangeException("Failure while loading vector vv3 with id: TypedFieldId [fieldIds=[0], remainder=null].");
}
vv3 = ((NullableVarCharVector) tmp5);
/** start SETUP for function concat **/
{
NullableVarCharHolder left = constant2;
DrillBuf buffer = work0;
StringFunctions$ConcatBothNullInput_setup:
{}
work0 = buffer;
}
/** end SETUP for function concat **/
int[] fieldIds9 = new int[ 1 ] ;
fieldIds9 [ 0 ] = 0;
Object tmp10 = (outgoing).getValueAccessorById(VarCharVector.class, fieldIds9).getValueVector();
if (tmp10 == null) {
throw new SchemaChangeException("Failure while loading vector vv8 with id: TypedFieldId [fieldIds=[0], remainder=null].");
}
vv8 = ((VarCharVector) tmp10);
}
}
public void __DRILL_INIT__()
throws SchemaChangeException
{
}
}
```
As you can see, the buffer is requested from the fragment context in the doSetup method (which in turn is called within setup). This happens once for each new schema in **ProjectRecordBatch#setupNewSchemaFromInput**
When calculating the function body, the buffer usage pattern is as follows:
1) Extract the current buffer from the class field to a local variable
2) Execute the function body
3) Save the value of the local buffer variable in the class field
In the current implementation, managed buffer is "reused" for each subsequent row. Therefore, the result of reallocIfNeeded must be stored in the buffer field declared within the function.
2. You are not quite right when you say that the result of left. end - left. start will always be 0. For example, the same implementation of the function will be used in the case when the first argument is the nullable field of the parquet file, in which not all values will be null.
P.S. Perhaps I did not understand you correctly
3. Unfortunately, I can't use the sample request given in the bug, as it uses a file that is missing from the drill project. In addition, this file is uploaded to MapR jira, where I do not have access. If you can attach a file to the specified Drill bug, I will happily replace my request with the request given in the bug.
P.S. I'm sorry for my English
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] oleg-zinovev edited a comment on pull request #2183: DRILL-6547: IllegalStateException: Tried to remove unmanaged buffer in concat function
Posted by GitBox <gi...@apache.org>.
oleg-zinovev edited a comment on pull request #2183:
URL: https://github.com/apache/drill/pull/2183#issuecomment-815969446
@paul-rogers , @luocooong , @cgivre
Thank you for the additional explanations. I have changed the **ConcatLeftNullInput**, **ConcatBothNullInput** and **ConcatRightNullInput** functions based on your comments.
I have also fixed the same error when using a buffer in the **binary_string** and **string_binary** functions.
I have removed the buffer in the **right**, **regexp_matches**, **substring**, **left** functions, since it was not used in this functions bodies.
And last but not least, I have found buffer overflow error in **replace** function. Should I open another PR, or I can commit fix in current one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] oleg-zinovev commented on pull request #2183: DRILL-6547: IllegalStateException: Tried to remove unmanaged buffer in concat function
Posted by GitBox <gi...@apache.org>.
oleg-zinovev commented on pull request #2183:
URL: https://github.com/apache/drill/pull/2183#issuecomment-840687215
> I would suggest scanning as many functions as you can for this error and fixing them all here. We know about the problem; it is a systematic oversight of the earlier authors, and so we might as well fix the whole thing rather than doing each case one PR at a time.
Fixed all uses of reallocIfNeeded that I could find
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] luocooong commented on pull request #2183: DRILL-6547: IllegalStateException: Tried to remove unmanaged buffer in concat function
Posted by GitBox <gi...@apache.org>.
luocooong commented on pull request #2183:
URL: https://github.com/apache/drill/pull/2183#issuecomment-811846074
@paul-rogers Thanks for the review. It's time to merge this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] oleg-zinovev edited a comment on pull request #2183: DRILL-6547: IllegalStateException: Tried to remove unmanaged buffer in concat function
Posted by GitBox <gi...@apache.org>.
oleg-zinovev edited a comment on pull request #2183:
URL: https://github.com/apache/drill/pull/2183#issuecomment-791903981
@paul-rogers
Thank you so much for reviewing the fix.
1. Here are the code generated to perform the projection:
```java
public class ProjectorGen0
extends ProjectorTemplate
{
DrillBuf work0;
NullableVarCharHolder constant2;
NullableVarCharVector vv3;
VarCharVector vv8;
public ProjectorGen0() {
try {
__DRILL_INIT__();
} catch (SchemaChangeException e) {
throw new UnsupportedOperationException(e);
}
}
public void doEval(int inIndex, int outIndex)
throws SchemaChangeException
{
{
NullableVarCharHolder out6 = new NullableVarCharHolder();
{
out6 .isSet = vv3 .getAccessor().isSet((inIndex));
if (out6 .isSet == 1) {
out6 .buffer = vv3 .getBuffer();
long startEnd = vv3 .getAccessor().getStartEnd((inIndex));
out6 .start = ((int) startEnd);
out6 .end = ((int)(startEnd >> 32));
}
}
//---- start of eval portion of concat function. ----//
VarCharHolder out7 = new VarCharHolder();
{
final VarCharHolder out = new VarCharHolder();
NullableVarCharHolder left = constant2;
NullableVarCharHolder right = out6;
DrillBuf buffer = work0;
StringFunctions$ConcatBothNullInput_eval:
{
out.buffer = buffer = buffer.reallocIfNeeded((left.end - left.start) + (right.end - right.start));
out.start = out.end = 0;
int id = 0;
if (left.isSet == 1) {
for (id = left.start; id < left.end; id++) {
out.buffer.setByte(out.end++, left.buffer.getByte(id));
}
}
if (right.isSet == 1) {
for (id = right.start; id < right.end; id++) {
out.buffer.setByte(out.end++, right.buffer.getByte(id));
}
}
}
work0 = buffer;
out7 .start = out.start;
out7 .end = out.end;
out7 .buffer = out.buffer;
}
//---- end of eval portion of concat function. ----//
vv8 .getMutator().setSafe((outIndex), out7 .start, out7 .end, out7 .buffer);
}
}
public void doSetup(FragmentContext context, RecordBatch incoming, RecordBatch outgoing)
throws SchemaChangeException
{
{
work0 = (incoming).getContext().getManagedBuffer();
NullableVarCharHolder out1 = new NullableVarCharHolder();
constant2 = new NullableVarCharHolder();
constant2 .isSet = out1 .isSet;
constant2 .start = out1 .start;
constant2 .end = out1 .end;
constant2 .buffer = out1 .buffer;
int[] fieldIds4 = new int[ 1 ] ;
fieldIds4 [ 0 ] = 0;
Object tmp5 = (incoming).getValueAccessorById(NullableVarCharVector.class, fieldIds4).getValueVector();
if (tmp5 == null) {
throw new SchemaChangeException("Failure while loading vector vv3 with id: TypedFieldId [fieldIds=[0], remainder=null].");
}
vv3 = ((NullableVarCharVector) tmp5);
/** start SETUP for function concat **/
{
NullableVarCharHolder left = constant2;
DrillBuf buffer = work0;
StringFunctions$ConcatBothNullInput_setup:
{}
work0 = buffer;
}
/** end SETUP for function concat **/
int[] fieldIds9 = new int[ 1 ] ;
fieldIds9 [ 0 ] = 0;
Object tmp10 = (outgoing).getValueAccessorById(VarCharVector.class, fieldIds9).getValueVector();
if (tmp10 == null) {
throw new SchemaChangeException("Failure while loading vector vv8 with id: TypedFieldId [fieldIds=[0], remainder=null].");
}
vv8 = ((VarCharVector) tmp10);
}
}
public void __DRILL_INIT__()
throws SchemaChangeException
{
}
}
```
As you can see, the buffer is requested from the fragment context in the doSetup method (which in turn is called within setup). This happens once for each new schema in **ProjectRecordBatch#setupNewSchemaFromInput**
When calculating the function body, the following buffer usage pattern is used:
1) Extract the current buffer from the class field to a local variable
2) Execute the function body
3) Save the value of the local buffer variable in the class field
In the current implementation, managed buffer is "reused" for each subsequent row. Therefore, the result of reallocIfNeeded must be stored in the buffer field declared within the function.
2. You are not quite right when you say that the result of left. end - left. start will always be 0. For example, the same implementation of the function will be used in the case when the first argument is the nullable field of the parquet file, in which not all values will be null.
P.S. Perhaps I did not understand you correctly
3. Unfortunately, I can't use the sample request given in the bug, as it uses a file that is missing from the drill project. In addition, this file is uploaded to MapR jira, where I do not have access. If you can attach a file to the specified Drill bug, I will happily replace my request with the request given in the bug.
P.S. I'm sorry for my English
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] oleg-zinovev edited a comment on pull request #2183: DRILL-6547: IllegalStateException: Tried to remove unmanaged buffer in concat function
Posted by GitBox <gi...@apache.org>.
oleg-zinovev edited a comment on pull request #2183:
URL: https://github.com/apache/drill/pull/2183#issuecomment-791903981
@paul-rogers
Thank you so much for reviewing the fix.
1. Here are the code generated to perform the projection:
```java
public class ProjectorGen0
extends ProjectorTemplate
{
DrillBuf work0;
NullableVarCharHolder constant2;
NullableVarCharVector vv3;
VarCharVector vv8;
public ProjectorGen0() {
try {
__DRILL_INIT__();
} catch (SchemaChangeException e) {
throw new UnsupportedOperationException(e);
}
}
public void doEval(int inIndex, int outIndex)
throws SchemaChangeException
{
{
NullableVarCharHolder out6 = new NullableVarCharHolder();
{
out6 .isSet = vv3 .getAccessor().isSet((inIndex));
if (out6 .isSet == 1) {
out6 .buffer = vv3 .getBuffer();
long startEnd = vv3 .getAccessor().getStartEnd((inIndex));
out6 .start = ((int) startEnd);
out6 .end = ((int)(startEnd >> 32));
}
}
//---- start of eval portion of concat function. ----//
VarCharHolder out7 = new VarCharHolder();
{
final VarCharHolder out = new VarCharHolder();
NullableVarCharHolder left = constant2;
NullableVarCharHolder right = out6;
DrillBuf buffer = work0;
StringFunctions$ConcatBothNullInput_eval:
{
out.buffer = buffer = buffer.reallocIfNeeded((left.end - left.start) + (right.end - right.start));
out.start = out.end = 0;
int id = 0;
if (left.isSet == 1) {
for (id = left.start; id < left.end; id++) {
out.buffer.setByte(out.end++, left.buffer.getByte(id));
}
}
if (right.isSet == 1) {
for (id = right.start; id < right.end; id++) {
out.buffer.setByte(out.end++, right.buffer.getByte(id));
}
}
}
work0 = buffer;
out7 .start = out.start;
out7 .end = out.end;
out7 .buffer = out.buffer;
}
//---- end of eval portion of concat function. ----//
vv8 .getMutator().setSafe((outIndex), out7 .start, out7 .end, out7 .buffer);
}
}
public void doSetup(FragmentContext context, RecordBatch incoming, RecordBatch outgoing)
throws SchemaChangeException
{
{
work0 = (incoming).getContext().getManagedBuffer();
NullableVarCharHolder out1 = new NullableVarCharHolder();
constant2 = new NullableVarCharHolder();
constant2 .isSet = out1 .isSet;
constant2 .start = out1 .start;
constant2 .end = out1 .end;
constant2 .buffer = out1 .buffer;
int[] fieldIds4 = new int[ 1 ] ;
fieldIds4 [ 0 ] = 0;
Object tmp5 = (incoming).getValueAccessorById(NullableVarCharVector.class, fieldIds4).getValueVector();
if (tmp5 == null) {
throw new SchemaChangeException("Failure while loading vector vv3 with id: TypedFieldId [fieldIds=[0], remainder=null].");
}
vv3 = ((NullableVarCharVector) tmp5);
/** start SETUP for function concat **/
{
NullableVarCharHolder left = constant2;
DrillBuf buffer = work0;
StringFunctions$ConcatBothNullInput_setup:
{}
work0 = buffer;
}
/** end SETUP for function concat **/
int[] fieldIds9 = new int[ 1 ] ;
fieldIds9 [ 0 ] = 0;
Object tmp10 = (outgoing).getValueAccessorById(VarCharVector.class, fieldIds9).getValueVector();
if (tmp10 == null) {
throw new SchemaChangeException("Failure while loading vector vv8 with id: TypedFieldId [fieldIds=[0], remainder=null].");
}
vv8 = ((VarCharVector) tmp10);
}
}
public void __DRILL_INIT__()
throws SchemaChangeException
{
}
}
```
As you can see, the buffer is requested from the fragment context in the doSetup method (which in turn is called within setup). This happens once for each new schema in **ProjectRecordBatch#setupNewSchemaFromInput**
When calculating the function body, the following buffer usage pattern is used:
- Extract the current buffer from the class field to a local variable
- Execute the function body
- Save the value of the local buffer variable in the class field
In the current implementation, managed buffer is "reused" for each subsequent row. Therefore, the result of reallocIfNeeded must be stored in the buffer field declared within the function.
2. You are not quite right when you say that the result of left. end - left. start will always be 0. For example, the same implementation of the function will be used in the case when the first argument is the nullable field of the parquet file, in which not all values will be null.
P.S. Perhaps I did not understand you correctly
3. Unfortunately, I can't use the sample request given in the bug, as it uses a file that is missing from the drill project. In addition, this file is uploaded to MapR jira, where I do not have access. If you can attach a file to the specified Drill bug, I will happily replace my request with the request given in the bug.
P.S. I'm sorry for my English
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] oleg-zinovev commented on pull request #2183: DRILL-6547: IllegalStateException: Tried to remove unmanaged buffer in concat function
Posted by GitBox <gi...@apache.org>.
oleg-zinovev commented on pull request #2183:
URL: https://github.com/apache/drill/pull/2183#issuecomment-815969446
@paul-rogers , @luocooong , @cgivre
Thank you for the additional explanations. I have changed the **ConcatLeftNullInput**, **ConcatBothNullInput** and **ConcatRightNullInput** functions based on your comments.
I have also fixed the same error when using a buffer in the **binary_string** and **string_binary** functions.
I have removed the buffer in the **right**, **regexp_matches**, **substring**, **left** functions, since it was not used in this functions bodies.
And last but not least, I have found buffer overflow error in **replace** function. Should I open another PR, or I can commit changes in current one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] oleg-zinovev edited a comment on pull request #2183: DRILL-6547: IllegalStateException: Tried to remove unmanaged buffer in concat function
Posted by GitBox <gi...@apache.org>.
oleg-zinovev edited a comment on pull request #2183:
URL: https://github.com/apache/drill/pull/2183#issuecomment-815969446
@paul-rogers , @luocooong , @cgivre
Thank you for the additional explanations. I have changed the **ConcatLeftNullInput**, **ConcatBothNullInput** and **ConcatRightNullInput** functions based on your comments.
I have also fixed the same error when using a buffer in the **binary_string** and **string_binary** functions.
I have removed the buffer in the **right**, **regexp_matches**, **substring**, **left** functions, since it was not used in this functions bodies.
P.S. I have found buffer overflow error in **replace** function. Should I open another PR, or I can commit fix in current one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org