You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@drill.apache.org by Nirav Shah <ni...@games24x7.com> on 2016/01/05 05:04:06 UTC
Null Return
Hi,
May be this is silly question, Please let me know how to return null string
from drill custom function.
Regards,
Nirav
Re: Null Return
Posted by Jason Altekruse <al...@gmail.com>.
Nirav,
Were you able to get your function working?
On Wed, Jan 6, 2016 at 6:42 AM, Jason Altekruse <al...@gmail.com>
wrote:
> I believe you are hitting a case that Jacques was trying to describe with
> his last message.
>
> Drill currently matches functions on both data type (int, varchar,
> boolean) and data mode (nullable or required).
>
> If you look at the error message, you can see that the data types that
> drill is receiving for your inputs are all required, not nullable. Your
> function has all nullable inputs. You will need to define a version that
> takes the non-nullable Holders.
>
> That being said, you probably don't want to just replace your existing one
> with different inputs, because then it will fail to run if you have
> nullable input data in a different query. That is why we use code
> generation with freemarker to explode out all of the combinations of
> nullable and non-nullable inputs. If you don't want to mess with code
> generation you could just duplicate the code yourself, but obviously that
> may be harder to maintain.
>
> Missing function implementation:
> [utmfuncgen(VARCHAR-REQUIRED, VARCHAR-REQUIRED, VARCHAR-REQUIRED,
> VARCHAR-REQUIRED)
>
> On Wed, Jan 6, 2016 at 4:45 AM, Nirav Shah <ni...@games24x7.com>
> wrote:
>
>> I tried but I am getting below issue.
>>
>>
>> *Code:*
>>
>> @FunctionTemplate(name = "UTMFuncGen", scope =
>> FunctionTemplate.FunctionScope.SIMPLE, nulls =
>> FunctionTemplate.NullHandling.INTERNAL)
>> public class UTMFuncGen implements DrillSimpleFunc {
>>
>> @Param
>> NullableVarCharHolder key;
>> @Param
>> NullableVarCharHolder value;
>> @Param
>> NullableVarCharHolder endPoint;
>> @Output
>> VarCharHolder outValue;
>> @Inject
>> DrillBuf buffer;
>>
>> public void setup() {
>> // TODO Auto-generated method stub
>>
>> }
>>
>> public void eval() {
>> String stringOutValue = "null";
>> if(value.isSet == 0){
>> stringOutValue = "null";
>> }
>> try {
>> String stringKey =
>>
>> org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.toStringFromUTF8(key.start,
>> key.end, key.buffer);
>> String stringValue =
>>
>> org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.toStringFromUTF8(value.start,
>> value.end, value.buffer);
>> String stringEndPoint =
>>
>> org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.toStringFromUTF8(endPoint.start,
>> endPoint.end, endPoint.buffer);
>> // String stringNullHandle =
>>
>> org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.toStringFromUTF8(nullHandle.start,
>> nullHandle.end, nullHandle.buffer);
>> int verOffset, verOffset1;
>> System.out.println("string value"+stringValue);
>> if ((verOffset = stringValue.indexOf(stringKey)) != -1) {
>> if ((verOffset1 = stringValue.indexOf(stringEndPoint,
>> verOffset + 1)) != -1) {
>> stringOutValue = stringValue.substring(verOffset +
>> stringKey.length(), verOffset1);
>> System.out.println("parsed"+stringOutValue);
>> } else {
>> verOffset1 = stringValue.length();
>> stringOutValue = stringValue.substring(verOffset +
>> stringKey.length(), verOffset1);
>> }
>>
>>
>>
>>
>> *.....*
>>
>> *Error:*
>> Error: SYSTEM ERROR: DrillRuntimeException: Failure while materializing
>> expression in constant expression evaluator [UTMFUNCGEN('cid=', '
>>
>> https://www.rummycircle.com/rummy-for-cash-is-fun.html?utm_sm_campaign=CT-FTM
>> ',
>> '&', 'cid=null')]. Errors:
>> Error in expression at index -1. Error: Missing function implementation:
>> [utmfuncgen(VARCHAR-REQUIRED, VARCHAR-REQUIRED, VARCHAR-REQUIRED,
>> VARCHAR-REQUIRED)]. Full expression: --UNKNOWN EXPRESSION--.
>>
>>
>>
>>
>> *My Requirement:*
>> I want to return Null as null not as string in all the cases but
>> 1) When input is null it returns null
>> 2) when input is there but not valid as per business logic need to return
>> null, I am not able to return null.
>>
>> Regards,
>> Nirav
>>
>>
>>
>> On Wed, Jan 6, 2016 at 11:23 AM, Nirav Shah <ni...@games24x7.com>
>> wrote:
>>
>> > Thanks Jacques. Appreciate your response.
>> >
>> > On Tue, Jan 5, 2016 at 10:59 PM, Jacques Nadeau <ja...@dremio.com>
>> > wrote:
>> >
>> >> The default behavior of Drill UDFs is NullHandling.NULL_IF_NULL
>> semantics.
>> >> This means that a the Drill engine manages checking nullability on
>> inputs.
>> >> If one or more inputs is null, then the output is null. You can also
>> >> implement NullHandling.INTERNAL as your null semantics in a UDF. This
>> >> allows you to set nullability on your output independent of the
>> >> nullability
>> >> of the inputs. An example is the isNull functions [1]. Note that the
>> >> complexity with using internal handling if you have to create
>> >> implementations for all variations of null inputs and outputs. This is
>> the
>> >> cartesian product of variations when you have multiple parameters. As
>> >> such,
>> >> we generally use freemarker templates (per the example below) to create
>> >> the
>> >> variations.
>> >>
>> >> [1]
>> >>
>> >>
>> https://github.com/apache/drill/blob/master/exec/java-exec/src/main/codegen/templates/NullOperator.java
>> >>
>> >> --
>> >> Jacques Nadeau
>> >> CTO and Co-Founder, Dremio
>> >>
>> >> On Mon, Jan 4, 2016 at 8:04 PM, Nirav Shah <ni...@games24x7.com>
>> >> wrote:
>> >>
>> >> > Hi,
>> >> > May be this is silly question, Please let me know how to return null
>> >> string
>> >> > from drill custom function.
>> >> >
>> >> >
>> >> > Regards,
>> >> > Nirav
>> >> >
>> >>
>> >
>> >
>>
>
>
Re: Null Return
Posted by Jason Altekruse <al...@gmail.com>.
I believe you are hitting a case that Jacques was trying to describe with
his last message.
Drill currently matches functions on both data type (int, varchar, boolean)
and data mode (nullable or required).
If you look at the error message, you can see that the data types that
drill is receiving for your inputs are all required, not nullable. Your
function has all nullable inputs. You will need to define a version that
takes the non-nullable Holders.
That being said, you probably don't want to just replace your existing one
with different inputs, because then it will fail to run if you have
nullable input data in a different query. That is why we use code
generation with freemarker to explode out all of the combinations of
nullable and non-nullable inputs. If you don't want to mess with code
generation you could just duplicate the code yourself, but obviously that
may be harder to maintain.
Missing function implementation:
[utmfuncgen(VARCHAR-REQUIRED, VARCHAR-REQUIRED, VARCHAR-REQUIRED,
VARCHAR-REQUIRED)
On Wed, Jan 6, 2016 at 4:45 AM, Nirav Shah <ni...@games24x7.com> wrote:
> I tried but I am getting below issue.
>
>
> *Code:*
>
> @FunctionTemplate(name = "UTMFuncGen", scope =
> FunctionTemplate.FunctionScope.SIMPLE, nulls =
> FunctionTemplate.NullHandling.INTERNAL)
> public class UTMFuncGen implements DrillSimpleFunc {
>
> @Param
> NullableVarCharHolder key;
> @Param
> NullableVarCharHolder value;
> @Param
> NullableVarCharHolder endPoint;
> @Output
> VarCharHolder outValue;
> @Inject
> DrillBuf buffer;
>
> public void setup() {
> // TODO Auto-generated method stub
>
> }
>
> public void eval() {
> String stringOutValue = "null";
> if(value.isSet == 0){
> stringOutValue = "null";
> }
> try {
> String stringKey =
>
> org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.toStringFromUTF8(key.start,
> key.end, key.buffer);
> String stringValue =
>
> org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.toStringFromUTF8(value.start,
> value.end, value.buffer);
> String stringEndPoint =
>
> org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.toStringFromUTF8(endPoint.start,
> endPoint.end, endPoint.buffer);
> // String stringNullHandle =
>
> org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.toStringFromUTF8(nullHandle.start,
> nullHandle.end, nullHandle.buffer);
> int verOffset, verOffset1;
> System.out.println("string value"+stringValue);
> if ((verOffset = stringValue.indexOf(stringKey)) != -1) {
> if ((verOffset1 = stringValue.indexOf(stringEndPoint,
> verOffset + 1)) != -1) {
> stringOutValue = stringValue.substring(verOffset +
> stringKey.length(), verOffset1);
> System.out.println("parsed"+stringOutValue);
> } else {
> verOffset1 = stringValue.length();
> stringOutValue = stringValue.substring(verOffset +
> stringKey.length(), verOffset1);
> }
>
>
>
>
> *.....*
>
> *Error:*
> Error: SYSTEM ERROR: DrillRuntimeException: Failure while materializing
> expression in constant expression evaluator [UTMFUNCGEN('cid=', '
>
> https://www.rummycircle.com/rummy-for-cash-is-fun.html?utm_sm_campaign=CT-FTM
> ',
> '&', 'cid=null')]. Errors:
> Error in expression at index -1. Error: Missing function implementation:
> [utmfuncgen(VARCHAR-REQUIRED, VARCHAR-REQUIRED, VARCHAR-REQUIRED,
> VARCHAR-REQUIRED)]. Full expression: --UNKNOWN EXPRESSION--.
>
>
>
>
> *My Requirement:*
> I want to return Null as null not as string in all the cases but
> 1) When input is null it returns null
> 2) when input is there but not valid as per business logic need to return
> null, I am not able to return null.
>
> Regards,
> Nirav
>
>
>
> On Wed, Jan 6, 2016 at 11:23 AM, Nirav Shah <ni...@games24x7.com>
> wrote:
>
> > Thanks Jacques. Appreciate your response.
> >
> > On Tue, Jan 5, 2016 at 10:59 PM, Jacques Nadeau <ja...@dremio.com>
> > wrote:
> >
> >> The default behavior of Drill UDFs is NullHandling.NULL_IF_NULL
> semantics.
> >> This means that a the Drill engine manages checking nullability on
> inputs.
> >> If one or more inputs is null, then the output is null. You can also
> >> implement NullHandling.INTERNAL as your null semantics in a UDF. This
> >> allows you to set nullability on your output independent of the
> >> nullability
> >> of the inputs. An example is the isNull functions [1]. Note that the
> >> complexity with using internal handling if you have to create
> >> implementations for all variations of null inputs and outputs. This is
> the
> >> cartesian product of variations when you have multiple parameters. As
> >> such,
> >> we generally use freemarker templates (per the example below) to create
> >> the
> >> variations.
> >>
> >> [1]
> >>
> >>
> https://github.com/apache/drill/blob/master/exec/java-exec/src/main/codegen/templates/NullOperator.java
> >>
> >> --
> >> Jacques Nadeau
> >> CTO and Co-Founder, Dremio
> >>
> >> On Mon, Jan 4, 2016 at 8:04 PM, Nirav Shah <ni...@games24x7.com>
> >> wrote:
> >>
> >> > Hi,
> >> > May be this is silly question, Please let me know how to return null
> >> string
> >> > from drill custom function.
> >> >
> >> >
> >> > Regards,
> >> > Nirav
> >> >
> >>
> >
> >
>
Re: Null Return
Posted by Nirav Shah <ni...@games24x7.com>.
I tried but I am getting below issue.
*Code:*
@FunctionTemplate(name = "UTMFuncGen", scope =
FunctionTemplate.FunctionScope.SIMPLE, nulls =
FunctionTemplate.NullHandling.INTERNAL)
public class UTMFuncGen implements DrillSimpleFunc {
@Param
NullableVarCharHolder key;
@Param
NullableVarCharHolder value;
@Param
NullableVarCharHolder endPoint;
@Output
VarCharHolder outValue;
@Inject
DrillBuf buffer;
public void setup() {
// TODO Auto-generated method stub
}
public void eval() {
String stringOutValue = "null";
if(value.isSet == 0){
stringOutValue = "null";
}
try {
String stringKey =
org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.toStringFromUTF8(key.start,
key.end, key.buffer);
String stringValue =
org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.toStringFromUTF8(value.start,
value.end, value.buffer);
String stringEndPoint =
org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.toStringFromUTF8(endPoint.start,
endPoint.end, endPoint.buffer);
// String stringNullHandle =
org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.toStringFromUTF8(nullHandle.start,
nullHandle.end, nullHandle.buffer);
int verOffset, verOffset1;
System.out.println("string value"+stringValue);
if ((verOffset = stringValue.indexOf(stringKey)) != -1) {
if ((verOffset1 = stringValue.indexOf(stringEndPoint,
verOffset + 1)) != -1) {
stringOutValue = stringValue.substring(verOffset +
stringKey.length(), verOffset1);
System.out.println("parsed"+stringOutValue);
} else {
verOffset1 = stringValue.length();
stringOutValue = stringValue.substring(verOffset +
stringKey.length(), verOffset1);
}
*.....*
*Error:*
Error: SYSTEM ERROR: DrillRuntimeException: Failure while materializing
expression in constant expression evaluator [UTMFUNCGEN('cid=', '
https://www.rummycircle.com/rummy-for-cash-is-fun.html?utm_sm_campaign=CT-FTM',
'&', 'cid=null')]. Errors:
Error in expression at index -1. Error: Missing function implementation:
[utmfuncgen(VARCHAR-REQUIRED, VARCHAR-REQUIRED, VARCHAR-REQUIRED,
VARCHAR-REQUIRED)]. Full expression: --UNKNOWN EXPRESSION--.
*My Requirement:*
I want to return Null as null not as string in all the cases but
1) When input is null it returns null
2) when input is there but not valid as per business logic need to return
null, I am not able to return null.
Regards,
Nirav
On Wed, Jan 6, 2016 at 11:23 AM, Nirav Shah <ni...@games24x7.com>
wrote:
> Thanks Jacques. Appreciate your response.
>
> On Tue, Jan 5, 2016 at 10:59 PM, Jacques Nadeau <ja...@dremio.com>
> wrote:
>
>> The default behavior of Drill UDFs is NullHandling.NULL_IF_NULL semantics.
>> This means that a the Drill engine manages checking nullability on inputs.
>> If one or more inputs is null, then the output is null. You can also
>> implement NullHandling.INTERNAL as your null semantics in a UDF. This
>> allows you to set nullability on your output independent of the
>> nullability
>> of the inputs. An example is the isNull functions [1]. Note that the
>> complexity with using internal handling if you have to create
>> implementations for all variations of null inputs and outputs. This is the
>> cartesian product of variations when you have multiple parameters. As
>> such,
>> we generally use freemarker templates (per the example below) to create
>> the
>> variations.
>>
>> [1]
>>
>> https://github.com/apache/drill/blob/master/exec/java-exec/src/main/codegen/templates/NullOperator.java
>>
>> --
>> Jacques Nadeau
>> CTO and Co-Founder, Dremio
>>
>> On Mon, Jan 4, 2016 at 8:04 PM, Nirav Shah <ni...@games24x7.com>
>> wrote:
>>
>> > Hi,
>> > May be this is silly question, Please let me know how to return null
>> string
>> > from drill custom function.
>> >
>> >
>> > Regards,
>> > Nirav
>> >
>>
>
>
Re: Null Return
Posted by Nirav Shah <ni...@games24x7.com>.
Thanks Jacques. Appreciate your response.
On Tue, Jan 5, 2016 at 10:59 PM, Jacques Nadeau <ja...@dremio.com> wrote:
> The default behavior of Drill UDFs is NullHandling.NULL_IF_NULL semantics.
> This means that a the Drill engine manages checking nullability on inputs.
> If one or more inputs is null, then the output is null. You can also
> implement NullHandling.INTERNAL as your null semantics in a UDF. This
> allows you to set nullability on your output independent of the nullability
> of the inputs. An example is the isNull functions [1]. Note that the
> complexity with using internal handling if you have to create
> implementations for all variations of null inputs and outputs. This is the
> cartesian product of variations when you have multiple parameters. As such,
> we generally use freemarker templates (per the example below) to create the
> variations.
>
> [1]
>
> https://github.com/apache/drill/blob/master/exec/java-exec/src/main/codegen/templates/NullOperator.java
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Mon, Jan 4, 2016 at 8:04 PM, Nirav Shah <ni...@games24x7.com>
> wrote:
>
> > Hi,
> > May be this is silly question, Please let me know how to return null
> string
> > from drill custom function.
> >
> >
> > Regards,
> > Nirav
> >
>
Re: Null Return
Posted by Jacques Nadeau <ja...@dremio.com>.
The default behavior of Drill UDFs is NullHandling.NULL_IF_NULL semantics.
This means that a the Drill engine manages checking nullability on inputs.
If one or more inputs is null, then the output is null. You can also
implement NullHandling.INTERNAL as your null semantics in a UDF. This
allows you to set nullability on your output independent of the nullability
of the inputs. An example is the isNull functions [1]. Note that the
complexity with using internal handling if you have to create
implementations for all variations of null inputs and outputs. This is the
cartesian product of variations when you have multiple parameters. As such,
we generally use freemarker templates (per the example below) to create the
variations.
[1]
https://github.com/apache/drill/blob/master/exec/java-exec/src/main/codegen/templates/NullOperator.java
--
Jacques Nadeau
CTO and Co-Founder, Dremio
On Mon, Jan 4, 2016 at 8:04 PM, Nirav Shah <ni...@games24x7.com> wrote:
> Hi,
> May be this is silly question, Please let me know how to return null string
> from drill custom function.
>
>
> Regards,
> Nirav
>