You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@drill.apache.org by Julian Feinauer <ju...@web.de> on 2016/05/19 10:31:55 UTC

UDF is not recognized by drill / Validation Error

Dear folks,

I’m currently experimenting with user defined functions in drill but I’m not able to get them to work on my drillbits.
I always get the error: Error: VALIDATION ERROR: From line 1, column 8 to line 1, column 41: No match found for function signature myaddints(<ANY>, <ANY>).

I already went through all the tips I found in the mailing list.
The jar contains a drill-module.conf with the content:
drill.classpath.scanning.packages += "org.julian"
And the UDF is defined as:
package org.julian;

import ...

@FunctionTemplate(name = „myaddints", scope = FunctionTemplate.FunctionScope.SIMPLE, nulls = FunctionTemplate.NullHandling.NULL_IF_NULL)
public class IntIntAdd implements DrillSimpleFunc {

    @Param
    IntHolder in1;

    @Param
    IntHolder in2;

    @Output
    IntHolder out;

    public void setup() {};

    public void eval() {
        out.value = in1.value + in2.value;
    }

}
I tried to find the bug using the logs and encountered the following messages from the drillbit.log:
Base Configuration:
	- jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-common-1.6.0.jar!/drill-default.conf

Intermediate Configuration and Plugin files, in order of precedence:
	- jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-gis-1.6.0.jar!/drill-module.conf
	- jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-logical-1.6.0.jar!/drill-module.conf
	- jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-kudu-storage-1.6.0.jar!/drill-module.conf
	- jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/3rdparty/drill-memory-base-1.6.0.jar!/drill-module.conf
	- jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-hive-exec-shaded-1.6.0.jar!/drill-module.conf
	- jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-memory-base-1.6.0.jar!/drill-module.conf
	- jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-mongo-storage-1.6.0.jar!/drill-module.conf
	- jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-java-exec-1.6.0.jar!/drill-module.conf
	- jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-common-1.6.0.jar!/drill-module.conf
	- jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-storage-hbase-1.6.0.jar!/drill-module.conf
	- jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-jdbc-storage-1.6.0.jar!/drill-module.conf
	- jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-storage-hive-core-1.6.0.jar!/drill-module.conf

which let’s me think that my jar file (drillUDF-1.0-SNAPSHOT.jar) is not scanned altough it is located in jars/3rdparty.

Am I doing something wrong or am I missing something or has anyone a good hint for me?

Thank you already!
Julian

Re: UDF is not recognized by drill / Validation Error

Posted by rahul challapalli <ch...@gmail.com>.

I could be repeating a few steps from the documentation, but let me know if
you already tried the below

  1. Did you add both the sources jar file and the classes jar file?
  2. And did you add them on every node?
  3. Assuming you did a restart after the above steps, can you check if
there is something in the *drillbit.out* file?

- Rahul

On Thu, May 19, 2016 at 3:31 AM, Julian Feinauer <ju...@web.de>
wrote:

> Dear folks,
>
> I’m currently experimenting with user defined functions in drill but I’m
> not able to get them to work on my drillbits.
> I always get the error: Error: VALIDATION ERROR: From line 1, column 8 to
> line 1, column 41: No match found for function signature myaddints(<ANY>,
> <ANY>).
>
> I already went through all the tips I found in the mailing list.
> The jar contains a drill-module.conf with the content:
> drill.classpath.scanning.packages += "org.julian"
> And the UDF is defined as:
> package org.julian;
>
> import ...
>
> @FunctionTemplate(name = „myaddints", scope =
> FunctionTemplate.FunctionScope.SIMPLE, nulls =
> FunctionTemplate.NullHandling.NULL_IF_NULL)
> public class IntIntAdd implements DrillSimpleFunc {
>
>     @Param
>     IntHolder in1;
>
>     @Param
>     IntHolder in2;
>
>     @Output
>     IntHolder out;
>
>     public void setup() {};
>
>     public void eval() {
>         out.value = in1.value + in2.value;
>     }
>
> }
> I tried to find the bug using the logs and encountered the following
> messages from the drillbit.log:
> Base Configuration:
>         -
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-common-1.6.0.jar!/drill-default.conf
>
> Intermediate Configuration and Plugin files, in order of precedence:
>         -
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-gis-1.6.0.jar!/drill-module.conf
>         -
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-logical-1.6.0.jar!/drill-module.conf
>         -
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-kudu-storage-1.6.0.jar!/drill-module.conf
>         -
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/3rdparty/drill-memory-base-1.6.0.jar!/drill-module.conf
>         -
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-hive-exec-shaded-1.6.0.jar!/drill-module.conf
>         -
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-memory-base-1.6.0.jar!/drill-module.conf
>         -
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-mongo-storage-1.6.0.jar!/drill-module.conf
>         -
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-java-exec-1.6.0.jar!/drill-module.conf
>         -
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-common-1.6.0.jar!/drill-module.conf
>         -
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-storage-hbase-1.6.0.jar!/drill-module.conf
>         -
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-jdbc-storage-1.6.0.jar!/drill-module.conf
>         -
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-storage-hive-core-1.6.0.jar!/drill-module.conf
>
> which let’s me think that my jar file (drillUDF-1.0-SNAPSHOT.jar) is not
> scanned altough it is located in jars/3rdparty.
>
> Am I doing something wrong or am I missing something or has anyone a good
> hint for me?
>
> Thank you already!
> Julian

Re: UDF is not recognized by drill / Validation Error

Posted by Abdel Hakim Deneche <ad...@maprtech.com>.

Julian,

can you share the both jars (classes and sources) with us. I will give it a
try and see if I can figure out what's going on.

Thanks

On Fri, May 20, 2016 at 6:08 AM, Julian Feinauer <ju...@web.de>
wrote:

> Dear all,
>
> thank you very much for all your replies.
> I tried everything but it is still not working.
>
> 1. I copy both files (classes and sources) in the /jars/3rdparty directory
> 2. I restarted the drillbit after this (I use only one drillbit and
> drill-conf both running on my local machine)
> 3. I changed the class to a static subclass
> 4. I have the drill-module.conf in my ressources
> 4. The error appears in the drillbit.log as soon as i call the udf because
> it is not recognized by drill.
> On startup drillbit.log states all the packages and jars that are scanned
> and my custom jar is not listed there.
> Therefore I think it is a problem with the class loader or something
> related?
>
> Could this be possible?
>
> Greetings
> Julian
>
> > Am 20.05.2016 um 14:36 schrieb Tugdual Grall <tu...@gmail.com>:
> >
> > Hi
> >
> > Be sure you deploy on each nodes, 2 jars:
> > - the jar containing the classes
> > - the jar contains the sources
> >
> > The POM.xml in the simple function examples contains the maven
> > configuration to generate these 2 files, be sure you have the same in
> your
> > project:
> >
> https://github.com/tgrall/drill-simple-mask-function/blob/master/pom.xml#L24-L37
> >
> > and you have restarted the drillbit
> >
> > Regards
> > Tug
> > @tgrall
> >
> >
> >
> > On Fri, May 20, 2016 at 12:34 AM, Abdel Hakim Deneche <
> adeneche@maprtech.com
> >> wrote:
> >
> >> the example I gave you was incomplete, here is what I meant to send:
> >>
> >> public class MyUDF {
> >>
> >>   @FunctionTemplate(name = „myaddints", scope = FunctionTemplate.
> >>   FunctionScope.SIMPLE, nulls =
> >> FunctionTemplate.NullHandling.NULL_IF_NULL)
> >>   public *static *class IntIntAdd implements DrillSimpleFunc {
> >>      ...
> >>   }
> >>
> >> }
> >>
> >>
> >> On Thu, May 19, 2016 at 3:33 PM, Abdel Hakim Deneche <
> >> adeneche@maprtech.com>
> >> wrote:
> >>
> >>> Hey Julian,
> >>>
> >>> one more thing you could try out: declare the UDF as a static class
> >> inside
> >>> another class:
> >>>
> >>> public class MyUDF {
> >>>
> >>>   @FunctionTemplate(name = „myaddints", scope = FunctionTemplate.
> >>>   FunctionScope.SIMPLE, nulls = FunctionTemplate.NullHandling.
> >>> NULL_IF_NULL)
> >>>   public class IntIntAdd implements DrillSimpleFunc {
> >>>      ...
> >>>   }
> >>>
> >>> }
> >>>
> >>> Take a look at the following page to see an examples of UDFs:
> >>> http://drill.apache.org/docs/custom-function-interfaces/
> >>>
> >>> If this doesn't work check the drillbit log, it should print an error
> >>> message when it's starting up if something's wrong with your UDF.
> >>>
> >>> Thanks
> >>>
> >>>
> >>> On Thu, May 19, 2016 at 3:31 AM, Julian Feinauer <
> julian.feinauer@web.de
> >>>
> >>> wrote:
> >>>
> >>>> Dear folks,
> >>>>
> >>>> I’m currently experimenting with user defined functions in drill but
> I’m
> >>>> not able to get them to work on my drillbits.
> >>>> I always get the error: Error: VALIDATION ERROR: From line 1, column 8
> >> to
> >>>> line 1, column 41: No match found for function signature
> >> myaddints(<ANY>,
> >>>> <ANY>).
> >>>>
> >>>> I already went through all the tips I found in the mailing list.
> >>>> The jar contains a drill-module.conf with the content:
> >>>> drill.classpath.scanning.packages += "org.julian"
> >>>> And the UDF is defined as:
> >>>> package org.julian;
> >>>>
> >>>> import ...
> >>>>
> >>>> @FunctionTemplate(name = „myaddints", scope =
> >>>> FunctionTemplate.FunctionScope.SIMPLE, nulls =
> >>>> FunctionTemplate.NullHandling.NULL_IF_NULL)
> >>>> public class IntIntAdd implements DrillSimpleFunc {
> >>>>
> >>>>    @Param
> >>>>    IntHolder in1;
> >>>>
> >>>>    @Param
> >>>>    IntHolder in2;
> >>>>
> >>>>    @Output
> >>>>    IntHolder out;
> >>>>
> >>>>    public void setup() {};
> >>>>
> >>>>    public void eval() {
> >>>>        out.value = in1.value + in2.value;
> >>>>    }
> >>>>
> >>>> }
> >>>> I tried to find the bug using the logs and encountered the following
> >>>> messages from the drillbit.log:
> >>>> Base Configuration:
> >>>>        -
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-common-1.6.0.jar!/drill-default.conf
> >>>>
> >>>> Intermediate Configuration and Plugin files, in order of precedence:
> >>>>        -
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-gis-1.6.0.jar!/drill-module.conf
> >>>>        -
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-logical-1.6.0.jar!/drill-module.conf
> >>>>        -
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-kudu-storage-1.6.0.jar!/drill-module.conf
> >>>>        -
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/3rdparty/drill-memory-base-1.6.0.jar!/drill-module.conf
> >>>>        -
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-hive-exec-shaded-1.6.0.jar!/drill-module.conf
> >>>>        -
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-memory-base-1.6.0.jar!/drill-module.conf
> >>>>        -
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-mongo-storage-1.6.0.jar!/drill-module.conf
> >>>>        -
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-java-exec-1.6.0.jar!/drill-module.conf
> >>>>        -
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-common-1.6.0.jar!/drill-module.conf
> >>>>        -
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-storage-hbase-1.6.0.jar!/drill-module.conf
> >>>>        -
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-jdbc-storage-1.6.0.jar!/drill-module.conf
> >>>>        -
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-storage-hive-core-1.6.0.jar!/drill-module.conf
> >>>>
> >>>> which let’s me think that my jar file (drillUDF-1.0-SNAPSHOT.jar) is
> not
> >>>> scanned altough it is located in jars/3rdparty.
> >>>>
> >>>> Am I doing something wrong or am I missing something or has anyone a
> >> good
> >>>> hint for me?
> >>>>
> >>>> Thank you already!
> >>>> Julian
> >>>
> >>>
> >>>
> >>>
> >>> --
> >>>
> >>> Abdelhakim Deneche
> >>>
> >>> Software Engineer
> >>>
> >>>  <http://www.mapr.com/>
> >>>
> >>>
> >>> Now Available - Free Hadoop On-Demand Training
> >>> <
> >>
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >>>
> >>>
> >>
> >>
> >>
> >> --
> >>
> >> Abdelhakim Deneche
> >>
> >> Software Engineer
> >>
> >>  <http://www.mapr.com/>
> >>
> >>
> >> Now Available - Free Hadoop On-Demand Training
> >> <
> >>
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >>>
> >>
>
>


-- 

Abdelhakim Deneche

Software Engineer

  <http://www.mapr.com/>


Now Available - Free Hadoop On-Demand Training
<http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available>

Re: UDF is not recognized by drill / Validation Error

Posted by Julian Feinauer <ju...@web.de>.

Hey Guys,

I finally found the solution of my weird problem.
I’m using OS X and besides the drill installation according to the documentation I also had drill-embedded installed via homebrew (package apache-drill).
The homebrew package seemed to interact in a weird way with my „regular“ installation because all the problems (correct log level from logback appeared in the drillbit.log) as well as the UDF problem suddenly disappeared.

Nonetheless, thank you all for your kind help and your efforts!
Julian

> Am 20.05.2016 um 20:38 schrieb Julian Feinauer <ju...@web.de>:
> 
> Hey,
> 
> I’m using Drill 1.6.0.
> 
>> Am 20.05.2016 um 20:37 schrieb Abdel Hakim Deneche <ad...@maprtech.com>:
>> 
>> What version of Drill are you using ?
>> 
>> On Fri, May 20, 2016 at 11:28 AM, Julian Feinauer <ju...@web.de>
>> wrote:
>> 
>>> Hey Abdel,
>>> 
>>> I renamed it due to your previous hint with the static subclass.
>>> But the SQL Commands still returns:
>>> 0: jdbc:drill:> SELECT myintadd(CAST(position_id AS int),CAST(store_id AS
>>> int)) FROM cp.`employee.json`;
>>> Error: VALIDATION ERROR: From line 1, column 8 to line 1, column 63: No
>>> match found for function signature myintadd(<NUMERIC>, <NUMERIC>)
>>> 
>>> So this has to be something wrong with my drill setup?
>>> 
>>> All I do is to start my drillbit with drillbit.sh start (or restart) and
>>> then start drill-conf where the conf/drill-override.conf contains:
>>> drill.exec: {
>>> cluster-id: "drillbits1",
>>> zk.connect: "localhost:2181"
>>> }
>>> 
>>> There’s one other thing that seems strange to me with my setup:
>>> Although I set the loglevel in conf/logback.xml to trace the drillbit.log
>>> file does not show any trace outputs.
>>> 
>>> Am I doing something wrong with my setup?
>>> 
>>> Thanks
>>> Julian
>>> 
>>>> Am 20.05.2016 um 20:13 schrieb Abdel Hakim Deneche <
>>> adeneche@maprtech.com>:
>>>> 
>>>> You defined your function as myintadd:
>>>> 
>>>> 
>>>> @FunctionTemplate(name = "*myintadd*", scope ...
>>>> 
>>>> 
>>>> The following query worked fine for me:
>>>> 
>>>> SELECT *myintadd*(CAST(position_id AS int),CAST(store_id AS int)) FROM
>>>>> cp.`employee.json`;
>>>> 
>>>> 
>>>> Thanks
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On Fri, May 20, 2016 at 10:17 AM, Julian Feinauer <
>>> julian.feinauer@web.de>
>>>> wrote:
>>>> 
>>>>> Dear Jacques,
>>>>> Dear Abdel,
>>>>> 
>>>>> thanks for your kind help.
>>>>> 
>>>>> jar tf drillUDF-1.0-SNAPSHOT.jar returns:
>>>>> 
>>>>> META-INF/
>>>>> META-INF/MANIFEST.MF
>>>>> org/
>>>>> org/apache/
>>>>> org/apache/drill/
>>>>> org/apache/drill/contrib/
>>>>> org/apache/drill/contrib/function/
>>>>> org/julian/
>>>>> drill-module.conf
>>>>> IntIntAdd.class
>>>>> org/apache/drill/contrib/function/IntIntAdd.class
>>>>> org/julian/IntIntAdd$myintadd.class
>>>>> org/julian/IntIntAdd.class
>>>>> org/julian/IntIntAdd.java
>>>>> META-INF/maven/
>>>>> META-INF/maven/julian.test/
>>>>> META-INF/maven/julian.test/drillUDF/
>>>>> META-INF/maven/julian.test/drillUDF/pom.xml
>>>>> META-INF/maven/julian.test/drillUDF/pom.properties
>>>>> 
>>>>> And jar tf drillUDF-1.0-SNAPSHOT-sources.jar returns
>>>>> META-INF/
>>>>> META-INF/MANIFEST.MF
>>>>> org/
>>>>> org/apache/
>>>>> org/apache/drill/
>>>>> org/apache/drill/contrib/
>>>>> org/apache/drill/contrib/function/
>>>>> org/julian/
>>>>> org/julian/IntIntAdd.java
>>>>> drill-module.conf
>>>>> 
>>>>> so the drill-module.conf is in both roots.
>>>>> 
>>>>> @Abdel:
>>>>> How or where can I share the jar files with you?
>>>>> 
>>>>> Best regards
>>>>> Julian
>>>>> 
>>>>>> Am 20.05.2016 um 17:31 schrieb Jacques Nadeau <ja...@dremio.com>:
>>>>>> 
>>>>>> Can you run  jar tf myudf.jar against your jar files? Since Drill is
>>> not
>>>>>> detecting the jar file, we need to resolve that first. The
>>>>>> drill-module.conf must be in the root of each jar file that should be
>>>>>> included. Lets start by verifying that.
>>>>>> 
>>>>>> --
>>>>>> Jacques Nadeau
>>>>>> CTO and Co-Founder, Dremio
>>>>>> 
>>>>>> On Fri, May 20, 2016 at 6:08 AM, Julian Feinauer <
>>> julian.feinauer@web.de
>>>>>> 
>>>>>> wrote:
>>>>>> 
>>>>>>> Dear all,
>>>>>>> 
>>>>>>> thank you very much for all your replies.
>>>>>>> I tried everything but it is still not working.
>>>>>>> 
>>>>>>> 1. I copy both files (classes and sources) in the /jars/3rdparty
>>>>> directory
>>>>>>> 2. I restarted the drillbit after this (I use only one drillbit and
>>>>>>> drill-conf both running on my local machine)
>>>>>>> 3. I changed the class to a static subclass
>>>>>>> 4. I have the drill-module.conf in my ressources
>>>>>>> 4. The error appears in the drillbit.log as soon as i call the udf
>>>>> because
>>>>>>> it is not recognized by drill.
>>>>>>> On startup drillbit.log states all the packages and jars that are
>>>>> scanned
>>>>>>> and my custom jar is not listed there.
>>>>>>> Therefore I think it is a problem with the class loader or something
>>>>>>> related?
>>>>>>> 
>>>>>>> Could this be possible?
>>>>>>> 
>>>>>>> Greetings
>>>>>>> Julian
>>>>>>> 
>>>>>>>> Am 20.05.2016 um 14:36 schrieb Tugdual Grall <tu...@gmail.com>:
>>>>>>>> 
>>>>>>>> Hi
>>>>>>>> 
>>>>>>>> Be sure you deploy on each nodes, 2 jars:
>>>>>>>> - the jar containing the classes
>>>>>>>> - the jar contains the sources
>>>>>>>> 
>>>>>>>> The POM.xml in the simple function examples contains the maven
>>>>>>>> configuration to generate these 2 files, be sure you have the same in
>>>>>>> your
>>>>>>>> project:
>>>>>>>> 
>>>>>>> 
>>>>> 
>>> https://github.com/tgrall/drill-simple-mask-function/blob/master/pom.xml#L24-L37
>>>>>>>> 
>>>>>>>> and you have restarted the drillbit
>>>>>>>> 
>>>>>>>> Regards
>>>>>>>> Tug
>>>>>>>> @tgrall
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Fri, May 20, 2016 at 12:34 AM, Abdel Hakim Deneche <
>>>>>>> adeneche@maprtech.com
>>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> the example I gave you was incomplete, here is what I meant to send:
>>>>>>>>> 
>>>>>>>>> public class MyUDF {
>>>>>>>>> 
>>>>>>>>> @FunctionTemplate(name = „myaddints", scope = FunctionTemplate.
>>>>>>>>> FunctionScope.SIMPLE, nulls =
>>>>>>>>> FunctionTemplate.NullHandling.NULL_IF_NULL)
>>>>>>>>> public *static *class IntIntAdd implements DrillSimpleFunc {
>>>>>>>>>  ...
>>>>>>>>> }
>>>>>>>>> 
>>>>>>>>> }
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Thu, May 19, 2016 at 3:33 PM, Abdel Hakim Deneche <
>>>>>>>>> adeneche@maprtech.com>
>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Hey Julian,
>>>>>>>>>> 
>>>>>>>>>> one more thing you could try out: declare the UDF as a static class
>>>>>>>>> inside
>>>>>>>>>> another class:
>>>>>>>>>> 
>>>>>>>>>> public class MyUDF {
>>>>>>>>>> 
>>>>>>>>>> @FunctionTemplate(name = „myaddints", scope = FunctionTemplate.
>>>>>>>>>> FunctionScope.SIMPLE, nulls = FunctionTemplate.NullHandling.
>>>>>>>>>> NULL_IF_NULL)
>>>>>>>>>> public class IntIntAdd implements DrillSimpleFunc {
>>>>>>>>>>  ...
>>>>>>>>>> }
>>>>>>>>>> 
>>>>>>>>>> }
>>>>>>>>>> 
>>>>>>>>>> Take a look at the following page to see an examples of UDFs:
>>>>>>>>>> http://drill.apache.org/docs/custom-function-interfaces/
>>>>>>>>>> 
>>>>>>>>>> If this doesn't work check the drillbit log, it should print an
>>> error
>>>>>>>>>> message when it's starting up if something's wrong with your UDF.
>>>>>>>>>> 
>>>>>>>>>> Thanks
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On Thu, May 19, 2016 at 3:31 AM, Julian Feinauer <
>>>>>>> julian.feinauer@web.de
>>>>>>>>>> 
>>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Dear folks,
>>>>>>>>>>> 
>>>>>>>>>>> I’m currently experimenting with user defined functions in drill
>>> but
>>>>>>> I’m
>>>>>>>>>>> not able to get them to work on my drillbits.
>>>>>>>>>>> I always get the error: Error: VALIDATION ERROR: From line 1,
>>>>> column 8
>>>>>>>>> to
>>>>>>>>>>> line 1, column 41: No match found for function signature
>>>>>>>>> myaddints(<ANY>,
>>>>>>>>>>> <ANY>).
>>>>>>>>>>> 
>>>>>>>>>>> I already went through all the tips I found in the mailing list.
>>>>>>>>>>> The jar contains a drill-module.conf with the content:
>>>>>>>>>>> drill.classpath.scanning.packages += "org.julian"
>>>>>>>>>>> And the UDF is defined as:
>>>>>>>>>>> package org.julian;
>>>>>>>>>>> 
>>>>>>>>>>> import ...
>>>>>>>>>>> 
>>>>>>>>>>> @FunctionTemplate(name = „myaddints", scope =
>>>>>>>>>>> FunctionTemplate.FunctionScope.SIMPLE, nulls =
>>>>>>>>>>> FunctionTemplate.NullHandling.NULL_IF_NULL)
>>>>>>>>>>> public class IntIntAdd implements DrillSimpleFunc {
>>>>>>>>>>> 
>>>>>>>>>>> @Param
>>>>>>>>>>> IntHolder in1;
>>>>>>>>>>> 
>>>>>>>>>>> @Param
>>>>>>>>>>> IntHolder in2;
>>>>>>>>>>> 
>>>>>>>>>>> @Output
>>>>>>>>>>> IntHolder out;
>>>>>>>>>>> 
>>>>>>>>>>> public void setup() {};
>>>>>>>>>>> 
>>>>>>>>>>> public void eval() {
>>>>>>>>>>>    out.value = in1.value + in2.value;
>>>>>>>>>>> }
>>>>>>>>>>> 
>>>>>>>>>>> }
>>>>>>>>>>> I tried to find the bug using the logs and encountered the
>>> following
>>>>>>>>>>> messages from the drillbit.log:
>>>>>>>>>>> Base Configuration:
>>>>>>>>>>>    -
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>> 
>>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-common-1.6.0.jar!/drill-default.conf
>>>>>>>>>>> 
>>>>>>>>>>> Intermediate Configuration and Plugin files, in order of
>>> precedence:
>>>>>>>>>>>    -
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>> 
>>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-gis-1.6.0.jar!/drill-module.conf
>>>>>>>>>>>    -
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>> 
>>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-logical-1.6.0.jar!/drill-module.conf
>>>>>>>>>>>    -
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>> 
>>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-kudu-storage-1.6.0.jar!/drill-module.conf
>>>>>>>>>>>    -
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>> 
>>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/3rdparty/drill-memory-base-1.6.0.jar!/drill-module.conf
>>>>>>>>>>>    -
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>> 
>>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-hive-exec-shaded-1.6.0.jar!/drill-module.conf
>>>>>>>>>>>    -
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>> 
>>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-memory-base-1.6.0.jar!/drill-module.conf
>>>>>>>>>>>    -
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>> 
>>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-mongo-storage-1.6.0.jar!/drill-module.conf
>>>>>>>>>>>    -
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>> 
>>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-java-exec-1.6.0.jar!/drill-module.conf
>>>>>>>>>>>    -
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>> 
>>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-common-1.6.0.jar!/drill-module.conf
>>>>>>>>>>>    -
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>> 
>>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-storage-hbase-1.6.0.jar!/drill-module.conf
>>>>>>>>>>>    -
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>> 
>>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-jdbc-storage-1.6.0.jar!/drill-module.conf
>>>>>>>>>>>    -
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>> 
>>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-storage-hive-core-1.6.0.jar!/drill-module.conf
>>>>>>>>>>> 
>>>>>>>>>>> which let’s me think that my jar file (drillUDF-1.0-SNAPSHOT.jar)
>>> is
>>>>>>> not
>>>>>>>>>>> scanned altough it is located in jars/3rdparty.
>>>>>>>>>>> 
>>>>>>>>>>> Am I doing something wrong or am I missing something or has
>>> anyone a
>>>>>>>>> good
>>>>>>>>>>> hint for me?
>>>>>>>>>>> 
>>>>>>>>>>> Thank you already!
>>>>>>>>>>> Julian
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> --
>>>>>>>>>> 
>>>>>>>>>> Abdelhakim Deneche
>>>>>>>>>> 
>>>>>>>>>> Software Engineer
>>>>>>>>>> 
>>>>>>>>>> <http://www.mapr.com/>
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Now Available - Free Hadoop On-Demand Training
>>>>>>>>>> <
>>>>>>>>> 
>>>>>>> 
>>>>> 
>>> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> 
>>>>>>>>> Abdelhakim Deneche
>>>>>>>>> 
>>>>>>>>> Software Engineer
>>>>>>>>> 
>>>>>>>>> <http://www.mapr.com/>
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Now Available - Free Hadoop On-Demand Training
>>>>>>>>> <
>>>>>>>>> 
>>>>>>> 
>>>>> 
>>> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> --
>>>> 
>>>> Abdelhakim Deneche
>>>> 
>>>> Software Engineer
>>>> 
>>>> <http://www.mapr.com/>
>>>> 
>>>> 
>>>> Now Available - Free Hadoop On-Demand Training
>>>> <
>>> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
>>>> 
>>> 
>>> 
>> 
>> 
>> -- 
>> 
>> Abdelhakim Deneche
>> 
>> Software Engineer
>> 
>> <http://www.mapr.com/>
>> 
>> 
>> Now Available - Free Hadoop On-Demand Training
>> <http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available>
>

Re: UDF is not recognized by drill / Validation Error

Posted by Julian Feinauer <ju...@web.de>.

Hey,

I’m using Drill 1.6.0.

> Am 20.05.2016 um 20:37 schrieb Abdel Hakim Deneche <ad...@maprtech.com>:
> 
> What version of Drill are you using ?
> 
> On Fri, May 20, 2016 at 11:28 AM, Julian Feinauer <ju...@web.de>
> wrote:
> 
>> Hey Abdel,
>> 
>> I renamed it due to your previous hint with the static subclass.
>> But the SQL Commands still returns:
>> 0: jdbc:drill:> SELECT myintadd(CAST(position_id AS int),CAST(store_id AS
>> int)) FROM cp.`employee.json`;
>> Error: VALIDATION ERROR: From line 1, column 8 to line 1, column 63: No
>> match found for function signature myintadd(<NUMERIC>, <NUMERIC>)
>> 
>> So this has to be something wrong with my drill setup?
>> 
>> All I do is to start my drillbit with drillbit.sh start (or restart) and
>> then start drill-conf where the conf/drill-override.conf contains:
>> drill.exec: {
>>  cluster-id: "drillbits1",
>>  zk.connect: "localhost:2181"
>> }
>> 
>> There’s one other thing that seems strange to me with my setup:
>> Although I set the loglevel in conf/logback.xml to trace the drillbit.log
>> file does not show any trace outputs.
>> 
>> Am I doing something wrong with my setup?
>> 
>> Thanks
>> Julian
>> 
>>> Am 20.05.2016 um 20:13 schrieb Abdel Hakim Deneche <
>> adeneche@maprtech.com>:
>>> 
>>> You defined your function as myintadd:
>>> 
>>> 
>>> @FunctionTemplate(name = "*myintadd*", scope ...
>>> 
>>> 
>>> The following query worked fine for me:
>>> 
>>> SELECT *myintadd*(CAST(position_id AS int),CAST(store_id AS int)) FROM
>>>> cp.`employee.json`;
>>> 
>>> 
>>> Thanks
>>> 
>>> 
>>> 
>>> 
>>> 
>>> On Fri, May 20, 2016 at 10:17 AM, Julian Feinauer <
>> julian.feinauer@web.de>
>>> wrote:
>>> 
>>>> Dear Jacques,
>>>> Dear Abdel,
>>>> 
>>>> thanks for your kind help.
>>>> 
>>>> jar tf drillUDF-1.0-SNAPSHOT.jar returns:
>>>> 
>>>> META-INF/
>>>> META-INF/MANIFEST.MF
>>>> org/
>>>> org/apache/
>>>> org/apache/drill/
>>>> org/apache/drill/contrib/
>>>> org/apache/drill/contrib/function/
>>>> org/julian/
>>>> drill-module.conf
>>>> IntIntAdd.class
>>>> org/apache/drill/contrib/function/IntIntAdd.class
>>>> org/julian/IntIntAdd$myintadd.class
>>>> org/julian/IntIntAdd.class
>>>> org/julian/IntIntAdd.java
>>>> META-INF/maven/
>>>> META-INF/maven/julian.test/
>>>> META-INF/maven/julian.test/drillUDF/
>>>> META-INF/maven/julian.test/drillUDF/pom.xml
>>>> META-INF/maven/julian.test/drillUDF/pom.properties
>>>> 
>>>> And jar tf drillUDF-1.0-SNAPSHOT-sources.jar returns
>>>> META-INF/
>>>> META-INF/MANIFEST.MF
>>>> org/
>>>> org/apache/
>>>> org/apache/drill/
>>>> org/apache/drill/contrib/
>>>> org/apache/drill/contrib/function/
>>>> org/julian/
>>>> org/julian/IntIntAdd.java
>>>> drill-module.conf
>>>> 
>>>> so the drill-module.conf is in both roots.
>>>> 
>>>> @Abdel:
>>>> How or where can I share the jar files with you?
>>>> 
>>>> Best regards
>>>> Julian
>>>> 
>>>>> Am 20.05.2016 um 17:31 schrieb Jacques Nadeau <ja...@dremio.com>:
>>>>> 
>>>>> Can you run  jar tf myudf.jar against your jar files? Since Drill is
>> not
>>>>> detecting the jar file, we need to resolve that first. The
>>>>> drill-module.conf must be in the root of each jar file that should be
>>>>> included. Lets start by verifying that.
>>>>> 
>>>>> --
>>>>> Jacques Nadeau
>>>>> CTO and Co-Founder, Dremio
>>>>> 
>>>>> On Fri, May 20, 2016 at 6:08 AM, Julian Feinauer <
>> julian.feinauer@web.de
>>>>> 
>>>>> wrote:
>>>>> 
>>>>>> Dear all,
>>>>>> 
>>>>>> thank you very much for all your replies.
>>>>>> I tried everything but it is still not working.
>>>>>> 
>>>>>> 1. I copy both files (classes and sources) in the /jars/3rdparty
>>>> directory
>>>>>> 2. I restarted the drillbit after this (I use only one drillbit and
>>>>>> drill-conf both running on my local machine)
>>>>>> 3. I changed the class to a static subclass
>>>>>> 4. I have the drill-module.conf in my ressources
>>>>>> 4. The error appears in the drillbit.log as soon as i call the udf
>>>> because
>>>>>> it is not recognized by drill.
>>>>>> On startup drillbit.log states all the packages and jars that are
>>>> scanned
>>>>>> and my custom jar is not listed there.
>>>>>> Therefore I think it is a problem with the class loader or something
>>>>>> related?
>>>>>> 
>>>>>> Could this be possible?
>>>>>> 
>>>>>> Greetings
>>>>>> Julian
>>>>>> 
>>>>>>> Am 20.05.2016 um 14:36 schrieb Tugdual Grall <tu...@gmail.com>:
>>>>>>> 
>>>>>>> Hi
>>>>>>> 
>>>>>>> Be sure you deploy on each nodes, 2 jars:
>>>>>>> - the jar containing the classes
>>>>>>> - the jar contains the sources
>>>>>>> 
>>>>>>> The POM.xml in the simple function examples contains the maven
>>>>>>> configuration to generate these 2 files, be sure you have the same in
>>>>>> your
>>>>>>> project:
>>>>>>> 
>>>>>> 
>>>> 
>> https://github.com/tgrall/drill-simple-mask-function/blob/master/pom.xml#L24-L37
>>>>>>> 
>>>>>>> and you have restarted the drillbit
>>>>>>> 
>>>>>>> Regards
>>>>>>> Tug
>>>>>>> @tgrall
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Fri, May 20, 2016 at 12:34 AM, Abdel Hakim Deneche <
>>>>>> adeneche@maprtech.com
>>>>>>>> wrote:
>>>>>>> 
>>>>>>>> the example I gave you was incomplete, here is what I meant to send:
>>>>>>>> 
>>>>>>>> public class MyUDF {
>>>>>>>> 
>>>>>>>> @FunctionTemplate(name = „myaddints", scope = FunctionTemplate.
>>>>>>>> FunctionScope.SIMPLE, nulls =
>>>>>>>> FunctionTemplate.NullHandling.NULL_IF_NULL)
>>>>>>>> public *static *class IntIntAdd implements DrillSimpleFunc {
>>>>>>>>   ...
>>>>>>>> }
>>>>>>>> 
>>>>>>>> }
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Thu, May 19, 2016 at 3:33 PM, Abdel Hakim Deneche <
>>>>>>>> adeneche@maprtech.com>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Hey Julian,
>>>>>>>>> 
>>>>>>>>> one more thing you could try out: declare the UDF as a static class
>>>>>>>> inside
>>>>>>>>> another class:
>>>>>>>>> 
>>>>>>>>> public class MyUDF {
>>>>>>>>> 
>>>>>>>>> @FunctionTemplate(name = „myaddints", scope = FunctionTemplate.
>>>>>>>>> FunctionScope.SIMPLE, nulls = FunctionTemplate.NullHandling.
>>>>>>>>> NULL_IF_NULL)
>>>>>>>>> public class IntIntAdd implements DrillSimpleFunc {
>>>>>>>>>   ...
>>>>>>>>> }
>>>>>>>>> 
>>>>>>>>> }
>>>>>>>>> 
>>>>>>>>> Take a look at the following page to see an examples of UDFs:
>>>>>>>>> http://drill.apache.org/docs/custom-function-interfaces/
>>>>>>>>> 
>>>>>>>>> If this doesn't work check the drillbit log, it should print an
>> error
>>>>>>>>> message when it's starting up if something's wrong with your UDF.
>>>>>>>>> 
>>>>>>>>> Thanks
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Thu, May 19, 2016 at 3:31 AM, Julian Feinauer <
>>>>>> julian.feinauer@web.de
>>>>>>>>> 
>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Dear folks,
>>>>>>>>>> 
>>>>>>>>>> I’m currently experimenting with user defined functions in drill
>> but
>>>>>> I’m
>>>>>>>>>> not able to get them to work on my drillbits.
>>>>>>>>>> I always get the error: Error: VALIDATION ERROR: From line 1,
>>>> column 8
>>>>>>>> to
>>>>>>>>>> line 1, column 41: No match found for function signature
>>>>>>>> myaddints(<ANY>,
>>>>>>>>>> <ANY>).
>>>>>>>>>> 
>>>>>>>>>> I already went through all the tips I found in the mailing list.
>>>>>>>>>> The jar contains a drill-module.conf with the content:
>>>>>>>>>> drill.classpath.scanning.packages += "org.julian"
>>>>>>>>>> And the UDF is defined as:
>>>>>>>>>> package org.julian;
>>>>>>>>>> 
>>>>>>>>>> import ...
>>>>>>>>>> 
>>>>>>>>>> @FunctionTemplate(name = „myaddints", scope =
>>>>>>>>>> FunctionTemplate.FunctionScope.SIMPLE, nulls =
>>>>>>>>>> FunctionTemplate.NullHandling.NULL_IF_NULL)
>>>>>>>>>> public class IntIntAdd implements DrillSimpleFunc {
>>>>>>>>>> 
>>>>>>>>>> @Param
>>>>>>>>>> IntHolder in1;
>>>>>>>>>> 
>>>>>>>>>> @Param
>>>>>>>>>> IntHolder in2;
>>>>>>>>>> 
>>>>>>>>>> @Output
>>>>>>>>>> IntHolder out;
>>>>>>>>>> 
>>>>>>>>>> public void setup() {};
>>>>>>>>>> 
>>>>>>>>>> public void eval() {
>>>>>>>>>>     out.value = in1.value + in2.value;
>>>>>>>>>> }
>>>>>>>>>> 
>>>>>>>>>> }
>>>>>>>>>> I tried to find the bug using the logs and encountered the
>> following
>>>>>>>>>> messages from the drillbit.log:
>>>>>>>>>> Base Configuration:
>>>>>>>>>>     -
>>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-common-1.6.0.jar!/drill-default.conf
>>>>>>>>>> 
>>>>>>>>>> Intermediate Configuration and Plugin files, in order of
>> precedence:
>>>>>>>>>>     -
>>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-gis-1.6.0.jar!/drill-module.conf
>>>>>>>>>>     -
>>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-logical-1.6.0.jar!/drill-module.conf
>>>>>>>>>>     -
>>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-kudu-storage-1.6.0.jar!/drill-module.conf
>>>>>>>>>>     -
>>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/3rdparty/drill-memory-base-1.6.0.jar!/drill-module.conf
>>>>>>>>>>     -
>>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-hive-exec-shaded-1.6.0.jar!/drill-module.conf
>>>>>>>>>>     -
>>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-memory-base-1.6.0.jar!/drill-module.conf
>>>>>>>>>>     -
>>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-mongo-storage-1.6.0.jar!/drill-module.conf
>>>>>>>>>>     -
>>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-java-exec-1.6.0.jar!/drill-module.conf
>>>>>>>>>>     -
>>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-common-1.6.0.jar!/drill-module.conf
>>>>>>>>>>     -
>>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-storage-hbase-1.6.0.jar!/drill-module.conf
>>>>>>>>>>     -
>>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-jdbc-storage-1.6.0.jar!/drill-module.conf
>>>>>>>>>>     -
>>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-storage-hive-core-1.6.0.jar!/drill-module.conf
>>>>>>>>>> 
>>>>>>>>>> which let’s me think that my jar file (drillUDF-1.0-SNAPSHOT.jar)
>> is
>>>>>> not
>>>>>>>>>> scanned altough it is located in jars/3rdparty.
>>>>>>>>>> 
>>>>>>>>>> Am I doing something wrong or am I missing something or has
>> anyone a
>>>>>>>> good
>>>>>>>>>> hint for me?
>>>>>>>>>> 
>>>>>>>>>> Thank you already!
>>>>>>>>>> Julian
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> 
>>>>>>>>> Abdelhakim Deneche
>>>>>>>>> 
>>>>>>>>> Software Engineer
>>>>>>>>> 
>>>>>>>>> <http://www.mapr.com/>
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Now Available - Free Hadoop On-Demand Training
>>>>>>>>> <
>>>>>>>> 
>>>>>> 
>>>> 
>> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> --
>>>>>>>> 
>>>>>>>> Abdelhakim Deneche
>>>>>>>> 
>>>>>>>> Software Engineer
>>>>>>>> 
>>>>>>>> <http://www.mapr.com/>
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Now Available - Free Hadoop On-Demand Training
>>>>>>>> <
>>>>>>>> 
>>>>>> 
>>>> 
>> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> --
>>> 
>>> Abdelhakim Deneche
>>> 
>>> Software Engineer
>>> 
>>> <http://www.mapr.com/>
>>> 
>>> 
>>> Now Available - Free Hadoop On-Demand Training
>>> <
>> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
>>> 
>> 
>> 
> 
> 
> -- 
> 
> Abdelhakim Deneche
> 
> Software Engineer
> 
>  <http://www.mapr.com/>
> 
> 
> Now Available - Free Hadoop On-Demand Training
> <http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available>

Re: UDF is not recognized by drill / Validation Error

Posted by Abdel Hakim Deneche <ad...@maprtech.com>.

What version of Drill are you using ?

On Fri, May 20, 2016 at 11:28 AM, Julian Feinauer <ju...@web.de>
wrote:

> Hey Abdel,
>
> I renamed it due to your previous hint with the static subclass.
> But the SQL Commands still returns:
> 0: jdbc:drill:> SELECT myintadd(CAST(position_id AS int),CAST(store_id AS
> int)) FROM cp.`employee.json`;
> Error: VALIDATION ERROR: From line 1, column 8 to line 1, column 63: No
> match found for function signature myintadd(<NUMERIC>, <NUMERIC>)
>
> So this has to be something wrong with my drill setup?
>
> All I do is to start my drillbit with drillbit.sh start (or restart) and
> then start drill-conf where the conf/drill-override.conf contains:
> drill.exec: {
>   cluster-id: "drillbits1",
>   zk.connect: "localhost:2181"
> }
>
> There’s one other thing that seems strange to me with my setup:
> Although I set the loglevel in conf/logback.xml to trace the drillbit.log
> file does not show any trace outputs.
>
> Am I doing something wrong with my setup?
>
> Thanks
> Julian
>
> > Am 20.05.2016 um 20:13 schrieb Abdel Hakim Deneche <
> adeneche@maprtech.com>:
> >
> > You defined your function as myintadd:
> >
> >
> > @FunctionTemplate(name = "*myintadd*", scope ...
> >
> >
> > The following query worked fine for me:
> >
> > SELECT *myintadd*(CAST(position_id AS int),CAST(store_id AS int)) FROM
> >> cp.`employee.json`;
> >
> >
> > Thanks
> >
> >
> >
> >
> >
> > On Fri, May 20, 2016 at 10:17 AM, Julian Feinauer <
> julian.feinauer@web.de>
> > wrote:
> >
> >> Dear Jacques,
> >> Dear Abdel,
> >>
> >> thanks for your kind help.
> >>
> >> jar tf drillUDF-1.0-SNAPSHOT.jar returns:
> >>
> >> META-INF/
> >> META-INF/MANIFEST.MF
> >> org/
> >> org/apache/
> >> org/apache/drill/
> >> org/apache/drill/contrib/
> >> org/apache/drill/contrib/function/
> >> org/julian/
> >> drill-module.conf
> >> IntIntAdd.class
> >> org/apache/drill/contrib/function/IntIntAdd.class
> >> org/julian/IntIntAdd$myintadd.class
> >> org/julian/IntIntAdd.class
> >> org/julian/IntIntAdd.java
> >> META-INF/maven/
> >> META-INF/maven/julian.test/
> >> META-INF/maven/julian.test/drillUDF/
> >> META-INF/maven/julian.test/drillUDF/pom.xml
> >> META-INF/maven/julian.test/drillUDF/pom.properties
> >>
> >> And jar tf drillUDF-1.0-SNAPSHOT-sources.jar returns
> >> META-INF/
> >> META-INF/MANIFEST.MF
> >> org/
> >> org/apache/
> >> org/apache/drill/
> >> org/apache/drill/contrib/
> >> org/apache/drill/contrib/function/
> >> org/julian/
> >> org/julian/IntIntAdd.java
> >> drill-module.conf
> >>
> >> so the drill-module.conf is in both roots.
> >>
> >> @Abdel:
> >> How or where can I share the jar files with you?
> >>
> >> Best regards
> >> Julian
> >>
> >>> Am 20.05.2016 um 17:31 schrieb Jacques Nadeau <ja...@dremio.com>:
> >>>
> >>> Can you run  jar tf myudf.jar against your jar files? Since Drill is
> not
> >>> detecting the jar file, we need to resolve that first. The
> >>> drill-module.conf must be in the root of each jar file that should be
> >>> included. Lets start by verifying that.
> >>>
> >>> --
> >>> Jacques Nadeau
> >>> CTO and Co-Founder, Dremio
> >>>
> >>> On Fri, May 20, 2016 at 6:08 AM, Julian Feinauer <
> julian.feinauer@web.de
> >>>
> >>> wrote:
> >>>
> >>>> Dear all,
> >>>>
> >>>> thank you very much for all your replies.
> >>>> I tried everything but it is still not working.
> >>>>
> >>>> 1. I copy both files (classes and sources) in the /jars/3rdparty
> >> directory
> >>>> 2. I restarted the drillbit after this (I use only one drillbit and
> >>>> drill-conf both running on my local machine)
> >>>> 3. I changed the class to a static subclass
> >>>> 4. I have the drill-module.conf in my ressources
> >>>> 4. The error appears in the drillbit.log as soon as i call the udf
> >> because
> >>>> it is not recognized by drill.
> >>>> On startup drillbit.log states all the packages and jars that are
> >> scanned
> >>>> and my custom jar is not listed there.
> >>>> Therefore I think it is a problem with the class loader or something
> >>>> related?
> >>>>
> >>>> Could this be possible?
> >>>>
> >>>> Greetings
> >>>> Julian
> >>>>
> >>>>> Am 20.05.2016 um 14:36 schrieb Tugdual Grall <tu...@gmail.com>:
> >>>>>
> >>>>> Hi
> >>>>>
> >>>>> Be sure you deploy on each nodes, 2 jars:
> >>>>> - the jar containing the classes
> >>>>> - the jar contains the sources
> >>>>>
> >>>>> The POM.xml in the simple function examples contains the maven
> >>>>> configuration to generate these 2 files, be sure you have the same in
> >>>> your
> >>>>> project:
> >>>>>
> >>>>
> >>
> https://github.com/tgrall/drill-simple-mask-function/blob/master/pom.xml#L24-L37
> >>>>>
> >>>>> and you have restarted the drillbit
> >>>>>
> >>>>> Regards
> >>>>> Tug
> >>>>> @tgrall
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Fri, May 20, 2016 at 12:34 AM, Abdel Hakim Deneche <
> >>>> adeneche@maprtech.com
> >>>>>> wrote:
> >>>>>
> >>>>>> the example I gave you was incomplete, here is what I meant to send:
> >>>>>>
> >>>>>> public class MyUDF {
> >>>>>>
> >>>>>> @FunctionTemplate(name = „myaddints", scope = FunctionTemplate.
> >>>>>> FunctionScope.SIMPLE, nulls =
> >>>>>> FunctionTemplate.NullHandling.NULL_IF_NULL)
> >>>>>> public *static *class IntIntAdd implements DrillSimpleFunc {
> >>>>>>    ...
> >>>>>> }
> >>>>>>
> >>>>>> }
> >>>>>>
> >>>>>>
> >>>>>> On Thu, May 19, 2016 at 3:33 PM, Abdel Hakim Deneche <
> >>>>>> adeneche@maprtech.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Hey Julian,
> >>>>>>>
> >>>>>>> one more thing you could try out: declare the UDF as a static class
> >>>>>> inside
> >>>>>>> another class:
> >>>>>>>
> >>>>>>> public class MyUDF {
> >>>>>>>
> >>>>>>> @FunctionTemplate(name = „myaddints", scope = FunctionTemplate.
> >>>>>>> FunctionScope.SIMPLE, nulls = FunctionTemplate.NullHandling.
> >>>>>>> NULL_IF_NULL)
> >>>>>>> public class IntIntAdd implements DrillSimpleFunc {
> >>>>>>>    ...
> >>>>>>> }
> >>>>>>>
> >>>>>>> }
> >>>>>>>
> >>>>>>> Take a look at the following page to see an examples of UDFs:
> >>>>>>> http://drill.apache.org/docs/custom-function-interfaces/
> >>>>>>>
> >>>>>>> If this doesn't work check the drillbit log, it should print an
> error
> >>>>>>> message when it's starting up if something's wrong with your UDF.
> >>>>>>>
> >>>>>>> Thanks
> >>>>>>>
> >>>>>>>
> >>>>>>> On Thu, May 19, 2016 at 3:31 AM, Julian Feinauer <
> >>>> julian.feinauer@web.de
> >>>>>>>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Dear folks,
> >>>>>>>>
> >>>>>>>> I’m currently experimenting with user defined functions in drill
> but
> >>>> I’m
> >>>>>>>> not able to get them to work on my drillbits.
> >>>>>>>> I always get the error: Error: VALIDATION ERROR: From line 1,
> >> column 8
> >>>>>> to
> >>>>>>>> line 1, column 41: No match found for function signature
> >>>>>> myaddints(<ANY>,
> >>>>>>>> <ANY>).
> >>>>>>>>
> >>>>>>>> I already went through all the tips I found in the mailing list.
> >>>>>>>> The jar contains a drill-module.conf with the content:
> >>>>>>>> drill.classpath.scanning.packages += "org.julian"
> >>>>>>>> And the UDF is defined as:
> >>>>>>>> package org.julian;
> >>>>>>>>
> >>>>>>>> import ...
> >>>>>>>>
> >>>>>>>> @FunctionTemplate(name = „myaddints", scope =
> >>>>>>>> FunctionTemplate.FunctionScope.SIMPLE, nulls =
> >>>>>>>> FunctionTemplate.NullHandling.NULL_IF_NULL)
> >>>>>>>> public class IntIntAdd implements DrillSimpleFunc {
> >>>>>>>>
> >>>>>>>>  @Param
> >>>>>>>>  IntHolder in1;
> >>>>>>>>
> >>>>>>>>  @Param
> >>>>>>>>  IntHolder in2;
> >>>>>>>>
> >>>>>>>>  @Output
> >>>>>>>>  IntHolder out;
> >>>>>>>>
> >>>>>>>>  public void setup() {};
> >>>>>>>>
> >>>>>>>>  public void eval() {
> >>>>>>>>      out.value = in1.value + in2.value;
> >>>>>>>>  }
> >>>>>>>>
> >>>>>>>> }
> >>>>>>>> I tried to find the bug using the logs and encountered the
> following
> >>>>>>>> messages from the drillbit.log:
> >>>>>>>> Base Configuration:
> >>>>>>>>      -
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-common-1.6.0.jar!/drill-default.conf
> >>>>>>>>
> >>>>>>>> Intermediate Configuration and Plugin files, in order of
> precedence:
> >>>>>>>>      -
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-gis-1.6.0.jar!/drill-module.conf
> >>>>>>>>      -
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-logical-1.6.0.jar!/drill-module.conf
> >>>>>>>>      -
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-kudu-storage-1.6.0.jar!/drill-module.conf
> >>>>>>>>      -
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/3rdparty/drill-memory-base-1.6.0.jar!/drill-module.conf
> >>>>>>>>      -
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-hive-exec-shaded-1.6.0.jar!/drill-module.conf
> >>>>>>>>      -
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-memory-base-1.6.0.jar!/drill-module.conf
> >>>>>>>>      -
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-mongo-storage-1.6.0.jar!/drill-module.conf
> >>>>>>>>      -
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-java-exec-1.6.0.jar!/drill-module.conf
> >>>>>>>>      -
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-common-1.6.0.jar!/drill-module.conf
> >>>>>>>>      -
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-storage-hbase-1.6.0.jar!/drill-module.conf
> >>>>>>>>      -
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-jdbc-storage-1.6.0.jar!/drill-module.conf
> >>>>>>>>      -
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-storage-hive-core-1.6.0.jar!/drill-module.conf
> >>>>>>>>
> >>>>>>>> which let’s me think that my jar file (drillUDF-1.0-SNAPSHOT.jar)
> is
> >>>> not
> >>>>>>>> scanned altough it is located in jars/3rdparty.
> >>>>>>>>
> >>>>>>>> Am I doing something wrong or am I missing something or has
> anyone a
> >>>>>> good
> >>>>>>>> hint for me?
> >>>>>>>>
> >>>>>>>> Thank you already!
> >>>>>>>> Julian
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>>
> >>>>>>> Abdelhakim Deneche
> >>>>>>>
> >>>>>>> Software Engineer
> >>>>>>>
> >>>>>>> <http://www.mapr.com/>
> >>>>>>>
> >>>>>>>
> >>>>>>> Now Available - Free Hadoop On-Demand Training
> >>>>>>> <
> >>>>>>
> >>>>
> >>
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>>
> >>>>>> Abdelhakim Deneche
> >>>>>>
> >>>>>> Software Engineer
> >>>>>>
> >>>>>> <http://www.mapr.com/>
> >>>>>>
> >>>>>>
> >>>>>> Now Available - Free Hadoop On-Demand Training
> >>>>>> <
> >>>>>>
> >>>>
> >>
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >>>>>>>
> >>>>>>
> >>>>
> >>>>
> >>
> >>
> >
> >
> > --
> >
> > Abdelhakim Deneche
> >
> > Software Engineer
> >
> >  <http://www.mapr.com/>
> >
> >
> > Now Available - Free Hadoop On-Demand Training
> > <
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >
>
>


-- 

Abdelhakim Deneche

Software Engineer

  <http://www.mapr.com/>


Now Available - Free Hadoop On-Demand Training
<http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available>

Re: UDF is not recognized by drill / Validation Error

Posted by Julian Feinauer <ju...@web.de>.

Hey Abdel,

I renamed it due to your previous hint with the static subclass.
But the SQL Commands still returns:
0: jdbc:drill:> SELECT myintadd(CAST(position_id AS int),CAST(store_id AS int)) FROM cp.`employee.json`;
Error: VALIDATION ERROR: From line 1, column 8 to line 1, column 63: No match found for function signature myintadd(<NUMERIC>, <NUMERIC>)

So this has to be something wrong with my drill setup?

All I do is to start my drillbit with drillbit.sh start (or restart) and then start drill-conf where the conf/drill-override.conf contains:
drill.exec: {
  cluster-id: "drillbits1",
  zk.connect: "localhost:2181"
}

There’s one other thing that seems strange to me with my setup:
Although I set the loglevel in conf/logback.xml to trace the drillbit.log file does not show any trace outputs.

Am I doing something wrong with my setup?

Thanks
Julian

> Am 20.05.2016 um 20:13 schrieb Abdel Hakim Deneche <ad...@maprtech.com>:
> 
> You defined your function as myintadd:
> 
> 
> @FunctionTemplate(name = "*myintadd*", scope ...
> 
> 
> The following query worked fine for me:
> 
> SELECT *myintadd*(CAST(position_id AS int),CAST(store_id AS int)) FROM
>> cp.`employee.json`;
> 
> 
> Thanks
> 
> 
> 
> 
> 
> On Fri, May 20, 2016 at 10:17 AM, Julian Feinauer <ju...@web.de>
> wrote:
> 
>> Dear Jacques,
>> Dear Abdel,
>> 
>> thanks for your kind help.
>> 
>> jar tf drillUDF-1.0-SNAPSHOT.jar returns:
>> 
>> META-INF/
>> META-INF/MANIFEST.MF
>> org/
>> org/apache/
>> org/apache/drill/
>> org/apache/drill/contrib/
>> org/apache/drill/contrib/function/
>> org/julian/
>> drill-module.conf
>> IntIntAdd.class
>> org/apache/drill/contrib/function/IntIntAdd.class
>> org/julian/IntIntAdd$myintadd.class
>> org/julian/IntIntAdd.class
>> org/julian/IntIntAdd.java
>> META-INF/maven/
>> META-INF/maven/julian.test/
>> META-INF/maven/julian.test/drillUDF/
>> META-INF/maven/julian.test/drillUDF/pom.xml
>> META-INF/maven/julian.test/drillUDF/pom.properties
>> 
>> And jar tf drillUDF-1.0-SNAPSHOT-sources.jar returns
>> META-INF/
>> META-INF/MANIFEST.MF
>> org/
>> org/apache/
>> org/apache/drill/
>> org/apache/drill/contrib/
>> org/apache/drill/contrib/function/
>> org/julian/
>> org/julian/IntIntAdd.java
>> drill-module.conf
>> 
>> so the drill-module.conf is in both roots.
>> 
>> @Abdel:
>> How or where can I share the jar files with you?
>> 
>> Best regards
>> Julian
>> 
>>> Am 20.05.2016 um 17:31 schrieb Jacques Nadeau <ja...@dremio.com>:
>>> 
>>> Can you run  jar tf myudf.jar against your jar files? Since Drill is not
>>> detecting the jar file, we need to resolve that first. The
>>> drill-module.conf must be in the root of each jar file that should be
>>> included. Lets start by verifying that.
>>> 
>>> --
>>> Jacques Nadeau
>>> CTO and Co-Founder, Dremio
>>> 
>>> On Fri, May 20, 2016 at 6:08 AM, Julian Feinauer <julian.feinauer@web.de
>>> 
>>> wrote:
>>> 
>>>> Dear all,
>>>> 
>>>> thank you very much for all your replies.
>>>> I tried everything but it is still not working.
>>>> 
>>>> 1. I copy both files (classes and sources) in the /jars/3rdparty
>> directory
>>>> 2. I restarted the drillbit after this (I use only one drillbit and
>>>> drill-conf both running on my local machine)
>>>> 3. I changed the class to a static subclass
>>>> 4. I have the drill-module.conf in my ressources
>>>> 4. The error appears in the drillbit.log as soon as i call the udf
>> because
>>>> it is not recognized by drill.
>>>> On startup drillbit.log states all the packages and jars that are
>> scanned
>>>> and my custom jar is not listed there.
>>>> Therefore I think it is a problem with the class loader or something
>>>> related?
>>>> 
>>>> Could this be possible?
>>>> 
>>>> Greetings
>>>> Julian
>>>> 
>>>>> Am 20.05.2016 um 14:36 schrieb Tugdual Grall <tu...@gmail.com>:
>>>>> 
>>>>> Hi
>>>>> 
>>>>> Be sure you deploy on each nodes, 2 jars:
>>>>> - the jar containing the classes
>>>>> - the jar contains the sources
>>>>> 
>>>>> The POM.xml in the simple function examples contains the maven
>>>>> configuration to generate these 2 files, be sure you have the same in
>>>> your
>>>>> project:
>>>>> 
>>>> 
>> https://github.com/tgrall/drill-simple-mask-function/blob/master/pom.xml#L24-L37
>>>>> 
>>>>> and you have restarted the drillbit
>>>>> 
>>>>> Regards
>>>>> Tug
>>>>> @tgrall
>>>>> 
>>>>> 
>>>>> 
>>>>> On Fri, May 20, 2016 at 12:34 AM, Abdel Hakim Deneche <
>>>> adeneche@maprtech.com
>>>>>> wrote:
>>>>> 
>>>>>> the example I gave you was incomplete, here is what I meant to send:
>>>>>> 
>>>>>> public class MyUDF {
>>>>>> 
>>>>>> @FunctionTemplate(name = „myaddints", scope = FunctionTemplate.
>>>>>> FunctionScope.SIMPLE, nulls =
>>>>>> FunctionTemplate.NullHandling.NULL_IF_NULL)
>>>>>> public *static *class IntIntAdd implements DrillSimpleFunc {
>>>>>>    ...
>>>>>> }
>>>>>> 
>>>>>> }
>>>>>> 
>>>>>> 
>>>>>> On Thu, May 19, 2016 at 3:33 PM, Abdel Hakim Deneche <
>>>>>> adeneche@maprtech.com>
>>>>>> wrote:
>>>>>> 
>>>>>>> Hey Julian,
>>>>>>> 
>>>>>>> one more thing you could try out: declare the UDF as a static class
>>>>>> inside
>>>>>>> another class:
>>>>>>> 
>>>>>>> public class MyUDF {
>>>>>>> 
>>>>>>> @FunctionTemplate(name = „myaddints", scope = FunctionTemplate.
>>>>>>> FunctionScope.SIMPLE, nulls = FunctionTemplate.NullHandling.
>>>>>>> NULL_IF_NULL)
>>>>>>> public class IntIntAdd implements DrillSimpleFunc {
>>>>>>>    ...
>>>>>>> }
>>>>>>> 
>>>>>>> }
>>>>>>> 
>>>>>>> Take a look at the following page to see an examples of UDFs:
>>>>>>> http://drill.apache.org/docs/custom-function-interfaces/
>>>>>>> 
>>>>>>> If this doesn't work check the drillbit log, it should print an error
>>>>>>> message when it's starting up if something's wrong with your UDF.
>>>>>>> 
>>>>>>> Thanks
>>>>>>> 
>>>>>>> 
>>>>>>> On Thu, May 19, 2016 at 3:31 AM, Julian Feinauer <
>>>> julian.feinauer@web.de
>>>>>>> 
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> Dear folks,
>>>>>>>> 
>>>>>>>> I’m currently experimenting with user defined functions in drill but
>>>> I’m
>>>>>>>> not able to get them to work on my drillbits.
>>>>>>>> I always get the error: Error: VALIDATION ERROR: From line 1,
>> column 8
>>>>>> to
>>>>>>>> line 1, column 41: No match found for function signature
>>>>>> myaddints(<ANY>,
>>>>>>>> <ANY>).
>>>>>>>> 
>>>>>>>> I already went through all the tips I found in the mailing list.
>>>>>>>> The jar contains a drill-module.conf with the content:
>>>>>>>> drill.classpath.scanning.packages += "org.julian"
>>>>>>>> And the UDF is defined as:
>>>>>>>> package org.julian;
>>>>>>>> 
>>>>>>>> import ...
>>>>>>>> 
>>>>>>>> @FunctionTemplate(name = „myaddints", scope =
>>>>>>>> FunctionTemplate.FunctionScope.SIMPLE, nulls =
>>>>>>>> FunctionTemplate.NullHandling.NULL_IF_NULL)
>>>>>>>> public class IntIntAdd implements DrillSimpleFunc {
>>>>>>>> 
>>>>>>>>  @Param
>>>>>>>>  IntHolder in1;
>>>>>>>> 
>>>>>>>>  @Param
>>>>>>>>  IntHolder in2;
>>>>>>>> 
>>>>>>>>  @Output
>>>>>>>>  IntHolder out;
>>>>>>>> 
>>>>>>>>  public void setup() {};
>>>>>>>> 
>>>>>>>>  public void eval() {
>>>>>>>>      out.value = in1.value + in2.value;
>>>>>>>>  }
>>>>>>>> 
>>>>>>>> }
>>>>>>>> I tried to find the bug using the logs and encountered the following
>>>>>>>> messages from the drillbit.log:
>>>>>>>> Base Configuration:
>>>>>>>>      -
>>>>>>>> 
>>>>>> 
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-common-1.6.0.jar!/drill-default.conf
>>>>>>>> 
>>>>>>>> Intermediate Configuration and Plugin files, in order of precedence:
>>>>>>>>      -
>>>>>>>> 
>>>>>> 
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-gis-1.6.0.jar!/drill-module.conf
>>>>>>>>      -
>>>>>>>> 
>>>>>> 
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-logical-1.6.0.jar!/drill-module.conf
>>>>>>>>      -
>>>>>>>> 
>>>>>> 
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-kudu-storage-1.6.0.jar!/drill-module.conf
>>>>>>>>      -
>>>>>>>> 
>>>>>> 
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/3rdparty/drill-memory-base-1.6.0.jar!/drill-module.conf
>>>>>>>>      -
>>>>>>>> 
>>>>>> 
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-hive-exec-shaded-1.6.0.jar!/drill-module.conf
>>>>>>>>      -
>>>>>>>> 
>>>>>> 
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-memory-base-1.6.0.jar!/drill-module.conf
>>>>>>>>      -
>>>>>>>> 
>>>>>> 
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-mongo-storage-1.6.0.jar!/drill-module.conf
>>>>>>>>      -
>>>>>>>> 
>>>>>> 
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-java-exec-1.6.0.jar!/drill-module.conf
>>>>>>>>      -
>>>>>>>> 
>>>>>> 
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-common-1.6.0.jar!/drill-module.conf
>>>>>>>>      -
>>>>>>>> 
>>>>>> 
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-storage-hbase-1.6.0.jar!/drill-module.conf
>>>>>>>>      -
>>>>>>>> 
>>>>>> 
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-jdbc-storage-1.6.0.jar!/drill-module.conf
>>>>>>>>      -
>>>>>>>> 
>>>>>> 
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-storage-hive-core-1.6.0.jar!/drill-module.conf
>>>>>>>> 
>>>>>>>> which let’s me think that my jar file (drillUDF-1.0-SNAPSHOT.jar) is
>>>> not
>>>>>>>> scanned altough it is located in jars/3rdparty.
>>>>>>>> 
>>>>>>>> Am I doing something wrong or am I missing something or has anyone a
>>>>>> good
>>>>>>>> hint for me?
>>>>>>>> 
>>>>>>>> Thank you already!
>>>>>>>> Julian
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> --
>>>>>>> 
>>>>>>> Abdelhakim Deneche
>>>>>>> 
>>>>>>> Software Engineer
>>>>>>> 
>>>>>>> <http://www.mapr.com/>
>>>>>>> 
>>>>>>> 
>>>>>>> Now Available - Free Hadoop On-Demand Training
>>>>>>> <
>>>>>> 
>>>> 
>> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> 
>>>>>> Abdelhakim Deneche
>>>>>> 
>>>>>> Software Engineer
>>>>>> 
>>>>>> <http://www.mapr.com/>
>>>>>> 
>>>>>> 
>>>>>> Now Available - Free Hadoop On-Demand Training
>>>>>> <
>>>>>> 
>>>> 
>> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
>>>>>>> 
>>>>>> 
>>>> 
>>>> 
>> 
>> 
> 
> 
> -- 
> 
> Abdelhakim Deneche
> 
> Software Engineer
> 
>  <http://www.mapr.com/>
> 
> 
> Now Available - Free Hadoop On-Demand Training
> <http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available>

Re: UDF is not recognized by drill / Validation Error

Posted by Abdel Hakim Deneche <ad...@maprtech.com>.

You defined your function as myintadd:


@FunctionTemplate(name = "*myintadd*", scope ...


The following query worked fine for me:

SELECT *myintadd*(CAST(position_id AS int),CAST(store_id AS int)) FROM
> cp.`employee.json`;


Thanks





On Fri, May 20, 2016 at 10:17 AM, Julian Feinauer <ju...@web.de>
wrote:

> Dear Jacques,
> Dear Abdel,
>
> thanks for your kind help.
>
> jar tf drillUDF-1.0-SNAPSHOT.jar returns:
>
> META-INF/
> META-INF/MANIFEST.MF
> org/
> org/apache/
> org/apache/drill/
> org/apache/drill/contrib/
> org/apache/drill/contrib/function/
> org/julian/
> drill-module.conf
> IntIntAdd.class
> org/apache/drill/contrib/function/IntIntAdd.class
> org/julian/IntIntAdd$myintadd.class
> org/julian/IntIntAdd.class
> org/julian/IntIntAdd.java
> META-INF/maven/
> META-INF/maven/julian.test/
> META-INF/maven/julian.test/drillUDF/
> META-INF/maven/julian.test/drillUDF/pom.xml
> META-INF/maven/julian.test/drillUDF/pom.properties
>
> And jar tf drillUDF-1.0-SNAPSHOT-sources.jar returns
> META-INF/
> META-INF/MANIFEST.MF
> org/
> org/apache/
> org/apache/drill/
> org/apache/drill/contrib/
> org/apache/drill/contrib/function/
> org/julian/
> org/julian/IntIntAdd.java
> drill-module.conf
>
> so the drill-module.conf is in both roots.
>
> @Abdel:
> How or where can I share the jar files with you?
>
> Best regards
> Julian
>
> > Am 20.05.2016 um 17:31 schrieb Jacques Nadeau <ja...@dremio.com>:
> >
> > Can you run  jar tf myudf.jar against your jar files? Since Drill is not
> > detecting the jar file, we need to resolve that first. The
> > drill-module.conf must be in the root of each jar file that should be
> > included. Lets start by verifying that.
> >
> > --
> > Jacques Nadeau
> > CTO and Co-Founder, Dremio
> >
> > On Fri, May 20, 2016 at 6:08 AM, Julian Feinauer <julian.feinauer@web.de
> >
> > wrote:
> >
> >> Dear all,
> >>
> >> thank you very much for all your replies.
> >> I tried everything but it is still not working.
> >>
> >> 1. I copy both files (classes and sources) in the /jars/3rdparty
> directory
> >> 2. I restarted the drillbit after this (I use only one drillbit and
> >> drill-conf both running on my local machine)
> >> 3. I changed the class to a static subclass
> >> 4. I have the drill-module.conf in my ressources
> >> 4. The error appears in the drillbit.log as soon as i call the udf
> because
> >> it is not recognized by drill.
> >> On startup drillbit.log states all the packages and jars that are
> scanned
> >> and my custom jar is not listed there.
> >> Therefore I think it is a problem with the class loader or something
> >> related?
> >>
> >> Could this be possible?
> >>
> >> Greetings
> >> Julian
> >>
> >>> Am 20.05.2016 um 14:36 schrieb Tugdual Grall <tu...@gmail.com>:
> >>>
> >>> Hi
> >>>
> >>> Be sure you deploy on each nodes, 2 jars:
> >>> - the jar containing the classes
> >>> - the jar contains the sources
> >>>
> >>> The POM.xml in the simple function examples contains the maven
> >>> configuration to generate these 2 files, be sure you have the same in
> >> your
> >>> project:
> >>>
> >>
> https://github.com/tgrall/drill-simple-mask-function/blob/master/pom.xml#L24-L37
> >>>
> >>> and you have restarted the drillbit
> >>>
> >>> Regards
> >>> Tug
> >>> @tgrall
> >>>
> >>>
> >>>
> >>> On Fri, May 20, 2016 at 12:34 AM, Abdel Hakim Deneche <
> >> adeneche@maprtech.com
> >>>> wrote:
> >>>
> >>>> the example I gave you was incomplete, here is what I meant to send:
> >>>>
> >>>> public class MyUDF {
> >>>>
> >>>>  @FunctionTemplate(name = „myaddints", scope = FunctionTemplate.
> >>>>  FunctionScope.SIMPLE, nulls =
> >>>> FunctionTemplate.NullHandling.NULL_IF_NULL)
> >>>>  public *static *class IntIntAdd implements DrillSimpleFunc {
> >>>>     ...
> >>>>  }
> >>>>
> >>>> }
> >>>>
> >>>>
> >>>> On Thu, May 19, 2016 at 3:33 PM, Abdel Hakim Deneche <
> >>>> adeneche@maprtech.com>
> >>>> wrote:
> >>>>
> >>>>> Hey Julian,
> >>>>>
> >>>>> one more thing you could try out: declare the UDF as a static class
> >>>> inside
> >>>>> another class:
> >>>>>
> >>>>> public class MyUDF {
> >>>>>
> >>>>>  @FunctionTemplate(name = „myaddints", scope = FunctionTemplate.
> >>>>>  FunctionScope.SIMPLE, nulls = FunctionTemplate.NullHandling.
> >>>>> NULL_IF_NULL)
> >>>>>  public class IntIntAdd implements DrillSimpleFunc {
> >>>>>     ...
> >>>>>  }
> >>>>>
> >>>>> }
> >>>>>
> >>>>> Take a look at the following page to see an examples of UDFs:
> >>>>> http://drill.apache.org/docs/custom-function-interfaces/
> >>>>>
> >>>>> If this doesn't work check the drillbit log, it should print an error
> >>>>> message when it's starting up if something's wrong with your UDF.
> >>>>>
> >>>>> Thanks
> >>>>>
> >>>>>
> >>>>> On Thu, May 19, 2016 at 3:31 AM, Julian Feinauer <
> >> julian.feinauer@web.de
> >>>>>
> >>>>> wrote:
> >>>>>
> >>>>>> Dear folks,
> >>>>>>
> >>>>>> I’m currently experimenting with user defined functions in drill but
> >> I’m
> >>>>>> not able to get them to work on my drillbits.
> >>>>>> I always get the error: Error: VALIDATION ERROR: From line 1,
> column 8
> >>>> to
> >>>>>> line 1, column 41: No match found for function signature
> >>>> myaddints(<ANY>,
> >>>>>> <ANY>).
> >>>>>>
> >>>>>> I already went through all the tips I found in the mailing list.
> >>>>>> The jar contains a drill-module.conf with the content:
> >>>>>> drill.classpath.scanning.packages += "org.julian"
> >>>>>> And the UDF is defined as:
> >>>>>> package org.julian;
> >>>>>>
> >>>>>> import ...
> >>>>>>
> >>>>>> @FunctionTemplate(name = „myaddints", scope =
> >>>>>> FunctionTemplate.FunctionScope.SIMPLE, nulls =
> >>>>>> FunctionTemplate.NullHandling.NULL_IF_NULL)
> >>>>>> public class IntIntAdd implements DrillSimpleFunc {
> >>>>>>
> >>>>>>   @Param
> >>>>>>   IntHolder in1;
> >>>>>>
> >>>>>>   @Param
> >>>>>>   IntHolder in2;
> >>>>>>
> >>>>>>   @Output
> >>>>>>   IntHolder out;
> >>>>>>
> >>>>>>   public void setup() {};
> >>>>>>
> >>>>>>   public void eval() {
> >>>>>>       out.value = in1.value + in2.value;
> >>>>>>   }
> >>>>>>
> >>>>>> }
> >>>>>> I tried to find the bug using the logs and encountered the following
> >>>>>> messages from the drillbit.log:
> >>>>>> Base Configuration:
> >>>>>>       -
> >>>>>>
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-common-1.6.0.jar!/drill-default.conf
> >>>>>>
> >>>>>> Intermediate Configuration and Plugin files, in order of precedence:
> >>>>>>       -
> >>>>>>
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-gis-1.6.0.jar!/drill-module.conf
> >>>>>>       -
> >>>>>>
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-logical-1.6.0.jar!/drill-module.conf
> >>>>>>       -
> >>>>>>
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-kudu-storage-1.6.0.jar!/drill-module.conf
> >>>>>>       -
> >>>>>>
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/3rdparty/drill-memory-base-1.6.0.jar!/drill-module.conf
> >>>>>>       -
> >>>>>>
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-hive-exec-shaded-1.6.0.jar!/drill-module.conf
> >>>>>>       -
> >>>>>>
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-memory-base-1.6.0.jar!/drill-module.conf
> >>>>>>       -
> >>>>>>
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-mongo-storage-1.6.0.jar!/drill-module.conf
> >>>>>>       -
> >>>>>>
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-java-exec-1.6.0.jar!/drill-module.conf
> >>>>>>       -
> >>>>>>
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-common-1.6.0.jar!/drill-module.conf
> >>>>>>       -
> >>>>>>
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-storage-hbase-1.6.0.jar!/drill-module.conf
> >>>>>>       -
> >>>>>>
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-jdbc-storage-1.6.0.jar!/drill-module.conf
> >>>>>>       -
> >>>>>>
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-storage-hive-core-1.6.0.jar!/drill-module.conf
> >>>>>>
> >>>>>> which let’s me think that my jar file (drillUDF-1.0-SNAPSHOT.jar) is
> >> not
> >>>>>> scanned altough it is located in jars/3rdparty.
> >>>>>>
> >>>>>> Am I doing something wrong or am I missing something or has anyone a
> >>>> good
> >>>>>> hint for me?
> >>>>>>
> >>>>>> Thank you already!
> >>>>>> Julian
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>>
> >>>>> Abdelhakim Deneche
> >>>>>
> >>>>> Software Engineer
> >>>>>
> >>>>> <http://www.mapr.com/>
> >>>>>
> >>>>>
> >>>>> Now Available - Free Hadoop On-Demand Training
> >>>>> <
> >>>>
> >>
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>>
> >>>> Abdelhakim Deneche
> >>>>
> >>>> Software Engineer
> >>>>
> >>>> <http://www.mapr.com/>
> >>>>
> >>>>
> >>>> Now Available - Free Hadoop On-Demand Training
> >>>> <
> >>>>
> >>
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >>>>>
> >>>>
> >>
> >>
>
>


-- 

Abdelhakim Deneche

Software Engineer

  <http://www.mapr.com/>


Now Available - Free Hadoop On-Demand Training
<http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available>

Re: UDF is not recognized by drill / Validation Error

Posted by Julian Feinauer <ju...@web.de>.

Dear Jacques,
Dear Abdel,

thanks for your kind help.

jar tf drillUDF-1.0-SNAPSHOT.jar returns:

META-INF/
META-INF/MANIFEST.MF
org/
org/apache/
org/apache/drill/
org/apache/drill/contrib/
org/apache/drill/contrib/function/
org/julian/
drill-module.conf
IntIntAdd.class
org/apache/drill/contrib/function/IntIntAdd.class
org/julian/IntIntAdd$myintadd.class
org/julian/IntIntAdd.class
org/julian/IntIntAdd.java
META-INF/maven/
META-INF/maven/julian.test/
META-INF/maven/julian.test/drillUDF/
META-INF/maven/julian.test/drillUDF/pom.xml
META-INF/maven/julian.test/drillUDF/pom.properties

And jar tf drillUDF-1.0-SNAPSHOT-sources.jar returns
META-INF/
META-INF/MANIFEST.MF
org/
org/apache/
org/apache/drill/
org/apache/drill/contrib/
org/apache/drill/contrib/function/
org/julian/
org/julian/IntIntAdd.java
drill-module.conf

so the drill-module.conf is in both roots.

@Abdel:
How or where can I share the jar files with you?

Best regards
Julian

> Am 20.05.2016 um 17:31 schrieb Jacques Nadeau <ja...@dremio.com>:
> 
> Can you run  jar tf myudf.jar against your jar files? Since Drill is not
> detecting the jar file, we need to resolve that first. The
> drill-module.conf must be in the root of each jar file that should be
> included. Lets start by verifying that.
> 
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
> 
> On Fri, May 20, 2016 at 6:08 AM, Julian Feinauer <ju...@web.de>
> wrote:
> 
>> Dear all,
>> 
>> thank you very much for all your replies.
>> I tried everything but it is still not working.
>> 
>> 1. I copy both files (classes and sources) in the /jars/3rdparty directory
>> 2. I restarted the drillbit after this (I use only one drillbit and
>> drill-conf both running on my local machine)
>> 3. I changed the class to a static subclass
>> 4. I have the drill-module.conf in my ressources
>> 4. The error appears in the drillbit.log as soon as i call the udf because
>> it is not recognized by drill.
>> On startup drillbit.log states all the packages and jars that are scanned
>> and my custom jar is not listed there.
>> Therefore I think it is a problem with the class loader or something
>> related?
>> 
>> Could this be possible?
>> 
>> Greetings
>> Julian
>> 
>>> Am 20.05.2016 um 14:36 schrieb Tugdual Grall <tu...@gmail.com>:
>>> 
>>> Hi
>>> 
>>> Be sure you deploy on each nodes, 2 jars:
>>> - the jar containing the classes
>>> - the jar contains the sources
>>> 
>>> The POM.xml in the simple function examples contains the maven
>>> configuration to generate these 2 files, be sure you have the same in
>> your
>>> project:
>>> 
>> https://github.com/tgrall/drill-simple-mask-function/blob/master/pom.xml#L24-L37
>>> 
>>> and you have restarted the drillbit
>>> 
>>> Regards
>>> Tug
>>> @tgrall
>>> 
>>> 
>>> 
>>> On Fri, May 20, 2016 at 12:34 AM, Abdel Hakim Deneche <
>> adeneche@maprtech.com
>>>> wrote:
>>> 
>>>> the example I gave you was incomplete, here is what I meant to send:
>>>> 
>>>> public class MyUDF {
>>>> 
>>>>  @FunctionTemplate(name = „myaddints", scope = FunctionTemplate.
>>>>  FunctionScope.SIMPLE, nulls =
>>>> FunctionTemplate.NullHandling.NULL_IF_NULL)
>>>>  public *static *class IntIntAdd implements DrillSimpleFunc {
>>>>     ...
>>>>  }
>>>> 
>>>> }
>>>> 
>>>> 
>>>> On Thu, May 19, 2016 at 3:33 PM, Abdel Hakim Deneche <
>>>> adeneche@maprtech.com>
>>>> wrote:
>>>> 
>>>>> Hey Julian,
>>>>> 
>>>>> one more thing you could try out: declare the UDF as a static class
>>>> inside
>>>>> another class:
>>>>> 
>>>>> public class MyUDF {
>>>>> 
>>>>>  @FunctionTemplate(name = „myaddints", scope = FunctionTemplate.
>>>>>  FunctionScope.SIMPLE, nulls = FunctionTemplate.NullHandling.
>>>>> NULL_IF_NULL)
>>>>>  public class IntIntAdd implements DrillSimpleFunc {
>>>>>     ...
>>>>>  }
>>>>> 
>>>>> }
>>>>> 
>>>>> Take a look at the following page to see an examples of UDFs:
>>>>> http://drill.apache.org/docs/custom-function-interfaces/
>>>>> 
>>>>> If this doesn't work check the drillbit log, it should print an error
>>>>> message when it's starting up if something's wrong with your UDF.
>>>>> 
>>>>> Thanks
>>>>> 
>>>>> 
>>>>> On Thu, May 19, 2016 at 3:31 AM, Julian Feinauer <
>> julian.feinauer@web.de
>>>>> 
>>>>> wrote:
>>>>> 
>>>>>> Dear folks,
>>>>>> 
>>>>>> I’m currently experimenting with user defined functions in drill but
>> I’m
>>>>>> not able to get them to work on my drillbits.
>>>>>> I always get the error: Error: VALIDATION ERROR: From line 1, column 8
>>>> to
>>>>>> line 1, column 41: No match found for function signature
>>>> myaddints(<ANY>,
>>>>>> <ANY>).
>>>>>> 
>>>>>> I already went through all the tips I found in the mailing list.
>>>>>> The jar contains a drill-module.conf with the content:
>>>>>> drill.classpath.scanning.packages += "org.julian"
>>>>>> And the UDF is defined as:
>>>>>> package org.julian;
>>>>>> 
>>>>>> import ...
>>>>>> 
>>>>>> @FunctionTemplate(name = „myaddints", scope =
>>>>>> FunctionTemplate.FunctionScope.SIMPLE, nulls =
>>>>>> FunctionTemplate.NullHandling.NULL_IF_NULL)
>>>>>> public class IntIntAdd implements DrillSimpleFunc {
>>>>>> 
>>>>>>   @Param
>>>>>>   IntHolder in1;
>>>>>> 
>>>>>>   @Param
>>>>>>   IntHolder in2;
>>>>>> 
>>>>>>   @Output
>>>>>>   IntHolder out;
>>>>>> 
>>>>>>   public void setup() {};
>>>>>> 
>>>>>>   public void eval() {
>>>>>>       out.value = in1.value + in2.value;
>>>>>>   }
>>>>>> 
>>>>>> }
>>>>>> I tried to find the bug using the logs and encountered the following
>>>>>> messages from the drillbit.log:
>>>>>> Base Configuration:
>>>>>>       -
>>>>>> 
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-common-1.6.0.jar!/drill-default.conf
>>>>>> 
>>>>>> Intermediate Configuration and Plugin files, in order of precedence:
>>>>>>       -
>>>>>> 
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-gis-1.6.0.jar!/drill-module.conf
>>>>>>       -
>>>>>> 
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-logical-1.6.0.jar!/drill-module.conf
>>>>>>       -
>>>>>> 
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-kudu-storage-1.6.0.jar!/drill-module.conf
>>>>>>       -
>>>>>> 
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/3rdparty/drill-memory-base-1.6.0.jar!/drill-module.conf
>>>>>>       -
>>>>>> 
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-hive-exec-shaded-1.6.0.jar!/drill-module.conf
>>>>>>       -
>>>>>> 
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-memory-base-1.6.0.jar!/drill-module.conf
>>>>>>       -
>>>>>> 
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-mongo-storage-1.6.0.jar!/drill-module.conf
>>>>>>       -
>>>>>> 
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-java-exec-1.6.0.jar!/drill-module.conf
>>>>>>       -
>>>>>> 
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-common-1.6.0.jar!/drill-module.conf
>>>>>>       -
>>>>>> 
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-storage-hbase-1.6.0.jar!/drill-module.conf
>>>>>>       -
>>>>>> 
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-jdbc-storage-1.6.0.jar!/drill-module.conf
>>>>>>       -
>>>>>> 
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-storage-hive-core-1.6.0.jar!/drill-module.conf
>>>>>> 
>>>>>> which let’s me think that my jar file (drillUDF-1.0-SNAPSHOT.jar) is
>> not
>>>>>> scanned altough it is located in jars/3rdparty.
>>>>>> 
>>>>>> Am I doing something wrong or am I missing something or has anyone a
>>>> good
>>>>>> hint for me?
>>>>>> 
>>>>>> Thank you already!
>>>>>> Julian
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> 
>>>>> Abdelhakim Deneche
>>>>> 
>>>>> Software Engineer
>>>>> 
>>>>> <http://www.mapr.com/>
>>>>> 
>>>>> 
>>>>> Now Available - Free Hadoop On-Demand Training
>>>>> <
>>>> 
>> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> --
>>>> 
>>>> Abdelhakim Deneche
>>>> 
>>>> Software Engineer
>>>> 
>>>> <http://www.mapr.com/>
>>>> 
>>>> 
>>>> Now Available - Free Hadoop On-Demand Training
>>>> <
>>>> 
>> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
>>>>> 
>>>> 
>> 
>>

Re: UDF is not recognized by drill / Validation Error

Posted by Jacques Nadeau <ja...@dremio.com>.

Can you run  jar tf myudf.jar against your jar files? Since Drill is not
detecting the jar file, we need to resolve that first. The
drill-module.conf must be in the root of each jar file that should be
included. Lets start by verifying that.

--
Jacques Nadeau
CTO and Co-Founder, Dremio

On Fri, May 20, 2016 at 6:08 AM, Julian Feinauer <ju...@web.de>
wrote:

> Dear all,
>
> thank you very much for all your replies.
> I tried everything but it is still not working.
>
> 1. I copy both files (classes and sources) in the /jars/3rdparty directory
> 2. I restarted the drillbit after this (I use only one drillbit and
> drill-conf both running on my local machine)
> 3. I changed the class to a static subclass
> 4. I have the drill-module.conf in my ressources
> 4. The error appears in the drillbit.log as soon as i call the udf because
> it is not recognized by drill.
> On startup drillbit.log states all the packages and jars that are scanned
> and my custom jar is not listed there.
> Therefore I think it is a problem with the class loader or something
> related?
>
> Could this be possible?
>
> Greetings
> Julian
>
> > Am 20.05.2016 um 14:36 schrieb Tugdual Grall <tu...@gmail.com>:
> >
> > Hi
> >
> > Be sure you deploy on each nodes, 2 jars:
> > - the jar containing the classes
> > - the jar contains the sources
> >
> > The POM.xml in the simple function examples contains the maven
> > configuration to generate these 2 files, be sure you have the same in
> your
> > project:
> >
> https://github.com/tgrall/drill-simple-mask-function/blob/master/pom.xml#L24-L37
> >
> > and you have restarted the drillbit
> >
> > Regards
> > Tug
> > @tgrall
> >
> >
> >
> > On Fri, May 20, 2016 at 12:34 AM, Abdel Hakim Deneche <
> adeneche@maprtech.com
> >> wrote:
> >
> >> the example I gave you was incomplete, here is what I meant to send:
> >>
> >> public class MyUDF {
> >>
> >>   @FunctionTemplate(name = „myaddints", scope = FunctionTemplate.
> >>   FunctionScope.SIMPLE, nulls =
> >> FunctionTemplate.NullHandling.NULL_IF_NULL)
> >>   public *static *class IntIntAdd implements DrillSimpleFunc {
> >>      ...
> >>   }
> >>
> >> }
> >>
> >>
> >> On Thu, May 19, 2016 at 3:33 PM, Abdel Hakim Deneche <
> >> adeneche@maprtech.com>
> >> wrote:
> >>
> >>> Hey Julian,
> >>>
> >>> one more thing you could try out: declare the UDF as a static class
> >> inside
> >>> another class:
> >>>
> >>> public class MyUDF {
> >>>
> >>>   @FunctionTemplate(name = „myaddints", scope = FunctionTemplate.
> >>>   FunctionScope.SIMPLE, nulls = FunctionTemplate.NullHandling.
> >>> NULL_IF_NULL)
> >>>   public class IntIntAdd implements DrillSimpleFunc {
> >>>      ...
> >>>   }
> >>>
> >>> }
> >>>
> >>> Take a look at the following page to see an examples of UDFs:
> >>> http://drill.apache.org/docs/custom-function-interfaces/
> >>>
> >>> If this doesn't work check the drillbit log, it should print an error
> >>> message when it's starting up if something's wrong with your UDF.
> >>>
> >>> Thanks
> >>>
> >>>
> >>> On Thu, May 19, 2016 at 3:31 AM, Julian Feinauer <
> julian.feinauer@web.de
> >>>
> >>> wrote:
> >>>
> >>>> Dear folks,
> >>>>
> >>>> I’m currently experimenting with user defined functions in drill but
> I’m
> >>>> not able to get them to work on my drillbits.
> >>>> I always get the error: Error: VALIDATION ERROR: From line 1, column 8
> >> to
> >>>> line 1, column 41: No match found for function signature
> >> myaddints(<ANY>,
> >>>> <ANY>).
> >>>>
> >>>> I already went through all the tips I found in the mailing list.
> >>>> The jar contains a drill-module.conf with the content:
> >>>> drill.classpath.scanning.packages += "org.julian"
> >>>> And the UDF is defined as:
> >>>> package org.julian;
> >>>>
> >>>> import ...
> >>>>
> >>>> @FunctionTemplate(name = „myaddints", scope =
> >>>> FunctionTemplate.FunctionScope.SIMPLE, nulls =
> >>>> FunctionTemplate.NullHandling.NULL_IF_NULL)
> >>>> public class IntIntAdd implements DrillSimpleFunc {
> >>>>
> >>>>    @Param
> >>>>    IntHolder in1;
> >>>>
> >>>>    @Param
> >>>>    IntHolder in2;
> >>>>
> >>>>    @Output
> >>>>    IntHolder out;
> >>>>
> >>>>    public void setup() {};
> >>>>
> >>>>    public void eval() {
> >>>>        out.value = in1.value + in2.value;
> >>>>    }
> >>>>
> >>>> }
> >>>> I tried to find the bug using the logs and encountered the following
> >>>> messages from the drillbit.log:
> >>>> Base Configuration:
> >>>>        -
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-common-1.6.0.jar!/drill-default.conf
> >>>>
> >>>> Intermediate Configuration and Plugin files, in order of precedence:
> >>>>        -
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-gis-1.6.0.jar!/drill-module.conf
> >>>>        -
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-logical-1.6.0.jar!/drill-module.conf
> >>>>        -
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-kudu-storage-1.6.0.jar!/drill-module.conf
> >>>>        -
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/3rdparty/drill-memory-base-1.6.0.jar!/drill-module.conf
> >>>>        -
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-hive-exec-shaded-1.6.0.jar!/drill-module.conf
> >>>>        -
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-memory-base-1.6.0.jar!/drill-module.conf
> >>>>        -
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-mongo-storage-1.6.0.jar!/drill-module.conf
> >>>>        -
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-java-exec-1.6.0.jar!/drill-module.conf
> >>>>        -
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-common-1.6.0.jar!/drill-module.conf
> >>>>        -
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-storage-hbase-1.6.0.jar!/drill-module.conf
> >>>>        -
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-jdbc-storage-1.6.0.jar!/drill-module.conf
> >>>>        -
> >>>>
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-storage-hive-core-1.6.0.jar!/drill-module.conf
> >>>>
> >>>> which let’s me think that my jar file (drillUDF-1.0-SNAPSHOT.jar) is
> not
> >>>> scanned altough it is located in jars/3rdparty.
> >>>>
> >>>> Am I doing something wrong or am I missing something or has anyone a
> >> good
> >>>> hint for me?
> >>>>
> >>>> Thank you already!
> >>>> Julian
> >>>
> >>>
> >>>
> >>>
> >>> --
> >>>
> >>> Abdelhakim Deneche
> >>>
> >>> Software Engineer
> >>>
> >>>  <http://www.mapr.com/>
> >>>
> >>>
> >>> Now Available - Free Hadoop On-Demand Training
> >>> <
> >>
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >>>
> >>>
> >>
> >>
> >>
> >> --
> >>
> >> Abdelhakim Deneche
> >>
> >> Software Engineer
> >>
> >>  <http://www.mapr.com/>
> >>
> >>
> >> Now Available - Free Hadoop On-Demand Training
> >> <
> >>
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >>>
> >>
>
>

Re: UDF is not recognized by drill / Validation Error

Posted by Julian Feinauer <ju...@web.de>.

Dear all,

thank you very much for all your replies.
I tried everything but it is still not working.

1. I copy both files (classes and sources) in the /jars/3rdparty directory
2. I restarted the drillbit after this (I use only one drillbit and drill-conf both running on my local machine)
3. I changed the class to a static subclass
4. I have the drill-module.conf in my ressources
4. The error appears in the drillbit.log as soon as i call the udf because it is not recognized by drill.
On startup drillbit.log states all the packages and jars that are scanned and my custom jar is not listed there.
Therefore I think it is a problem with the class loader or something related?

Could this be possible?

Greetings
Julian

> Am 20.05.2016 um 14:36 schrieb Tugdual Grall <tu...@gmail.com>:
> 
> Hi
> 
> Be sure you deploy on each nodes, 2 jars:
> - the jar containing the classes
> - the jar contains the sources
> 
> The POM.xml in the simple function examples contains the maven
> configuration to generate these 2 files, be sure you have the same in your
> project:
> https://github.com/tgrall/drill-simple-mask-function/blob/master/pom.xml#L24-L37
> 
> and you have restarted the drillbit
> 
> Regards
> Tug
> @tgrall
> 
> 
> 
> On Fri, May 20, 2016 at 12:34 AM, Abdel Hakim Deneche <adeneche@maprtech.com
>> wrote:
> 
>> the example I gave you was incomplete, here is what I meant to send:
>> 
>> public class MyUDF {
>> 
>>   @FunctionTemplate(name = „myaddints", scope = FunctionTemplate.
>>   FunctionScope.SIMPLE, nulls =
>> FunctionTemplate.NullHandling.NULL_IF_NULL)
>>   public *static *class IntIntAdd implements DrillSimpleFunc {
>>      ...
>>   }
>> 
>> }
>> 
>> 
>> On Thu, May 19, 2016 at 3:33 PM, Abdel Hakim Deneche <
>> adeneche@maprtech.com>
>> wrote:
>> 
>>> Hey Julian,
>>> 
>>> one more thing you could try out: declare the UDF as a static class
>> inside
>>> another class:
>>> 
>>> public class MyUDF {
>>> 
>>>   @FunctionTemplate(name = „myaddints", scope = FunctionTemplate.
>>>   FunctionScope.SIMPLE, nulls = FunctionTemplate.NullHandling.
>>> NULL_IF_NULL)
>>>   public class IntIntAdd implements DrillSimpleFunc {
>>>      ...
>>>   }
>>> 
>>> }
>>> 
>>> Take a look at the following page to see an examples of UDFs:
>>> http://drill.apache.org/docs/custom-function-interfaces/
>>> 
>>> If this doesn't work check the drillbit log, it should print an error
>>> message when it's starting up if something's wrong with your UDF.
>>> 
>>> Thanks
>>> 
>>> 
>>> On Thu, May 19, 2016 at 3:31 AM, Julian Feinauer <julian.feinauer@web.de
>>> 
>>> wrote:
>>> 
>>>> Dear folks,
>>>> 
>>>> I’m currently experimenting with user defined functions in drill but I’m
>>>> not able to get them to work on my drillbits.
>>>> I always get the error: Error: VALIDATION ERROR: From line 1, column 8
>> to
>>>> line 1, column 41: No match found for function signature
>> myaddints(<ANY>,
>>>> <ANY>).
>>>> 
>>>> I already went through all the tips I found in the mailing list.
>>>> The jar contains a drill-module.conf with the content:
>>>> drill.classpath.scanning.packages += "org.julian"
>>>> And the UDF is defined as:
>>>> package org.julian;
>>>> 
>>>> import ...
>>>> 
>>>> @FunctionTemplate(name = „myaddints", scope =
>>>> FunctionTemplate.FunctionScope.SIMPLE, nulls =
>>>> FunctionTemplate.NullHandling.NULL_IF_NULL)
>>>> public class IntIntAdd implements DrillSimpleFunc {
>>>> 
>>>>    @Param
>>>>    IntHolder in1;
>>>> 
>>>>    @Param
>>>>    IntHolder in2;
>>>> 
>>>>    @Output
>>>>    IntHolder out;
>>>> 
>>>>    public void setup() {};
>>>> 
>>>>    public void eval() {
>>>>        out.value = in1.value + in2.value;
>>>>    }
>>>> 
>>>> }
>>>> I tried to find the bug using the logs and encountered the following
>>>> messages from the drillbit.log:
>>>> Base Configuration:
>>>>        -
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-common-1.6.0.jar!/drill-default.conf
>>>> 
>>>> Intermediate Configuration and Plugin files, in order of precedence:
>>>>        -
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-gis-1.6.0.jar!/drill-module.conf
>>>>        -
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-logical-1.6.0.jar!/drill-module.conf
>>>>        -
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-kudu-storage-1.6.0.jar!/drill-module.conf
>>>>        -
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/3rdparty/drill-memory-base-1.6.0.jar!/drill-module.conf
>>>>        -
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-hive-exec-shaded-1.6.0.jar!/drill-module.conf
>>>>        -
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-memory-base-1.6.0.jar!/drill-module.conf
>>>>        -
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-mongo-storage-1.6.0.jar!/drill-module.conf
>>>>        -
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-java-exec-1.6.0.jar!/drill-module.conf
>>>>        -
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-common-1.6.0.jar!/drill-module.conf
>>>>        -
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-storage-hbase-1.6.0.jar!/drill-module.conf
>>>>        -
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-jdbc-storage-1.6.0.jar!/drill-module.conf
>>>>        -
>>>> 
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-storage-hive-core-1.6.0.jar!/drill-module.conf
>>>> 
>>>> which let’s me think that my jar file (drillUDF-1.0-SNAPSHOT.jar) is not
>>>> scanned altough it is located in jars/3rdparty.
>>>> 
>>>> Am I doing something wrong or am I missing something or has anyone a
>> good
>>>> hint for me?
>>>> 
>>>> Thank you already!
>>>> Julian
>>> 
>>> 
>>> 
>>> 
>>> --
>>> 
>>> Abdelhakim Deneche
>>> 
>>> Software Engineer
>>> 
>>>  <http://www.mapr.com/>
>>> 
>>> 
>>> Now Available - Free Hadoop On-Demand Training
>>> <
>> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
>>> 
>>> 
>> 
>> 
>> 
>> --
>> 
>> Abdelhakim Deneche
>> 
>> Software Engineer
>> 
>>  <http://www.mapr.com/>
>> 
>> 
>> Now Available - Free Hadoop On-Demand Training
>> <
>> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
>>> 
>>

Re: UDF is not recognized by drill / Validation Error

Posted by Tugdual Grall <tu...@gmail.com>.

Hi

Be sure you deploy on each nodes, 2 jars:
 - the jar containing the classes
 - the jar contains the sources

The POM.xml in the simple function examples contains the maven
configuration to generate these 2 files, be sure you have the same in your
project:
https://github.com/tgrall/drill-simple-mask-function/blob/master/pom.xml#L24-L37

and you have restarted the drillbit

Regards
Tug
@tgrall



On Fri, May 20, 2016 at 12:34 AM, Abdel Hakim Deneche <adeneche@maprtech.com
> wrote:

> the example I gave you was incomplete, here is what I meant to send:
>
> public class MyUDF {
>
>    @FunctionTemplate(name = „myaddints", scope = FunctionTemplate.
>    FunctionScope.SIMPLE, nulls =
> FunctionTemplate.NullHandling.NULL_IF_NULL)
>    public *static *class IntIntAdd implements DrillSimpleFunc {
>       ...
>    }
>
> }
>
>
> On Thu, May 19, 2016 at 3:33 PM, Abdel Hakim Deneche <
> adeneche@maprtech.com>
> wrote:
>
> > Hey Julian,
> >
> > one more thing you could try out: declare the UDF as a static class
> inside
> > another class:
> >
> > public class MyUDF {
> >
> >    @FunctionTemplate(name = „myaddints", scope = FunctionTemplate.
> >    FunctionScope.SIMPLE, nulls = FunctionTemplate.NullHandling.
> > NULL_IF_NULL)
> >    public class IntIntAdd implements DrillSimpleFunc {
> >       ...
> >    }
> >
> > }
> >
> > Take a look at the following page to see an examples of UDFs:
> > http://drill.apache.org/docs/custom-function-interfaces/
> >
> > If this doesn't work check the drillbit log, it should print an error
> > message when it's starting up if something's wrong with your UDF.
> >
> > Thanks
> >
> >
> > On Thu, May 19, 2016 at 3:31 AM, Julian Feinauer <julian.feinauer@web.de
> >
> > wrote:
> >
> >> Dear folks,
> >>
> >> I’m currently experimenting with user defined functions in drill but I’m
> >> not able to get them to work on my drillbits.
> >> I always get the error: Error: VALIDATION ERROR: From line 1, column 8
> to
> >> line 1, column 41: No match found for function signature
> myaddints(<ANY>,
> >> <ANY>).
> >>
> >> I already went through all the tips I found in the mailing list.
> >> The jar contains a drill-module.conf with the content:
> >> drill.classpath.scanning.packages += "org.julian"
> >> And the UDF is defined as:
> >> package org.julian;
> >>
> >> import ...
> >>
> >> @FunctionTemplate(name = „myaddints", scope =
> >> FunctionTemplate.FunctionScope.SIMPLE, nulls =
> >> FunctionTemplate.NullHandling.NULL_IF_NULL)
> >> public class IntIntAdd implements DrillSimpleFunc {
> >>
> >>     @Param
> >>     IntHolder in1;
> >>
> >>     @Param
> >>     IntHolder in2;
> >>
> >>     @Output
> >>     IntHolder out;
> >>
> >>     public void setup() {};
> >>
> >>     public void eval() {
> >>         out.value = in1.value + in2.value;
> >>     }
> >>
> >> }
> >> I tried to find the bug using the logs and encountered the following
> >> messages from the drillbit.log:
> >> Base Configuration:
> >>         -
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-common-1.6.0.jar!/drill-default.conf
> >>
> >> Intermediate Configuration and Plugin files, in order of precedence:
> >>         -
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-gis-1.6.0.jar!/drill-module.conf
> >>         -
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-logical-1.6.0.jar!/drill-module.conf
> >>         -
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-kudu-storage-1.6.0.jar!/drill-module.conf
> >>         -
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/3rdparty/drill-memory-base-1.6.0.jar!/drill-module.conf
> >>         -
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-hive-exec-shaded-1.6.0.jar!/drill-module.conf
> >>         -
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-memory-base-1.6.0.jar!/drill-module.conf
> >>         -
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-mongo-storage-1.6.0.jar!/drill-module.conf
> >>         -
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-java-exec-1.6.0.jar!/drill-module.conf
> >>         -
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-common-1.6.0.jar!/drill-module.conf
> >>         -
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-storage-hbase-1.6.0.jar!/drill-module.conf
> >>         -
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-jdbc-storage-1.6.0.jar!/drill-module.conf
> >>         -
> >>
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-storage-hive-core-1.6.0.jar!/drill-module.conf
> >>
> >> which let’s me think that my jar file (drillUDF-1.0-SNAPSHOT.jar) is not
> >> scanned altough it is located in jars/3rdparty.
> >>
> >> Am I doing something wrong or am I missing something or has anyone a
> good
> >> hint for me?
> >>
> >> Thank you already!
> >> Julian
> >
> >
> >
> >
> > --
> >
> > Abdelhakim Deneche
> >
> > Software Engineer
> >
> >   <http://www.mapr.com/>
> >
> >
> > Now Available - Free Hadoop On-Demand Training
> > <
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >
> >
>
>
>
> --
>
> Abdelhakim Deneche
>
> Software Engineer
>
>   <http://www.mapr.com/>
>
>
> Now Available - Free Hadoop On-Demand Training
> <
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> >
>

Re: UDF is not recognized by drill / Validation Error

Posted by Abdel Hakim Deneche <ad...@maprtech.com>.

the example I gave you was incomplete, here is what I meant to send:

public class MyUDF {

   @FunctionTemplate(name = „myaddints", scope = FunctionTemplate.
   FunctionScope.SIMPLE, nulls = FunctionTemplate.NullHandling.NULL_IF_NULL)
   public *static *class IntIntAdd implements DrillSimpleFunc {
      ...
   }

}


On Thu, May 19, 2016 at 3:33 PM, Abdel Hakim Deneche <ad...@maprtech.com>
wrote:

> Hey Julian,
>
> one more thing you could try out: declare the UDF as a static class inside
> another class:
>
> public class MyUDF {
>
>    @FunctionTemplate(name = „myaddints", scope = FunctionTemplate.
>    FunctionScope.SIMPLE, nulls = FunctionTemplate.NullHandling.
> NULL_IF_NULL)
>    public class IntIntAdd implements DrillSimpleFunc {
>       ...
>    }
>
> }
>
> Take a look at the following page to see an examples of UDFs:
> http://drill.apache.org/docs/custom-function-interfaces/
>
> If this doesn't work check the drillbit log, it should print an error
> message when it's starting up if something's wrong with your UDF.
>
> Thanks
>
>
> On Thu, May 19, 2016 at 3:31 AM, Julian Feinauer <ju...@web.de>
> wrote:
>
>> Dear folks,
>>
>> I’m currently experimenting with user defined functions in drill but I’m
>> not able to get them to work on my drillbits.
>> I always get the error: Error: VALIDATION ERROR: From line 1, column 8 to
>> line 1, column 41: No match found for function signature myaddints(<ANY>,
>> <ANY>).
>>
>> I already went through all the tips I found in the mailing list.
>> The jar contains a drill-module.conf with the content:
>> drill.classpath.scanning.packages += "org.julian"
>> And the UDF is defined as:
>> package org.julian;
>>
>> import ...
>>
>> @FunctionTemplate(name = „myaddints", scope =
>> FunctionTemplate.FunctionScope.SIMPLE, nulls =
>> FunctionTemplate.NullHandling.NULL_IF_NULL)
>> public class IntIntAdd implements DrillSimpleFunc {
>>
>>     @Param
>>     IntHolder in1;
>>
>>     @Param
>>     IntHolder in2;
>>
>>     @Output
>>     IntHolder out;
>>
>>     public void setup() {};
>>
>>     public void eval() {
>>         out.value = in1.value + in2.value;
>>     }
>>
>> }
>> I tried to find the bug using the logs and encountered the following
>> messages from the drillbit.log:
>> Base Configuration:
>>         -
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-common-1.6.0.jar!/drill-default.conf
>>
>> Intermediate Configuration and Plugin files, in order of precedence:
>>         -
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-gis-1.6.0.jar!/drill-module.conf
>>         -
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-logical-1.6.0.jar!/drill-module.conf
>>         -
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-kudu-storage-1.6.0.jar!/drill-module.conf
>>         -
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/3rdparty/drill-memory-base-1.6.0.jar!/drill-module.conf
>>         -
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-hive-exec-shaded-1.6.0.jar!/drill-module.conf
>>         -
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-memory-base-1.6.0.jar!/drill-module.conf
>>         -
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-mongo-storage-1.6.0.jar!/drill-module.conf
>>         -
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-java-exec-1.6.0.jar!/drill-module.conf
>>         -
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-common-1.6.0.jar!/drill-module.conf
>>         -
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-storage-hbase-1.6.0.jar!/drill-module.conf
>>         -
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-jdbc-storage-1.6.0.jar!/drill-module.conf
>>         -
>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-storage-hive-core-1.6.0.jar!/drill-module.conf
>>
>> which let’s me think that my jar file (drillUDF-1.0-SNAPSHOT.jar) is not
>> scanned altough it is located in jars/3rdparty.
>>
>> Am I doing something wrong or am I missing something or has anyone a good
>> hint for me?
>>
>> Thank you already!
>> Julian
>
>
>
>
> --
>
> Abdelhakim Deneche
>
> Software Engineer
>
>   <http://www.mapr.com/>
>
>
> Now Available - Free Hadoop On-Demand Training
> <http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available>
>



-- 

Abdelhakim Deneche

Software Engineer

  <http://www.mapr.com/>


Now Available - Free Hadoop On-Demand Training
<http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available>

Re: UDF is not recognized by drill / Validation Error

Posted by Abdel Hakim Deneche <ad...@maprtech.com>.

Hey Julian,

one more thing you could try out: declare the UDF as a static class inside
another class:

public class MyUDF {

   @FunctionTemplate(name = „myaddints", scope = FunctionTemplate.
   FunctionScope.SIMPLE, nulls = FunctionTemplate.NullHandling.NULL_IF_NULL)
   public class IntIntAdd implements DrillSimpleFunc {
      ...
   }

}

Take a look at the following page to see an examples of UDFs:
http://drill.apache.org/docs/custom-function-interfaces/

If this doesn't work check the drillbit log, it should print an error
message when it's starting up if something's wrong with your UDF.

Thanks


On Thu, May 19, 2016 at 3:31 AM, Julian Feinauer <ju...@web.de>
wrote:

> Dear folks,
>
> I’m currently experimenting with user defined functions in drill but I’m
> not able to get them to work on my drillbits.
> I always get the error: Error: VALIDATION ERROR: From line 1, column 8 to
> line 1, column 41: No match found for function signature myaddints(<ANY>,
> <ANY>).
>
> I already went through all the tips I found in the mailing list.
> The jar contains a drill-module.conf with the content:
> drill.classpath.scanning.packages += "org.julian"
> And the UDF is defined as:
> package org.julian;
>
> import ...
>
> @FunctionTemplate(name = „myaddints", scope =
> FunctionTemplate.FunctionScope.SIMPLE, nulls =
> FunctionTemplate.NullHandling.NULL_IF_NULL)
> public class IntIntAdd implements DrillSimpleFunc {
>
>     @Param
>     IntHolder in1;
>
>     @Param
>     IntHolder in2;
>
>     @Output
>     IntHolder out;
>
>     public void setup() {};
>
>     public void eval() {
>         out.value = in1.value + in2.value;
>     }
>
> }
> I tried to find the bug using the logs and encountered the following
> messages from the drillbit.log:
> Base Configuration:
>         -
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-common-1.6.0.jar!/drill-default.conf
>
> Intermediate Configuration and Plugin files, in order of precedence:
>         -
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-gis-1.6.0.jar!/drill-module.conf
>         -
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-logical-1.6.0.jar!/drill-module.conf
>         -
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-kudu-storage-1.6.0.jar!/drill-module.conf
>         -
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/3rdparty/drill-memory-base-1.6.0.jar!/drill-module.conf
>         -
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-hive-exec-shaded-1.6.0.jar!/drill-module.conf
>         -
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-memory-base-1.6.0.jar!/drill-module.conf
>         -
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-mongo-storage-1.6.0.jar!/drill-module.conf
>         -
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-java-exec-1.6.0.jar!/drill-module.conf
>         -
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-common-1.6.0.jar!/drill-module.conf
>         -
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-storage-hbase-1.6.0.jar!/drill-module.conf
>         -
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-jdbc-storage-1.6.0.jar!/drill-module.conf
>         -
> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-storage-hive-core-1.6.0.jar!/drill-module.conf
>
> which let’s me think that my jar file (drillUDF-1.0-SNAPSHOT.jar) is not
> scanned altough it is located in jars/3rdparty.
>
> Am I doing something wrong or am I missing something or has anyone a good
> hint for me?
>
> Thank you already!
> Julian




-- 

Abdelhakim Deneche

Software Engineer

  <http://www.mapr.com/>


Now Available - Free Hadoop On-Demand Training
<http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available>