You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@drill.apache.org by Stefán Baxter <st...@activitystream.com> on 2015/11/09 19:56:29 UTC
UDFs and 1.3
Hi,
I have a small set of UDFs that I have been running with Drill 1.1/1.2
which I'm trying to get working with 1.3 to no avail.
It's as if the library is no picked up correctly even though the error I
get indicates a missing function signature (variant):
Error: VALIDATION ERROR: From line 1, column 129 to line 1, column 164: No
match found for function signature roundTimeStamp(<NUMERIC>, <CHARACTER>)
Have the rules for UDFs changed at all for the 1.3 snapshot?
Regards,
-Stefan
Re: UDFs and 1.3
Posted by Julien Le Dem <ju...@dremio.com>.
Hi Stefan,
This is correct : you need to add your package to drill.classpath.scanning.
packages
However, you don't need to add your classes to drill.classpath.scanning.
base.classes
This is because they extend the base classes that drill already defines
here:
https://github.com/apache/drill/blob/e78e28661e921c6bd07ba7bb95d91f9adeb5a247/exec/java-exec/src/main/resources/drill-module.conf#L22
The Hive plugin configuration that Jacques sent as an example defines new
base classes because it uses the same mechanism to scan for Hive UDFs.
On Tue, Nov 10, 2015 at 2:46 AM, Stefán Baxter <st...@activitystream.com>
wrote:
> Hi,
>
> I re did everything and update the drill-module.conf with this:
>
> drill {
>
> classpath.scanning {
> base.classes : ${?drill.classpath.scanning.base.classes} [
> com.activitystream.drill.DirUtils,
> com.activitystream.drill.IPUtils,
> com.activitystream.drill.ListUtils,
> com.activitystream.drill.TimestampUtils
> ],
> packages : ${?drill.classpath.scanning.packages} [
> com.activitystream.drill
> ]
> }
> }
>
> The previous one had a spelling error in it.
>
> Thank you, everything is running as it should now
>
> Regards, -Stefán
>
>
> On Mon, Nov 9, 2015 at 8:13 PM, Stefán Baxter <st...@activitystream.com>
> wrote:
>
> > So, here is my setup:
> >
> > - I have two .jar files in ./jars/3rdpary
> > - one containing compiled classes and the other containing source
> > (Generated by source plugin)
> > - both jars have contained an empty drill-module.conf (was encouraged
> > for 1.1) (did not work)
> > - both jars have contained a drill-module.conf with this (which tries
> > to emulate the example (that I do not understand)):
> >
> > drill {
> > classpath.scanning {
> > base.classes : ${?drill.classpath.scanning.base.classes} [
> > com.acticitystream.drill.DirUtils,
> > com.acticitystream.drill.IPUtils,
> > com.acticitystream.drill.ListUtils,
> > com.acticitystream.drill.TimestampUtils
> > ],
> >
> > packages : ${?drill.classpath.scanning.packages} [
> > com.acticitystream.drill
> > ]
> > }
> > }
> >
> > - conf/drill-override.conf contains this:
> >
> >
> drill.logical.function.package+=["org.apache.drill.exec.expr.fn.impl","com.activitystream.drill"]
> > - worked in 1.1 / 1.2 but not in 1.3
> >
> > Any clues?
> >
> > Regards,
> > -Stefan
> >
> >
> >
> > On Mon, Nov 9, 2015 at 7:51 PM, Jacques Nadeau <ja...@dremio.com>
> wrote:
> >
> >> I don't think the compilation needs to be changed as there were no
> >> interface changes.
> >>
> >> However, we did get much more strict about making sure your
> >> drill-module.conf points at the right package names.
> >>
> >> Note here, specifically the addition of package names:
> >>
> >>
> >>
> https://github.com/apache/drill/blob/master/contrib/storage-hive/core/src/main/resources/drill-module.conf
> >>
> >>
> >>
> >> --
> >> Jacques Nadeau
> >> CTO and Co-Founder, Dremio
> >>
> >> On Mon, Nov 9, 2015 at 11:30 AM, Stefán Baxter <
> stefan@activitystream.com
> >> >
> >> wrote:
> >>
> >> > Hi,
> >> >
> >> > Now they are but the outcome remains the same.
> >> >
> >> > Any additional pointers?
> >> >
> >> > Regards,
> >> > -Stefan
> >> >
> >> > On Mon, Nov 9, 2015 at 7:21 PM, Stefán Baxter <
> >> stefan@activitystream.com>
> >> > wrote:
> >> >
> >> > > Hi Nathan,
> >> > >
> >> > > thank you for a prompt reply.
> >> > >
> >> > > I thought the were but they are in fact compiled with the 1.2
> >> dependency
> >> > > (from maven) but they are placed in the correct place :)
> >> > >
> >> > > I will correct this and continue.
> >> > >
> >> > > Regards,
> >> > > -Stefán
> >> > >
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > On Mon, Nov 9, 2015 at 7:15 PM, Nathan Griffith <
> ngriffith@dremio.com
> >> >
> >> > > wrote:
> >> > >
> >> > >> Hi Stefan,
> >> > >>
> >> > >> A couple things:
> >> > >>
> >> > >> 1.) Have the UDFs been recompiled using the 1.3 dependency?
> >> > >>
> >> > >> 2.) Have the jars for the UDFs definitely been placed in the new
> 1.3
> >> > >> install's jars/3rdparty directory?
> >> > >>
> >> > >> Forgive me if these suggestions are too rudimentary, but it's
> always
> >> > >> nice to start with some sanity checks :)
> >> > >>
> >> > >> Best,
> >> > >> Nathan
> >> > >>
> >> > >> On Mon, Nov 9, 2015 at 10:56 AM, Stefán Baxter
> >> > >> <st...@activitystream.com> wrote:
> >> > >> > Hi,
> >> > >> >
> >> > >> > I have a small set of UDFs that I have been running with Drill
> >> 1.1/1.2
> >> > >> > which I'm trying to get working with 1.3 to no avail.
> >> > >> >
> >> > >> > It's as if the library is no picked up correctly even though the
> >> > error I
> >> > >> > get indicates a missing function signature (variant):
> >> > >> >
> >> > >> > Error: VALIDATION ERROR: From line 1, column 129 to line 1,
> column
> >> > 164:
> >> > >> No
> >> > >> > match found for function signature roundTimeStamp(<NUMERIC>,
> >> > >> <CHARACTER>)
> >> > >> >
> >> > >> > Have the rules for UDFs changed at all for the 1.3 snapshot?
> >> > >> >
> >> > >> > Regards,
> >> > >> > -Stefan
> >> > >>
> >> > >
> >> > >
> >> >
> >>
> >
> >
>
--
Julien
Re: UDFs and 1.3
Posted by Stefán Baxter <st...@activitystream.com>.
Hi,
I re did everything and update the drill-module.conf with this:
drill {
classpath.scanning {
base.classes : ${?drill.classpath.scanning.base.classes} [
com.activitystream.drill.DirUtils,
com.activitystream.drill.IPUtils,
com.activitystream.drill.ListUtils,
com.activitystream.drill.TimestampUtils
],
packages : ${?drill.classpath.scanning.packages} [
com.activitystream.drill
]
}
}
The previous one had a spelling error in it.
Thank you, everything is running as it should now
Regards, -Stefán
On Mon, Nov 9, 2015 at 8:13 PM, Stefán Baxter <st...@activitystream.com>
wrote:
> So, here is my setup:
>
> - I have two .jar files in ./jars/3rdpary
> - one containing compiled classes and the other containing source
> (Generated by source plugin)
> - both jars have contained an empty drill-module.conf (was encouraged
> for 1.1) (did not work)
> - both jars have contained a drill-module.conf with this (which tries
> to emulate the example (that I do not understand)):
>
> drill {
> classpath.scanning {
> base.classes : ${?drill.classpath.scanning.base.classes} [
> com.acticitystream.drill.DirUtils,
> com.acticitystream.drill.IPUtils,
> com.acticitystream.drill.ListUtils,
> com.acticitystream.drill.TimestampUtils
> ],
>
> packages : ${?drill.classpath.scanning.packages} [
> com.acticitystream.drill
> ]
> }
> }
>
> - conf/drill-override.conf contains this:
>
> drill.logical.function.package+=["org.apache.drill.exec.expr.fn.impl","com.activitystream.drill"]
> - worked in 1.1 / 1.2 but not in 1.3
>
> Any clues?
>
> Regards,
> -Stefan
>
>
>
> On Mon, Nov 9, 2015 at 7:51 PM, Jacques Nadeau <ja...@dremio.com> wrote:
>
>> I don't think the compilation needs to be changed as there were no
>> interface changes.
>>
>> However, we did get much more strict about making sure your
>> drill-module.conf points at the right package names.
>>
>> Note here, specifically the addition of package names:
>>
>>
>> https://github.com/apache/drill/blob/master/contrib/storage-hive/core/src/main/resources/drill-module.conf
>>
>>
>>
>> --
>> Jacques Nadeau
>> CTO and Co-Founder, Dremio
>>
>> On Mon, Nov 9, 2015 at 11:30 AM, Stefán Baxter <stefan@activitystream.com
>> >
>> wrote:
>>
>> > Hi,
>> >
>> > Now they are but the outcome remains the same.
>> >
>> > Any additional pointers?
>> >
>> > Regards,
>> > -Stefan
>> >
>> > On Mon, Nov 9, 2015 at 7:21 PM, Stefán Baxter <
>> stefan@activitystream.com>
>> > wrote:
>> >
>> > > Hi Nathan,
>> > >
>> > > thank you for a prompt reply.
>> > >
>> > > I thought the were but they are in fact compiled with the 1.2
>> dependency
>> > > (from maven) but they are placed in the correct place :)
>> > >
>> > > I will correct this and continue.
>> > >
>> > > Regards,
>> > > -Stefán
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > On Mon, Nov 9, 2015 at 7:15 PM, Nathan Griffith <ngriffith@dremio.com
>> >
>> > > wrote:
>> > >
>> > >> Hi Stefan,
>> > >>
>> > >> A couple things:
>> > >>
>> > >> 1.) Have the UDFs been recompiled using the 1.3 dependency?
>> > >>
>> > >> 2.) Have the jars for the UDFs definitely been placed in the new 1.3
>> > >> install's jars/3rdparty directory?
>> > >>
>> > >> Forgive me if these suggestions are too rudimentary, but it's always
>> > >> nice to start with some sanity checks :)
>> > >>
>> > >> Best,
>> > >> Nathan
>> > >>
>> > >> On Mon, Nov 9, 2015 at 10:56 AM, Stefán Baxter
>> > >> <st...@activitystream.com> wrote:
>> > >> > Hi,
>> > >> >
>> > >> > I have a small set of UDFs that I have been running with Drill
>> 1.1/1.2
>> > >> > which I'm trying to get working with 1.3 to no avail.
>> > >> >
>> > >> > It's as if the library is no picked up correctly even though the
>> > error I
>> > >> > get indicates a missing function signature (variant):
>> > >> >
>> > >> > Error: VALIDATION ERROR: From line 1, column 129 to line 1, column
>> > 164:
>> > >> No
>> > >> > match found for function signature roundTimeStamp(<NUMERIC>,
>> > >> <CHARACTER>)
>> > >> >
>> > >> > Have the rules for UDFs changed at all for the 1.3 snapshot?
>> > >> >
>> > >> > Regards,
>> > >> > -Stefan
>> > >>
>> > >
>> > >
>> >
>>
>
>
Re: UDFs and 1.3
Posted by Stefán Baxter <st...@activitystream.com>.
So, here is my setup:
- I have two .jar files in ./jars/3rdpary
- one containing compiled classes and the other containing source
(Generated by source plugin)
- both jars have contained an empty drill-module.conf (was encouraged
for 1.1) (did not work)
- both jars have contained a drill-module.conf with this (which tries to
emulate the example (that I do not understand)):
drill {
classpath.scanning {
base.classes : ${?drill.classpath.scanning.base.classes} [
com.acticitystream.drill.DirUtils,
com.acticitystream.drill.IPUtils,
com.acticitystream.drill.ListUtils,
com.acticitystream.drill.TimestampUtils
],
packages : ${?drill.classpath.scanning.packages} [
com.acticitystream.drill
]
}
}
- conf/drill-override.conf contains this:
drill.logical.function.package+=["org.apache.drill.exec.expr.fn.impl","com.activitystream.drill"]
- worked in 1.1 / 1.2 but not in 1.3
Any clues?
Regards,
-Stefan
On Mon, Nov 9, 2015 at 7:51 PM, Jacques Nadeau <ja...@dremio.com> wrote:
> I don't think the compilation needs to be changed as there were no
> interface changes.
>
> However, we did get much more strict about making sure your
> drill-module.conf points at the right package names.
>
> Note here, specifically the addition of package names:
>
>
> https://github.com/apache/drill/blob/master/contrib/storage-hive/core/src/main/resources/drill-module.conf
>
>
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Mon, Nov 9, 2015 at 11:30 AM, Stefán Baxter <st...@activitystream.com>
> wrote:
>
> > Hi,
> >
> > Now they are but the outcome remains the same.
> >
> > Any additional pointers?
> >
> > Regards,
> > -Stefan
> >
> > On Mon, Nov 9, 2015 at 7:21 PM, Stefán Baxter <stefan@activitystream.com
> >
> > wrote:
> >
> > > Hi Nathan,
> > >
> > > thank you for a prompt reply.
> > >
> > > I thought the were but they are in fact compiled with the 1.2
> dependency
> > > (from maven) but they are placed in the correct place :)
> > >
> > > I will correct this and continue.
> > >
> > > Regards,
> > > -Stefán
> > >
> > >
> > >
> > >
> > >
> > > On Mon, Nov 9, 2015 at 7:15 PM, Nathan Griffith <ng...@dremio.com>
> > > wrote:
> > >
> > >> Hi Stefan,
> > >>
> > >> A couple things:
> > >>
> > >> 1.) Have the UDFs been recompiled using the 1.3 dependency?
> > >>
> > >> 2.) Have the jars for the UDFs definitely been placed in the new 1.3
> > >> install's jars/3rdparty directory?
> > >>
> > >> Forgive me if these suggestions are too rudimentary, but it's always
> > >> nice to start with some sanity checks :)
> > >>
> > >> Best,
> > >> Nathan
> > >>
> > >> On Mon, Nov 9, 2015 at 10:56 AM, Stefán Baxter
> > >> <st...@activitystream.com> wrote:
> > >> > Hi,
> > >> >
> > >> > I have a small set of UDFs that I have been running with Drill
> 1.1/1.2
> > >> > which I'm trying to get working with 1.3 to no avail.
> > >> >
> > >> > It's as if the library is no picked up correctly even though the
> > error I
> > >> > get indicates a missing function signature (variant):
> > >> >
> > >> > Error: VALIDATION ERROR: From line 1, column 129 to line 1, column
> > 164:
> > >> No
> > >> > match found for function signature roundTimeStamp(<NUMERIC>,
> > >> <CHARACTER>)
> > >> >
> > >> > Have the rules for UDFs changed at all for the 1.3 snapshot?
> > >> >
> > >> > Regards,
> > >> > -Stefan
> > >>
> > >
> > >
> >
>
Re: UDFs and 1.3
Posted by Jacques Nadeau <ja...@dremio.com>.
I don't think the compilation needs to be changed as there were no
interface changes.
However, we did get much more strict about making sure your
drill-module.conf points at the right package names.
Note here, specifically the addition of package names:
https://github.com/apache/drill/blob/master/contrib/storage-hive/core/src/main/resources/drill-module.conf
--
Jacques Nadeau
CTO and Co-Founder, Dremio
On Mon, Nov 9, 2015 at 11:30 AM, Stefán Baxter <st...@activitystream.com>
wrote:
> Hi,
>
> Now they are but the outcome remains the same.
>
> Any additional pointers?
>
> Regards,
> -Stefan
>
> On Mon, Nov 9, 2015 at 7:21 PM, Stefán Baxter <st...@activitystream.com>
> wrote:
>
> > Hi Nathan,
> >
> > thank you for a prompt reply.
> >
> > I thought the were but they are in fact compiled with the 1.2 dependency
> > (from maven) but they are placed in the correct place :)
> >
> > I will correct this and continue.
> >
> > Regards,
> > -Stefán
> >
> >
> >
> >
> >
> > On Mon, Nov 9, 2015 at 7:15 PM, Nathan Griffith <ng...@dremio.com>
> > wrote:
> >
> >> Hi Stefan,
> >>
> >> A couple things:
> >>
> >> 1.) Have the UDFs been recompiled using the 1.3 dependency?
> >>
> >> 2.) Have the jars for the UDFs definitely been placed in the new 1.3
> >> install's jars/3rdparty directory?
> >>
> >> Forgive me if these suggestions are too rudimentary, but it's always
> >> nice to start with some sanity checks :)
> >>
> >> Best,
> >> Nathan
> >>
> >> On Mon, Nov 9, 2015 at 10:56 AM, Stefán Baxter
> >> <st...@activitystream.com> wrote:
> >> > Hi,
> >> >
> >> > I have a small set of UDFs that I have been running with Drill 1.1/1.2
> >> > which I'm trying to get working with 1.3 to no avail.
> >> >
> >> > It's as if the library is no picked up correctly even though the
> error I
> >> > get indicates a missing function signature (variant):
> >> >
> >> > Error: VALIDATION ERROR: From line 1, column 129 to line 1, column
> 164:
> >> No
> >> > match found for function signature roundTimeStamp(<NUMERIC>,
> >> <CHARACTER>)
> >> >
> >> > Have the rules for UDFs changed at all for the 1.3 snapshot?
> >> >
> >> > Regards,
> >> > -Stefan
> >>
> >
> >
>
Re: UDFs and 1.3
Posted by Stefán Baxter <st...@activitystream.com>.
Hi,
Now they are but the outcome remains the same.
Any additional pointers?
Regards,
-Stefan
On Mon, Nov 9, 2015 at 7:21 PM, Stefán Baxter <st...@activitystream.com>
wrote:
> Hi Nathan,
>
> thank you for a prompt reply.
>
> I thought the were but they are in fact compiled with the 1.2 dependency
> (from maven) but they are placed in the correct place :)
>
> I will correct this and continue.
>
> Regards,
> -Stefán
>
>
>
>
>
> On Mon, Nov 9, 2015 at 7:15 PM, Nathan Griffith <ng...@dremio.com>
> wrote:
>
>> Hi Stefan,
>>
>> A couple things:
>>
>> 1.) Have the UDFs been recompiled using the 1.3 dependency?
>>
>> 2.) Have the jars for the UDFs definitely been placed in the new 1.3
>> install's jars/3rdparty directory?
>>
>> Forgive me if these suggestions are too rudimentary, but it's always
>> nice to start with some sanity checks :)
>>
>> Best,
>> Nathan
>>
>> On Mon, Nov 9, 2015 at 10:56 AM, Stefán Baxter
>> <st...@activitystream.com> wrote:
>> > Hi,
>> >
>> > I have a small set of UDFs that I have been running with Drill 1.1/1.2
>> > which I'm trying to get working with 1.3 to no avail.
>> >
>> > It's as if the library is no picked up correctly even though the error I
>> > get indicates a missing function signature (variant):
>> >
>> > Error: VALIDATION ERROR: From line 1, column 129 to line 1, column 164:
>> No
>> > match found for function signature roundTimeStamp(<NUMERIC>,
>> <CHARACTER>)
>> >
>> > Have the rules for UDFs changed at all for the 1.3 snapshot?
>> >
>> > Regards,
>> > -Stefan
>>
>
>
Re: UDFs and 1.3
Posted by Stefán Baxter <st...@activitystream.com>.
Hi Nathan,
thank you for a prompt reply.
I thought the were but they are in fact compiled with the 1.2 dependency
(from maven) but they are placed in the correct place :)
I will correct this and continue.
Regards,
-Stefán
On Mon, Nov 9, 2015 at 7:15 PM, Nathan Griffith <ng...@dremio.com>
wrote:
> Hi Stefan,
>
> A couple things:
>
> 1.) Have the UDFs been recompiled using the 1.3 dependency?
>
> 2.) Have the jars for the UDFs definitely been placed in the new 1.3
> install's jars/3rdparty directory?
>
> Forgive me if these suggestions are too rudimentary, but it's always
> nice to start with some sanity checks :)
>
> Best,
> Nathan
>
> On Mon, Nov 9, 2015 at 10:56 AM, Stefán Baxter
> <st...@activitystream.com> wrote:
> > Hi,
> >
> > I have a small set of UDFs that I have been running with Drill 1.1/1.2
> > which I'm trying to get working with 1.3 to no avail.
> >
> > It's as if the library is no picked up correctly even though the error I
> > get indicates a missing function signature (variant):
> >
> > Error: VALIDATION ERROR: From line 1, column 129 to line 1, column 164:
> No
> > match found for function signature roundTimeStamp(<NUMERIC>, <CHARACTER>)
> >
> > Have the rules for UDFs changed at all for the 1.3 snapshot?
> >
> > Regards,
> > -Stefan
>
Re: UDFs and 1.3
Posted by Nathan Griffith <ng...@dremio.com>.
Hi Stefan,
A couple things:
1.) Have the UDFs been recompiled using the 1.3 dependency?
2.) Have the jars for the UDFs definitely been placed in the new 1.3
install's jars/3rdparty directory?
Forgive me if these suggestions are too rudimentary, but it's always
nice to start with some sanity checks :)
Best,
Nathan
On Mon, Nov 9, 2015 at 10:56 AM, Stefán Baxter
<st...@activitystream.com> wrote:
> Hi,
>
> I have a small set of UDFs that I have been running with Drill 1.1/1.2
> which I'm trying to get working with 1.3 to no avail.
>
> It's as if the library is no picked up correctly even though the error I
> get indicates a missing function signature (variant):
>
> Error: VALIDATION ERROR: From line 1, column 129 to line 1, column 164: No
> match found for function signature roundTimeStamp(<NUMERIC>, <CHARACTER>)
>
> Have the rules for UDFs changed at all for the 1.3 snapshot?
>
> Regards,
> -Stefan