Classpath scanning & udfs

[rahul challapalli <ch...@gmail.com>]

Before 1.2, my udfs project contained an empty drill-override.conf file and
I used to update the drill-override.conf on all the drillbits to specify
the package of my UDF. This is no longer working for me. I tried a few
things and below is how my drill-override.conf file looks now

drill.classpath.scanning.packages : ${?drill.classpath.scanning.packages} [
org.apache.drill.udfs ]
drill.exec: {
  cluster-id: "rahul_cluster_com-drillbits",
  zk.connect: "localhost:5181"
}

When I restart the drillbits, I get this strange error " Caused by:
com.fasterxml.jackson.databind.JsonMappingException: Could not resolve type
id 'hbase' into a subtype of [simple type, class
org.apache.drill.common.logical.StoragePluginConfig]"

If I moved the package information to the drill-module.conf in my udf's
project, then things are working fine. However this requires re-compiling
the udfs which is not desirable. Is there any other way around this ?

- Rahul

Re: Classpath scanning & udfs

[rahul challapalli <ch...@gmail.com>]

Adding the drill-module.conf file in the drill-conf directory with the
custom package names worked for me. Thanks for the assistance Jason, Julien
and Jacques.

- Rahul

On Tue, Jan 12, 2016 at 8:09 AM, Jason Altekruse <al...@gmail.com>
wrote:

> Copying info over from Slack.
>
> For anyone who finds this thread, empty drill-module.conf files do not
> cause problems. The issue was a misunderstanding about the function of
> drill-override.conf. Values cannot be added to existing property lists
> using this file, it is designed for overriding the default values.
>
> If you put your UDFs in a package that is already being scanned as part of
> the default list then there is no need to add a new package to the list.
> The default list is generated by merging the drill-module.conf files in all
> of the default Drill packages. Here is one example from the drill exec
> module [1]
>
> To augment the list, add a drill-module.conf file to the classpath, the
> easiest way to do this is just include it in your jar.
>
> [1] -
>
> https://github.com/apache/drill/blob/master/exec/java-exec/src/main/resources/drill-module.conf
> <
> https://github.com/apache/drill/blob/master/exec/java-exec/src/main/resources/drill-module.conf
> >
>
> - Jason
>
> On Mon, Jan 11, 2016 at 11:24 AM, rahul challapalli <
> challapallirahul@gmail.com> wrote:
>
> > Sure!
> >
> > On Mon, Jan 11, 2016 at 11:06 AM, Jason Altekruse <
> > altekrusejason@gmail.com>
> > wrote:
> >
> > > The error you posted originally is from deserializing a storage plugin
> > > config, not the drill-override or drill-module files. That being said,
> we
> > > find the list of available storage plugins using classpath scanning.
> > >
> > > I don't know why the inclusion or exclusion of contents inside of a
> > > module.conf file would impact our scanning for storage plugins, but I
> > > believe this is the issue you are seeing. Somehow the package that
> > contains
> > > the hbase plugin is being removed from the list of packages to scan
> when
> > > you add the empty file.
> > >
> > > Do you want to jump on slack to chat about this?
> > >
> > > - Jason
> > >
> > > On Mon, Jan 11, 2016 at 10:52 AM, rahul challapalli <
> > > challapallirahul@gmail.com> wrote:
> > >
> > > > Julien,
> > > >
> > > > I have an empty drill-module.conf file in the udf jar file. Below are
> > the
> > > > contents of my global drill-override.conf file
> > > >
> > > > drill.classpath.scanning.packages :
> > > ${?drill.classpath.scanning.packages} [
> > > > org.apache.drill.udfs ]
> > > > drill.exec: {
> > > >   cluster-id: "rahul_cluster_com-drillbits",
> > > >   zk.connect: "localhost:5181"
> > > > }
> > > >
> > > > With this, the drillbit fails to start with the error I posted in the
> > > first
> > > > email. May be I am getting the syntax wrong?
> > > >
> > > > The reason I am insisting an empty drill-module.conf file is because,
> > > udf's
> > > > developed prior to the "classpath scanning change" had an empty
> > > > drill-module.conf file and we used to override the udf package name
> in
> > > the
> > > > global conf file.
> > > >
> > > > - Rahul
> > > >
> > > > On Mon, Jan 11, 2016 at 10:46 AM, Julien Le Dem <ju...@dremio.com>
> > > wrote:
> > > >
> > > > > Yes I believe that should work:
> > > > > -  add an empty drill-module.conf in the root of the udf jar
> > > > > -  add the package to drill.classpath.scanning.packages in the
> drill
> > > conf
> > > > > (possibly using drill-override.conf)
> > > > >
> > > > > However if you are adding the drill-module.conf file to the jar,
> you
> > > > might
> > > > > as well add the package in it. (unless there's some other reason)
> > > > >
> > > > > On Mon, Jan 11, 2016 at 10:28 AM, rahul challapalli <
> > > > > challapallirahul@gmail.com> wrote:
> > > > >
> > > > > > Just to be sure, If I have an empty drill-module.conf in the root
> > of
> > > my
> > > > > udf
> > > > > > jar, then there is no way to add the package information to the
> > > global
> > > > > > drill-override.conf file?
> > > > > >
> > > > > > On Mon, Jan 11, 2016 at 10:26 AM, Julien Le Dem <
> julien@dremio.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > You are correct:
> > > > > > > The jar containing the UDFs should have a drill-module.conf at
> > the
> > > > root
> > > > > > > adding your package to the property
> > > drill.classpath.scanning.packages
> > > > > for
> > > > > > > scanning
> > > > > > > drill.classpath.scanning.packages :
> > > > > > ${?drill.classpath.scanning.packages} [
> > > > > > >         my.package.containing.my.udfs
> > > > > > >     ]
> > > > > > > Jars that don't contain a drill-module.conf will not get
> scanned.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Jan 11, 2016 at 10:17 AM, rahul challapalli <
> > > > > > > challapallirahul@gmail.com> wrote:
> > > > > > >
> > > > > > > > Thanks for your reply Jason.
> > > > > > > >
> > > > > > > > If we cannot override the global configuration file, then for
> > > > > existing
> > > > > > > > UDF's we have to re-compile them by modifying the
> > > drill-module.conf
> > > > > > file.
> > > > > > > > If so our UDF's are not backward compatible. Appreciate it if
> > > > someone
> > > > > > can
> > > > > > > > confirm this.
> > > > > > > >
> > > > > > > > - Rahul
> > > > > > > >
> > > > > > > > On Mon, Jan 11, 2016 at 9:59 AM, Jason Altekruse <
> > > > > > > altekrusejason@gmail.com
> > > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Rahul,
> > > > > > > > >
> > > > > > > > > The error message you are seeing is in reading a storage
> > plugin
> > > > > > > > > configuration file. I am planning to fix these kinds of
> > > messages
> > > > to
> > > > > > > > > actually direct users at the file that is failing parsing.
> I
> > > have
> > > > > > seen
> > > > > > > > this
> > > > > > > > > in the past when the classpath was incorrect and one of the
> > > > plugins
> > > > > > > (like
> > > > > > > > > Hbase) was not included.
> > > > > > > > >
> > > > > > > > > Julien can confirm, but I think this might be intentional
> to
> > > have
> > > > > the
> > > > > > > > paths
> > > > > > > > > read out of the modules configuration rather than the
> global
> > > one
> > > > to
> > > > > > > save
> > > > > > > > > time when scanning the path (rather than scanning all of
> the
> > > jars
> > > > > for
> > > > > > > all
> > > > > > > > > paths given in the override file).
> > > > > > > > >
> > > > > > > > > On Fri, Jan 8, 2016 at 4:32 PM, rahul challapalli <
> > > > > > > > > challapallirahul@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > > Before 1.2, my udfs project contained an empty
> > > > > drill-override.conf
> > > > > > > file
> > > > > > > > > and
> > > > > > > > > > I used to update the drill-override.conf on all the
> > drillbits
> > > > to
> > > > > > > > specify
> > > > > > > > > > the package of my UDF. This is no longer working for me.
> I
> > > > tried
> > > > > a
> > > > > > > few
> > > > > > > > > > things and below is how my drill-override.conf file looks
> > now
> > > > > > > > > >
> > > > > > > > > > drill.classpath.scanning.packages :
> > > > > > > > > ${?drill.classpath.scanning.packages} [
> > > > > > > > > > org.apache.drill.udfs ]
> > > > > > > > > > drill.exec: {
> > > > > > > > > >   cluster-id: "rahul_cluster_com-drillbits",
> > > > > > > > > >   zk.connect: "localhost:5181"
> > > > > > > > > > }
> > > > > > > > > >
> > > > > > > > > > When I restart the drillbits, I get this strange error "
> > > Caused
> > > > > by:
> > > > > > > > > > com.fasterxml.jackson.databind.JsonMappingException:
> Could
> > > not
> > > > > > > resolve
> > > > > > > > > type
> > > > > > > > > > id 'hbase' into a subtype of [simple type, class
> > > > > > > > > > org.apache.drill.common.logical.StoragePluginConfig]"
> > > > > > > > > >
> > > > > > > > > > If I moved the package information to the
> drill-module.conf
> > > in
> > > > my
> > > > > > > udf's
> > > > > > > > > > project, then things are working fine. However this
> > requires
> > > > > > > > re-compiling
> > > > > > > > > > the udfs which is not desirable. Is there any other way
> > > around
> > > > > > this ?
> > > > > > > > > >
> > > > > > > > > > - Rahul
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Julien
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Julien
> > > > >
> > > >
> > >
> >
>

Re: Classpath scanning & udfs

[rahul challapalli <ch...@gmail.com>]

Adding the drill-module.conf file in the drill-conf directory with the
custom package names worked for me. Thanks for the assistance Jason, Julien
and Jacques.

- Rahul

On Tue, Jan 12, 2016 at 8:09 AM, Jason Altekruse <al...@gmail.com>
wrote:

> Copying info over from Slack.
>
> For anyone who finds this thread, empty drill-module.conf files do not
> cause problems. The issue was a misunderstanding about the function of
> drill-override.conf. Values cannot be added to existing property lists
> using this file, it is designed for overriding the default values.
>
> If you put your UDFs in a package that is already being scanned as part of
> the default list then there is no need to add a new package to the list.
> The default list is generated by merging the drill-module.conf files in all
> of the default Drill packages. Here is one example from the drill exec
> module [1]
>
> To augment the list, add a drill-module.conf file to the classpath, the
> easiest way to do this is just include it in your jar.
>
> [1] -
>
> https://github.com/apache/drill/blob/master/exec/java-exec/src/main/resources/drill-module.conf
> <
> https://github.com/apache/drill/blob/master/exec/java-exec/src/main/resources/drill-module.conf
> >
>
> - Jason
>
> On Mon, Jan 11, 2016 at 11:24 AM, rahul challapalli <
> challapallirahul@gmail.com> wrote:
>
> > Sure!
> >
> > On Mon, Jan 11, 2016 at 11:06 AM, Jason Altekruse <
> > altekrusejason@gmail.com>
> > wrote:
> >
> > > The error you posted originally is from deserializing a storage plugin
> > > config, not the drill-override or drill-module files. That being said,
> we
> > > find the list of available storage plugins using classpath scanning.
> > >
> > > I don't know why the inclusion or exclusion of contents inside of a
> > > module.conf file would impact our scanning for storage plugins, but I
> > > believe this is the issue you are seeing. Somehow the package that
> > contains
> > > the hbase plugin is being removed from the list of packages to scan
> when
> > > you add the empty file.
> > >
> > > Do you want to jump on slack to chat about this?
> > >
> > > - Jason
> > >
> > > On Mon, Jan 11, 2016 at 10:52 AM, rahul challapalli <
> > > challapallirahul@gmail.com> wrote:
> > >
> > > > Julien,
> > > >
> > > > I have an empty drill-module.conf file in the udf jar file. Below are
> > the
> > > > contents of my global drill-override.conf file
> > > >
> > > > drill.classpath.scanning.packages :
> > > ${?drill.classpath.scanning.packages} [
> > > > org.apache.drill.udfs ]
> > > > drill.exec: {
> > > >   cluster-id: "rahul_cluster_com-drillbits",
> > > >   zk.connect: "localhost:5181"
> > > > }
> > > >
> > > > With this, the drillbit fails to start with the error I posted in the
> > > first
> > > > email. May be I am getting the syntax wrong?
> > > >
> > > > The reason I am insisting an empty drill-module.conf file is because,
> > > udf's
> > > > developed prior to the "classpath scanning change" had an empty
> > > > drill-module.conf file and we used to override the udf package name
> in
> > > the
> > > > global conf file.
> > > >
> > > > - Rahul
> > > >
> > > > On Mon, Jan 11, 2016 at 10:46 AM, Julien Le Dem <ju...@dremio.com>
> > > wrote:
> > > >
> > > > > Yes I believe that should work:
> > > > > -  add an empty drill-module.conf in the root of the udf jar
> > > > > -  add the package to drill.classpath.scanning.packages in the
> drill
> > > conf
> > > > > (possibly using drill-override.conf)
> > > > >
> > > > > However if you are adding the drill-module.conf file to the jar,
> you
> > > > might
> > > > > as well add the package in it. (unless there's some other reason)
> > > > >
> > > > > On Mon, Jan 11, 2016 at 10:28 AM, rahul challapalli <
> > > > > challapallirahul@gmail.com> wrote:
> > > > >
> > > > > > Just to be sure, If I have an empty drill-module.conf in the root
> > of
> > > my
> > > > > udf
> > > > > > jar, then there is no way to add the package information to the
> > > global
> > > > > > drill-override.conf file?
> > > > > >
> > > > > > On Mon, Jan 11, 2016 at 10:26 AM, Julien Le Dem <
> julien@dremio.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > You are correct:
> > > > > > > The jar containing the UDFs should have a drill-module.conf at
> > the
> > > > root
> > > > > > > adding your package to the property
> > > drill.classpath.scanning.packages
> > > > > for
> > > > > > > scanning
> > > > > > > drill.classpath.scanning.packages :
> > > > > > ${?drill.classpath.scanning.packages} [
> > > > > > >         my.package.containing.my.udfs
> > > > > > >     ]
> > > > > > > Jars that don't contain a drill-module.conf will not get
> scanned.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Jan 11, 2016 at 10:17 AM, rahul challapalli <
> > > > > > > challapallirahul@gmail.com> wrote:
> > > > > > >
> > > > > > > > Thanks for your reply Jason.
> > > > > > > >
> > > > > > > > If we cannot override the global configuration file, then for
> > > > > existing
> > > > > > > > UDF's we have to re-compile them by modifying the
> > > drill-module.conf
> > > > > > file.
> > > > > > > > If so our UDF's are not backward compatible. Appreciate it if
> > > > someone
> > > > > > can
> > > > > > > > confirm this.
> > > > > > > >
> > > > > > > > - Rahul
> > > > > > > >
> > > > > > > > On Mon, Jan 11, 2016 at 9:59 AM, Jason Altekruse <
> > > > > > > altekrusejason@gmail.com
> > > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Rahul,
> > > > > > > > >
> > > > > > > > > The error message you are seeing is in reading a storage
> > plugin
> > > > > > > > > configuration file. I am planning to fix these kinds of
> > > messages
> > > > to
> > > > > > > > > actually direct users at the file that is failing parsing.
> I
> > > have
> > > > > > seen
> > > > > > > > this
> > > > > > > > > in the past when the classpath was incorrect and one of the
> > > > plugins
> > > > > > > (like
> > > > > > > > > Hbase) was not included.
> > > > > > > > >
> > > > > > > > > Julien can confirm, but I think this might be intentional
> to
> > > have
> > > > > the
> > > > > > > > paths
> > > > > > > > > read out of the modules configuration rather than the
> global
> > > one
> > > > to
> > > > > > > save
> > > > > > > > > time when scanning the path (rather than scanning all of
> the
> > > jars
> > > > > for
> > > > > > > all
> > > > > > > > > paths given in the override file).
> > > > > > > > >
> > > > > > > > > On Fri, Jan 8, 2016 at 4:32 PM, rahul challapalli <
> > > > > > > > > challapallirahul@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > > Before 1.2, my udfs project contained an empty
> > > > > drill-override.conf
> > > > > > > file
> > > > > > > > > and
> > > > > > > > > > I used to update the drill-override.conf on all the
> > drillbits
> > > > to
> > > > > > > > specify
> > > > > > > > > > the package of my UDF. This is no longer working for me.
> I
> > > > tried
> > > > > a
> > > > > > > few
> > > > > > > > > > things and below is how my drill-override.conf file looks
> > now
> > > > > > > > > >
> > > > > > > > > > drill.classpath.scanning.packages :
> > > > > > > > > ${?drill.classpath.scanning.packages} [
> > > > > > > > > > org.apache.drill.udfs ]
> > > > > > > > > > drill.exec: {
> > > > > > > > > >   cluster-id: "rahul_cluster_com-drillbits",
> > > > > > > > > >   zk.connect: "localhost:5181"
> > > > > > > > > > }
> > > > > > > > > >
> > > > > > > > > > When I restart the drillbits, I get this strange error "
> > > Caused
> > > > > by:
> > > > > > > > > > com.fasterxml.jackson.databind.JsonMappingException:
> Could
> > > not
> > > > > > > resolve
> > > > > > > > > type
> > > > > > > > > > id 'hbase' into a subtype of [simple type, class
> > > > > > > > > > org.apache.drill.common.logical.StoragePluginConfig]"
> > > > > > > > > >
> > > > > > > > > > If I moved the package information to the
> drill-module.conf
> > > in
> > > > my
> > > > > > > udf's
> > > > > > > > > > project, then things are working fine. However this
> > requires
> > > > > > > > re-compiling
> > > > > > > > > > the udfs which is not desirable. Is there any other way
> > > around
> > > > > > this ?
> > > > > > > > > >
> > > > > > > > > > - Rahul
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Julien
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Julien
> > > > >
> > > >
> > >
> >
>

Re: Classpath scanning & udfs

[Jason Altekruse <al...@gmail.com>]

Copying info over from Slack.

For anyone who finds this thread, empty drill-module.conf files do not
cause problems. The issue was a misunderstanding about the function of
drill-override.conf. Values cannot be added to existing property lists
using this file, it is designed for overriding the default values.

If you put your UDFs in a package that is already being scanned as part of
the default list then there is no need to add a new package to the list.
The default list is generated by merging the drill-module.conf files in all
of the default Drill packages. Here is one example from the drill exec
module [1]

To augment the list, add a drill-module.conf file to the classpath, the
easiest way to do this is just include it in your jar.

[1] -
https://github.com/apache/drill/blob/master/exec/java-exec/src/main/resources/drill-module.conf
<https://github.com/apache/drill/blob/master/exec/java-exec/src/main/resources/drill-module.conf>

- Jason

On Mon, Jan 11, 2016 at 11:24 AM, rahul challapalli <
challapallirahul@gmail.com> wrote:

> Sure!
>
> On Mon, Jan 11, 2016 at 11:06 AM, Jason Altekruse <
> altekrusejason@gmail.com>
> wrote:
>
> > The error you posted originally is from deserializing a storage plugin
> > config, not the drill-override or drill-module files. That being said, we
> > find the list of available storage plugins using classpath scanning.
> >
> > I don't know why the inclusion or exclusion of contents inside of a
> > module.conf file would impact our scanning for storage plugins, but I
> > believe this is the issue you are seeing. Somehow the package that
> contains
> > the hbase plugin is being removed from the list of packages to scan when
> > you add the empty file.
> >
> > Do you want to jump on slack to chat about this?
> >
> > - Jason
> >
> > On Mon, Jan 11, 2016 at 10:52 AM, rahul challapalli <
> > challapallirahul@gmail.com> wrote:
> >
> > > Julien,
> > >
> > > I have an empty drill-module.conf file in the udf jar file. Below are
> the
> > > contents of my global drill-override.conf file
> > >
> > > drill.classpath.scanning.packages :
> > ${?drill.classpath.scanning.packages} [
> > > org.apache.drill.udfs ]
> > > drill.exec: {
> > >   cluster-id: "rahul_cluster_com-drillbits",
> > >   zk.connect: "localhost:5181"
> > > }
> > >
> > > With this, the drillbit fails to start with the error I posted in the
> > first
> > > email. May be I am getting the syntax wrong?
> > >
> > > The reason I am insisting an empty drill-module.conf file is because,
> > udf's
> > > developed prior to the "classpath scanning change" had an empty
> > > drill-module.conf file and we used to override the udf package name in
> > the
> > > global conf file.
> > >
> > > - Rahul
> > >
> > > On Mon, Jan 11, 2016 at 10:46 AM, Julien Le Dem <ju...@dremio.com>
> > wrote:
> > >
> > > > Yes I believe that should work:
> > > > -  add an empty drill-module.conf in the root of the udf jar
> > > > -  add the package to drill.classpath.scanning.packages in the drill
> > conf
> > > > (possibly using drill-override.conf)
> > > >
> > > > However if you are adding the drill-module.conf file to the jar, you
> > > might
> > > > as well add the package in it. (unless there's some other reason)
> > > >
> > > > On Mon, Jan 11, 2016 at 10:28 AM, rahul challapalli <
> > > > challapallirahul@gmail.com> wrote:
> > > >
> > > > > Just to be sure, If I have an empty drill-module.conf in the root
> of
> > my
> > > > udf
> > > > > jar, then there is no way to add the package information to the
> > global
> > > > > drill-override.conf file?
> > > > >
> > > > > On Mon, Jan 11, 2016 at 10:26 AM, Julien Le Dem <julien@dremio.com
> >
> > > > wrote:
> > > > >
> > > > > > You are correct:
> > > > > > The jar containing the UDFs should have a drill-module.conf at
> the
> > > root
> > > > > > adding your package to the property
> > drill.classpath.scanning.packages
> > > > for
> > > > > > scanning
> > > > > > drill.classpath.scanning.packages :
> > > > > ${?drill.classpath.scanning.packages} [
> > > > > >         my.package.containing.my.udfs
> > > > > >     ]
> > > > > > Jars that don't contain a drill-module.conf will not get scanned.
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Jan 11, 2016 at 10:17 AM, rahul challapalli <
> > > > > > challapallirahul@gmail.com> wrote:
> > > > > >
> > > > > > > Thanks for your reply Jason.
> > > > > > >
> > > > > > > If we cannot override the global configuration file, then for
> > > > existing
> > > > > > > UDF's we have to re-compile them by modifying the
> > drill-module.conf
> > > > > file.
> > > > > > > If so our UDF's are not backward compatible. Appreciate it if
> > > someone
> > > > > can
> > > > > > > confirm this.
> > > > > > >
> > > > > > > - Rahul
> > > > > > >
> > > > > > > On Mon, Jan 11, 2016 at 9:59 AM, Jason Altekruse <
> > > > > > altekrusejason@gmail.com
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Rahul,
> > > > > > > >
> > > > > > > > The error message you are seeing is in reading a storage
> plugin
> > > > > > > > configuration file. I am planning to fix these kinds of
> > messages
> > > to
> > > > > > > > actually direct users at the file that is failing parsing. I
> > have
> > > > > seen
> > > > > > > this
> > > > > > > > in the past when the classpath was incorrect and one of the
> > > plugins
> > > > > > (like
> > > > > > > > Hbase) was not included.
> > > > > > > >
> > > > > > > > Julien can confirm, but I think this might be intentional to
> > have
> > > > the
> > > > > > > paths
> > > > > > > > read out of the modules configuration rather than the global
> > one
> > > to
> > > > > > save
> > > > > > > > time when scanning the path (rather than scanning all of the
> > jars
> > > > for
> > > > > > all
> > > > > > > > paths given in the override file).
> > > > > > > >
> > > > > > > > On Fri, Jan 8, 2016 at 4:32 PM, rahul challapalli <
> > > > > > > > challapallirahul@gmail.com> wrote:
> > > > > > > >
> > > > > > > > > Before 1.2, my udfs project contained an empty
> > > > drill-override.conf
> > > > > > file
> > > > > > > > and
> > > > > > > > > I used to update the drill-override.conf on all the
> drillbits
> > > to
> > > > > > > specify
> > > > > > > > > the package of my UDF. This is no longer working for me. I
> > > tried
> > > > a
> > > > > > few
> > > > > > > > > things and below is how my drill-override.conf file looks
> now
> > > > > > > > >
> > > > > > > > > drill.classpath.scanning.packages :
> > > > > > > > ${?drill.classpath.scanning.packages} [
> > > > > > > > > org.apache.drill.udfs ]
> > > > > > > > > drill.exec: {
> > > > > > > > >   cluster-id: "rahul_cluster_com-drillbits",
> > > > > > > > >   zk.connect: "localhost:5181"
> > > > > > > > > }
> > > > > > > > >
> > > > > > > > > When I restart the drillbits, I get this strange error "
> > Caused
> > > > by:
> > > > > > > > > com.fasterxml.jackson.databind.JsonMappingException: Could
> > not
> > > > > > resolve
> > > > > > > > type
> > > > > > > > > id 'hbase' into a subtype of [simple type, class
> > > > > > > > > org.apache.drill.common.logical.StoragePluginConfig]"
> > > > > > > > >
> > > > > > > > > If I moved the package information to the drill-module.conf
> > in
> > > my
> > > > > > udf's
> > > > > > > > > project, then things are working fine. However this
> requires
> > > > > > > re-compiling
> > > > > > > > > the udfs which is not desirable. Is there any other way
> > around
> > > > > this ?
> > > > > > > > >
> > > > > > > > > - Rahul
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Julien
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Julien
> > > >
> > >
> >
>

Re: Classpath scanning & udfs

[Jason Altekruse <al...@gmail.com>]

Copying info over from Slack.

For anyone who finds this thread, empty drill-module.conf files do not
cause problems. The issue was a misunderstanding about the function of
drill-override.conf. Values cannot be added to existing property lists
using this file, it is designed for overriding the default values.

If you put your UDFs in a package that is already being scanned as part of
the default list then there is no need to add a new package to the list.
The default list is generated by merging the drill-module.conf files in all
of the default Drill packages. Here is one example from the drill exec
module [1]

To augment the list, add a drill-module.conf file to the classpath, the
easiest way to do this is just include it in your jar.

[1] -
https://github.com/apache/drill/blob/master/exec/java-exec/src/main/resources/drill-module.conf
<https://github.com/apache/drill/blob/master/exec/java-exec/src/main/resources/drill-module.conf>

- Jason

On Mon, Jan 11, 2016 at 11:24 AM, rahul challapalli <
challapallirahul@gmail.com> wrote:

> Sure!
>
> On Mon, Jan 11, 2016 at 11:06 AM, Jason Altekruse <
> altekrusejason@gmail.com>
> wrote:
>
> > The error you posted originally is from deserializing a storage plugin
> > config, not the drill-override or drill-module files. That being said, we
> > find the list of available storage plugins using classpath scanning.
> >
> > I don't know why the inclusion or exclusion of contents inside of a
> > module.conf file would impact our scanning for storage plugins, but I
> > believe this is the issue you are seeing. Somehow the package that
> contains
> > the hbase plugin is being removed from the list of packages to scan when
> > you add the empty file.
> >
> > Do you want to jump on slack to chat about this?
> >
> > - Jason
> >
> > On Mon, Jan 11, 2016 at 10:52 AM, rahul challapalli <
> > challapallirahul@gmail.com> wrote:
> >
> > > Julien,
> > >
> > > I have an empty drill-module.conf file in the udf jar file. Below are
> the
> > > contents of my global drill-override.conf file
> > >
> > > drill.classpath.scanning.packages :
> > ${?drill.classpath.scanning.packages} [
> > > org.apache.drill.udfs ]
> > > drill.exec: {
> > >   cluster-id: "rahul_cluster_com-drillbits",
> > >   zk.connect: "localhost:5181"
> > > }
> > >
> > > With this, the drillbit fails to start with the error I posted in the
> > first
> > > email. May be I am getting the syntax wrong?
> > >
> > > The reason I am insisting an empty drill-module.conf file is because,
> > udf's
> > > developed prior to the "classpath scanning change" had an empty
> > > drill-module.conf file and we used to override the udf package name in
> > the
> > > global conf file.
> > >
> > > - Rahul
> > >
> > > On Mon, Jan 11, 2016 at 10:46 AM, Julien Le Dem <ju...@dremio.com>
> > wrote:
> > >
> > > > Yes I believe that should work:
> > > > -  add an empty drill-module.conf in the root of the udf jar
> > > > -  add the package to drill.classpath.scanning.packages in the drill
> > conf
> > > > (possibly using drill-override.conf)
> > > >
> > > > However if you are adding the drill-module.conf file to the jar, you
> > > might
> > > > as well add the package in it. (unless there's some other reason)
> > > >
> > > > On Mon, Jan 11, 2016 at 10:28 AM, rahul challapalli <
> > > > challapallirahul@gmail.com> wrote:
> > > >
> > > > > Just to be sure, If I have an empty drill-module.conf in the root
> of
> > my
> > > > udf
> > > > > jar, then there is no way to add the package information to the
> > global
> > > > > drill-override.conf file?
> > > > >
> > > > > On Mon, Jan 11, 2016 at 10:26 AM, Julien Le Dem <julien@dremio.com
> >
> > > > wrote:
> > > > >
> > > > > > You are correct:
> > > > > > The jar containing the UDFs should have a drill-module.conf at
> the
> > > root
> > > > > > adding your package to the property
> > drill.classpath.scanning.packages
> > > > for
> > > > > > scanning
> > > > > > drill.classpath.scanning.packages :
> > > > > ${?drill.classpath.scanning.packages} [
> > > > > >         my.package.containing.my.udfs
> > > > > >     ]
> > > > > > Jars that don't contain a drill-module.conf will not get scanned.
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Jan 11, 2016 at 10:17 AM, rahul challapalli <
> > > > > > challapallirahul@gmail.com> wrote:
> > > > > >
> > > > > > > Thanks for your reply Jason.
> > > > > > >
> > > > > > > If we cannot override the global configuration file, then for
> > > > existing
> > > > > > > UDF's we have to re-compile them by modifying the
> > drill-module.conf
> > > > > file.
> > > > > > > If so our UDF's are not backward compatible. Appreciate it if
> > > someone
> > > > > can
> > > > > > > confirm this.
> > > > > > >
> > > > > > > - Rahul
> > > > > > >
> > > > > > > On Mon, Jan 11, 2016 at 9:59 AM, Jason Altekruse <
> > > > > > altekrusejason@gmail.com
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Rahul,
> > > > > > > >
> > > > > > > > The error message you are seeing is in reading a storage
> plugin
> > > > > > > > configuration file. I am planning to fix these kinds of
> > messages
> > > to
> > > > > > > > actually direct users at the file that is failing parsing. I
> > have
> > > > > seen
> > > > > > > this
> > > > > > > > in the past when the classpath was incorrect and one of the
> > > plugins
> > > > > > (like
> > > > > > > > Hbase) was not included.
> > > > > > > >
> > > > > > > > Julien can confirm, but I think this might be intentional to
> > have
> > > > the
> > > > > > > paths
> > > > > > > > read out of the modules configuration rather than the global
> > one
> > > to
> > > > > > save
> > > > > > > > time when scanning the path (rather than scanning all of the
> > jars
> > > > for
> > > > > > all
> > > > > > > > paths given in the override file).
> > > > > > > >
> > > > > > > > On Fri, Jan 8, 2016 at 4:32 PM, rahul challapalli <
> > > > > > > > challapallirahul@gmail.com> wrote:
> > > > > > > >
> > > > > > > > > Before 1.2, my udfs project contained an empty
> > > > drill-override.conf
> > > > > > file
> > > > > > > > and
> > > > > > > > > I used to update the drill-override.conf on all the
> drillbits
> > > to
> > > > > > > specify
> > > > > > > > > the package of my UDF. This is no longer working for me. I
> > > tried
> > > > a
> > > > > > few
> > > > > > > > > things and below is how my drill-override.conf file looks
> now
> > > > > > > > >
> > > > > > > > > drill.classpath.scanning.packages :
> > > > > > > > ${?drill.classpath.scanning.packages} [
> > > > > > > > > org.apache.drill.udfs ]
> > > > > > > > > drill.exec: {
> > > > > > > > >   cluster-id: "rahul_cluster_com-drillbits",
> > > > > > > > >   zk.connect: "localhost:5181"
> > > > > > > > > }
> > > > > > > > >
> > > > > > > > > When I restart the drillbits, I get this strange error "
> > Caused
> > > > by:
> > > > > > > > > com.fasterxml.jackson.databind.JsonMappingException: Could
> > not
> > > > > > resolve
> > > > > > > > type
> > > > > > > > > id 'hbase' into a subtype of [simple type, class
> > > > > > > > > org.apache.drill.common.logical.StoragePluginConfig]"
> > > > > > > > >
> > > > > > > > > If I moved the package information to the drill-module.conf
> > in
> > > my
> > > > > > udf's
> > > > > > > > > project, then things are working fine. However this
> requires
> > > > > > > re-compiling
> > > > > > > > > the udfs which is not desirable. Is there any other way
> > around
> > > > > this ?
> > > > > > > > >
> > > > > > > > > - Rahul
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Julien
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Julien
> > > >
> > >
> >
>

Re: Classpath scanning & udfs

[rahul challapalli <ch...@gmail.com>]

Sure!

On Mon, Jan 11, 2016 at 11:06 AM, Jason Altekruse <al...@gmail.com>
wrote:

> The error you posted originally is from deserializing a storage plugin
> config, not the drill-override or drill-module files. That being said, we
> find the list of available storage plugins using classpath scanning.
>
> I don't know why the inclusion or exclusion of contents inside of a
> module.conf file would impact our scanning for storage plugins, but I
> believe this is the issue you are seeing. Somehow the package that contains
> the hbase plugin is being removed from the list of packages to scan when
> you add the empty file.
>
> Do you want to jump on slack to chat about this?
>
> - Jason
>
> On Mon, Jan 11, 2016 at 10:52 AM, rahul challapalli <
> challapallirahul@gmail.com> wrote:
>
> > Julien,
> >
> > I have an empty drill-module.conf file in the udf jar file. Below are the
> > contents of my global drill-override.conf file
> >
> > drill.classpath.scanning.packages :
> ${?drill.classpath.scanning.packages} [
> > org.apache.drill.udfs ]
> > drill.exec: {
> >   cluster-id: "rahul_cluster_com-drillbits",
> >   zk.connect: "localhost:5181"
> > }
> >
> > With this, the drillbit fails to start with the error I posted in the
> first
> > email. May be I am getting the syntax wrong?
> >
> > The reason I am insisting an empty drill-module.conf file is because,
> udf's
> > developed prior to the "classpath scanning change" had an empty
> > drill-module.conf file and we used to override the udf package name in
> the
> > global conf file.
> >
> > - Rahul
> >
> > On Mon, Jan 11, 2016 at 10:46 AM, Julien Le Dem <ju...@dremio.com>
> wrote:
> >
> > > Yes I believe that should work:
> > > -  add an empty drill-module.conf in the root of the udf jar
> > > -  add the package to drill.classpath.scanning.packages in the drill
> conf
> > > (possibly using drill-override.conf)
> > >
> > > However if you are adding the drill-module.conf file to the jar, you
> > might
> > > as well add the package in it. (unless there's some other reason)
> > >
> > > On Mon, Jan 11, 2016 at 10:28 AM, rahul challapalli <
> > > challapallirahul@gmail.com> wrote:
> > >
> > > > Just to be sure, If I have an empty drill-module.conf in the root of
> my
> > > udf
> > > > jar, then there is no way to add the package information to the
> global
> > > > drill-override.conf file?
> > > >
> > > > On Mon, Jan 11, 2016 at 10:26 AM, Julien Le Dem <ju...@dremio.com>
> > > wrote:
> > > >
> > > > > You are correct:
> > > > > The jar containing the UDFs should have a drill-module.conf at the
> > root
> > > > > adding your package to the property
> drill.classpath.scanning.packages
> > > for
> > > > > scanning
> > > > > drill.classpath.scanning.packages :
> > > > ${?drill.classpath.scanning.packages} [
> > > > >         my.package.containing.my.udfs
> > > > >     ]
> > > > > Jars that don't contain a drill-module.conf will not get scanned.
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Jan 11, 2016 at 10:17 AM, rahul challapalli <
> > > > > challapallirahul@gmail.com> wrote:
> > > > >
> > > > > > Thanks for your reply Jason.
> > > > > >
> > > > > > If we cannot override the global configuration file, then for
> > > existing
> > > > > > UDF's we have to re-compile them by modifying the
> drill-module.conf
> > > > file.
> > > > > > If so our UDF's are not backward compatible. Appreciate it if
> > someone
> > > > can
> > > > > > confirm this.
> > > > > >
> > > > > > - Rahul
> > > > > >
> > > > > > On Mon, Jan 11, 2016 at 9:59 AM, Jason Altekruse <
> > > > > altekrusejason@gmail.com
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Rahul,
> > > > > > >
> > > > > > > The error message you are seeing is in reading a storage plugin
> > > > > > > configuration file. I am planning to fix these kinds of
> messages
> > to
> > > > > > > actually direct users at the file that is failing parsing. I
> have
> > > > seen
> > > > > > this
> > > > > > > in the past when the classpath was incorrect and one of the
> > plugins
> > > > > (like
> > > > > > > Hbase) was not included.
> > > > > > >
> > > > > > > Julien can confirm, but I think this might be intentional to
> have
> > > the
> > > > > > paths
> > > > > > > read out of the modules configuration rather than the global
> one
> > to
> > > > > save
> > > > > > > time when scanning the path (rather than scanning all of the
> jars
> > > for
> > > > > all
> > > > > > > paths given in the override file).
> > > > > > >
> > > > > > > On Fri, Jan 8, 2016 at 4:32 PM, rahul challapalli <
> > > > > > > challapallirahul@gmail.com> wrote:
> > > > > > >
> > > > > > > > Before 1.2, my udfs project contained an empty
> > > drill-override.conf
> > > > > file
> > > > > > > and
> > > > > > > > I used to update the drill-override.conf on all the drillbits
> > to
> > > > > > specify
> > > > > > > > the package of my UDF. This is no longer working for me. I
> > tried
> > > a
> > > > > few
> > > > > > > > things and below is how my drill-override.conf file looks now
> > > > > > > >
> > > > > > > > drill.classpath.scanning.packages :
> > > > > > > ${?drill.classpath.scanning.packages} [
> > > > > > > > org.apache.drill.udfs ]
> > > > > > > > drill.exec: {
> > > > > > > >   cluster-id: "rahul_cluster_com-drillbits",
> > > > > > > >   zk.connect: "localhost:5181"
> > > > > > > > }
> > > > > > > >
> > > > > > > > When I restart the drillbits, I get this strange error "
> Caused
> > > by:
> > > > > > > > com.fasterxml.jackson.databind.JsonMappingException: Could
> not
> > > > > resolve
> > > > > > > type
> > > > > > > > id 'hbase' into a subtype of [simple type, class
> > > > > > > > org.apache.drill.common.logical.StoragePluginConfig]"
> > > > > > > >
> > > > > > > > If I moved the package information to the drill-module.conf
> in
> > my
> > > > > udf's
> > > > > > > > project, then things are working fine. However this requires
> > > > > > re-compiling
> > > > > > > > the udfs which is not desirable. Is there any other way
> around
> > > > this ?
> > > > > > > >
> > > > > > > > - Rahul
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Julien
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Julien
> > >
> >
>

Re: Classpath scanning & udfs

[rahul challapalli <ch...@gmail.com>]

Sure!

On Mon, Jan 11, 2016 at 11:06 AM, Jason Altekruse <al...@gmail.com>
wrote:

> The error you posted originally is from deserializing a storage plugin
> config, not the drill-override or drill-module files. That being said, we
> find the list of available storage plugins using classpath scanning.
>
> I don't know why the inclusion or exclusion of contents inside of a
> module.conf file would impact our scanning for storage plugins, but I
> believe this is the issue you are seeing. Somehow the package that contains
> the hbase plugin is being removed from the list of packages to scan when
> you add the empty file.
>
> Do you want to jump on slack to chat about this?
>
> - Jason
>
> On Mon, Jan 11, 2016 at 10:52 AM, rahul challapalli <
> challapallirahul@gmail.com> wrote:
>
> > Julien,
> >
> > I have an empty drill-module.conf file in the udf jar file. Below are the
> > contents of my global drill-override.conf file
> >
> > drill.classpath.scanning.packages :
> ${?drill.classpath.scanning.packages} [
> > org.apache.drill.udfs ]
> > drill.exec: {
> >   cluster-id: "rahul_cluster_com-drillbits",
> >   zk.connect: "localhost:5181"
> > }
> >
> > With this, the drillbit fails to start with the error I posted in the
> first
> > email. May be I am getting the syntax wrong?
> >
> > The reason I am insisting an empty drill-module.conf file is because,
> udf's
> > developed prior to the "classpath scanning change" had an empty
> > drill-module.conf file and we used to override the udf package name in
> the
> > global conf file.
> >
> > - Rahul
> >
> > On Mon, Jan 11, 2016 at 10:46 AM, Julien Le Dem <ju...@dremio.com>
> wrote:
> >
> > > Yes I believe that should work:
> > > -  add an empty drill-module.conf in the root of the udf jar
> > > -  add the package to drill.classpath.scanning.packages in the drill
> conf
> > > (possibly using drill-override.conf)
> > >
> > > However if you are adding the drill-module.conf file to the jar, you
> > might
> > > as well add the package in it. (unless there's some other reason)
> > >
> > > On Mon, Jan 11, 2016 at 10:28 AM, rahul challapalli <
> > > challapallirahul@gmail.com> wrote:
> > >
> > > > Just to be sure, If I have an empty drill-module.conf in the root of
> my
> > > udf
> > > > jar, then there is no way to add the package information to the
> global
> > > > drill-override.conf file?
> > > >
> > > > On Mon, Jan 11, 2016 at 10:26 AM, Julien Le Dem <ju...@dremio.com>
> > > wrote:
> > > >
> > > > > You are correct:
> > > > > The jar containing the UDFs should have a drill-module.conf at the
> > root
> > > > > adding your package to the property
> drill.classpath.scanning.packages
> > > for
> > > > > scanning
> > > > > drill.classpath.scanning.packages :
> > > > ${?drill.classpath.scanning.packages} [
> > > > >         my.package.containing.my.udfs
> > > > >     ]
> > > > > Jars that don't contain a drill-module.conf will not get scanned.
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Jan 11, 2016 at 10:17 AM, rahul challapalli <
> > > > > challapallirahul@gmail.com> wrote:
> > > > >
> > > > > > Thanks for your reply Jason.
> > > > > >
> > > > > > If we cannot override the global configuration file, then for
> > > existing
> > > > > > UDF's we have to re-compile them by modifying the
> drill-module.conf
> > > > file.
> > > > > > If so our UDF's are not backward compatible. Appreciate it if
> > someone
> > > > can
> > > > > > confirm this.
> > > > > >
> > > > > > - Rahul
> > > > > >
> > > > > > On Mon, Jan 11, 2016 at 9:59 AM, Jason Altekruse <
> > > > > altekrusejason@gmail.com
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Rahul,
> > > > > > >
> > > > > > > The error message you are seeing is in reading a storage plugin
> > > > > > > configuration file. I am planning to fix these kinds of
> messages
> > to
> > > > > > > actually direct users at the file that is failing parsing. I
> have
> > > > seen
> > > > > > this
> > > > > > > in the past when the classpath was incorrect and one of the
> > plugins
> > > > > (like
> > > > > > > Hbase) was not included.
> > > > > > >
> > > > > > > Julien can confirm, but I think this might be intentional to
> have
> > > the
> > > > > > paths
> > > > > > > read out of the modules configuration rather than the global
> one
> > to
> > > > > save
> > > > > > > time when scanning the path (rather than scanning all of the
> jars
> > > for
> > > > > all
> > > > > > > paths given in the override file).
> > > > > > >
> > > > > > > On Fri, Jan 8, 2016 at 4:32 PM, rahul challapalli <
> > > > > > > challapallirahul@gmail.com> wrote:
> > > > > > >
> > > > > > > > Before 1.2, my udfs project contained an empty
> > > drill-override.conf
> > > > > file
> > > > > > > and
> > > > > > > > I used to update the drill-override.conf on all the drillbits
> > to
> > > > > > specify
> > > > > > > > the package of my UDF. This is no longer working for me. I
> > tried
> > > a
> > > > > few
> > > > > > > > things and below is how my drill-override.conf file looks now
> > > > > > > >
> > > > > > > > drill.classpath.scanning.packages :
> > > > > > > ${?drill.classpath.scanning.packages} [
> > > > > > > > org.apache.drill.udfs ]
> > > > > > > > drill.exec: {
> > > > > > > >   cluster-id: "rahul_cluster_com-drillbits",
> > > > > > > >   zk.connect: "localhost:5181"
> > > > > > > > }
> > > > > > > >
> > > > > > > > When I restart the drillbits, I get this strange error "
> Caused
> > > by:
> > > > > > > > com.fasterxml.jackson.databind.JsonMappingException: Could
> not
> > > > > resolve
> > > > > > > type
> > > > > > > > id 'hbase' into a subtype of [simple type, class
> > > > > > > > org.apache.drill.common.logical.StoragePluginConfig]"
> > > > > > > >
> > > > > > > > If I moved the package information to the drill-module.conf
> in
> > my
> > > > > udf's
> > > > > > > > project, then things are working fine. However this requires
> > > > > > re-compiling
> > > > > > > > the udfs which is not desirable. Is there any other way
> around
> > > > this ?
> > > > > > > >
> > > > > > > > - Rahul
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Julien
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Julien
> > >
> >
>

Re: Classpath scanning & udfs

[Jason Altekruse <al...@gmail.com>]

The error you posted originally is from deserializing a storage plugin
config, not the drill-override or drill-module files. That being said, we
find the list of available storage plugins using classpath scanning.

I don't know why the inclusion or exclusion of contents inside of a
module.conf file would impact our scanning for storage plugins, but I
believe this is the issue you are seeing. Somehow the package that contains
the hbase plugin is being removed from the list of packages to scan when
you add the empty file.

Do you want to jump on slack to chat about this?

- Jason

On Mon, Jan 11, 2016 at 10:52 AM, rahul challapalli <
challapallirahul@gmail.com> wrote:

> Julien,
>
> I have an empty drill-module.conf file in the udf jar file. Below are the
> contents of my global drill-override.conf file
>
> drill.classpath.scanning.packages : ${?drill.classpath.scanning.packages} [
> org.apache.drill.udfs ]
> drill.exec: {
>   cluster-id: "rahul_cluster_com-drillbits",
>   zk.connect: "localhost:5181"
> }
>
> With this, the drillbit fails to start with the error I posted in the first
> email. May be I am getting the syntax wrong?
>
> The reason I am insisting an empty drill-module.conf file is because, udf's
> developed prior to the "classpath scanning change" had an empty
> drill-module.conf file and we used to override the udf package name in the
> global conf file.
>
> - Rahul
>
> On Mon, Jan 11, 2016 at 10:46 AM, Julien Le Dem <ju...@dremio.com> wrote:
>
> > Yes I believe that should work:
> > -  add an empty drill-module.conf in the root of the udf jar
> > -  add the package to drill.classpath.scanning.packages in the drill conf
> > (possibly using drill-override.conf)
> >
> > However if you are adding the drill-module.conf file to the jar, you
> might
> > as well add the package in it. (unless there's some other reason)
> >
> > On Mon, Jan 11, 2016 at 10:28 AM, rahul challapalli <
> > challapallirahul@gmail.com> wrote:
> >
> > > Just to be sure, If I have an empty drill-module.conf in the root of my
> > udf
> > > jar, then there is no way to add the package information to the global
> > > drill-override.conf file?
> > >
> > > On Mon, Jan 11, 2016 at 10:26 AM, Julien Le Dem <ju...@dremio.com>
> > wrote:
> > >
> > > > You are correct:
> > > > The jar containing the UDFs should have a drill-module.conf at the
> root
> > > > adding your package to the property drill.classpath.scanning.packages
> > for
> > > > scanning
> > > > drill.classpath.scanning.packages :
> > > ${?drill.classpath.scanning.packages} [
> > > >         my.package.containing.my.udfs
> > > >     ]
> > > > Jars that don't contain a drill-module.conf will not get scanned.
> > > >
> > > >
> > > >
> > > > On Mon, Jan 11, 2016 at 10:17 AM, rahul challapalli <
> > > > challapallirahul@gmail.com> wrote:
> > > >
> > > > > Thanks for your reply Jason.
> > > > >
> > > > > If we cannot override the global configuration file, then for
> > existing
> > > > > UDF's we have to re-compile them by modifying the drill-module.conf
> > > file.
> > > > > If so our UDF's are not backward compatible. Appreciate it if
> someone
> > > can
> > > > > confirm this.
> > > > >
> > > > > - Rahul
> > > > >
> > > > > On Mon, Jan 11, 2016 at 9:59 AM, Jason Altekruse <
> > > > altekrusejason@gmail.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Rahul,
> > > > > >
> > > > > > The error message you are seeing is in reading a storage plugin
> > > > > > configuration file. I am planning to fix these kinds of messages
> to
> > > > > > actually direct users at the file that is failing parsing. I have
> > > seen
> > > > > this
> > > > > > in the past when the classpath was incorrect and one of the
> plugins
> > > > (like
> > > > > > Hbase) was not included.
> > > > > >
> > > > > > Julien can confirm, but I think this might be intentional to have
> > the
> > > > > paths
> > > > > > read out of the modules configuration rather than the global one
> to
> > > > save
> > > > > > time when scanning the path (rather than scanning all of the jars
> > for
> > > > all
> > > > > > paths given in the override file).
> > > > > >
> > > > > > On Fri, Jan 8, 2016 at 4:32 PM, rahul challapalli <
> > > > > > challapallirahul@gmail.com> wrote:
> > > > > >
> > > > > > > Before 1.2, my udfs project contained an empty
> > drill-override.conf
> > > > file
> > > > > > and
> > > > > > > I used to update the drill-override.conf on all the drillbits
> to
> > > > > specify
> > > > > > > the package of my UDF. This is no longer working for me. I
> tried
> > a
> > > > few
> > > > > > > things and below is how my drill-override.conf file looks now
> > > > > > >
> > > > > > > drill.classpath.scanning.packages :
> > > > > > ${?drill.classpath.scanning.packages} [
> > > > > > > org.apache.drill.udfs ]
> > > > > > > drill.exec: {
> > > > > > >   cluster-id: "rahul_cluster_com-drillbits",
> > > > > > >   zk.connect: "localhost:5181"
> > > > > > > }
> > > > > > >
> > > > > > > When I restart the drillbits, I get this strange error " Caused
> > by:
> > > > > > > com.fasterxml.jackson.databind.JsonMappingException: Could not
> > > > resolve
> > > > > > type
> > > > > > > id 'hbase' into a subtype of [simple type, class
> > > > > > > org.apache.drill.common.logical.StoragePluginConfig]"
> > > > > > >
> > > > > > > If I moved the package information to the drill-module.conf in
> my
> > > > udf's
> > > > > > > project, then things are working fine. However this requires
> > > > > re-compiling
> > > > > > > the udfs which is not desirable. Is there any other way around
> > > this ?
> > > > > > >
> > > > > > > - Rahul
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Julien
> > > >
> > >
> >
> >
> >
> > --
> > Julien
> >
>

Re: Classpath scanning & udfs

[Jason Altekruse <al...@gmail.com>]

The error you posted originally is from deserializing a storage plugin
config, not the drill-override or drill-module files. That being said, we
find the list of available storage plugins using classpath scanning.

I don't know why the inclusion or exclusion of contents inside of a
module.conf file would impact our scanning for storage plugins, but I
believe this is the issue you are seeing. Somehow the package that contains
the hbase plugin is being removed from the list of packages to scan when
you add the empty file.

Do you want to jump on slack to chat about this?

- Jason

On Mon, Jan 11, 2016 at 10:52 AM, rahul challapalli <
challapallirahul@gmail.com> wrote:

> Julien,
>
> I have an empty drill-module.conf file in the udf jar file. Below are the
> contents of my global drill-override.conf file
>
> drill.classpath.scanning.packages : ${?drill.classpath.scanning.packages} [
> org.apache.drill.udfs ]
> drill.exec: {
>   cluster-id: "rahul_cluster_com-drillbits",
>   zk.connect: "localhost:5181"
> }
>
> With this, the drillbit fails to start with the error I posted in the first
> email. May be I am getting the syntax wrong?
>
> The reason I am insisting an empty drill-module.conf file is because, udf's
> developed prior to the "classpath scanning change" had an empty
> drill-module.conf file and we used to override the udf package name in the
> global conf file.
>
> - Rahul
>
> On Mon, Jan 11, 2016 at 10:46 AM, Julien Le Dem <ju...@dremio.com> wrote:
>
> > Yes I believe that should work:
> > -  add an empty drill-module.conf in the root of the udf jar
> > -  add the package to drill.classpath.scanning.packages in the drill conf
> > (possibly using drill-override.conf)
> >
> > However if you are adding the drill-module.conf file to the jar, you
> might
> > as well add the package in it. (unless there's some other reason)
> >
> > On Mon, Jan 11, 2016 at 10:28 AM, rahul challapalli <
> > challapallirahul@gmail.com> wrote:
> >
> > > Just to be sure, If I have an empty drill-module.conf in the root of my
> > udf
> > > jar, then there is no way to add the package information to the global
> > > drill-override.conf file?
> > >
> > > On Mon, Jan 11, 2016 at 10:26 AM, Julien Le Dem <ju...@dremio.com>
> > wrote:
> > >
> > > > You are correct:
> > > > The jar containing the UDFs should have a drill-module.conf at the
> root
> > > > adding your package to the property drill.classpath.scanning.packages
> > for
> > > > scanning
> > > > drill.classpath.scanning.packages :
> > > ${?drill.classpath.scanning.packages} [
> > > >         my.package.containing.my.udfs
> > > >     ]
> > > > Jars that don't contain a drill-module.conf will not get scanned.
> > > >
> > > >
> > > >
> > > > On Mon, Jan 11, 2016 at 10:17 AM, rahul challapalli <
> > > > challapallirahul@gmail.com> wrote:
> > > >
> > > > > Thanks for your reply Jason.
> > > > >
> > > > > If we cannot override the global configuration file, then for
> > existing
> > > > > UDF's we have to re-compile them by modifying the drill-module.conf
> > > file.
> > > > > If so our UDF's are not backward compatible. Appreciate it if
> someone
> > > can
> > > > > confirm this.
> > > > >
> > > > > - Rahul
> > > > >
> > > > > On Mon, Jan 11, 2016 at 9:59 AM, Jason Altekruse <
> > > > altekrusejason@gmail.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Rahul,
> > > > > >
> > > > > > The error message you are seeing is in reading a storage plugin
> > > > > > configuration file. I am planning to fix these kinds of messages
> to
> > > > > > actually direct users at the file that is failing parsing. I have
> > > seen
> > > > > this
> > > > > > in the past when the classpath was incorrect and one of the
> plugins
> > > > (like
> > > > > > Hbase) was not included.
> > > > > >
> > > > > > Julien can confirm, but I think this might be intentional to have
> > the
> > > > > paths
> > > > > > read out of the modules configuration rather than the global one
> to
> > > > save
> > > > > > time when scanning the path (rather than scanning all of the jars
> > for
> > > > all
> > > > > > paths given in the override file).
> > > > > >
> > > > > > On Fri, Jan 8, 2016 at 4:32 PM, rahul challapalli <
> > > > > > challapallirahul@gmail.com> wrote:
> > > > > >
> > > > > > > Before 1.2, my udfs project contained an empty
> > drill-override.conf
> > > > file
> > > > > > and
> > > > > > > I used to update the drill-override.conf on all the drillbits
> to
> > > > > specify
> > > > > > > the package of my UDF. This is no longer working for me. I
> tried
> > a
> > > > few
> > > > > > > things and below is how my drill-override.conf file looks now
> > > > > > >
> > > > > > > drill.classpath.scanning.packages :
> > > > > > ${?drill.classpath.scanning.packages} [
> > > > > > > org.apache.drill.udfs ]
> > > > > > > drill.exec: {
> > > > > > >   cluster-id: "rahul_cluster_com-drillbits",
> > > > > > >   zk.connect: "localhost:5181"
> > > > > > > }
> > > > > > >
> > > > > > > When I restart the drillbits, I get this strange error " Caused
> > by:
> > > > > > > com.fasterxml.jackson.databind.JsonMappingException: Could not
> > > > resolve
> > > > > > type
> > > > > > > id 'hbase' into a subtype of [simple type, class
> > > > > > > org.apache.drill.common.logical.StoragePluginConfig]"
> > > > > > >
> > > > > > > If I moved the package information to the drill-module.conf in
> my
> > > > udf's
> > > > > > > project, then things are working fine. However this requires
> > > > > re-compiling
> > > > > > > the udfs which is not desirable. Is there any other way around
> > > this ?
> > > > > > >
> > > > > > > - Rahul
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Julien
> > > >
> > >
> >
> >
> >
> > --
> > Julien
> >
>

Re: Classpath scanning & udfs

[rahul challapalli <ch...@gmail.com>]

Julien,

I have an empty drill-module.conf file in the udf jar file. Below are the
contents of my global drill-override.conf file

drill.classpath.scanning.packages : ${?drill.classpath.scanning.packages} [
org.apache.drill.udfs ]
drill.exec: {
  cluster-id: "rahul_cluster_com-drillbits",
  zk.connect: "localhost:5181"
}

With this, the drillbit fails to start with the error I posted in the first
email. May be I am getting the syntax wrong?

The reason I am insisting an empty drill-module.conf file is because, udf's
developed prior to the "classpath scanning change" had an empty
drill-module.conf file and we used to override the udf package name in the
global conf file.

- Rahul

On Mon, Jan 11, 2016 at 10:46 AM, Julien Le Dem <ju...@dremio.com> wrote:

> Yes I believe that should work:
> -  add an empty drill-module.conf in the root of the udf jar
> -  add the package to drill.classpath.scanning.packages in the drill conf
> (possibly using drill-override.conf)
>
> However if you are adding the drill-module.conf file to the jar, you might
> as well add the package in it. (unless there's some other reason)
>
> On Mon, Jan 11, 2016 at 10:28 AM, rahul challapalli <
> challapallirahul@gmail.com> wrote:
>
> > Just to be sure, If I have an empty drill-module.conf in the root of my
> udf
> > jar, then there is no way to add the package information to the global
> > drill-override.conf file?
> >
> > On Mon, Jan 11, 2016 at 10:26 AM, Julien Le Dem <ju...@dremio.com>
> wrote:
> >
> > > You are correct:
> > > The jar containing the UDFs should have a drill-module.conf at the root
> > > adding your package to the property drill.classpath.scanning.packages
> for
> > > scanning
> > > drill.classpath.scanning.packages :
> > ${?drill.classpath.scanning.packages} [
> > >         my.package.containing.my.udfs
> > >     ]
> > > Jars that don't contain a drill-module.conf will not get scanned.
> > >
> > >
> > >
> > > On Mon, Jan 11, 2016 at 10:17 AM, rahul challapalli <
> > > challapallirahul@gmail.com> wrote:
> > >
> > > > Thanks for your reply Jason.
> > > >
> > > > If we cannot override the global configuration file, then for
> existing
> > > > UDF's we have to re-compile them by modifying the drill-module.conf
> > file.
> > > > If so our UDF's are not backward compatible. Appreciate it if someone
> > can
> > > > confirm this.
> > > >
> > > > - Rahul
> > > >
> > > > On Mon, Jan 11, 2016 at 9:59 AM, Jason Altekruse <
> > > altekrusejason@gmail.com
> > > > >
> > > > wrote:
> > > >
> > > > > Rahul,
> > > > >
> > > > > The error message you are seeing is in reading a storage plugin
> > > > > configuration file. I am planning to fix these kinds of messages to
> > > > > actually direct users at the file that is failing parsing. I have
> > seen
> > > > this
> > > > > in the past when the classpath was incorrect and one of the plugins
> > > (like
> > > > > Hbase) was not included.
> > > > >
> > > > > Julien can confirm, but I think this might be intentional to have
> the
> > > > paths
> > > > > read out of the modules configuration rather than the global one to
> > > save
> > > > > time when scanning the path (rather than scanning all of the jars
> for
> > > all
> > > > > paths given in the override file).
> > > > >
> > > > > On Fri, Jan 8, 2016 at 4:32 PM, rahul challapalli <
> > > > > challapallirahul@gmail.com> wrote:
> > > > >
> > > > > > Before 1.2, my udfs project contained an empty
> drill-override.conf
> > > file
> > > > > and
> > > > > > I used to update the drill-override.conf on all the drillbits to
> > > > specify
> > > > > > the package of my UDF. This is no longer working for me. I tried
> a
> > > few
> > > > > > things and below is how my drill-override.conf file looks now
> > > > > >
> > > > > > drill.classpath.scanning.packages :
> > > > > ${?drill.classpath.scanning.packages} [
> > > > > > org.apache.drill.udfs ]
> > > > > > drill.exec: {
> > > > > >   cluster-id: "rahul_cluster_com-drillbits",
> > > > > >   zk.connect: "localhost:5181"
> > > > > > }
> > > > > >
> > > > > > When I restart the drillbits, I get this strange error " Caused
> by:
> > > > > > com.fasterxml.jackson.databind.JsonMappingException: Could not
> > > resolve
> > > > > type
> > > > > > id 'hbase' into a subtype of [simple type, class
> > > > > > org.apache.drill.common.logical.StoragePluginConfig]"
> > > > > >
> > > > > > If I moved the package information to the drill-module.conf in my
> > > udf's
> > > > > > project, then things are working fine. However this requires
> > > > re-compiling
> > > > > > the udfs which is not desirable. Is there any other way around
> > this ?
> > > > > >
> > > > > > - Rahul
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Julien
> > >
> >
>
>
>
> --
> Julien
>

Re: Classpath scanning & udfs

[rahul challapalli <ch...@gmail.com>]

Julien,

I have an empty drill-module.conf file in the udf jar file. Below are the
contents of my global drill-override.conf file

drill.classpath.scanning.packages : ${?drill.classpath.scanning.packages} [
org.apache.drill.udfs ]
drill.exec: {
  cluster-id: "rahul_cluster_com-drillbits",
  zk.connect: "localhost:5181"
}

With this, the drillbit fails to start with the error I posted in the first
email. May be I am getting the syntax wrong?

The reason I am insisting an empty drill-module.conf file is because, udf's
developed prior to the "classpath scanning change" had an empty
drill-module.conf file and we used to override the udf package name in the
global conf file.

- Rahul

On Mon, Jan 11, 2016 at 10:46 AM, Julien Le Dem <ju...@dremio.com> wrote:

> Yes I believe that should work:
> -  add an empty drill-module.conf in the root of the udf jar
> -  add the package to drill.classpath.scanning.packages in the drill conf
> (possibly using drill-override.conf)
>
> However if you are adding the drill-module.conf file to the jar, you might
> as well add the package in it. (unless there's some other reason)
>
> On Mon, Jan 11, 2016 at 10:28 AM, rahul challapalli <
> challapallirahul@gmail.com> wrote:
>
> > Just to be sure, If I have an empty drill-module.conf in the root of my
> udf
> > jar, then there is no way to add the package information to the global
> > drill-override.conf file?
> >
> > On Mon, Jan 11, 2016 at 10:26 AM, Julien Le Dem <ju...@dremio.com>
> wrote:
> >
> > > You are correct:
> > > The jar containing the UDFs should have a drill-module.conf at the root
> > > adding your package to the property drill.classpath.scanning.packages
> for
> > > scanning
> > > drill.classpath.scanning.packages :
> > ${?drill.classpath.scanning.packages} [
> > >         my.package.containing.my.udfs
> > >     ]
> > > Jars that don't contain a drill-module.conf will not get scanned.
> > >
> > >
> > >
> > > On Mon, Jan 11, 2016 at 10:17 AM, rahul challapalli <
> > > challapallirahul@gmail.com> wrote:
> > >
> > > > Thanks for your reply Jason.
> > > >
> > > > If we cannot override the global configuration file, then for
> existing
> > > > UDF's we have to re-compile them by modifying the drill-module.conf
> > file.
> > > > If so our UDF's are not backward compatible. Appreciate it if someone
> > can
> > > > confirm this.
> > > >
> > > > - Rahul
> > > >
> > > > On Mon, Jan 11, 2016 at 9:59 AM, Jason Altekruse <
> > > altekrusejason@gmail.com
> > > > >
> > > > wrote:
> > > >
> > > > > Rahul,
> > > > >
> > > > > The error message you are seeing is in reading a storage plugin
> > > > > configuration file. I am planning to fix these kinds of messages to
> > > > > actually direct users at the file that is failing parsing. I have
> > seen
> > > > this
> > > > > in the past when the classpath was incorrect and one of the plugins
> > > (like
> > > > > Hbase) was not included.
> > > > >
> > > > > Julien can confirm, but I think this might be intentional to have
> the
> > > > paths
> > > > > read out of the modules configuration rather than the global one to
> > > save
> > > > > time when scanning the path (rather than scanning all of the jars
> for
> > > all
> > > > > paths given in the override file).
> > > > >
> > > > > On Fri, Jan 8, 2016 at 4:32 PM, rahul challapalli <
> > > > > challapallirahul@gmail.com> wrote:
> > > > >
> > > > > > Before 1.2, my udfs project contained an empty
> drill-override.conf
> > > file
> > > > > and
> > > > > > I used to update the drill-override.conf on all the drillbits to
> > > > specify
> > > > > > the package of my UDF. This is no longer working for me. I tried
> a
> > > few
> > > > > > things and below is how my drill-override.conf file looks now
> > > > > >
> > > > > > drill.classpath.scanning.packages :
> > > > > ${?drill.classpath.scanning.packages} [
> > > > > > org.apache.drill.udfs ]
> > > > > > drill.exec: {
> > > > > >   cluster-id: "rahul_cluster_com-drillbits",
> > > > > >   zk.connect: "localhost:5181"
> > > > > > }
> > > > > >
> > > > > > When I restart the drillbits, I get this strange error " Caused
> by:
> > > > > > com.fasterxml.jackson.databind.JsonMappingException: Could not
> > > resolve
> > > > > type
> > > > > > id 'hbase' into a subtype of [simple type, class
> > > > > > org.apache.drill.common.logical.StoragePluginConfig]"
> > > > > >
> > > > > > If I moved the package information to the drill-module.conf in my
> > > udf's
> > > > > > project, then things are working fine. However this requires
> > > > re-compiling
> > > > > > the udfs which is not desirable. Is there any other way around
> > this ?
> > > > > >
> > > > > > - Rahul
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Julien
> > >
> >
>
>
>
> --
> Julien
>

Re: Classpath scanning & udfs

[Julien Le Dem <ju...@dremio.com>]

Yes I believe that should work:
-  add an empty drill-module.conf in the root of the udf jar
-  add the package to drill.classpath.scanning.packages in the drill conf
(possibly using drill-override.conf)

However if you are adding the drill-module.conf file to the jar, you might
as well add the package in it. (unless there's some other reason)

On Mon, Jan 11, 2016 at 10:28 AM, rahul challapalli <
challapallirahul@gmail.com> wrote:

> Just to be sure, If I have an empty drill-module.conf in the root of my udf
> jar, then there is no way to add the package information to the global
> drill-override.conf file?
>
> On Mon, Jan 11, 2016 at 10:26 AM, Julien Le Dem <ju...@dremio.com> wrote:
>
> > You are correct:
> > The jar containing the UDFs should have a drill-module.conf at the root
> > adding your package to the property drill.classpath.scanning.packages for
> > scanning
> > drill.classpath.scanning.packages :
> ${?drill.classpath.scanning.packages} [
> >         my.package.containing.my.udfs
> >     ]
> > Jars that don't contain a drill-module.conf will not get scanned.
> >
> >
> >
> > On Mon, Jan 11, 2016 at 10:17 AM, rahul challapalli <
> > challapallirahul@gmail.com> wrote:
> >
> > > Thanks for your reply Jason.
> > >
> > > If we cannot override the global configuration file, then for existing
> > > UDF's we have to re-compile them by modifying the drill-module.conf
> file.
> > > If so our UDF's are not backward compatible. Appreciate it if someone
> can
> > > confirm this.
> > >
> > > - Rahul
> > >
> > > On Mon, Jan 11, 2016 at 9:59 AM, Jason Altekruse <
> > altekrusejason@gmail.com
> > > >
> > > wrote:
> > >
> > > > Rahul,
> > > >
> > > > The error message you are seeing is in reading a storage plugin
> > > > configuration file. I am planning to fix these kinds of messages to
> > > > actually direct users at the file that is failing parsing. I have
> seen
> > > this
> > > > in the past when the classpath was incorrect and one of the plugins
> > (like
> > > > Hbase) was not included.
> > > >
> > > > Julien can confirm, but I think this might be intentional to have the
> > > paths
> > > > read out of the modules configuration rather than the global one to
> > save
> > > > time when scanning the path (rather than scanning all of the jars for
> > all
> > > > paths given in the override file).
> > > >
> > > > On Fri, Jan 8, 2016 at 4:32 PM, rahul challapalli <
> > > > challapallirahul@gmail.com> wrote:
> > > >
> > > > > Before 1.2, my udfs project contained an empty drill-override.conf
> > file
> > > > and
> > > > > I used to update the drill-override.conf on all the drillbits to
> > > specify
> > > > > the package of my UDF. This is no longer working for me. I tried a
> > few
> > > > > things and below is how my drill-override.conf file looks now
> > > > >
> > > > > drill.classpath.scanning.packages :
> > > > ${?drill.classpath.scanning.packages} [
> > > > > org.apache.drill.udfs ]
> > > > > drill.exec: {
> > > > >   cluster-id: "rahul_cluster_com-drillbits",
> > > > >   zk.connect: "localhost:5181"
> > > > > }
> > > > >
> > > > > When I restart the drillbits, I get this strange error " Caused by:
> > > > > com.fasterxml.jackson.databind.JsonMappingException: Could not
> > resolve
> > > > type
> > > > > id 'hbase' into a subtype of [simple type, class
> > > > > org.apache.drill.common.logical.StoragePluginConfig]"
> > > > >
> > > > > If I moved the package information to the drill-module.conf in my
> > udf's
> > > > > project, then things are working fine. However this requires
> > > re-compiling
> > > > > the udfs which is not desirable. Is there any other way around
> this ?
> > > > >
> > > > > - Rahul
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Julien
> >
>



-- 
Julien

Re: Classpath scanning & udfs

[Julien Le Dem <ju...@dremio.com>]

Yes I believe that should work:
-  add an empty drill-module.conf in the root of the udf jar
-  add the package to drill.classpath.scanning.packages in the drill conf
(possibly using drill-override.conf)

However if you are adding the drill-module.conf file to the jar, you might
as well add the package in it. (unless there's some other reason)

On Mon, Jan 11, 2016 at 10:28 AM, rahul challapalli <
challapallirahul@gmail.com> wrote:

> Just to be sure, If I have an empty drill-module.conf in the root of my udf
> jar, then there is no way to add the package information to the global
> drill-override.conf file?
>
> On Mon, Jan 11, 2016 at 10:26 AM, Julien Le Dem <ju...@dremio.com> wrote:
>
> > You are correct:
> > The jar containing the UDFs should have a drill-module.conf at the root
> > adding your package to the property drill.classpath.scanning.packages for
> > scanning
> > drill.classpath.scanning.packages :
> ${?drill.classpath.scanning.packages} [
> >         my.package.containing.my.udfs
> >     ]
> > Jars that don't contain a drill-module.conf will not get scanned.
> >
> >
> >
> > On Mon, Jan 11, 2016 at 10:17 AM, rahul challapalli <
> > challapallirahul@gmail.com> wrote:
> >
> > > Thanks for your reply Jason.
> > >
> > > If we cannot override the global configuration file, then for existing
> > > UDF's we have to re-compile them by modifying the drill-module.conf
> file.
> > > If so our UDF's are not backward compatible. Appreciate it if someone
> can
> > > confirm this.
> > >
> > > - Rahul
> > >
> > > On Mon, Jan 11, 2016 at 9:59 AM, Jason Altekruse <
> > altekrusejason@gmail.com
> > > >
> > > wrote:
> > >
> > > > Rahul,
> > > >
> > > > The error message you are seeing is in reading a storage plugin
> > > > configuration file. I am planning to fix these kinds of messages to
> > > > actually direct users at the file that is failing parsing. I have
> seen
> > > this
> > > > in the past when the classpath was incorrect and one of the plugins
> > (like
> > > > Hbase) was not included.
> > > >
> > > > Julien can confirm, but I think this might be intentional to have the
> > > paths
> > > > read out of the modules configuration rather than the global one to
> > save
> > > > time when scanning the path (rather than scanning all of the jars for
> > all
> > > > paths given in the override file).
> > > >
> > > > On Fri, Jan 8, 2016 at 4:32 PM, rahul challapalli <
> > > > challapallirahul@gmail.com> wrote:
> > > >
> > > > > Before 1.2, my udfs project contained an empty drill-override.conf
> > file
> > > > and
> > > > > I used to update the drill-override.conf on all the drillbits to
> > > specify
> > > > > the package of my UDF. This is no longer working for me. I tried a
> > few
> > > > > things and below is how my drill-override.conf file looks now
> > > > >
> > > > > drill.classpath.scanning.packages :
> > > > ${?drill.classpath.scanning.packages} [
> > > > > org.apache.drill.udfs ]
> > > > > drill.exec: {
> > > > >   cluster-id: "rahul_cluster_com-drillbits",
> > > > >   zk.connect: "localhost:5181"
> > > > > }
> > > > >
> > > > > When I restart the drillbits, I get this strange error " Caused by:
> > > > > com.fasterxml.jackson.databind.JsonMappingException: Could not
> > resolve
> > > > type
> > > > > id 'hbase' into a subtype of [simple type, class
> > > > > org.apache.drill.common.logical.StoragePluginConfig]"
> > > > >
> > > > > If I moved the package information to the drill-module.conf in my
> > udf's
> > > > > project, then things are working fine. However this requires
> > > re-compiling
> > > > > the udfs which is not desirable. Is there any other way around
> this ?
> > > > >
> > > > > - Rahul
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Julien
> >
>



-- 
Julien

Re: Classpath scanning & udfs

[rahul challapalli <ch...@gmail.com>]

Just to be sure, If I have an empty drill-module.conf in the root of my udf
jar, then there is no way to add the package information to the global
drill-override.conf file?

On Mon, Jan 11, 2016 at 10:26 AM, Julien Le Dem <ju...@dremio.com> wrote:

> You are correct:
> The jar containing the UDFs should have a drill-module.conf at the root
> adding your package to the property drill.classpath.scanning.packages for
> scanning
> drill.classpath.scanning.packages : ${?drill.classpath.scanning.packages} [
>         my.package.containing.my.udfs
>     ]
> Jars that don't contain a drill-module.conf will not get scanned.
>
>
>
> On Mon, Jan 11, 2016 at 10:17 AM, rahul challapalli <
> challapallirahul@gmail.com> wrote:
>
> > Thanks for your reply Jason.
> >
> > If we cannot override the global configuration file, then for existing
> > UDF's we have to re-compile them by modifying the drill-module.conf file.
> > If so our UDF's are not backward compatible. Appreciate it if someone can
> > confirm this.
> >
> > - Rahul
> >
> > On Mon, Jan 11, 2016 at 9:59 AM, Jason Altekruse <
> altekrusejason@gmail.com
> > >
> > wrote:
> >
> > > Rahul,
> > >
> > > The error message you are seeing is in reading a storage plugin
> > > configuration file. I am planning to fix these kinds of messages to
> > > actually direct users at the file that is failing parsing. I have seen
> > this
> > > in the past when the classpath was incorrect and one of the plugins
> (like
> > > Hbase) was not included.
> > >
> > > Julien can confirm, but I think this might be intentional to have the
> > paths
> > > read out of the modules configuration rather than the global one to
> save
> > > time when scanning the path (rather than scanning all of the jars for
> all
> > > paths given in the override file).
> > >
> > > On Fri, Jan 8, 2016 at 4:32 PM, rahul challapalli <
> > > challapallirahul@gmail.com> wrote:
> > >
> > > > Before 1.2, my udfs project contained an empty drill-override.conf
> file
> > > and
> > > > I used to update the drill-override.conf on all the drillbits to
> > specify
> > > > the package of my UDF. This is no longer working for me. I tried a
> few
> > > > things and below is how my drill-override.conf file looks now
> > > >
> > > > drill.classpath.scanning.packages :
> > > ${?drill.classpath.scanning.packages} [
> > > > org.apache.drill.udfs ]
> > > > drill.exec: {
> > > >   cluster-id: "rahul_cluster_com-drillbits",
> > > >   zk.connect: "localhost:5181"
> > > > }
> > > >
> > > > When I restart the drillbits, I get this strange error " Caused by:
> > > > com.fasterxml.jackson.databind.JsonMappingException: Could not
> resolve
> > > type
> > > > id 'hbase' into a subtype of [simple type, class
> > > > org.apache.drill.common.logical.StoragePluginConfig]"
> > > >
> > > > If I moved the package information to the drill-module.conf in my
> udf's
> > > > project, then things are working fine. However this requires
> > re-compiling
> > > > the udfs which is not desirable. Is there any other way around this ?
> > > >
> > > > - Rahul
> > > >
> > >
> >
>
>
>
> --
> Julien
>

Re: Classpath scanning & udfs

[rahul challapalli <ch...@gmail.com>]

Just to be sure, If I have an empty drill-module.conf in the root of my udf
jar, then there is no way to add the package information to the global
drill-override.conf file?

On Mon, Jan 11, 2016 at 10:26 AM, Julien Le Dem <ju...@dremio.com> wrote:

> You are correct:
> The jar containing the UDFs should have a drill-module.conf at the root
> adding your package to the property drill.classpath.scanning.packages for
> scanning
> drill.classpath.scanning.packages : ${?drill.classpath.scanning.packages} [
>         my.package.containing.my.udfs
>     ]
> Jars that don't contain a drill-module.conf will not get scanned.
>
>
>
> On Mon, Jan 11, 2016 at 10:17 AM, rahul challapalli <
> challapallirahul@gmail.com> wrote:
>
> > Thanks for your reply Jason.
> >
> > If we cannot override the global configuration file, then for existing
> > UDF's we have to re-compile them by modifying the drill-module.conf file.
> > If so our UDF's are not backward compatible. Appreciate it if someone can
> > confirm this.
> >
> > - Rahul
> >
> > On Mon, Jan 11, 2016 at 9:59 AM, Jason Altekruse <
> altekrusejason@gmail.com
> > >
> > wrote:
> >
> > > Rahul,
> > >
> > > The error message you are seeing is in reading a storage plugin
> > > configuration file. I am planning to fix these kinds of messages to
> > > actually direct users at the file that is failing parsing. I have seen
> > this
> > > in the past when the classpath was incorrect and one of the plugins
> (like
> > > Hbase) was not included.
> > >
> > > Julien can confirm, but I think this might be intentional to have the
> > paths
> > > read out of the modules configuration rather than the global one to
> save
> > > time when scanning the path (rather than scanning all of the jars for
> all
> > > paths given in the override file).
> > >
> > > On Fri, Jan 8, 2016 at 4:32 PM, rahul challapalli <
> > > challapallirahul@gmail.com> wrote:
> > >
> > > > Before 1.2, my udfs project contained an empty drill-override.conf
> file
> > > and
> > > > I used to update the drill-override.conf on all the drillbits to
> > specify
> > > > the package of my UDF. This is no longer working for me. I tried a
> few
> > > > things and below is how my drill-override.conf file looks now
> > > >
> > > > drill.classpath.scanning.packages :
> > > ${?drill.classpath.scanning.packages} [
> > > > org.apache.drill.udfs ]
> > > > drill.exec: {
> > > >   cluster-id: "rahul_cluster_com-drillbits",
> > > >   zk.connect: "localhost:5181"
> > > > }
> > > >
> > > > When I restart the drillbits, I get this strange error " Caused by:
> > > > com.fasterxml.jackson.databind.JsonMappingException: Could not
> resolve
> > > type
> > > > id 'hbase' into a subtype of [simple type, class
> > > > org.apache.drill.common.logical.StoragePluginConfig]"
> > > >
> > > > If I moved the package information to the drill-module.conf in my
> udf's
> > > > project, then things are working fine. However this requires
> > re-compiling
> > > > the udfs which is not desirable. Is there any other way around this ?
> > > >
> > > > - Rahul
> > > >
> > >
> >
>
>
>
> --
> Julien
>

Re: Classpath scanning & udfs

[Julien Le Dem <ju...@dremio.com>]

You are correct:
The jar containing the UDFs should have a drill-module.conf at the root
adding your package to the property drill.classpath.scanning.packages for
scanning
drill.classpath.scanning.packages : ${?drill.classpath.scanning.packages} [
        my.package.containing.my.udfs
    ]
Jars that don't contain a drill-module.conf will not get scanned.



On Mon, Jan 11, 2016 at 10:17 AM, rahul challapalli <
challapallirahul@gmail.com> wrote:

> Thanks for your reply Jason.
>
> If we cannot override the global configuration file, then for existing
> UDF's we have to re-compile them by modifying the drill-module.conf file.
> If so our UDF's are not backward compatible. Appreciate it if someone can
> confirm this.
>
> - Rahul
>
> On Mon, Jan 11, 2016 at 9:59 AM, Jason Altekruse <altekrusejason@gmail.com
> >
> wrote:
>
> > Rahul,
> >
> > The error message you are seeing is in reading a storage plugin
> > configuration file. I am planning to fix these kinds of messages to
> > actually direct users at the file that is failing parsing. I have seen
> this
> > in the past when the classpath was incorrect and one of the plugins (like
> > Hbase) was not included.
> >
> > Julien can confirm, but I think this might be intentional to have the
> paths
> > read out of the modules configuration rather than the global one to save
> > time when scanning the path (rather than scanning all of the jars for all
> > paths given in the override file).
> >
> > On Fri, Jan 8, 2016 at 4:32 PM, rahul challapalli <
> > challapallirahul@gmail.com> wrote:
> >
> > > Before 1.2, my udfs project contained an empty drill-override.conf file
> > and
> > > I used to update the drill-override.conf on all the drillbits to
> specify
> > > the package of my UDF. This is no longer working for me. I tried a few
> > > things and below is how my drill-override.conf file looks now
> > >
> > > drill.classpath.scanning.packages :
> > ${?drill.classpath.scanning.packages} [
> > > org.apache.drill.udfs ]
> > > drill.exec: {
> > >   cluster-id: "rahul_cluster_com-drillbits",
> > >   zk.connect: "localhost:5181"
> > > }
> > >
> > > When I restart the drillbits, I get this strange error " Caused by:
> > > com.fasterxml.jackson.databind.JsonMappingException: Could not resolve
> > type
> > > id 'hbase' into a subtype of [simple type, class
> > > org.apache.drill.common.logical.StoragePluginConfig]"
> > >
> > > If I moved the package information to the drill-module.conf in my udf's
> > > project, then things are working fine. However this requires
> re-compiling
> > > the udfs which is not desirable. Is there any other way around this ?
> > >
> > > - Rahul
> > >
> >
>



-- 
Julien

Re: Classpath scanning & udfs

[Julien Le Dem <ju...@dremio.com>]

You are correct:
The jar containing the UDFs should have a drill-module.conf at the root
adding your package to the property drill.classpath.scanning.packages for
scanning
drill.classpath.scanning.packages : ${?drill.classpath.scanning.packages} [
        my.package.containing.my.udfs
    ]
Jars that don't contain a drill-module.conf will not get scanned.



On Mon, Jan 11, 2016 at 10:17 AM, rahul challapalli <
challapallirahul@gmail.com> wrote:

> Thanks for your reply Jason.
>
> If we cannot override the global configuration file, then for existing
> UDF's we have to re-compile them by modifying the drill-module.conf file.
> If so our UDF's are not backward compatible. Appreciate it if someone can
> confirm this.
>
> - Rahul
>
> On Mon, Jan 11, 2016 at 9:59 AM, Jason Altekruse <altekrusejason@gmail.com
> >
> wrote:
>
> > Rahul,
> >
> > The error message you are seeing is in reading a storage plugin
> > configuration file. I am planning to fix these kinds of messages to
> > actually direct users at the file that is failing parsing. I have seen
> this
> > in the past when the classpath was incorrect and one of the plugins (like
> > Hbase) was not included.
> >
> > Julien can confirm, but I think this might be intentional to have the
> paths
> > read out of the modules configuration rather than the global one to save
> > time when scanning the path (rather than scanning all of the jars for all
> > paths given in the override file).
> >
> > On Fri, Jan 8, 2016 at 4:32 PM, rahul challapalli <
> > challapallirahul@gmail.com> wrote:
> >
> > > Before 1.2, my udfs project contained an empty drill-override.conf file
> > and
> > > I used to update the drill-override.conf on all the drillbits to
> specify
> > > the package of my UDF. This is no longer working for me. I tried a few
> > > things and below is how my drill-override.conf file looks now
> > >
> > > drill.classpath.scanning.packages :
> > ${?drill.classpath.scanning.packages} [
> > > org.apache.drill.udfs ]
> > > drill.exec: {
> > >   cluster-id: "rahul_cluster_com-drillbits",
> > >   zk.connect: "localhost:5181"
> > > }
> > >
> > > When I restart the drillbits, I get this strange error " Caused by:
> > > com.fasterxml.jackson.databind.JsonMappingException: Could not resolve
> > type
> > > id 'hbase' into a subtype of [simple type, class
> > > org.apache.drill.common.logical.StoragePluginConfig]"
> > >
> > > If I moved the package information to the drill-module.conf in my udf's
> > > project, then things are working fine. However this requires
> re-compiling
> > > the udfs which is not desirable. Is there any other way around this ?
> > >
> > > - Rahul
> > >
> >
>



-- 
Julien

Re: Classpath scanning & udfs

[Julien Le Dem <ju...@dremio.com>]

Jinfeng:
It seems that step 1. 2. 3. in the page you link are correct.
Quote bellow:

   1. Add the sources JAR file and the classes JAR file for the custom
   function to the Drill classpath on all nodes running a Drillbit. To add the
   JAR files, copy them to <drill installation directory>/jars/3rdparty.
   2. Your class jar file should contain a drill-module.conf file at its
   root.
   3.

   The drill-module.conf file should contain the packages to scan for
   functionsdrill.classpath.scanning.package+=. Separate package names with
   a comma.


On Mon, Jan 11, 2016 at 10:28 AM, Jinfeng Ni <ji...@gmail.com> wrote:

> If what Rahul described is true, I think at minimum we should document
> the backward compatible issue in Drill new release doc. For today,
> drill doc seems to still use the old way [1]
>
> [1] https://drill.apache.org/docs/adding-custom-functions-to-drill/
>
>
> On Mon, Jan 11, 2016 at 10:17 AM, rahul challapalli
> <ch...@gmail.com> wrote:
> > Thanks for your reply Jason.
> >
> > If we cannot override the global configuration file, then for existing
> > UDF's we have to re-compile them by modifying the drill-module.conf file.
> > If so our UDF's are not backward compatible. Appreciate it if someone can
> > confirm this.
> >
> > - Rahul
> >
> > On Mon, Jan 11, 2016 at 9:59 AM, Jason Altekruse <
> altekrusejason@gmail.com>
> > wrote:
> >
> >> Rahul,
> >>
> >> The error message you are seeing is in reading a storage plugin
> >> configuration file. I am planning to fix these kinds of messages to
> >> actually direct users at the file that is failing parsing. I have seen
> this
> >> in the past when the classpath was incorrect and one of the plugins
> (like
> >> Hbase) was not included.
> >>
> >> Julien can confirm, but I think this might be intentional to have the
> paths
> >> read out of the modules configuration rather than the global one to save
> >> time when scanning the path (rather than scanning all of the jars for
> all
> >> paths given in the override file).
> >>
> >> On Fri, Jan 8, 2016 at 4:32 PM, rahul challapalli <
> >> challapallirahul@gmail.com> wrote:
> >>
> >> > Before 1.2, my udfs project contained an empty drill-override.conf
> file
> >> and
> >> > I used to update the drill-override.conf on all the drillbits to
> specify
> >> > the package of my UDF. This is no longer working for me. I tried a few
> >> > things and below is how my drill-override.conf file looks now
> >> >
> >> > drill.classpath.scanning.packages :
> >> ${?drill.classpath.scanning.packages} [
> >> > org.apache.drill.udfs ]
> >> > drill.exec: {
> >> >   cluster-id: "rahul_cluster_com-drillbits",
> >> >   zk.connect: "localhost:5181"
> >> > }
> >> >
> >> > When I restart the drillbits, I get this strange error " Caused by:
> >> > com.fasterxml.jackson.databind.JsonMappingException: Could not resolve
> >> type
> >> > id 'hbase' into a subtype of [simple type, class
> >> > org.apache.drill.common.logical.StoragePluginConfig]"
> >> >
> >> > If I moved the package information to the drill-module.conf in my
> udf's
> >> > project, then things are working fine. However this requires
> re-compiling
> >> > the udfs which is not desirable. Is there any other way around this ?
> >> >
> >> > - Rahul
> >> >
> >>
>



-- 
Julien

Re: Classpath scanning & udfs

[Jinfeng Ni <ji...@gmail.com>]

If what Rahul described is true, I think at minimum we should document
the backward compatible issue in Drill new release doc. For today,
drill doc seems to still use the old way [1]

[1] https://drill.apache.org/docs/adding-custom-functions-to-drill/


On Mon, Jan 11, 2016 at 10:17 AM, rahul challapalli
<ch...@gmail.com> wrote:
> Thanks for your reply Jason.
>
> If we cannot override the global configuration file, then for existing
> UDF's we have to re-compile them by modifying the drill-module.conf file.
> If so our UDF's are not backward compatible. Appreciate it if someone can
> confirm this.
>
> - Rahul
>
> On Mon, Jan 11, 2016 at 9:59 AM, Jason Altekruse <al...@gmail.com>
> wrote:
>
>> Rahul,
>>
>> The error message you are seeing is in reading a storage plugin
>> configuration file. I am planning to fix these kinds of messages to
>> actually direct users at the file that is failing parsing. I have seen this
>> in the past when the classpath was incorrect and one of the plugins (like
>> Hbase) was not included.
>>
>> Julien can confirm, but I think this might be intentional to have the paths
>> read out of the modules configuration rather than the global one to save
>> time when scanning the path (rather than scanning all of the jars for all
>> paths given in the override file).
>>
>> On Fri, Jan 8, 2016 at 4:32 PM, rahul challapalli <
>> challapallirahul@gmail.com> wrote:
>>
>> > Before 1.2, my udfs project contained an empty drill-override.conf file
>> and
>> > I used to update the drill-override.conf on all the drillbits to specify
>> > the package of my UDF. This is no longer working for me. I tried a few
>> > things and below is how my drill-override.conf file looks now
>> >
>> > drill.classpath.scanning.packages :
>> ${?drill.classpath.scanning.packages} [
>> > org.apache.drill.udfs ]
>> > drill.exec: {
>> >   cluster-id: "rahul_cluster_com-drillbits",
>> >   zk.connect: "localhost:5181"
>> > }
>> >
>> > When I restart the drillbits, I get this strange error " Caused by:
>> > com.fasterxml.jackson.databind.JsonMappingException: Could not resolve
>> type
>> > id 'hbase' into a subtype of [simple type, class
>> > org.apache.drill.common.logical.StoragePluginConfig]"
>> >
>> > If I moved the package information to the drill-module.conf in my udf's
>> > project, then things are working fine. However this requires re-compiling
>> > the udfs which is not desirable. Is there any other way around this ?
>> >
>> > - Rahul
>> >
>>

Re: Classpath scanning & udfs

[Jinfeng Ni <ji...@gmail.com>]

If what Rahul described is true, I think at minimum we should document
the backward compatible issue in Drill new release doc. For today,
drill doc seems to still use the old way [1]

[1] https://drill.apache.org/docs/adding-custom-functions-to-drill/


On Mon, Jan 11, 2016 at 10:17 AM, rahul challapalli
<ch...@gmail.com> wrote:
> Thanks for your reply Jason.
>
> If we cannot override the global configuration file, then for existing
> UDF's we have to re-compile them by modifying the drill-module.conf file.
> If so our UDF's are not backward compatible. Appreciate it if someone can
> confirm this.
>
> - Rahul
>
> On Mon, Jan 11, 2016 at 9:59 AM, Jason Altekruse <al...@gmail.com>
> wrote:
>
>> Rahul,
>>
>> The error message you are seeing is in reading a storage plugin
>> configuration file. I am planning to fix these kinds of messages to
>> actually direct users at the file that is failing parsing. I have seen this
>> in the past when the classpath was incorrect and one of the plugins (like
>> Hbase) was not included.
>>
>> Julien can confirm, but I think this might be intentional to have the paths
>> read out of the modules configuration rather than the global one to save
>> time when scanning the path (rather than scanning all of the jars for all
>> paths given in the override file).
>>
>> On Fri, Jan 8, 2016 at 4:32 PM, rahul challapalli <
>> challapallirahul@gmail.com> wrote:
>>
>> > Before 1.2, my udfs project contained an empty drill-override.conf file
>> and
>> > I used to update the drill-override.conf on all the drillbits to specify
>> > the package of my UDF. This is no longer working for me. I tried a few
>> > things and below is how my drill-override.conf file looks now
>> >
>> > drill.classpath.scanning.packages :
>> ${?drill.classpath.scanning.packages} [
>> > org.apache.drill.udfs ]
>> > drill.exec: {
>> >   cluster-id: "rahul_cluster_com-drillbits",
>> >   zk.connect: "localhost:5181"
>> > }
>> >
>> > When I restart the drillbits, I get this strange error " Caused by:
>> > com.fasterxml.jackson.databind.JsonMappingException: Could not resolve
>> type
>> > id 'hbase' into a subtype of [simple type, class
>> > org.apache.drill.common.logical.StoragePluginConfig]"
>> >
>> > If I moved the package information to the drill-module.conf in my udf's
>> > project, then things are working fine. However this requires re-compiling
>> > the udfs which is not desirable. Is there any other way around this ?
>> >
>> > - Rahul
>> >
>>

Re: Classpath scanning & udfs

[rahul challapalli <ch...@gmail.com>]

Thanks for your reply Jason.

If we cannot override the global configuration file, then for existing
UDF's we have to re-compile them by modifying the drill-module.conf file.
If so our UDF's are not backward compatible. Appreciate it if someone can
confirm this.

- Rahul

On Mon, Jan 11, 2016 at 9:59 AM, Jason Altekruse <al...@gmail.com>
wrote:

> Rahul,
>
> The error message you are seeing is in reading a storage plugin
> configuration file. I am planning to fix these kinds of messages to
> actually direct users at the file that is failing parsing. I have seen this
> in the past when the classpath was incorrect and one of the plugins (like
> Hbase) was not included.
>
> Julien can confirm, but I think this might be intentional to have the paths
> read out of the modules configuration rather than the global one to save
> time when scanning the path (rather than scanning all of the jars for all
> paths given in the override file).
>
> On Fri, Jan 8, 2016 at 4:32 PM, rahul challapalli <
> challapallirahul@gmail.com> wrote:
>
> > Before 1.2, my udfs project contained an empty drill-override.conf file
> and
> > I used to update the drill-override.conf on all the drillbits to specify
> > the package of my UDF. This is no longer working for me. I tried a few
> > things and below is how my drill-override.conf file looks now
> >
> > drill.classpath.scanning.packages :
> ${?drill.classpath.scanning.packages} [
> > org.apache.drill.udfs ]
> > drill.exec: {
> >   cluster-id: "rahul_cluster_com-drillbits",
> >   zk.connect: "localhost:5181"
> > }
> >
> > When I restart the drillbits, I get this strange error " Caused by:
> > com.fasterxml.jackson.databind.JsonMappingException: Could not resolve
> type
> > id 'hbase' into a subtype of [simple type, class
> > org.apache.drill.common.logical.StoragePluginConfig]"
> >
> > If I moved the package information to the drill-module.conf in my udf's
> > project, then things are working fine. However this requires re-compiling
> > the udfs which is not desirable. Is there any other way around this ?
> >
> > - Rahul
> >
>

Re: Classpath scanning & udfs

[rahul challapalli <ch...@gmail.com>]

Thanks for your reply Jason.

If we cannot override the global configuration file, then for existing
UDF's we have to re-compile them by modifying the drill-module.conf file.
If so our UDF's are not backward compatible. Appreciate it if someone can
confirm this.

- Rahul

On Mon, Jan 11, 2016 at 9:59 AM, Jason Altekruse <al...@gmail.com>
wrote:

> Rahul,
>
> The error message you are seeing is in reading a storage plugin
> configuration file. I am planning to fix these kinds of messages to
> actually direct users at the file that is failing parsing. I have seen this
> in the past when the classpath was incorrect and one of the plugins (like
> Hbase) was not included.
>
> Julien can confirm, but I think this might be intentional to have the paths
> read out of the modules configuration rather than the global one to save
> time when scanning the path (rather than scanning all of the jars for all
> paths given in the override file).
>
> On Fri, Jan 8, 2016 at 4:32 PM, rahul challapalli <
> challapallirahul@gmail.com> wrote:
>
> > Before 1.2, my udfs project contained an empty drill-override.conf file
> and
> > I used to update the drill-override.conf on all the drillbits to specify
> > the package of my UDF. This is no longer working for me. I tried a few
> > things and below is how my drill-override.conf file looks now
> >
> > drill.classpath.scanning.packages :
> ${?drill.classpath.scanning.packages} [
> > org.apache.drill.udfs ]
> > drill.exec: {
> >   cluster-id: "rahul_cluster_com-drillbits",
> >   zk.connect: "localhost:5181"
> > }
> >
> > When I restart the drillbits, I get this strange error " Caused by:
> > com.fasterxml.jackson.databind.JsonMappingException: Could not resolve
> type
> > id 'hbase' into a subtype of [simple type, class
> > org.apache.drill.common.logical.StoragePluginConfig]"
> >
> > If I moved the package information to the drill-module.conf in my udf's
> > project, then things are working fine. However this requires re-compiling
> > the udfs which is not desirable. Is there any other way around this ?
> >
> > - Rahul
> >
>

Re: Classpath scanning & udfs

[Jason Altekruse <al...@gmail.com>]

Rahul,

The error message you are seeing is in reading a storage plugin
configuration file. I am planning to fix these kinds of messages to
actually direct users at the file that is failing parsing. I have seen this
in the past when the classpath was incorrect and one of the plugins (like
Hbase) was not included.

Julien can confirm, but I think this might be intentional to have the paths
read out of the modules configuration rather than the global one to save
time when scanning the path (rather than scanning all of the jars for all
paths given in the override file).

On Fri, Jan 8, 2016 at 4:32 PM, rahul challapalli <
challapallirahul@gmail.com> wrote:

> Before 1.2, my udfs project contained an empty drill-override.conf file and
> I used to update the drill-override.conf on all the drillbits to specify
> the package of my UDF. This is no longer working for me. I tried a few
> things and below is how my drill-override.conf file looks now
>
> drill.classpath.scanning.packages : ${?drill.classpath.scanning.packages} [
> org.apache.drill.udfs ]
> drill.exec: {
>   cluster-id: "rahul_cluster_com-drillbits",
>   zk.connect: "localhost:5181"
> }
>
> When I restart the drillbits, I get this strange error " Caused by:
> com.fasterxml.jackson.databind.JsonMappingException: Could not resolve type
> id 'hbase' into a subtype of [simple type, class
> org.apache.drill.common.logical.StoragePluginConfig]"
>
> If I moved the package information to the drill-module.conf in my udf's
> project, then things are working fine. However this requires re-compiling
> the udfs which is not desirable. Is there any other way around this ?
>
> - Rahul
>

Re: Classpath scanning & udfs

[Jason Altekruse <al...@gmail.com>]

Rahul,

The error message you are seeing is in reading a storage plugin
configuration file. I am planning to fix these kinds of messages to
actually direct users at the file that is failing parsing. I have seen this
in the past when the classpath was incorrect and one of the plugins (like
Hbase) was not included.

Julien can confirm, but I think this might be intentional to have the paths
read out of the modules configuration rather than the global one to save
time when scanning the path (rather than scanning all of the jars for all
paths given in the override file).

On Fri, Jan 8, 2016 at 4:32 PM, rahul challapalli <
challapallirahul@gmail.com> wrote:

> Before 1.2, my udfs project contained an empty drill-override.conf file and
> I used to update the drill-override.conf on all the drillbits to specify
> the package of my UDF. This is no longer working for me. I tried a few
> things and below is how my drill-override.conf file looks now
>
> drill.classpath.scanning.packages : ${?drill.classpath.scanning.packages} [
> org.apache.drill.udfs ]
> drill.exec: {
>   cluster-id: "rahul_cluster_com-drillbits",
>   zk.connect: "localhost:5181"
> }
>
> When I restart the drillbits, I get this strange error " Caused by:
> com.fasterxml.jackson.databind.JsonMappingException: Could not resolve type
> id 'hbase' into a subtype of [simple type, class
> org.apache.drill.common.logical.StoragePluginConfig]"
>
> If I moved the package information to the drill-module.conf in my udf's
> project, then things are working fine. However this requires re-compiling
> the udfs which is not desirable. Is there any other way around this ?
>
> - Rahul
>